It's funny, when we were acquired they started moving us to Github actions but it seems that maybe we should stay on our old crusty self-hosted Jenkins setup...

Same thoughts - we use an action to ship to production, its builds an image, pushes it to ECS which triggers a deployment.

We can't be blocked here. Seems silly what we settled on this, but for a long time GitHub had been reliable enough for many years, but things are sliding down the pan as of late.

Sounds like a very easy process to rewrite in bash/python and have it on hand if needed.

You should never entirely depend on a third party service for deployments.

Been burned too many times on that one.

Ok.

Move to EC2.

Darn AWS is down.

Alright, run it on a Mac Mini in your basement. Ahh dawn, your ISP is having issues. Good thing you have a backup 5G hotspot.

Ohh no, the power is out.

Eventually you have to trust someone else.

GitHub is a tragedy of the Commons. Too many people are using it, and Microsoft isn't willing to handle it correctly.

Feels like a very good business opportunity. Minimum 50k yearly contracts, GitHub with actual uptime. GitPro ?

We’re actually moving back to redundant data centres due to all of those problems.

Aggregate risk is too high.

It's almost as though GitHub should never have let itself be sold to Microsoft...

I'm sure the VCs who invested in GitHub disagree.

This is supposed to be Hacker News! Who is coming up with a startup to fill the gap !

Maybe we need a split between source management and distribution? The former looks like git[hub] to me, the latter maybe more like a Linux distro repo?

We could still deploy manually but it's suboptimal! And we're 'flying blind' without CI runs

> And we're 'flying blind' without CI runs

You should never entirely depend on a third party service to run your tests, either.

   make test

Should work without CI

It is a control pain

./deploy.sh

Self host gitlab. If you already host runners it’s not a big lift.

Even if there is features that are similar, most of gitlabs features are for paying customers only.

OP said they already pay for GitHub. We pay for the premium tier of Gitlab at my work and it’s definitely worth it.

It's always best to be portable - always be able to do builds and releases locally (at least, once you get the keys - it shouldn't be possible by default), then add things like github actions on top as convenience.

Same here. You’d think they could at least separate out the GitHub-hosted and self-hosted runners, so you’re still able to dispatch jobs if the self-hosted runners are down.

If the job queue is down, that wouldn't help, would it?

On my repo the jobs do not get scheduled on the PRs at all, so I assume that separation wouldn't help for todays issue.

They have the github enterprise domain separated out and its working fine right now https://us.githubstatus.com/posts/dashboard

I’m not convinced they actually do, because GHE on the cloud tends to have the same problems as the main outages. Probably costs extra to be “single tenant” or whatever

Depending on how many thousands of $ per year, it would probably be cheaper and more reliable to self-host GitLab. It's better in terms of organisational structure (you can have one, including access and secret inheritance), and (personal view) Gitlab-CI is better than GitHub Actions because it doesn't push you towards a JavaScript/NPM style dependency hell. And it's actually fairly easy to self-hosted, with options from a single machine with an omnibus package that handles everything to a full blown autoscaling Kubernetes deployment.

Sounds good until you see their cvedetails page

Hide it behind VPN, so it's not accessible from outside.

Now patching becomes a responsibility, unless your organisation is willing to run knowingly vulnerable software.

When you own it you can just limit it into vpn-ed company users, that significantly cuts down on the area that can be hit

I mean, the GitHub Actions supply chain risks and attacks definitely compensate for any GitLab security vulnerabilities you can think of.

> For a service we're paying thousands of $ per year for even though we host runners ourselves...

Wait until you charge you for self-hosting runners.

Oh wait. They already tried.

Sure. Don't use GitHub.

You can now hire me as an overpriced consultant instead of paying Microsoft.