> How are these clowns deploying stuff on a Friday, it is unbelievable to me
I have stopped fighting this battle at work (not Cloudflare). Despite Friday being one of the most important days of the week for our customers, people still deploy the latest commit 10 minutes before they leave in the afternoon. Going on a weekend trip home to your family? No problem, just deploy and be offline for hours while you are traveling...
The response was that my way of thinking is "old school". Modern development is "fail fast" and that CI/CD with good tests and rollback fixes everything. Being afraid of deploys is "so last decade"... The problem is that our tests don't cover everything, it may not fail fast, and not all deploys can be rolled back quickly and the person who knows what their huge commit that touches multiple files actually does is unavailable!
We have had multiple issues with late afternoon deploys, but somehow we keep doing this. Funnily enough, I have noticed a pattern. Most experienced devs stops doing this after causing a couple of major downtimes, due to the massive backlash from customers while they are scrambling to fix the bug. So gradually they learn to deploy at less busy times and monitor the logs to be able to catch potential bugs early.
The problem is that not enough has learned this lesson because they have been lucky that their bugs have not been critical, or because they are too invested in their point of view to change. It seems that some individuals learn the hard way, but the organization has not learned or is reluctant to push for a change due to office politics. I decided to keep my head low and let things play out, as I simply no longer care as long as the management don't care either.
- Affecting both shopify.com and claude.ai, so no phased deployment
- Takes 30 minutes to remediate
If they would've just deployed to a single of their high-value customers at once, they could've spared shopify.com an hour of downtime and maybe millions in abandoned shopping carts.
I have stopped fighting this battle at work (not Cloudflare). Despite Friday being one of the most important days of the week for our customers, people still deploy the latest commit 10 minutes before they leave in the afternoon. Going on a weekend trip home to your family? No problem, just deploy and be offline for hours while you are traveling...
The response was that my way of thinking is "old school". Modern development is "fail fast" and that CI/CD with good tests and rollback fixes everything. Being afraid of deploys is "so last decade"... The problem is that our tests don't cover everything, it may not fail fast, and not all deploys can be rolled back quickly and the person who knows what their huge commit that touches multiple files actually does is unavailable!
We have had multiple issues with late afternoon deploys, but somehow we keep doing this. Funnily enough, I have noticed a pattern. Most experienced devs stops doing this after causing a couple of major downtimes, due to the massive backlash from customers while they are scrambling to fix the bug. So gradually they learn to deploy at less busy times and monitor the logs to be able to catch potential bugs early.
The problem is that not enough has learned this lesson because they have been lucky that their bugs have not been critical, or because they are too invested in their point of view to change. It seems that some individuals learn the hard way, but the organization has not learned or is reluctant to push for a change due to office politics. I decided to keep my head low and let things play out, as I simply no longer care as long as the management don't care either.