FYI, PagerDuty is built on top of AWS. They used to do some multi-cloud stuff, but no longer the case (too expensive, too complex, causing more issues than it solved). Source: worked there for 2 years.
Alerting or control plane or both? If alerting is AWS only, I'm very sad. What happens if there is a global AWS outage? Never been one yet, but never say never.
If your service needs to survive a global aws outage, you just can't run with any saas. So many of these companies are single regioned in AWS. Auth0, Okta, Datadog, many others put customers in a regional box, and if that region goes down, all of those customers go down.
I'm working on the infrastructure of a startup (Whova). Our backend is fully in Django, so our background tasks are run with celery since that's the main tool for that in the python community and a lot of legacy code is built around that. We process millions of tasks per day, for various things: cpu-bound logic, io (email/push notifications, 3rd part api calls, ...), scheduled tasks, ... These tasks are split across 20 queues processed by multiple workers on 6 dedicated machines.
My celery experience so far has been quite awful to say the least.
From the infrastructure point of view, celery has been the less reliable component of our stack by far.
As others have mentioned, one problem is that celery is frequently used for what it is not meant to be. And that's true for us too. If you check the documentation, deep down, celery was originally designed for short lived tasks, cpu bound, but turned out to be used for long lived tasks. And starting to process long lived tasks is the root of many problems until you find the correct settings to make it work. Of course, these settings are either not documented, or the documentation is useless at best (it sometimes creates even more confusion). There are also very few good quality resources online regarding this.
For several months, we dealt with celery workers getting stuck and not processing tasks, celery workers running out of memory, ... until we found the correct solution. And even now, we still have some random issues we have difficulty to track down due to the poor quality of monitoring around celery.
It actually made me smile a few weeks ago when the engineering team of DoorDash released a blog article about celery in which they mentioned several issues we encountered, including some they still have no clue but managed to mitigate (in particular, the stuck celery queue: they need to use -Ofair to fix the scheduling algorithm!) [1]
It's also very easy for developers to make mistake with celery: celery routing in Django is messy (routing of individual tasks and scheduled tasks), adding new queues need some coordination upon deployment until you automate it, generating too many scheduled tasks can make your workers run out memory, ... Celery definitely requires a solid training for all the engineers that will work with it. To be fair, this is very likely to be a true for any backgroubd processing tools: it usually is a critical part of the tech stack, but resources/training about that are less.
We are still using Celery 3. We few months ago, when they released celery 4, we looked into upgrading, but it was way more work that expected as the entire configuration syntax was broken. The testing needed to deploy that to production was not worth the shot, especially when factoring it took us months to find some tricky settings to get celery to finally be somewhat stable, so why risk losing that. Now, they already are at celery 5.0, and they plan to release even more breaking updates: seriously, WTF! And if you try to report issues but you use celery 3, you'll just be told to upgrade.
To be frank, I believe celery is a good project. They aren't many alternatives in python anyway. But they don't seem to listen to what their users need. It really seems that there is a gap between what they expect people to do with celery and what people do with it. I understand it's hard to provide a good default configuration suiting everyone, but then provide the appropriate documentation about how you can tune celery based on your use case, or clearly state the intented use case and limitations. So, the last thing we need is more breaking versions with more uncertainty about celery, but more documentation!
If they really go on that path, it's clear that we will eventually ditch celery for something else. Celery, from our experience, is not production friendly unless you put major efforts into it, or unless your project is fairly simple.
I can definitely confirm the issues. We had memory issues, stuck tasks and lost tasks (despite late ACK). It really requires a lot of fine tuning to get right. We started switching to AWS SQS four years ago and created a thin Django/Python wrapper [0] for it. Now we handle tasks in the hundred of millions a day and I don't want to think about how that would have turned out if we didn't switch.
But honestly, people working on Celery in their free time (i think) so who am I to complain. I can see that monetary support would be necessary to make it better. But in my opinion the project just got too big and it will be difficult to fix all the underlying issues.
Whova | Infrastructure Engineer | San Diego, CA, USA | ONSITE FULL-TIME VISA | 1Y+ experience | 90K+ |
Whova is growing fast, especially our recent shifts toward virtual events, and we need to scale our infrastructure!
We are looking for a new infrastructure engineer to join our team in San Diego and tackle many scalability-related challenges.
Our infrastructure team is small, so the job responsibilities are various (operations, team processes improvements, application-level performance improvements, various infrastructure-level projects like setting up docker/k8s or adding support for clusters/replicas, ...).
In short, the position is a unique opportunity to build the foundations of a growing infrastructure.
1 year of experience working on infrastructure is needed.
If you are interested to learn more, feel free to drop me an email at simon.ninon@whova.com.
Whova | Full-Stack Engineer (React & Django) | San Diego, CA, USA | ONSITE FULL-TIME NEWGRAD VISA
Whova is looking for new full-stack engineers to join our growing team in San Diego (growing 2X every year)!
The position involves developing new features from scratch in Django and React to our core product.
Prior experience in Django/React/Web/... is not needed: we are looking for people willing and capable to learn, and seeking interesting challenges to solve.
If you are interested to learn more, feel free to drop me an email at simon.ninon@whova.com.
---------------------------------------------
Whova | QA Engineers | San Diego, CA, USA | ONSITE FULL-TIME NEWGRAD VISA
Whova is looking for new QA engineers to join our growing team in San Diego (growing 2X every year)!
The position involves developing new automation tests for our web and mobile products, improving our QA testing processes and developing tools to increase automation tests throughput and accuracy.
We are looking for people willing and capable to learn, and seeking interesting challenges to solve.
If you are interested to learn more, feel free to drop me an email at simon.ninon@whova.com.
Whova | Full-Stack Engineer (React & Django) | San Diego, CA, USA | ONSITE FULL-TIME NEWGRAD INTERNS VISA
Whova is looking for new full-stack engineers to join our growing team in San Diego (growing 2X every year)!
The position involves developing new features from scratch in Django and React to our core product.
Prior experience in Django/React/Web/... is not needed: we are looking for people willing and capable to learn, and seeking interesting challenges to solve.
Thus, this position is open to anyone interested: newgrad, senior, intern, ...
If you are interested to learn more, feel free to drop me an email at simon.ninon@whova.com.
Whova | San Diego, California | Full time | Backend/Infrastructure Software Engineer
Whova is creating a new team in charge of developing its distributed development and production infrastructure to support our continuous growth.
We are looking for a talented software engineer interested in pushing our infrastructure to the next level, as well as working on our mobile backend. The position is basically a mix of SRE and backend development.
That's a brand new team, so that's a great opportunity for career advancement, learning and solving unique challenges.
New grads are welcomed and we offer free pizza on Fridays :)
Whova | San Diego, California | Full time | iOS/Android Software Engineer
Whova is looking for a talented iOS or Android engineer to work on our main even app.
This is a great opportunity as our user base is growing ~2X per year and you will be directly contributing to our most-used product. There are also interesting challenges to solve to help improve performance issues or even help our team to scale.
New grads are welcomed and we offer free pizza on Fridays :)
Whova | San Diego, California | Full time | Backend/Infrastructure Software Engineer
Whova is creating a new team in charge of developing its distributed development and production infrastructure to support our continuous growth.
We are looking for a talented software engineer interested in pushing our infrastructure to the next level, as well as working on our mobile backend. The position is basically a mix of SRE and backend development.
That's a brand new team, so that's a great opportunity for career advancement, learning and solving unique challenges.
New grads are welcomed and we offer free pizza on Fridays :)
Whova | San Diego, California | Full time | iOS/Android Software Engineer
Whova is looking for a talented iOS or Android engineer to work on our main even app.
This is a great opportunity as our user base is growing ~2X per year and you will be directly contributing to our most-used product.
There are also interesting challenges to solve to help improve performance issues or even help our team to scale.
New grads are welcomed and we offer free pizza on Fridays :)
Whova | San Diego, California | Full time | Backend/Infrastructure Software Engineer
Whova is creating a new team in charge of developing its distributed development and production infrastructure to support our continuous growth.
We are looking for a talented software engineer interested in pushing our infrastructure to the next level, as well as working on our mobile backend. The position is basically a mix of SRE and backend development.
That's a brand new team, so that's a great opportunity for career advancement, learning and solving unique challenges.
New grads are welcomed and we offer free pizza on Fridays :)
Whova | San Diego, California | Full time | Backend/Infrastructure Software Engineer
Whova is creating a new team in charge of developing its distributed development and production infrastructure to support our continuous growth.
We are looking for a talented software engineer interested in pushing our infrastructure to the next level, as well as working on our mobile backend. The position is basically a mix of SRE and backend development.
That's a brand new team, so that's a great opportunity for career advancement, learning and solving unique challenges.
New grads are welcomed and we offer free pizza on Fridays :)
Whova | San Diego, CA | INTERNS, PART TIME, FULL TIME
Company:
Whova provides an all-in-one event management solutions including a mobile event app to support conferences, expos, tradeshows, networking and business events. Whova is a fast-growing company with customers in more than 50 countries. Whova is proud to be a winner of the San Diego Business Journal’s the Best Places to Work in San Diego in 2016, 2017 and 2018, three years in a row. Whova also won the Fastest Growing Private Companies award from the San Diego Business Journal in 2018.
