This article is wildly off base, particularly when you look at the examples the article cites.
WhatsApp was able to scale to so many users while keeping the engineering team small precisely because they hired great engineers who wrote an extremely efficient system in Erlang. (Does he think commodity coders know Erlang? Really!?) They heavily tuned their architecture and built efficient paradigms which kept developer productivity & performance from degrading as their user base grew. [1] Commodity engineers don't write that sort of code.
Yes, we increasingly have tools which allow mediocre coders to get stuff done. (For example, Ruby on Rails.) But the inevitable byproduct of letting mediocre coders loose on a stack they only partially understand is a mess of spaghetti code which takes even more developers to maintain. This is already what you see with non-tech corporations who employ dozens/hundreds of developers to maintain Drupal/RoR/Java apps which are significantly simpler in scale & complexity than WhatsApp or Instagram. If you want to use tools well, you need to hire people who understand the tools.
Yes, the tooling has improved dramatically. A 10x engineer of yesteryear might actually be a 100x engineer today. But if you give a bad engineer modern tools, it just allows them to fail faster.
I might be able to build a house with legos, but I expect to be spending a lot more time repairing it than if I'd just hired a carpenter in the first place.
>WhatsApp was able to scale to so many users while keeping the engineering team small precisely because they hired great engineers who wrote an extremely efficient system in Erlang. They heavily tuned their architecture and build efficient paradigms which kept developer productivity & performance from degrading as their user base grew. [1] Commodity engineers don't write that sort of code.
The problem is they did it with only a few (16?) "great engineers". That's the premise of the article -- our world today might need 100,000 great engineers to solve great problems. Tomorrow 1000 might be enough.
As for the "near great" they would too be less needed, and commodity engineers would be the equivalent of burger flipers.
>Yes, the tooling has improved dramatically. A 10x engineer of yesteryear might actually be a 100x engineer today. But if you give a bad engineer modern tools, it just allow.
The first yes, the second not really. A bad engineer can get enormously more done with modern tools than he could 10 or 30 years ago. Essentially a bad engineer today can build stuff that took high level senior engineers to build -- and do it in less time too. It might be a little crude, but it will work.
E.g a bad engineer can automate a workflow that would have taken expert engineers, custom software and several man-years to do 20 years ago, and can do it in a day.
And because it's so abstracted, building something like that needs little influence from his skills, so his "badness" doesn't show up much.
> The problem is they did it with only a few (16?) "great engineers". That's the premise of the article -- our world today might need 100,000 great engineers to solve great problems. Tomorrow 1000 might be enough.
The premise of the article seems to be that we no longer need great engineers at all. Or that they only need to work at infrastructure companies (not application companies like WhatsApp). The very example he chose shows that's empirically untrue.
The fact that a small team of amazing engineers can accomplish huge things (given some ops support) is not new. The original Macintosh team was also tiny.
There's no evidence that we will ever stop needing great engineers, particularly as more and more applications require huge scale while maintaining agility.
>The premise of the article seems to be that we no longer need great engineers at all.
Well, not really, because somebody will have to build all those elaborate abstraction layers people will use.
The core idea is that we'll need much fewer great engineers -- which is another way of saying we won't need "great engineers" at all. Read "not need them at all" liberally (very few), not literally (nobody).
After all if job demand goes from 1,000,000 people to 1000 people (arbitrary numbers), it's like it's the end of that job, even if technically people are still employed (there are after all horse cabbies and film lab technicians and letterpress printers employed still today).
This is absurd. How many people where creating their own languages 20 years ago? Everyone and their grand ma is building new languages on top of javascript. The expertise level of software engineers is going up! 20 years ago, how many people could build a VM from the ground up? Have you visited github lately? Sure, an average engineer can do much more today, and the reason is because of these great engineers, and they are not fewer by any means.
>How many people where creating their own languages 20 years ago?
20 years ago? Far less. That means writing a language is getting more commoditized -- and we work in a higher, easier, level of abstraction. That's what TFA says about other spheres of development and deployment too.
>Sure, an average engineer can do much more today, and the reason is because of these great engineers, and they are not fewer by any means.
Neither I, nor the article, said there are fewer great engineers. Only that fewer great engineers are (and fewer still will be) needed. Both as a percentage of all great engineers, and in absolute numbers.
I can't substantiate it, but there were in fact a lot of languages being created through the 60's to 90's. One could argue the language implementors of old were quiet daring, push boundaries much more than today (see APL and ICON as examples). sed and awk arose in an era of language explosion, make, sql, the list goes on.
> Well, not really, because somebody will have to build all those elaborate abstraction layers people will use.
It's almost as though you completely ignored my next sentence, which clarified that great engineers continue to be valued outside of abstraction/infrastructure companies.
Well, I don't find it valid to be frank. They might be valued, but they are valued less and less, and will be valued even less in the future.
An enterprise that used to need 100 engineers, with the cloud, SaaS, PaaS etc, can even today make do with 1/5 of that. Is there any trend showing companies needing MORE software engineers?
I would say yes. More and more stuff is getting done by computer. The majority of IT staff are still building / maintaining in house applications as far as I am aware, not building teh next Facebook / Whatsapp / etc.
The problem with the article's conclusion (and title) is that they don't stop with 100,000 -> 1,000 engineers.
"Tomorrow, that billion-dollar startup acquisition might not need an engineer at all."
The article's claim is not that startups are going to keep having smaller and smaller teams of higher-value engineers, they're saying that they're going to have zero engineers. Which I don't see at all from the evidence provided.
Well, they might technically have an engineer, but it could be some average joe, using a "21st century" equivalent of Hypercard -- building a whole startup with some composable cloud components and some logic.
After all, the basic skills for a successful startup, even today, are not hardcore engineering skills, but idea, business, finances, etc.
I mean, most startups bought for multi billion dollars work on BS trivial engineering problems (mostly: make a social web service scalable), compared to what talented engineers do elsewhere (in research, in space exploration, in robotics, in DSP, etc).
Heck, wasn't Facebook started with crappy PHP code a 15 year old could write (and a 20 year old did wrote)? Or, for a smaller scale example, Derek Shivers (which I admire) sold his CDBaby startup for a nice sum, and that was also started and based on amateurish code at best that he wrote himself.
It's a problem if you are an engineer. It's the kind of "going homeless" fuckery.
It's also a problem, longer term, if you are an entrepreneur. Initially it starts as "lower costs", but with an impoverished middle class, you soon find there are not that many people to buy your fancy products and services.
Automation is only good is it's combined with a scheme for the surplus of labor -- e.g less hours work week, creation of new kind of jobs, etc. Without that is a huge creator of poverty and inequality, and eventually the collapse of a market economy.
If I remember correctly from a high scalability post about WhatsApp it was also on bare metal. So more complexity to manage then a simple AWS setup where you can tear up/down in seconds.
Whatsapp also makes design decisions that I would call 'engineer-easy'. Their iOS app is pretty much all default system widgets and behavior for example, which is significantly less code to maintain than something that was customized heavily.
On the server side they don't retain conversations, have multi-session usage (many clients, one user), etc which simplifies what is required.
I haven't used whatsapp in a while so things might of changed, but before they were acquired by Facebook, that was definitely their direction.
WhatsApp was able to scale to so many users while keeping the engineering team small precisely because they hired great engineers who wrote an extremely efficient system in Erlang. (Does he think commodity coders know Erlang? Really!?) They heavily tuned their architecture and built efficient paradigms which kept developer productivity & performance from degrading as their user base grew. [1] Commodity engineers don't write that sort of code.
Yes, we increasingly have tools which allow mediocre coders to get stuff done. (For example, Ruby on Rails.) But the inevitable byproduct of letting mediocre coders loose on a stack they only partially understand is a mess of spaghetti code which takes even more developers to maintain. This is already what you see with non-tech corporations who employ dozens/hundreds of developers to maintain Drupal/RoR/Java apps which are significantly simpler in scale & complexity than WhatsApp or Instagram. If you want to use tools well, you need to hire people who understand the tools.
Yes, the tooling has improved dramatically. A 10x engineer of yesteryear might actually be a 100x engineer today. But if you give a bad engineer modern tools, it just allows them to fail faster.
I might be able to build a house with legos, but I expect to be spending a lot more time repairing it than if I'd just hired a carpenter in the first place.
[1] http://highscalability.com/blog/2014/2/26/the-whatsapp-archi...