Agree. Counting tokens today is like counting lines of code submitted to prove productivity. Can be completely gamed and diminishing proof productivity (aka, having any lines of code usually shows more competence than having none, but after a certain point there is no correlation and maybe a negative correlation).
What do people think of tools like www.aiqrank.com which measure on agent orchestration use, skill use etc?
I have found good success with Claude Code/AgentOS. The real question - is Elixir the best language to develop with using AI code generators? It may be?
Thank you Simon! Too many people conflate non-engineer vibe coding with engineers using ai to make themselves much more productive. We need different terms!
If the software is doing complicated integrations, that may be a barrier as said in the article.
And to be clear, this is people using teams of Claude Code agents (either Sonnet 4.5 or Sonnet 5 and 5.5 in the future). Reliability/scale can be mitigated with a combination of a senior engineer or two, AI Coding tools like the latest Claude Code and the right language and frameworks. (Depending on the scale of course) It no longer takes a team senior and mid-level engineers many months. The barriers even for that have been reduced.
Completely agree that using Lovable, Bolt, etc aren't going to compete except as part of noise, but that's not what this article is saying.
I think this puts the onus in the wrong direction. I _love_ LLM coding and write probably 70% of my code that way. But having seen its (current) limits, and building a few toy apps myself, I'd love to see examples of successful, complex products that are mostly vibe coded. Until I see that, I'll continue to believe the current crop of LLM is best suited for building prototypes, helping get initial ideas shipped, and helping speed up very experienced developers working in well trodden ground (i.e. mostly CRUD in popular languages / frameworks), because that's what most peoples experience is (at best - many here wouldn't be nearly as generous as my takes).
Look at the AI visibility tools. They all integrate with multiple LLM models, include scheduling, management of multiple external processes, data parsing, site-scraping, graphs, as well as multiple database structures. They need retry and error logic, real-time displays and updates, and multiple flow UX's, and Stripe integration with webhooks, and subscription management.
Same thing with competitor monitoring. These tools require scraping multiple sites, checking X, Facebook, Jobs sites, Crunchbase, etc, aggregating data and displaying and making sense of changes. And the same multi-process management, queuing, and Stripe integrations.
A few years ago, these would both fit into businesses requiring many months of development to get it all running. Now we are seeing dozens of companies emerging in each of these categories each month as they take weeks to build. And if one finds a cool aha (a new integration or graph or UX flow or positioning) the others can quickly follow in a week or less of AI-agent coding.
There are dozens of other categories where this is happening too.
The hard part of figuring out the nuances of the APIs and integrations and retries and AWS integrations and Rabbit MQ configurations and corner cases can all be done by AI with the right context.
On a tangent, I sometimes avoid correcting typos and awkward expressions because it adds non-ai signal. I do t intentionally add any, but I let them be.
LOL. AI is currently like a mid-level average to good engr who can write good code but ocassionally goes off the rails. Any engr on a team with those characteristics would be heavily vetted in reviews. Almost like a smart CS intern.
If AI was amazing senior level engr, it would be a different story.
What do people think of tools like www.aiqrank.com which measure on agent orchestration use, skill use etc?
reply