Hi, I say this so much that it is almost a revolving PSA at this point, but here we go again:
Big reminder that the Bitter Lesson is _not_ saying "just scale your methods and they work". What the Bitter Lesson _is_, however, is "work on methods that scale". There is a _huge_ distinction between the two, in my opinion.
Effectively, if I can add my layer of (re?)interpretation on it, it's saying that specialized boutique, hand-designed solutions don't play very well in the long-term in an arena with Moore's Law and money. But, what it's not saying, is that just making an algorithm bigger is the solution. This is how I see most people missing its interpretation.
For an algorithm to be effective, it needs 3 things in my opinion:
1. It needs to scale (measurable by some factor)
2. Because we have 1, it needs to have an implementation with extraordinarily rapid iteration time
3. If we have 2, we need an implementation that is extraordinarily lightweight (enough to run on consumer machines)
These three factors together, in my personal estimation, unlock algorithmic research progress in an area.
An ancillary fourth rule that really drives progress IMO is competition, formalized or otherwise, that is 1. open, 2. well-known, 3. incentivized and 4. accessible.
This is my personal opinion of course, and bound to be flawed in some way, but -- from personal experience, at least -- when I've seen this field (or research fields in general) align with this kind of method of research is where the speed of algorithmic research has absolutely exploded. :) <3
Big reminder that the Bitter Lesson is _not_ saying "just scale your methods and they work". What the Bitter Lesson _is_, however, is "work on methods that scale". There is a _huge_ distinction between the two, in my opinion.
Effectively, if I can add my layer of (re?)interpretation on it, it's saying that specialized boutique, hand-designed solutions don't play very well in the long-term in an arena with Moore's Law and money. But, what it's not saying, is that just making an algorithm bigger is the solution. This is how I see most people missing its interpretation.
For an algorithm to be effective, it needs 3 things in my opinion: 1. It needs to scale (measurable by some factor) 2. Because we have 1, it needs to have an implementation with extraordinarily rapid iteration time 3. If we have 2, we need an implementation that is extraordinarily lightweight (enough to run on consumer machines)
These three factors together, in my personal estimation, unlock algorithmic research progress in an area.
An ancillary fourth rule that really drives progress IMO is competition, formalized or otherwise, that is 1. open, 2. well-known, 3. incentivized and 4. accessible.
This is my personal opinion of course, and bound to be flawed in some way, but -- from personal experience, at least -- when I've seen this field (or research fields in general) align with this kind of method of research is where the speed of algorithmic research has absolutely exploded. :) <3