> The second general point to be learned from the bitter lesson is that the actual contents of minds are tremendously, irredeemably complex; we should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries.
I agree and think “ML explainability” efforts are doomed to fail as ML becomes increasingly more effective. There is no a priori reason that the human brain should be capable of intuitively grokking sufficiently advanced general learners. We can invent them and improve them, but saying that we will be able to understand what the myriad matrix multiplications are “doing” will be like saying we understand the human brain because we can model the physics of its constituent atoms. The emergent complexity is too high for us to make any sense of it.
There is also no a-priori reason why we shouldn’t be able to understand how the higher-level behavior emerges. And without such an understanding, trying to improve or control the behavior is like poking around in the dark.
> trying to improve or control the behavior is like poking around in the dark
Which is basically what we've been doing with AI so far, as the paper notes (among other fields, like medicine). Have you heard the joke "grad student descent"?
This (popular) opinion makes no sense. How are you going to improve something you dont understand? Throwing ramdom software pieces to see what sticks? The recent progress was possible because people understood intuitively enough of the limitations of earlier models to think and invent a remedy.
Not that I necessarily agree with the above poster, but this:
> How are you going to improve something you dont understand?
Is just nonsense. Evolution understands nothing, yet produced a mind. Closer to us, the early people who produced all the crops that led to the shift to agriculture, and it's later improvements, absolutely did not understand how any of it worked.
Evaluation and selection are sufficient to improve things. Understanding is useful, but optional.
These are just absurdly remote analogies that have nothing to with how ML algorithms have developed thus far, nor how they will develop in the the immediate future.
So how would you "improve" here an now any algorithm. Create and insert random code and evaluate? Jeepers people are losing touch with reality.
> This (popular) opinion makes no sense. How are you going to improve something you dont understand?
How does that opinion not make sense? There are numerous things humans have invented for which we have little understanding of how they work: medical drugs, anesthesia, certain quantum phenomena utilized in semiconductors, etc.
I would argue that for current state-of-the-art LLMs, the implementation is likewise ahead of the theory at the moment.
"Understanding" does not need internalizing a process or algorithm at low level. We are not stochastic parrots. It is simply possesing sufficient insight so as to be able to explore nearby designs, formulate and test hypotheses, narrow the search space etc.
There is a strange emerging AI cult that is also in force here in HN that seems to believe these algorithms have evolved themselves or were some random trial and error. Ergo, they can keep evolving and the researchers dont need to understand a thing about how they work.
Serendipity in combining ideas that prove effective plays a role but within a fairly well defined conceptual sandbox. But progress with AI is more or less conditional on people having sufficient understanding to coax algorithmic structures in the desired direction.
For decades ML researchers were walking on eggshells, thinking that they are going to run into overfitting and bias/variance tradeoffs, adding regularisation, worrying about local maxima, etc. Then one day a certain company kept making their models bigger without any concern and suddenly everyone thinks that there are unlimited returns to scale, when in reality you merely bought yourself a one time victory.
I agree and think “ML explainability” efforts are doomed to fail as ML becomes increasingly more effective. There is no a priori reason that the human brain should be capable of intuitively grokking sufficiently advanced general learners. We can invent them and improve them, but saying that we will be able to understand what the myriad matrix multiplications are “doing” will be like saying we understand the human brain because we can model the physics of its constituent atoms. The emergent complexity is too high for us to make any sense of it.