Not really. I can obviously say something, like you learn which features the models are able to actually implement, and you learn how to phrase and approach trickier features to get the model too do what you want.
And that's not really explainable without exploring specific examples. And now we're in thousands of words of explanation territory, hence my decision to say it's hard to put it into words.
I think you’re handwaving away vague, ungrounded intuition and calling it learning.
For instance, if I say “I noticed I run better in my blue shoes than my red shoes” I did not learn anything. If I examine my shoes and notice that my blue shoes have a cushioned sole, while my red shoes are flat, I can combine that with thinking about how I run and learn that cushioned soles cause less fatigue to the muscles in my feet and ankles.
The reason the difference matters is because if I don’t do the learning step, when buy another pair of blue shoes but they’re flat soled, I’m back to square one.
Back to the real scenario, if you hold on to your ungrounded intuition re what tricks and phrasing work without understanding why, you may find those don’t work at all on a new model version or when forced to change to a different product due to price, insolvency, etc.
You're always free to stop at the level of abstraction at which you find a certain answer to be satisfying, but you can also keep digging. Why are flat shoes better? Well, it's to do with my gait. Ok, but why is my gait like that? Something-something musculoskeletal. Why is my body that way? Something-something genetic. OK, but why is that? And so on.
Pursued far enough, any line of thought will reach something non-deterministic - or, simply, That's The Way It Is - however unsatisfying that is to those of us who crave straightforward answers. Like it or not, our ground truth as human beings ultimately rests on intuition. (Feel free to say, "No, it's physics", or "No, it's maths", but I'll ask you if you're doing those calculations in your head as you run!)
It is very silly to treat zero grounding the same as accepting core, proven concepts. Your PoV here is no different than saying "It rains because god is sad and crying" is an appropriate thing to believe.
If you want to say "god is responsible for creating the precipitation cycle", sure. But we don't disregard understanding that exists to substitute intuition.
We're talking past each other, and mixing up some concepts, most of which is my fault for not writing particularly clearly.
Yeah, "God did it" is the first of those answer layers at which some people stop interrogating the world around them, just like "that's just the way I am" is where some people stop developing their self-understanding. Neither of those answers advance civilization / ourselves any further than the status quo. They're terrible answers! Everyone should be digging deeper.
However, I would not use the word "understanding" in opposition to "intuition". Someone who can generate a ballistics chart understands trajectories, but so does someone who can reliably put a basketball through a hoop or a bullet on target. I would set "analysis" against "intuition" (or "instinct", if you prefer), but they're not in opposition: instead, they reinforce each other. We're all familiar with the scientists and mathematicians who ride a hunch to a ground-breaking discovery, which is then validated by exhaustive analysis. From the other direction, athletes and musicians analyze their technique in minute detail, and practice incessantly, in order to ingrain analytical insights into instinct. (Or, if you prefer a less physical example, programmers study algorithms so that they can intuit which to apply to a particular problem.)
My point - badly expressed in my earlier comment - is that as humans we exist moment-by-moment, and as such react, in each moment, by intuition. As important as analysis is, we cannot live in analytical mode: it lags too much! Furthermore, approximately none of us will ever make a groundbreaking discovery in any field, far less in all of the areas to which we can (and should!) direct our analytical energy. At some point we have to stop (even if we are a groundbreaking genius in one area, we'll have to in all of the others), and accept the answer that satisfys our purpose or exhausts our motivation.
And that's not really explainable without exploring specific examples. And now we're in thousands of words of explanation territory, hence my decision to say it's hard to put it into words.