Are the relative rankings in search results (e.g. moving from 25th to 15th or 2nd to 1st) for a set of pre-defined searches considered poor metrics?
Sure it will be hard to determine if the improvements are due to the cause of SEO or some other update, but using revenue as a metric has the same issue.
(Ideally one would use a "difference in differences"[1] approach, but it could be difficult without good comparators).
Any metric that you choose will have the same problem. The metric may improve over a period because of actions that you take, or they may improve because of something entirely unrelated. This is an important thing to be aware of.
However, there is another problem with metrics. Sometimes, it is too easy to pay too much attention to vanity metrics that won't add anything to your bottom line. Will moving from a 25th place to 1st place for a search help your business? That depends - does that search generate traffic, and does that traffic generate revenue? On the other hand, increasing revenue is never a bad thing.
I'd argue that unless decision makers have too much time on their hands, they're better off focusing on metrics like revenue than metrics like search results. Validity will be a problem across all metrics, but at least there isn't a tendency to optimize useless metrics.
Sure it will be hard to determine if the improvements are due to the cause of SEO or some other update, but using revenue as a metric has the same issue.
(Ideally one would use a "difference in differences"[1] approach, but it could be difficult without good comparators).
[1] https://en.wikipedia.org/wiki/Difference_in_differences