I don't think YAGNI ignores this so much as explicitly rejects it. I've participated in the type of rewrites you mention here. Yes, they suck. Yes, they happen far too often. But I've also observed that they seem to happen regardless of how much you plan ahead to future requirements: even if you are exhaustively brainstorming possible future directions and are absolutely sure you're going to want something, you're very often wrong. And the inevitable rewrite that happens is a lot more painful when you have to carryover requirements that you don't actually need.
YAGNI fundamentally is a statement about costs and benefits. And it's a statement about personal experiences of costs and benefits, with a counterintuitive conclusion. I've found, however, that the teams I've worked on that just accept that they'll have to rewrite or throw away 90% of what they write end up performing at a much higher level (in terms of their impact on the broader industry) than the teams who figure "But if we could just get that 90% down to 50%, we'll be 5x more effective than other teams!"
Another way to look at this is in terms of external vs. internal drivers of success. YAGNI makes sense when the primary drivers of success are external and you need to quickly react to changing market conditions or customer requirements. It doesn't when the primary drivers of success are internal and you need to quickly act to get from known point A to known point B as efficiently as possible. Some engineers are lucky (unlucky?) enough to work on the latter, but typically it only happens when you either have a monopoly or you're deep in the bowels of a corporation and only need to report up to an executive who never changes her mind.
To use the example from the article - if the biggest risk or change in the external environment you'll face is your software, go ahead and build the feature into it. But who knows? You may be able to close a round of venture funding in 2 months and then hire the Gondor navy to eliminate piracy. Or Gondor may enter into a trade agreement with Rohan and redoing all your contracts takes primacy. Or Aragorn may arrive with the Army of the Dead and suddenly piracy is not a problem anymore, but a lucrative business in life insurance may pick up.
I think there's a difference between going ahead and implementing piracy risk immediately, vs determining all the requirements of piracy risk and using them when designing your fundamental architecture, with hooks left in place for extensibility that simply aren't actually implemented yet. Maybe the Gondor navy will destroy the pirates, but then you have to worry about Corruption risk because you now may need to bribe the navy or risk having your cargo impounded (so you have to balance the risk of impounding vs the cost of bribing). Sure it's not the same thing as piracy risk, but it's similar, and because you designed your architecture from the get-go to enable piracy risk and other such extensions, you can now implement navy bribes pretty easily.
Meanwhile, if you'd said YAGNI to piracy risk and just implemented support for storm risk, you may find that you can't easily implement navy bribes without re-doing much of the work you already did for storm risk.
As saganus said[1], this is more of a craft than a science. You need to plan ahead with your architectural decisions, and they need to be made using the actual requirements you expect to encounter (as opposed to theoretical requirements, which aren't really much of a use to anyone), but that doesn't mean you need to actually implement everything immediately. Just enough to be satisfied that your architecture will suffice.
And this is why a lot of software written by the agile teams in the companies I've worked for looks like a messy accumulation of ad hoc solutions, with too little shared code and too much duplication.
Even if you are not implementing a feature right now, you still need to have as much information as possible about what might be needed in the future and how it will fit in in your solution.
No. You need to have information about what you're implementing right now, and the willingness to write it well.
What you generally see is that if you have a complex code base with lots of plan-for-the-future abstractions in it, refactoring it in any non-trivial way is really hard[1], so people don't, and you end up with hacks.
Whereas if the system is as simple as possible, you can make bigger changes more easily, and the code can stay cleaner.
That's no guarantee that it will -- code quality still requires discipline and sound engineering -- but it's a lot more likely with YAGNI than without.
[1] "Hey, we need to make our pricing system handle multiple currencies."
"Oh geez, that's going to wreak havoc with our risk plugin system, that change will take at least three weeks."
"Oh yeah, and we'll definitely need that risk plugin system when we get to the piracy risk feature later this year... what if we just kind of hack currency systems up by [doing something awful]?"
"Sure, we can do that in a week. I don't like it, but it's the only choice given our deadline right now."
The software I'm working on now is a messy accumulation of ad-hoc solutions but not because of YAGNI - it's because the real world of selling a system to multiple customers who get to demand things like "but we want our adverts to be blue on the second Tuesday of every third month" tends to makes software a mess.
YAGNI fundamentally is a statement about costs and benefits. And it's a statement about personal experiences of costs and benefits, with a counterintuitive conclusion. I've found, however, that the teams I've worked on that just accept that they'll have to rewrite or throw away 90% of what they write end up performing at a much higher level (in terms of their impact on the broader industry) than the teams who figure "But if we could just get that 90% down to 50%, we'll be 5x more effective than other teams!"
Another way to look at this is in terms of external vs. internal drivers of success. YAGNI makes sense when the primary drivers of success are external and you need to quickly react to changing market conditions or customer requirements. It doesn't when the primary drivers of success are internal and you need to quickly act to get from known point A to known point B as efficiently as possible. Some engineers are lucky (unlucky?) enough to work on the latter, but typically it only happens when you either have a monopoly or you're deep in the bowels of a corporation and only need to report up to an executive who never changes her mind.
To use the example from the article - if the biggest risk or change in the external environment you'll face is your software, go ahead and build the feature into it. But who knows? You may be able to close a round of venture funding in 2 months and then hire the Gondor navy to eliminate piracy. Or Gondor may enter into a trade agreement with Rohan and redoing all your contracts takes primacy. Or Aragorn may arrive with the Army of the Dead and suddenly piracy is not a problem anymore, but a lucrative business in life insurance may pick up.