There was a famous (in the diy audio community at least) blind test where music was played in audiophile equipment with some premium audio cable that cost a lot, and then the same music was played with a bent coathanger used as a cable. The order was randomized, and the audiophiles couldn't tell the difference. There have been other blind tests with different equipment, like hi end amplifiers vs mid-level consumer grade equipment (like your average Sony) with same results. You can search audio forums for the details. From all that I've read over the years, and from my own experience of quite a few years listening to the best recorded classical music, pretty much every good consumer player, amplifier, dac today is good enough for every audiophile. Where you'll find the difference is in the speakers, and to a lesser extent, headphones.
TL;DR: If I had 5,000 dollars to spend on high end audio equipment for my living room, I'd spend 3,500-4,000 on the speakers. But I wouldn't, personally. I'd spend half of that and get 98% of the quality.
I think there is varying degrees of craziness with audiophiles and then they all get lumped together making them look worse.
For example you have the Japanese audiophiles spending 100,000 on speakers but live in a 10x10 room, or install power lines as they say they can hear the difference.
Audio reaches diminishing returns pretty quickly, you aren't going to see massive audio gains by upping your budget from $2,000 to $20,000 but you will by going from $200 to $2,000. At a certain point you are paying for a sound curve that suits you over the end all be all music reproduction.
A lot of stuff is getting cheaper too with good class D amplifiers with very low distortion being available to DIY and custom builders for relatively cheap (as long as you aren't in the class D is garbage audiophile crowd, which I think is similar to the my $1,000 dollar wire is better than your $50 dollar wires).
TL;DR: If I had 5,000 dollars to spend on high end audio equipment for my living room, I'd spend 3,500-4,000 on the speakers. But I wouldn't, personally. I'd spend half of that and get 98% of the quality.