Those compilers don't do nearly as much optimization as LLVM does. It's hard to overstate how wide the gap between the classical compilers you like and LLVM/GCC truly is. LLVM has no fewer than three IRs, each with a full suite of compiler optimizations (four if you count ScheduleDAG). LICM, just to name one example, runs at each stage (see MachineLICM.cpp). Turbo Pascal did nowhere near that level of optimization, and it produced unacceptably slow programs by modern standards.
Also I never had 4h builds with single core processors back when I was working full time with C++. The slowest ones were around 2h.
A core duo would run circles around those computer systems, hence my dismay how much hardware LLVM requires for building.