A different OS instance for every program? That seems like a pretty kludgy way to do it.
Kludgy or not, there have been some moves in that direction. People often distribute "VMWare appliances", i.e., a single application packaged up as a VMWare instance; and FreeBSD system administrators often run different services in separate jails.
There are other constraints on the size of a program, though. All other things equal, larger programs tend to be harder to maintain, and proportionally buggier. This is completely independent of the cost of memory, the size of processor caches, etc. -- There's nothing like Moore's law affecting the human brain's ability to understand sprawling, tangled code bases.
Also: Bugs in device drivers tend to affect stability much more than in userland programs.
Kludgy or not, there have been some moves in that direction. People often distribute "VMWare appliances", i.e., a single application packaged up as a VMWare instance; and FreeBSD system administrators often run different services in separate jails.