The main advantage of Selenium is that it allows you to test cross-platform compatibility, since it can be configured to run on pretty much every major browser/OS combo.
For everything other than that, though, PhantomJS is fantastic. It's infinitely more pleasant to deal with than Selenium.
To make a comparison for our heavy Selenium use, I can't clearly see what is lacking in PhantomJS that would be a show stopper.
Does anybody know if other than lack of Flash support, it covers all aspects of what the end user would experience ?