I do the same. The one disadvantage is you lose the ability to click on overlayed links to related videos that some people often put at the end of their videos.
I'd say it depends entirely on the video/channel. For instance, on Brady Haran channels (Periodic Videos, Sixy Symbols, Numberphile, etc) the overlays are very useful as they link to videos about tangentially related topics, followup videos on the same topic, or even other videos on the same topic featuring a different professor explaining things a different way.
