YouTube prefers VP9, when available. You can try disabling VP9 in Firefox by setting the about:config pref "media.mediasource.webm.enabled" = false. Depending on your GPU, Firefox may be able to decode H.264 using a hardware decoder.

This YouTube page will show which codecs are currently enabled in your browser: https://www.youtube.com/html5