Great idea. One thing I see is that you're graphing FPS, which means that your data becomes useless once enough of your target devices are able to sustain 60fps.
What I did for my HTML5 benchmarking solution for JSIL games is to instead record the timing in MS for the update and draw stages of every frame. This ensures that even if the benchmark is throttled to 60hz (Chrome does this no matter how you schedule ticks) you can compare timings across browsers on the same configuration and tell whether things have gotten faster.
Click 'Show Log' to see the timing data. It's been a very useful tool for evaluating improvements in JITcode quality, GC performance, canvas performance, and various JIT regressions for complex JavaScript. Just running it periodically has revealed dozens of regressions in Firefox and Chrome.
Hopefully with more usable data, your benchmark will contribute to spotting regressions too :D
P.S. feel free to harvest stuff from the JSIL benchmarking code. Everything is open-source.
record the timing in MS for the update and draw stages of every frame
This tells you nothing about when the frame actually makes it to the screen. It's ok for optimizing your own code, not so useful for comparing across devices, platforms & browsers.
Frame scheduling delay doesn't matter for performance benchmarking. Modern browser compositors literally grab the frame you hand them and display it at next vsync. It's irrelevant.
Note that a browser can also run rAF ticks at 60hz even if it's dropping half your frames to hit 30hz vsync. You won't be able to tell.
It is the case that some browser compositors fuck up the presentation of frames (hi Chrome), and that sucks, but you can't programmatically detect it.
What I did for my HTML5 benchmarking solution for JSIL games is to instead record the timing in MS for the update and draw stages of every frame. This ensures that even if the benchmark is throttled to 60hz (Chrome does this no matter how you schedule ticks) you can compare timings across browsers on the same configuration and tell whether things have gotten faster.
You can run the benchmark for yourself here: http://www.playescapegoat.com/?replayURI=/stages2and3.replay...
Click 'Show Log' to see the timing data. It's been a very useful tool for evaluating improvements in JITcode quality, GC performance, canvas performance, and various JIT regressions for complex JavaScript. Just running it periodically has revealed dozens of regressions in Firefox and Chrome.
Hopefully with more usable data, your benchmark will contribute to spotting regressions too :D
P.S. feel free to harvest stuff from the JSIL benchmarking code. Everything is open-source.