This benchmark gets harder the longer it runs. Which is not in itself terribly unusual. It is pretty smooth for the first bit, where the benchmark runs, but right about where things get hard on my iPad Air is when the test stops.
So this raises a question about how the test endpoint was decided. Because of course if you run longer, the performance would be worse, if you run shorter, the performance will be better, etc.
Now according to the previous article [1], this endpoint is exactly 60 seconds, and an average is taken over that entire period except some time at the beginning for JIT. Now 60 is a suspiciously round number to arrive at by trying to hit X many sprites on the screen or numerically deriving some benchmark objective. Probably, somebody just eyeballed what a Space Blaster game looked like and one minute of runtime was about right.
That probably is a reasonable predictor for Space Blaster, but it is quite the leap to use these results to predict "most other HTML5 games" as asserted in this article. Some games will have half the sprites and others will have double, some games will be more spikey, and others will be more well-behaved, etc.
If I was going to use this benchmark to predict my game's performance, I would do what the benchmark designer probably did. I would ignore the results table, run the benchmark until the sprite count/size looked about right, and take my own average of the FPS indicator around that time.
I'd also be concerned that the benchmark doesn't present a realistic performance profile, either in CPU or GPU terms.
It appears to have a low entity count up until the end, and not have any considerable amount of collision detection or other cpu-intensive entity behaviors. This means it's not really stressing a JS runtime and could probably perform great under an interpreter (as demonstrated by Ejecta getting a good framerate)
It also doesn't appear to produce a taxing GPU profile even on a canvas implementation. It draws very few objects, and most of them share the same texture and rendering state, ensuring that even a simple rendering implementation can batch them up into a single draw call with very few vertices. The framebuffer resolution is low and the overdraw is low as well.
A real-world game, even one that isn't 'too complicated' for mobile, would have a dramatically different performance profile.
I think this is a solid benchmark, but it's irresponsible to make claims like 'if sbperftest runs well, most other HTML5 games should run decently as well' and 'The next most common case is usually someone designing or porting such a complex game that even a native engine would struggle on the limited, battery-powered mobile hardware.'
You simply can't make those claims without a broader benchmark suite (which I should note most of the professional mobile/desktop benchmarking suites come closer to offering, with a handful of different benchmarks based on different workloads).
Author here. Perhaps the "if sbperftest runs well, most other HTML5 games should run decently as well" statement should have been inverted, so it reads "if sbperftest doesn't run well, few other HTML5 games will". We develop a major HTML5 game engine, and this statement is backed by quite a bit of testing and profiling of real-world games.
The test is deliberately based on a real-world game (it was adapted from an interactive game) so it's not a synthetic benchmark. No it is not enormously onerous for the CPU or GPU, but if you look at the results many mobile devices still can't quite manage a rock-solid 60 FPS, so I think it serves pretty well to see if mobile devices are fast enough for relatively simple games. There is actually a lot of collision testing in the laser-to-enemy intersection test - all of the objects in the game have custom collision polygons, it's not a simple bounding box test. The explosions use additive blending (or "lighter" composite mode) which actually seems to stress many canvas2d implementations but generally is a lot smoother with WebGL. So it's not that simple a test.
I'm not sure why the 60 second period is important; the most intensive part is the waves of green sprites which should come well within the measured period, and it gets quieter after that when the test ends, so it's not skipping the hard bit.
There are constant-FPS tests, that create more and more sprites until your FPS falls below a certain target. Your score is the number of sprites you can handle above that target.
Great idea. One thing I see is that you're graphing FPS, which means that your data becomes useless once enough of your target devices are able to sustain 60fps.
What I did for my HTML5 benchmarking solution for JSIL games is to instead record the timing in MS for the update and draw stages of every frame. This ensures that even if the benchmark is throttled to 60hz (Chrome does this no matter how you schedule ticks) you can compare timings across browsers on the same configuration and tell whether things have gotten faster.
Click 'Show Log' to see the timing data. It's been a very useful tool for evaluating improvements in JITcode quality, GC performance, canvas performance, and various JIT regressions for complex JavaScript. Just running it periodically has revealed dozens of regressions in Firefox and Chrome.
Hopefully with more usable data, your benchmark will contribute to spotting regressions too :D
P.S. feel free to harvest stuff from the JSIL benchmarking code. Everything is open-source.
record the timing in MS for the update and draw stages of every frame
This tells you nothing about when the frame actually makes it to the screen. It's ok for optimizing your own code, not so useful for comparing across devices, platforms & browsers.
Frame scheduling delay doesn't matter for performance benchmarking. Modern browser compositors literally grab the frame you hand them and display it at next vsync. It's irrelevant.
Note that a browser can also run rAF ticks at 60hz even if it's dropping half your frames to hit 30hz vsync. You won't be able to tell.
It is the case that some browser compositors fuck up the presentation of frames (hi Chrome), and that sucks, but you can't programmatically detect it.
The main bottleneck in WebKit was Object.prototype.hasOwnProperty(). It wasn't optimized for the case where the argument is a concatenation of strings, e.g hasOwnProperty("foo" + someString).
Honestly, I was hoping to see an FxOS phone on the charts. I mean, the whole thing is about the web apps, so what else could be a better way to measure it's gaming capabilities than the Construct 2? And vice versa, the FxOS seems to be an ideal (or at least, logical) target for Scirra.
Sounds like software rendering, did you try other browsers or disabling the blacklist? It's actually relatively common that high-end mobiles like the Nexus 5 can outperform low-end or software-rendering laptops/desktops.
FPS isn't really an issue for web games, it's input latency. Current browsers add something like a 60ms delay to every input event, and there isn't any fast path for translating inputs to events on screen. This makes it very difficult right now to create fun and fast paced interactive games in HTML5.
Input latency is an issue, sure, but saying that framerate doesn't matter is just ignorant. The existence of attempts at improving throughput (like asm.js and NaCL and Dart) demonstrates that performance is still a serious concern for developers.
Also, keep in mind that high throughput - necessary for delivering a consistent 60FPS - will naturally reduce input latency by reducing the length of frames and the amount of time spent not pumping input messages.
P.S. It's my understanding that gamepad input in both Firefox and Chrome is not buffered in any fashion, so that's going to be low-latency.
This benchmark gets harder the longer it runs. Which is not in itself terribly unusual. It is pretty smooth for the first bit, where the benchmark runs, but right about where things get hard on my iPad Air is when the test stops.
So this raises a question about how the test endpoint was decided. Because of course if you run longer, the performance would be worse, if you run shorter, the performance will be better, etc.
Now according to the previous article [1], this endpoint is exactly 60 seconds, and an average is taken over that entire period except some time at the beginning for JIT. Now 60 is a suspiciously round number to arrive at by trying to hit X many sprites on the screen or numerically deriving some benchmark objective. Probably, somebody just eyeballed what a Space Blaster game looked like and one minute of runtime was about right.
That probably is a reasonable predictor for Space Blaster, but it is quite the leap to use these results to predict "most other HTML5 games" as asserted in this article. Some games will have half the sprites and others will have double, some games will be more spikey, and others will be more well-behaved, etc.
If I was going to use this benchmark to predict my game's performance, I would do what the benchmark designer probably did. I would ignore the results table, run the benchmark until the sprite count/size looked about right, and take my own average of the FPS indicator around that time.
[1] https://www.scirra.com/blog/107/boosting-mobile-html5-game-p...