Very agreed with all this. I've been learning the same lessons working on more robust computer vision for biomedical imaging. I bet unsupervised predictive pretraining could be adapted to (static) 3d image volumes. The z axis replaces the t axis, and you predict the next 2d slice from previous ones. Hmm...

As an aside - from the paper it looks like you worked at Brain Corp a few years back. Any thoughts on them and what they're doing these days? I'll be looking for a job again soon and i see a lot of ads for them.