Math reasoning is still a non solved problem even if the rest of the capabilities are getting better. This means the transformers architecture may not be the best way to approach all problems