I'm not too worried, since a) that makes it about as fast as cpython and b) there is a lot of low-hanging fruit. There is work being done on integrating the Cleavir compiler which does a number of tricks. In particular escape analysis and lambda lifting should both help a lot on certain types of code.
To write a compiler (and not a interpreter) it's indeed almost impossible to reach the slowness of CPython. Especially because it's hard to create a complex language in just a weekend.