Yes exactly. But how was it when they first appeared in the human population? We can't know for sure, but most likely they caused a heavy pandemic in the first years and then over time became "harmless".

Just think about how bacteria/viruses which were not really dangerous for Europeans wiped out entire populations in Middle and South America.