Exactly this.

You can ruin entire filesystem, because not only file blocks are cached, but filesystem blocks too.

If you still do not want to eject the drive beforehand, turn on Optimize for quick removal option (should be default for flash drives now). You will lose write cache though (the write performance will be drastically worse).