Well then that's your responsibility, as a COVID "skeptic", to make the distinction clear -- especially if you invoke phrases such as "the Cathedral" as a big nebulous conspiracy. People who think the vaccine gives you autism also consider themselves COVID (vaccine) skeptics.
It would be far healthier for discussions to separate skeptics and obviously crazy or idiotic. Those are two very different groups.