I think you're saying that people who harass should be able to be removed absent a code of conduct. I agree. However, harassment isn't always the overt act you may be picturing, it can be more subtle. Code of Conducts give a single canonical state for what's considered acceptable, but more importantly, give organizers something tangible to point to when there is a violation. In addition to that, they inform people (even the harassed, who may not have considered what's happening to them harassment!) of what I'd acceptable.
As a white, male engineer, I haven't ever found a need for codes of conduct because I've had the privilege of never having been harassed. This isn't the case for all groups, so it's best to have one.
> Code of Conducts give a single canonical state for what's considered acceptable
This is what they're supposed to do. In reality, this is impossible because so many of the rules in a CoC are subjective, and therefore are enforced based on the opinions of the people enforcing the CoC.
> but more importantly, give organizers something tangible to point to when there is a violation.
This just gives the enforcers a sense of moral authority to impose their opinions, nothing more. I don't see that as a benefit of what is inevitably an incomplete document, and frequently poorly thought out as well.
As a white, male engineer, I haven't ever found a need for codes of conduct because I've had the privilege of never having been harassed. This isn't the case for all groups, so it's best to have one.