If it turns out to be a big enough problem, Google could probably add a way for people to run Glass off of a throat mike [0], simultaneously solving the "crazy bluetooth headset person" and "hostile takeover" problems.
Given that there's a bone conduction speaker, I rather assumed that Glass was going to pick up on bone-conducted vibrations to make sure the wearer is the one speaking.
0: http://en.wikipedia.org/wiki/Laryngophone