You may not think your event needs a code of conduct, but it does. They express, in simple language, the required behaviour for all attendees, speakers and organisational staff. They help people of under-represented groups feel safe in the knowledge that harassment will not be tolerated. They help to build trust in your event.
You're not the first person to want to create a set of clear rules. A good place to start is http://confcodeofconduct.com.
When people break the rules (and they eventually will), you want to hold them accountable. You can only do that if it has been clear what the Code of Conduct is. Don't hide it away. Link to it in the footer. Talk about it (even a little) on your Call for Papers. Mention it on your conference/meet-up home page. You're telling attendees that you care about their safety. You're telling everyone they need to behave.
The only way you can enforce a Code of Conduct is to allow anybody to contact you when they really need to. List contact details of you, and at least one other person (who knows how to contact you when all other methods fail). Many meet-up groups/conferences etc. have multiple organisers so that shouldn't be too hard to do.
Your highest priority, as far as a Code of Conduct is concerned, is making sure that people are safe. Sometimes that will mean asking someone to leave. Sometimes that will mean reporting harassment to an authority. Don't wait to act. Make people safe and then seek out the facts. Hopefully it's all a misunderstanding. But you don't want something serious to go unattended.