Biggest worry for me is impostor syndrome. I'd be terrified of turning up and being absolutely useless and achieving nothing. I don't go to conferences for the same reason.
There are a lot of people who feel like that, me included (at times I kinda expect my parents to tell me that I have been playing grown up long enough and that it is time to come home and go to bed at my old bed time).
My best recommendation is to go there anyway, and you will see that you can more than you will ever imagine (useful advice, right? just telling you to do it anyway). Also remember that it is absolutely okay to be scared out of your wits (relevant blog post http://headrush.typepad.com/creating_passionate_users/2007/0...) and do it anyway.
We're all humans and we all have some fears and insecurities. One practical suggestion: organize a private hackathon with people you trust like your friends or colleagues. This might help to build confidence for larger events.
Just by being there you get aligned with the culture of that project and exposed to the current challenges. That alone has value in preparing you for contributions, or allowing you to decide the project isn't a good fit.
