Not Drinking in Work Settings

So, I travel a little for work... and invariably am in situations where people are out having drinks. It is always so conspicuous that I am not drinking (I just choose not to drink for personal reasons I'd rather not share). Now for all of you who are going to tell me that no one notices or cares - unless I'm out with my friends who realize I don't drink, I get comments - usually just a few and minor, but it makes me feel awkward. Today I just told people I was going to be working out after the reception (which is true, I just worked out), but there are only so many excuses you can give... and if I say I don't drink then people nearly always ask why and/or get defensive about their drinking (and I could care less what they drink, my not drinking is no reflection on my feelings about others drinking). That is definitely not what I want to get into in a professional setting.

In a personal setting I am happy to explain myself - I just feel odd in a work environment, especially when people drink heavily at conferences or other functions.

How do you handle this? Anyone else feel the same way?