At times, yes, people do judge you for it. Now, I find that the people who will go so far as to actually make negative comments about it are generally people whose opinions I respect, but not all of us are to the point that we can just choose to not care if crappy people judge us. I know I'm not.
Most people tend to work by association, which tends to include generalization and stereotyping... not just with anime, but everything. Someone who expresses an interest in anime can be associated with ideas such as lazy, childish, unable to deal with the "real world," socially inept, and the plethora of other prejudices held against "otaku" without any other prompting.

Because I've generally been around judgmental people, I try to be fairly subtle about what I display. I do carry around a Laughing Man keychain from GitS (I'm also a Salinger fan, so it fits), and I find that it tends to be just obscure enough that only an anime fan can pick it out.

I also sport anime wallpapers on my PC... although I make sure they're thoroughly de-sexified. Because for some reason, in the context of anime, an inch and a half of cleavage makes you a pervert. >_>

Asians generally can get away with it better, being a guy or girl, just how it works. It's also depending on what community you live in. If you live by a large Asian community, no one is going to judge you, because either them, or their sons/daughters/cousins etc are into that stuff, and Asians generally think it's normal to have cute stuff around.

For example, I live by a Korean community while going to college. I wore my lady imp hoodie by Yahtzee and the Korean lady at the sushi restaurant smiled at me and said how cute it was.

I went home about an hour away, went to Walmart, and had this white-middle-aged lady give me the nastiest look ever, just depends on location I guess ;x.

My professor, (well accomplished, married, good-looking, etc) is into anime, and isn't afraid to hide it. He also is into gaming XD.

I would also go to the Japanese market and sometimes see Caucasians dressed in decora/lolita/cosplay, and everyone else who is Asian doesn't really care or make a comment about it.

Anime has become a large part in Asian pop culture. I'd say that anime is actually well-liked for people ages 24 and under in native countries and in other foreign countries, (such as the US, Canada, and so on.) Internet access plays a HUGE part in this. The younger generations have become more apt in technology, not to mention it's become more readily available.

Anime's funnily become part of modernization as well, which I see a lot happening in Vietnam. Vietnam is speedily building its way up in hoping to match its great East Asian neighbors ;P. Asians also tend to ride the trends, and Korea is playing a huge part in it at the moment with its K-pop groups K-dramas XD.

The older population of all the East Asian countries obviously have stigma in one way or another with each other, (for obvious reasons XP,) but the younger generations could care less, and I've seen a lot of Japanese admiring the Koreans, and people in China and Vietnam obviously loving Japan, and so forth.

In the past, I've been criticized because of my love for anime. I went to a predominantly black high school (and am black as well) and so bringing manga in was seen as... well.. strange and 'white'. There were few of us who were gamers/anime lovers and we took quite a bit of abuse---verbally and in some cases physically. Still, we stuck together and did our own thing despite all the heckling it got us.

I also saw the same thing among some coworkers of mines---mind you this was about 9 or 10 years down the road from the high school incident. It wasn't nearly as bad as high school though. People just picked on me because I would bring in manga or watch anime on my ipod, but this time around I was much older and could see the dumbassery and properly ignore it. Why would I value the opinion of someone who would say things like, 'So can you can understand that Chinese they're talkin'?'. PLUS they always thought that whatever anime I watched was some kind of porn. Why though?

I likes what I likes and am not ashamed of it one bit. People are gonna judge you no matter WHAT you do in life, so why not just enjoy yourself before you become worm food?

Yes, which is why I keep it a secret to the people I know in real life. The only people that know I like anime are on here

This is generally my reaction when somebody says something about my love of anime