Several reasons, but I think the thing that bears mentioning most is that most casual fans of something don't go on about it in conversation, so you don't grow to identify them as an (insert media here) fan. It's much easier to associate the behaviour of people who are obsessive over something and wear it on their wrist with the fan base as a whole, and they tend to be the weird ones. Another thing worth thinking about is that people who are openly enthusiastic about something that makes you a "nerd" or something like that are often instantly dismissed as the typical undesirable nerd by a lot of people, so it's difficult for dedicated fans of a lot of things to have as complete a social life as most people, typically staying within their own little cliques; a big problem for manga/anime fans, as that means they're amongst a lot of other people who are likely to take social cues from anime and manga and accept the quirks of people who do the same, which isn't really conducive to a normal social life.
I know a lot of hardcore anime fans, but honestly I think its just a matter of how you perceive them. Like, I think the ones you find annoying are the ones that stand out the most, so immediately you pay attention to them more and you completely ignore the others that you would find cool/alright.
Like I have friends who are anime fans and those who I dislike. They talk about anime a lot but I guess the way they talk and the way they do things makes a difference.
I know a lot of anime fans that seem to be recluse in terms of their demeanor, but actually they're very social if you spend time with them. A lot of them do talk about anime a lot but usually they only do so to people who they think/know are interested in anime. They would have other interests aside from that, its just the matter of finding other things that interest them.
Its the same for a lot of people who are interested in many different things. For example those who are into medieval things, or DnD, or whatever.
Personally I dont like girls who talk about Yaoi and hot anime boys because it reminds me of my 13 year old self... HAH. (Also partially because I know girls that should start liking real boys and not have like an uber high standard where their types are the same as the ones in anime, so they can stop whining about not having any boyfriends... Honestly, like facepalm their life.)