Several reasons, but I think the thing that bears mentioning most is that most casual fans of something don't go on about it in conversation, so you don't grow to identify them as an (insert media here) fan. It's much easier to associate the behaviour of people who are obsessive over something and wear it on their wrist with the fan base as a whole, and they tend to be the weird ones. Another thing worth thinking about is that people who are openly enthusiastic about something that makes you a "nerd" or something like that are often instantly dismissed as the typical undesirable nerd by a lot of people, so it's difficult for dedicated fans of a lot of things to have as complete a social life as most people, typically staying within their own little cliques; a big problem for manga/anime fans, as that means they're amongst a lot of other people who are likely to take social cues from anime and manga and accept the quirks of people who do the same, which isn't really conducive to a normal social life.