Yes, entirely depends on who it's coming from. If another pothead called me one then obviously it wasn't meant to be an insult... Even if a straightedge person calls me one I don't necessarily take it as an insult.. I'm just like ... yeah ... I guess I am a pothead? Some people do imply it as a bad thing though and it's easy to tell if they are.
it depends who's saying it. if it's my parents, yes I would be offended. my friends and the such, I would just reply with "I know", smile, then take a hit from a joint...