A closet full of clothes and ripe weed smell ... damn dude, you should drill a hole in your grow wall and turn that shit into a glory hole.
Fun aside, I'd be paranoid too as you may not smell it, but to other people you stank like a stoner etc. What about those vacuum bags you get for clothes? They'd not only offer some degree of protection/freshness, but it would mean the clothes took up like 70% less space, meaning more room for plants