i worry more about the residuals left behind on the walls and lights. all that nasty nicotine on the bulb can't be good. i do find myself smoking in my room though.
the only thing to be very careful of is if your smoking mids or anything with seeds, spores from your bag can be transfered to your flowering plants producing seeds.