First of all, it is shitty and not your fault that you have to work in a society where "your ass is on the street", and you have to be so paranoid about not getting fired from your current job.
It's entirely believable what you said about smelling like weed even after some minor contact with the plants. Especially people who are non-smokers (of tobacco or cannabis) will very easily be able to pick up the smell. I've also found that the smell is quite "reluctant" to leave, even after some good hand washing.
I guess you will have to suit up before tending to the garden. That is inconvenient, but consider that some people have to do that routinely for work every day (e.g. one of my friends now works in pharma production, and she takes 30 minutes every day to suit up before getting to her station; or think about the surgeons who will have to do that all the time).
If I was you, I would only go near the garden AFTER work, or at times when you're partaking anyway, and the smell from the plants won't make a difference. Automate the garden (esp. the watering) as far as you can, so there will be less reason to get in contact with the plants.