..... even the tomatoes i germinated in there got some sort of fungus on the soil after a few weeks.
What did you germinate them in ? If it was regular potting soil I strongly suggest you try again with Jiffy Seed Starting Mix in the same closet as a test. I have been growing several kinds of peppers for years and some take a very long time to germ compared to other veggies. When I used potting mix, they would damp off and some pots would get some kind of fungus etc... Ever since I made the switch to Jiffy years ago my success rate has went way up and no more fungus...ever. It is more sterile than soil and if you can't tell by now I swear by it.... and I grow all kinds of stuff. I really doubt that it is the closets fault.