Hairs in the buds will likely happen, I don't give a fuck about a couple hairs in my buds if that means my pet(s) can chill with me while I work, but the main issue is pests. They can harbor and transport some nasties in to your grow room, especially considering the season. And this isn't limited to cats (or other pets) that go outside, indoors pet can have mites, for example. So hit you pets with some frontline, comb them a bit, and just generally keep them clean and it should not be a real issue.
Also watch out for a cat that will eat your shit, my cat will never knock anything over, but he will eat my plants, and drink water from my rez (clean water only, but it's still lame). So I don't let him in there, but my dogs chill with me all the time, never had a problem, I think lol...