I wouldn't assume that the cracks will be enough, depends on how well your house is built really. The problem with that is you have no control over where the air comes from. The air leaking in would be going through the walls, warming up slightly as it goes, making it less efficient than having some way of ducting in cool air. Why not just get an air con with a fresh air intake? That's the best way, cooling the air as it enters the room rather than letting it seep in and then get cooled. I thought that was what you meant with the switchable panels before tbh.
Don't get an intake fan, if it is smaller than the exhaust it will limit the amount of air coming in to whatever the fan can blow + whatever gets through the cracks. If you move air out, more will come in if it has a way to get in in sufficient quantities. How leaky your room is compared to how powerful your fan is will determine how much the pressure is lowered.
If the pressure drops significantly then the air becomes thinner, just as if you go high up a mountain. So it contains less carbon dioxide and oxygen atoms per cubic foot. I haven't come across this situation because everyone I have seen growing either has a way for air to come in, or they have an overheating room, but I would speculate if you were able to move enough of the hot air out to keep temps in check then you could conceivably approach a point where the plants begin to suffer as a result of the lower co2 levels. What that point would be, I wouldn't like to guess and I'm to wrecked to do the maths right now.
I could be wrong, it could be that by lowering the pressure in the room the air coming in would simply travel at greater velocity and at least come close to equalizing the pressure, keeping it at a constant, workable level. I have a friend who's last grow, before he moved, had such a big fan in it (bloody great steel industrial unit, 12" ductwork) that when you opened the door, inwards, you had to make sure to hold on tightly to the handle as you turned it, otherwise the door would fly open if the fan happened to be running, and that was with a decent amount of passive intake. That fan was the very definition of overkill though, I doubt you'll be extracting quite so much air.
I just re-read your q and if the fan is only coming on at intervals then the pressure would equalise in the time the fan was off, meaning most of this post is irrelavent. I'll leave my thought process there for you to see, I've typed it now...