Bathroom vent fans are not worth a shit unless they are mounted properly & the base plate is used,without the base plate very little air will be pulled from the room,the base plates are designed to create a vaccume,without it the blade spins with no suction.
I'd look into how you have it installed before i changed anything, but be forewarned,bathroom fans are not designed for constant useage & are known to catch fire.
Anyhow whatever you decide to do about ventilation you must have 2 things,powered exhaust (you allready know this) & at the very least a passive intake to allow fresh air to be brought in,relying on air to be sucked in the cracks is not the most efficient approach & will cut the cfm's of your exhaust,if your going to be running a 4" inline fan then you should have a 1" passive intake cut in the cabinet but preferabally a 2" hole for an intake,if the light is bothersome you can make a light trap quickly & easily out of pvc plumbing parts from any hardware store,the trap can be made exactly like whats under the kitchen sink,no light will escape but due to the negative pressure in the grow room being created from the powered exhaust fresh air will be oulled through the light trap.
The fan in the pic is a good reliable fan but its noisy as it sits out of the box,i measured the noise level of that fan at 71 dbl's which is a pretty irritating sound level,if you wrap the outer hull of the fan in several layers of quality heat resistant duct tape it will lower the sound level by 10 dbl's,every 10 db's the noise level you hear is doubled, so by lowering the noise level those 10 db's you'll be cuttting the noise you hear directly in half,its a cheap tweek that should not be overlooked.