Don't mess around with trying to find different parts, just get an exhaust fan that has the proper sized outlets/duct work. Square footage only gives one part of the equation, you need your total cubic feet. I dont' know what indoor growers use as a rule of thumb for exhaust rates, but in aquaria, depending on what you're keeping this is how we size pumps.
So, let's say your room is 30 cubic feet, I'm gonna guess that you would want the exhaust to turn over approximately 3x that volume, equaling 90cfm (I'm just pulling that number out of my head, you only gave square footage, not total volume). That gives you your base cfm, yeah? Googling 'best price' + the cfm you want will give you lots of results, especially of bathroom fans that have the 4" ducting already installed.
Also, there's no reason why you can't work with something like Schedule 40 pvc to make the duct work, is there? Should be able to find reducers that would take you from 4" to 6".. now that I think about it you would probably find that in the black pvc section of your local hardware shop, and I can't remember what the schedule of that stuff is, but you get what I mean, yeah?