Thanks for the input. I started to work on it today. I had some egg crate foam and some one inch thick styrofoam sheets that are like 3'x6'. I tried to insulate better in between the joyces above the fan and I wrapped the fan with the foam. I think I had minimal success so far. I need to build a box for the fan I think.
The fan is setup on the outside wall of the grow room, I just cut a 6" hole to put the flange through and then I mounted the bracket on the joyce above. Then the carbon filter is right on the fan also on the outside of the room with the air being sucked out of the room, if that helps any...