You should have a place where air comes in and a place for air to go out. It's best if you can blow air in from the bottom and exhaust heat out near the top, since the heat will rise anyway. Or you could move to a better growing location that you could ventilate easier, like you said.
If you have a fan right up against the wall, it is not going to work very well, the fan needs space behind it so it can pull air through to blow forward. Try to have a fan blowing between the tops of the plants and the lights, blowing hot air up away from the plant tops and upward to an exhaust at the top of the grow area.