Put a pvc 4 way on the pole. 1 side holds the fan. Other side, put a cap on it and thread a knob head bolt into it. If you size everything right you can just tighten the knob to clamp the 4 way to the pole. I've had some pretty strong threads in PVC caps on other projects, I think it could work.
www.rollitup.org