When you transplant the plant will basically stop growing for a short period. It is going to happen no matter when you transplant though. Basically the shock of being pulled out of the ground(the pot) and put somewhere else takes a toll on the plant but it will recover and growth will start again. If you plant looks like crap for a little while after transplanting don't be worried it will spring back. Its ultimately up to you, I personally wait till flower begins(mid week 3) because I can not afford the extra space/soil that transplanting males would require. To each his own though, transplanting can be a little scary!