Actually it does matter...a bit but it does. Best time to harvest is after their dark cycle, preferably a longer one than usual. Trichomes (where the THC is mainly stored) degrade during the light cycle, once in the dark they start to regenerate, starting to degrade again once the lights come on; so the best time to harvest would be in the early morning hours after your dark cycle, trichs at this point will be at their max potential. Also dry your bud in a dark place to prolong the potency potential. Hope it helps mate.
P.S.: It's not like it's a huge diference but for ME it's enough to make the effort and wake up early. Last harvest was at 5:30 am, just when the lights were about to come on.