You need to remove worms by hand on the plant still out there. It's a pain but worth it. Once you cut the buds the worms will vacate on their own - you'll see them hanging by webs from the drying buds.
Wont be able to make it up this weekend. My wife gave me a bad cold.
If the worms are bad cut it now and save what you have. Early bud is better than worm eaten bud. And it looks great. The buds could definitely still mature a bit more from the pics but they are mature enough for an early cut with reduced yield. Do you know how many weeks they've been flowering?