Definitely the bud worms. You will see pure white butterflies buzzing around during the day and those are the ones I have issues with. Maybe moths at night too but I see the white butterflies all the time in this area. They leave small translucent yellow/white eggs on the leaves and buds so if you can keep an eye out for them and destroy them before they hatch it helps a lot.
If you see any spot that looks like the pistils have been munched on, or if there is any damage to the buds at all. If a sugar leaf starts dying or you see a patch of mold, peak inside the bud to check for worms. They will be eating away inside the bud and you won't even know they are there until they have fucked up an entire top and eaten their way out. It's brutal. I would spray Spinosad and or BT weekly. The BT is special for budworms. Also look for bug poop in the buds and on the leaves. Find the poop and you can track the worms down quick.
In my experience with how dry it is here, spraying weekly won't cause mold, it's the worms that cause mold. Once it gets humid late in flower and the rains come, anywhere a worm was will start to mold up from the bacteria in their poop and from the damage they did to the plant. So basically spraying weekly will lower your risk of mold.