If you are a small time grower with a few plants, they would probably just bust you as soon as they had enough to get a warrant. I doubt they are going to go to the trouble to try and time it 'just right'. If they wait, you might stop growing for a while, or move. There's no reason to think you are in any special kind of danger.
You do read about busts in this forum but considering the number of people growing, its pretty rare and almost always because the grower broke some a stealth rule, usualy the 'don't tell no one' rule. Sure you could be really unlucky. But if you are going to be really unlucky there's all sorts of disasters waiting in the wings. Are you fretting about those too?
This is fear and the only way to deal with it is courage. You've decided to do something risky and you have no choice but to live with the uncertainty. All you can do is take the right precautions and hope it's enough. If you let fear get a grip of your head, it will start churning these paranoid fantasies such as "...I'm busted. They are just letting me stew until the right moment...."
You probably drive your vehicle everyday. This is a really dangerous thing to do. There is a significant risk of serious injury or death even a short car trip. How do you handle this? You do your best to make your driving safe and for the most part you don't worry about it. What good would it do to worry? You've decided you are going to drive. You've done all you can to minimize the risks. After that whatever is going to happen will happen and worrying does nothing except frighten you.