Now you got me wondering??? Are you growing in pots or raised beds, where you finding them worms??
If you have a worm farm than your looking for the best environment for them. Heat is not good for worms, thats why you don't see many during the day sunning themselves LOL serious, they like it cool. Living in a pot plant pot is the kiss of death most likely.