If you're growing outdoors as well, you should try talking to him about it. If he's collecting pollen and knows what he's doing, it's probable your plants will not be affected. If indoors, keep your trap, as well as any windows pointing toward his plants shut. Don't look to take any highground regarding your own grow though, males are equally important in the world of marijuana, and the best growers out there make full use of them.