Definitely let 'em grow. My first grow, bagseed, 6 plants, two of them just like yours. Kept them and removed the "balls". Of course when you remove them, more will appear but as long as you manage to remove them it's ok. I got a few seeds in two of the branches but the weed tastes just the same and I didn't notice any difference in potency. Plus, if you have a female that is accidentally polinated by a hermie, you'll most probably get feminised seeds (that will still probably have the hermie trait, though).
If you can keep your growroom at a decent temperature without ventilation at the plant level, that will help too as the pollen won't spread that much in case a pollen sac opens (keep in mind that when your buds get fatter it will be hard, I'd say even impossible, to locate all of the pollen sacs. You're now 20 days into flowering so I guess you won't have much of a probem if you keep them. Cheers and good luck!
