That's unfortunately not how breeding works. What you have is a cross, not even an F1 line.
I've used hermie pollen on many occasions and have rarely seems problems doing so. Sometimes hermaphrodism happens due to light feeding, over feeding, light interruption in flower and climate issues...