Go for it, they wont get any smaller, you wont lose anything.
Keep an eye out for bananas though, confusing light periods is the main cause of hermaphrodites.
If you see a girl growing balls, you might want to decide it's time to harvest before any seeds appear.
Or, you might want to stick it out and try to pluck off as many of the balls as you can.