It can take anywhere from 2days to 3 weeks for your plant to actually switch to flowering. SOMETIMES LONGER.
You're fine switching back. Just try not to let it happen over and over, you don't want to confuse your plant.
But from what you said, you are 100% in the clear.