When my plants are in veg I hardly need to adjust pH, it's when they are blooming and gaining weight fast that the plants acidify my nutrient solution FAST. My pH will drop from 6.0 to 4.0 in 24 hours if I don't dose it with pH raise often and the controller is perfect for this. I don't know exactly why I see this pH drop but it only happens when the plants are doing great so don't mind. I wonder if the plants excrete acidic waste, or if they are just removing so much minerals by feeding that the pH drops? Anyone know?
When in veg, your plants are sucking down nitrogen; removing nitrogen (from ammonium nitrate or potassium nitrate) doesn't shift the pH quite so much. In these forms (ammonium and nitrate), nitrogen isn't much of a buffer system.
Phosphates, on the other hand, are very good for forming buffers. Start removing that component of the buffer solution (the "P" in N-P-K), and the pH will go whack. Of course, what does one look for when putting plants into bloom? High phosphorus- the plants are pulling more phosphorus at that phase of their growth.
Out of curiosity, it would be interesting to see what happened if you were to add primarily phosphorus during the flowering phase. Anyway- my answer would be that it's your second theory: the plants are removing specific minerals, and those happen to be part of a buffer system that maintains the pH at a higher level.