Gotta throw in my $0.02 on this one...
-Sure, waiting is better.
-Sure, it may cause some stress to the plant.
-Sure, it will have an impact on total yield.
-Sure, the under developed buds will not be as potent as more mature... Duh.
However, the simple fact is that many, many casual growers, fist time growers, etc will take numerous little "sample" buds from both a curiosity and experimental perspective from their plant, whether they admit it or not. It's not much different than taking a flower from any other "Annual" plant. The stress, assuming a healthy maturing outdoor plant, is negligible at best. A die hard pro may tell you otherwise and we are all entitled to our opinions and I respect theirs, it's just 'fun'damentally different with the casual and or first time growers, with the emphasis on 'fun'.
Using myself as an example... This is my first grow in several years, but I have grown many times in the past and this is aso my only current source of herb and I only grow for personal use. Have I been somewhat impatient? You bet. Is it fun? You bet. I have both this year and others taken samples with NO discernible impact at all, although in all fairness I did not have a 'control' plant for comparison, and I have also learned in the process which is a definite plus. While I am sure there is stress to the plant, again I view it as negligible. The way I see it, these are my plants that nobody else even knows about, and I grew them to provide me with some product, so why not since I understand the relatively small negative impact? I should also add that I have always been pleased at harvest time.
Of course I have been wrong before, and will be wrong again, but this is my opinion which is backed by personal experience.