Unless the soil you will be growing in comes from the planet Krypton or you will be growing next to a nuclear power plant then yes using water only and no ferts/nutes will effect both your plants growth and bud production in a negative way.
I'm really not sure how much smaller they would be man, is it true if you kept feeding your plant through out flowering that it would take longer to mature?