It basically just depends on how big your plants get (which wrectly affect yield). l have 4 white lavender clones in flower right now (like 5 days in flower) in 7 gallon smart pots.
Let me start from the beginning it'll be easier to explain. I took the clones and planted them in four 1 gallon smart pots. I let them veg for a month, then transplanted 2 of them to the 7 gallons (bc I only had enough soil for 2). Two weeks later (@ 6 weeks) I transplanted the other 2 into 7 gallons, and could already tell a size difference. Two weeks later (8 weeks veg) I put them in flower and the 2 I had transplanted were literally like twice the size. Granted, the smaller ones shop up pretty quick in the beginning of flower, and are almost as tall but just don't have as much mass, as many branches, etc.
So basically, yes a smaller pot can stunt your growth if a plant's root mass gets big enough. But, if you're only vegging for like 4 weeks then it won't matter as much a especially if you use a smart pot. They can hold bigger plants than regular plastic pots.