You could build it out the way you stated, then add another 2 or 3 feet in front of the veg and flower room to act as a hallway. It'll only look like it has 1 door, and it gives you some room to store your materials out of the way, and out of sight.
I think this pic shows what I mean pretty well.