Your yield depends on a number of things, of course, strain and cultivation as well as light. Growing with an HID lamp one expects to get about 0.5g per watt. Good growers can get 1g per watt. You are not able to make full use of your lamp with a flat canopy because of the narrowness but you can make up for some of that by putting reflective material on the two 4' long walls. HIDs are just great
You want your light as close to the plants as you can without heat stressing them. The hand test is too subjective. I start a bit high and lower the light a couple of inches a time over several days. When the top leaves begin to show the first signs of heat stress - edges start to curl up and the leaf may canoe - back off a bit. If you run cool air across the tops of the plants you can get closer and because of the inv sq effect, just one inch can make a significant difference.
One issue you haven't discussed is cooling. A 400W in a closet is going to need some kind of cooling
PS
Your plants look very good. Not a big yield but it looks like everything went right.