I like the fact that you are willing to question conventional wisdom and say "why not?" Here is my take.
1) Matter is never created nor destroyed - only moved. When you grow, all the stuff you add in to your soil has to go somewhere. Some gets absorbed into the plant material, some gets flushed out when watered and some get stored in the soil. The latter is especially true for salt based synthetic nutrients.
2) Your harvest physically removes plant material (stalk, buds, leaves, etc.). The building blocks used to make that material came from your nutrients and your soil mix. What remains after the harvest is a net delta of those building blocks that were removed.
3) The Nitrogen reserves that was present in the soil is used up during the flowering phase of the previous grow (by design).
4) Pathogens, molds, and other pests that thrive in the soil will persist after havest - some of it not ideal for growing (hence the reason to use sterilized soils).
Sooo... if you want to use the soil again, you will need to add the elements that were removed when you harvested. You will have to deal with the potential for the molds and bugs. You will also have to adjust your PH buffer as most of it would have been altered in the previous grow. All of this are possible - if - you could accurately measure and understand what was missing and replace those elements. Otherwise you run the risk of plant issues due to the missing components. Personally, I never reuse soil indoor potting soil because the cost/benefit of doing so versus the risk just does not make much sense to me.
Regarding the outdoor comparison - it is not a like for like comparison as with outdoor grows, you really do not have a closed system with a controlled in/out of added nutrients. Roots can extend well beneath the holes planted for them. The earth itself, rain, run off, etc. all mix in to even out the stores of nutrients that the plant can use. Ever see what happens after a crop is harvested? Farmers typically burn the fields and mulch it up so that that matter is broken down into something that is readily absorbed back into the earth on a macro level and able to sustain another harvest. We eat the food that is taken from the fields, and poop out the remains, which is added to the collective that makes it way back to the earth. Within a closed system, nothing is created or destroyed - just moved.