This can work with a living soil.
If you're running something more sterile, outdoor dirt invites guests such as weed seeds and bug eggs. Without a living soil to guard against, say, spider mites or aphids, it could become a hassle.
Either way, it can be a risk (and one which I take regularly).