Id definitely keep notes of what gets eaten outside. As long as what they don't eat is a good plant then I'd probably grow more of those and less of the tasty ones. What I have found when I deal with caterpillars is that they were all planted as eggs from a flying mother. Then when they start to hatch obviously they eat what they are on. When I have ran multiple outdoor plants I find each plant either have none, or they have a ton. As you follow them along you can see them growing in size too, usually starting at the size of a grain of rice.
They are extremely resourceful too once the foods gone, they will move on to other plants. I had an outdoor plant drying in my closet and I'll be damned if a worm didnt drop off the plant (it was almost dry so not very edible anymore) work his way at least 8 feet across the carpeted floor and onto a seedling that was only 3" tall. How thats even possible I still don't know. The fucker either saw it, or smelled it, or both... but theres no way I can see that he stumbled across it on accident. Emily and I both dealt with worms that year, it wasn't pretty
I've been spraying mine every week or 2 with neem and keep the surrounding area covered with deterrents.