I really hate that It's come to this, but I don't think I can pick out all the caterpillars by hand this season. They can be so annoying, but they grow up to be pollinators and are crucial for our environment, also they are rather cute and friendly when they're bumbling around in your palm and not eating your crops. I know that the large spooky horned ones grow up to be the beautiful sphinx / hummingbird moth. If I can see any, I will grab them and put them in the flower garden, but as of tonight I will be experimenting with an unconventional bacterial control agent. I'll try to post results, but harvest is still quite a way off, so no promises as of yet.
Protip: DO NOT REMOVE THE PAPER WASPS FROM AROUND YOUR HOME. Paper wasps prey upon caterpillars. If they are too close for comfort and they are attacking you, then you must do what you've got to do, but not many people realize that paper wasps are beneficial predatory insects. I have a ton that I was going to remove, but now I will not, as they are not bothering anyone, even though they scary to some visitors.