There is nothing illegal at all and your neighbor really has no options, you can't get in trouble. That said, he is your neighbor ... you have to live next to him and in my experience having pissed off shitty neighbors never ends well (my sister's asshole neighbor was so bad that it drove her to selling her home and moving out).
Maybe show him that you are trying, that you've invested money in filters (with pictures I guess). He sounds like he has uh other problems and you are probably just getting caught up in his shitty life, but if you at least show him that you're trying; if he is a reasonable human being perhaps that will help help him understand and mend relations.
Also I don't see how small claims court would get him anywhere because he would have to prove that the smell was coming from your house and how is he going to do that? Perhaps it's coming from his other neighbours house, or the one across the street, or maybe he has a family of skunks living under his porch? Also you have to prove damages and an annoying smell isn't a damage as far as I can tell.