Using a filter doesn't take much away from the smoke and if you've got some harsh smoke then using a filter makes it easier on your throat and lungs...
I don't know if anyone that is saying not to do it has ever done it... if you haven't then how would you know what the heck you're talking...