I don't know, it would have been tough for me to walk away a decade or so ago.
Nowadays, I would just reply with some sort of condesccending smart ass remark. Of course I'd have to make sure there were witness' in case the mentioned big, bad bully decides to punch a paraplegic. Now that would...