While leaf is a little harsher than bud, for sure, I also find that this is somewhat dependent on how long the leaf has cured.... well cured bud with a little leaf tends to still be pretty smooth, but uncured leaf is atrocious. I assume this is because of the cholorophyl content still present in the uncured leaf.
Personally, I always remove SOME leaf, for a handful of reasons...
I start smoking my weed right away and never really wait til after cure to start toking on it. (I cure as I go, obviously)
Bag appeal is also a big factor... many people live by the rule of thumb that they want their weed to have as much bag appeal as possible, even if they're not moving it, because of the show-off factor. If it IS moving, bag appeal has a low leaf content because, as a matter of rule, the leaf simply contains less THC than the bud, and is considered the inferior source of weight across the board in both the medicinal trade as well as the high end black market.
Also, it's fun to make extracts like hash and kief, and it feels more rewarding when you know you've picked off the less valuable part to do so, rather than sacrificed smokable buds.