I'd be inclined to leave them to drop off on their own. The plant will remove most of the nutrients and use it for new growth. The only time I'd consider removing the leaves is if they are blocking light from reaching the budding sites. Happy growing!