I guess if you happen to choose the wrong male, while only choosing one, then your seeds will reflect this and be a mediocre batch, possibly. Sure open pollinations create the greatest amount of genetic diversity, but is that what we really want? So if a few of the males are undesirable, by using a number of males, then you end up w progeny from varied males all jumbled together into the seed batch. Selection is already difficult enough. I'll also cite that DJ Short does not use multiple males on a project, perhaps for other reasons than mentioned but thought I'd just throw that out there.
My tendency is to work a line, pollenate all the best females, but with only one best male. [keep in mind, pollen can always be collected from other males and stored or a target application.] You'll still have plenty of diversity by only choosing one male along with multiple females for a project, without all the randomness multiple males bring. I believe this simplifies by providing a basis of some degree of male continuity while working towards an f2 or f3 version. Not everyone will agree with me, but I hope my perspective adds something.