You wouldnt be the ahole, they would. I wouldnt help anyone that said that to me. Live for your self and dont change your plans. In fact tell them to suck it when you go. If the people you work with actually believe you are the reason they dont get a vacation then they also have a problem.
RUN...