You did the right thing! I don't care who is knocking on my door at 12AM, but that door is not opening under my control. It could have been a serial rapist, a drunk asshat, or who knows? When you call the office in the morning, if it was somebody sent by them, express your extreme displeasure...