The first times I shroomed were after many, many months (close to 6) without smoking pot, and it didn't have a negative impact at all.
Many people shroom who don't even smoke pot, so it's not an issue. Having a bad trip doesn't come from external elements, but internal conflicts based upon your own psyche.