That's really a question that you should answer for yourself.
Do the plants smell? If so, how strong is the smell? If not, are the plants visible from a place in their driveway? If so, do you have somewhere else to move them to? If not, are you comfortable checking on your plants with them being so close to your neighbor's house? If so, leave them there. If not, move them.
I think we've covered all bases. Good luck making the right decision.