If I lived in a very small town(3000 or less) I would worry, other than that no. The postman if so inclined, could check the address from sender and narc you out. That is very unlikly imo, unless you keep getting envelopes one after the other from UK and he is a no life asshole.
I live in a city of I guess 20,000 and am a med patient, so I do not worry a lot about it. The DEA does not go after people that get caught importing small amounts of seeds, you just get a letter from customs saying they took your seeds.