Perhaps it's time to move into a single family house? I haven't lived in an apartment in years, but I remember I didn't like it very much back then. If it's not kids bothering you, then it's some drunk asshole working on his beat up Camaro with 20 friends spilling oil all over the driveway, or the girl next door that just loves to pick fights with her bf at 4:30am, or the barking dog that won't stop until his owner comes home, etc.
I assume you rent, that is. Can you find a small house for rent? Or a cheap foreclosure, and just buy it? Then you don't have to deal with it. I have no neighbors within viewing area, that's why I do what I do. I would never grow if I had people around me all the time. I have 8 gardens that I work on all summer, all fenced in. If I found any kids in my yard, I'll just let my dog at them. Considering he runs like a cheetah, no one will get very far.