No dont harvest early ,
Just keep treating with neem untill their gone ,
You will regret it if you harvest early ,
If you can get some organic neem if your worried about the taste , it works great , i got a bottle from lowes for less then $10 , It was the extract bottle , not ready to use...