I've harvest parts of plants before. Some stalks had buds that reached maturity before other stalks. So I'd take the mature ones, leave the rest to grow a little longer.
I'd suggest though, not speed drying them. You'll lose sooo much in flavor and high.
Also, keep in mind that even after you...