I've been fuckin with a dry-vape lately, and I get a really pronounced perfumey-soapy first rip, every time, with almost ANY herb. . .. I think I get what you're saying. That perfumey-soapy vibe tapers off right after that first rip as the temp of the flower increases.
How are you getting that flavor, out of curiosity? Joints? Pipe bowls, bongs? All the above?? What are you using to light them? Might need more heat? Another possibility; you might be harvesting your weed a lil too early, consider extending your flowering times? --since I get a similar vibe when dry-vaping my flower, I think this is a strong possibility because I myself am NOTORIOUS for early pulls. Just sayin.
In short: Burn ur weed hotter, and flower it longer (maybe)