I very rarely experienced anxiety, except back when I was younger but more social anxiety... I loved smoking from the minute I tried it, including by myself as I would self medicate from the first time when it took my migraine away. Since about 13 yrs old Ive only ever stopped smoking twice for an extended period of time... Both times the first time smoking since returning I would roll a fatty to the face probably not the best idea but you want to celebrate and I love joints/blunts... But either way, I feel like the tolerance break induced a small bit of panic those first times coming back to smoking and one time started to spiral out like a bad trip, I was able to stop it real quick... But I could easily see how something happens to someone, they attribute it to smoking, then stop for a bit, try it again and have a lower tolerance, then retriggers that panic attack, reconfirms their belief that its the weed causing it... I can easily see how it can spiral out from there. I have a really good friend through out high school we would meet every day and smoke at least 3-5grams between us every night after work for years and years... He got a panic attack one time at my house, got through it I didnt think it was big deal... he was convinced it was the weed... quit weed, started drinking.. he tried smoking a few months later and got super high, had another panic attack.. Although hes the hypochondriac type so I can see how it happened, but he was also someone I considered a heavy smoker, and someone I could hang out with and they would be able to keep up with being passed joints and blunts all night no problem, then out of no where thinks they're having a heart attack..
I've experienced it that little bit and it can be scary... My suggestion would be to practice snapping out of it any time a smaller panic attack comes up... and if you do smoke... take it very slow... like one or two hits and wait a while slow. Once your tolerance builds back up you can enjoy joints again.