So it had just rained when I took the pics. Should i remove the affected leaves
Affected how? Leaves with mild to medium necrotic lesions or anything still green(ish) leave on. Any leaf that's hosting PM with a "white dusting" is matured and capable of reproducing. These gotta go asap! Search around for a foliar solution product. Solutions with a high pH...ie: silica products, baking soda, ect.
There's also microbial products, mainly bacterial. These can also be included into an existing IPM program. If you choose microbial, I'd suggest combining the microbial solution into the silica and use as one. Microbial can be very low pH in origin and PM thrives in a acidic range. This is the reason why you have PM. Your plants internal pH is acidic, a result of its influencing conditions, (soil, nutrient availability, pH).
Edit: any leaf that is heavily affect with necrotic lesions remove as well. Helping to open up air flow throughout. Try to target overlaping leaves as well. Leaf laying on leaf will certainly trap moisture and this is where PM takes hold.
Address your soil as well. Take some pH readings. You definitely have nutrient availability issues going on. A plant in poor health will succumb to any foreign body.
