You are 100% good to go. Flower Week 5 through harvest SHOULD yellow out the fan leaves eventually as nitrogen is depleted in the soil. In fact, I would not pull those off but instead let them get used up until they are literally brown and getting crispy and hanging by a thread then go ahead and pull.