ESA - DWP ERROR & DEBT REPAYMENT

Hi, I applied for ESA when I was first signed off as Medically Retired from being diagnosed with Fibromyalgia. I gave them all the information and my Certificate from my Employer that said I was now retired on ill health and I would now be in receipt of my pension. I did a Benefits Check which said I would be eligible for PIP & ESA - the support element I think it was. Fast forward a couple of years when they first contacted me to say I was not eligible for the ESA Payments I had received after all. I rang and wrote back saying that it was DWP who said I was eligible and that I had told them that I had provided them with my Certificate that said I was medically retired and would now be in receipt of my pension. I heard nothing back after pointing out it was their mistake not mine so I should not be penalised for that. Again no further contact for a couple of years and I just assumed that, that was the end of it until I received a letter from them on Saturday saying that I needed to pay back the £7,772.60. Deductions would start in December but they have actually made the first deduction in November. This is their mistake not mine which they acknowledge and I know the Government conveniently changed legislation a few years ago that said even if an error has been made by DWP the claimant has to pay the money back. So my question really is what can I do to fight this? I can’t afford to lose £144.72 from my PIP every month for the next 4.5 years. Can anyone please help me on how I can fight this. Thanks