Hi Justme
There is actually little to worry about as you have not done anything wrong and official errors with ESA can't count against you.
You can tell if you have been receiving Contribution Based ESA by looking at the P60 that the DWP send you annually. If there is a figure in the box Pay and Taxable Benefits that will relate to Contribution Based ESA.
Let me know how you get on
David
Nothing on this board constitutes legal advice - always consult a professional about specific problems