in receipt of fit note and being hauled into the job center AGAIN...

Hi folks, I'm in receipt of a fit note for CFS and IBS. I have been left alone by Brighton job center for about three years with these difficulties going on.. I was called in randomly the week before last, for ten minutes, just to be asked - anything to report?
Odd that considering I have a UC journal to write in. I said a worsening of conditions and left. Anyway, I've now been informed that I have another ten minute in person appointment - quote: Attend your work search interview in a job center. This is to review what you're doing to meet your commitments and agree next steps to increase your chances of getting work or earning more. If you miss this mandatory UC appointment without what we believe to be a good reason, your payment will be affected. This is called a sanction.
So, at the moment I do not have any work search requirements, they are turned off. I've had brief appointments about every six months on the phone for three years.
I replied saying I'm concerned about getting to the appointment, because the day before I have a Dr.s appointment and I am likely to be wiped out after it for a number of days. I also asked if something had changed when - I got a reply saying would you like a work capability assessment then you won't have to attend the job center every two weeks. So I asked has something changed - I was not aware I had to do that? that has not been the case. - so maybe a mistake of that person or?? Are you folks at BW aware of any changes?
Two questions please, and anything else you consider helpful or relevant..
With a fit note am I going to face consequences/sanctions - when unable to get to the job center?
...I would give good reason as far as I could, like I'm wiped out, in pain, had two hours sleep, that kind of thing.
-What kind of protection does one have with a fit note?-
(I was doing a Very small amount of gardening work but have paused due to a knee injury flare up for the past three months.)
I just need and want to be left alone really, not prodded and poked and disbelieved by an assessment. I'm keen to work as much as I'm able to, and wondering what to do to get out of awful poverty, but I find the molestation, control and desperation of the job center staff to feast on my flesh unbearable. Like every second they want to control given the chance..
Thankyou for your time!