Hi Andrew
As it stands you do have to attend the appointment. If you try and cancel it, your papers will be returned to the DWP and you risk your case being closed.
I don't know why you were not granted a home visit. I don't know whether there was anything in what your careworker said about your difficulties that would have deterred them from offering you one. Some people with mental health difficulties are turned down for home visits. Your careworker could phone and ask.
If they have simply not taken on board the request and you have a supportive GP - then you should ask the GP if they will provide you with a letter saying that to go to an assessment centre would cause you "overwhelming psychological distress" (They need to use that term.) which would lead to a deterioration in your mental health.
If none of that is possible I hope you have someone who can support you on the visit.
BIS
Nothing on this board constitutes legal advice - always consult a professional about specific problems