PIP Mandatory Recon

Ok, here goes, this is all about my eldest brother. He and is his family are suffering from stress, worry and quite plainly I am very worried about my brother. As a child he was on medications for autism, and ADHD. He was in special schools and an residential patient at the Charles Burns Clinic in Birmingham around the age of 5. I applied for PIP for him since as a family we are all getting older and are struggling to give him the support he needs.
He wasn't even given an assessment for his PIP, because low and behold 50 years ago doctor's didn't put anything in his notes about his conditions.
So after putting in for a mandatory recon, we went to the doctor's, who incidentally is not sympathetic to him but she referred him further on.
After an assessment at one place PIP decided that he should have a f2f, but not at home as originally requested but in Birmingham, near rush hour time and with less than 2 weeks notice.
We emailed them giving my mum's phone number saying that it would cause him too much distress, and that he couldn't plan or carry the journey out without someone doing it for him. He has had a phone call to day from them asking him to go to the Walsall assessment centre to get a note off their doctor as to why he cannot attend. He has gone mad, very upset, he's really frightening and can be violent, he refused them and said he would go to his own ( unhelpful doctor) I am really worried for him. Why are they being so nasty? We are only trying to help my brother...
Any advice pls? When he was a kid he tried to commit suicide, this whole experience which I wish I had never started is pushing him over the edge.

Thank you Gordon for your prompt reply. We shall try to get a very clear to letter. Can they really ask him to go to a doctor of their choice to be assessed whether he is fit to attend a f2f? This is what they were pushing for, which really does pose the same problems.