missed tel assessment they don't have my number!

Hi, despite sending in a gp letter asking for a paper based esa review due to overwhelming psychological distress I was sent a letter informing me I had a telephone assessment on 21st September. I have never given them my number and the telephone number box was left blank. At the same time I had sent in a letter informing them that due to recieving a small inheritance I was now over the threshold. I then recieved a letter demanding I explained why I had not taken part in the telephone assessment. I have sent back a reply saying that 1.It was no longer applicable 2.Due to overwheming psychological distress I was not able to engage with them and 3.even if I was they have no telephone number. I am worried they may try to penalise me. Am I within my rights to refuse to give a number where they can contact me, because of psychological distress?