Severe disability premium and frequently visiting partner

Hi
I’ve just had a PIP assessment and was quizzed about how often my partner visits/ how long for.

I receive the ESA Severe Disability Premium as I have standard PIP and do not ‘live with’ my partner who lives a few hours away..(citizens advice helped me apply for this 6 years ago). But over the years he has stayed for longer periods.
Like my health condition this is a fluctuating situation as he stays for blocks of time, for less than half the time in a year but often for runs of many weeks, up to a few months in the summer sometimes. I’m now panicking because so far this year for various reasons (including a particularly bad period of health for me) he’s stayed for two very long blocks amounting to about 19 weeks, but by year’s end it will still most likely be just less than half. During the phone assessment I couldn’t think straight as it was towards the end and I mumbled about ‘a few weeks’ for his most recent stay - I couldn’t face the complications of a long explanation as I was exhausted by then (I have CFS). I added that his visits are in blocks of weeks but mostly unpredictable, and amount to less than half the time a year. (Last year he stayed for about 22 weeks in total). The only predictability is that he’s never here over winter (usually 3 or 4 months). Another complication is that he owns the house I live in and technically I’m a tenant paying a peppercorn rent although we are now civil partners., But our finances and daily decisions have always been completely separate. I pay all the bills here and always have. Likewise him where he lives.

Two things I’m freaking about:

1), could I be accused of some kind of fraud after not giving the correct info today? I’ve asked DWP in the past what the rules are and I’ve researched but all I’ve been told is ‘if he lived here for several nights a week for example we might have to investigate’. What does that even mean?!
2) If they remove the premium, can they backdate it and demand a huge amount in arrears? My last review was years ago (this one has been 18 months overdue).

Should I write to DWP and tell them how long he’s actually stayed for so far this year and projection to the end of the year, or just wait and see what happens? I can cope with losing the premium going forward, but not with being prosecuted for misleading info when I’m knackered and there are no clear guidelines anyway. (At least there won’t be room for me in jail!!)
Argh!