Hi Dan,
Welcome to the forum.
Don't worry about the guy suing you. Talk is cheap and people often make ridiculous statements especially when they don't know what they are talking about.
Since your carer resigned, he has no grounds to say you terminated him. Where is his proof? Did he resign in writing? Even if he consults a lawyer, a lawyer with even basic knowledge would advise, he has no grounds for action.
How much notice did the carer provide before he stopped working for you? I don't know if there are minimum requirements in the UK but if notice wasn't sufficient, if anything, you might have grounds for action against him.
Enjoy your new life!