Again as a former Elder - I agree with Earnest - they don't have anything on you unless you offer up more information.
To do anything they need Two witnesses, that can corroborate that you are often with the DFed person.
Being seen once with someone DFed is not enough to have a Judicial Case.
I would not engage with them, but if they contact you just be respectful and say you are appreciative of their concern for you, that you are ok, thank them, and exit.