Sounds like the lawyer is playing mind games. Also, go with your gut feeling. If you think your ex is telling lies, then he probably is. Mine did it to me, the worst garbage you could ever think of - and I was still a witness! If you know you've done nothing wrong, then don't sweat it. Hang tight. Make sure you've told your lawyer everything. Keep all documents, they will come in handy. The supporting evidence such as Doctor and employment records will be a lifesaver. A judge cant grant him things on the basis of you not being a witness, that would be ludicrous. So just sit tight, stay calm and if you come into contact with him before it goes to court, be wise with your words - he may just be taping you or having someone watch so they can be a 'witness' to your manner and actions.
You said you talked to him at a meeting. Are you looking to be reinstated?
Wishing you all the best...my turn for similar is next month.