While you have been seemingly more than casual friends for upwards of 6 years, your reluctance to be even seen by his parents and brother seems to imply they don't even know you exist and you realize that your presence would cause a huge problem.
Isn't it extremely strange that someone who is supposedly so important to him, is not known to his immediate family? You are being treated as a non-person yet, you continue to accept that treatment.
While you infer you want to move-on to find someone else, your statement: 'But he was supposed to let me free to move on and find a future for myself ...' implies he is in full control and you are helpless.
Regardless of whether or not this relationship has any hope to continue, take control of yourself... for your own self worth, health and happiness.
Are these self-worth issues a result of treatment by him?
While he implies wanting to delay until older, you are both over 21 so, how old is old enough? You are now adults and don't need anyone else to give permission to establish a relationship, or to at least talk in-person.
He appears to be postponing any commitment to either move forward or to end the relationship, giving you a false hope of a potential future with him.
Find someone who is worthy of your love and is proud to be in a relationship with you.