Because what you're describing is not clearly a religious symbol, I don't believe those elders have anything to stand on in this instance. If they decide to disfellowship, they will have to send this in a report to the branch office. In the report they will have to provide detailed evidence that supports their formation of a judicial committee and a decision to disfellowship. Also, they will have to provide even more detail in the narrative for the report.
The branch office can and will kick back disfellowshippings which have insufficient basis. I've seen more than one instance where the elders were hasty and acted without proper foundation and the branch office directed the elders to rescind the disfellowshipping and announce it to the congregation just that way.
An interesting thought exercise is to substitute another religion. Let's say that I bought white tennis shoes with red and yellow coloring (stripes or otherwise). Those ALSO happen to be the colors of the Catholic Church (white, yellow, and red). Does it follow that I'm now a practicing Catholic if I'm not attending mass or taking part in communion? Even if I used to be a Catholic, there is no basis for an apostasy charge in my opinion here.
If you are called to meet with a judicial committee mention very early in the hearing that you see no basis for the hearing and that you believe that the body of elders did not have proper foundation and evidence to form the judicial committee. These are code words to an experienced chairman that if they reprove or disfellowship that you believe there is basis for appeal. Also mention to them the option they have (a surprising number of elders do not realize this) that if they decide that the committee should not have been formed because the evidence is insufficient to justify a committee that this is authorized by policy in their instructions. If they decide to disfellowship, immediately request (mail a registered letter the same day if possible and hand the chairman of your committee a copy and tell the chairman orally that you will definitely be appealing the decision) an appeal hearing on the basis that there is no evidence of apostasy (you have to state why you are requesting the appeal) then if the appeal committee also upholds the decision exercise your right to send a companion letter with the report to the branch office to force the issue to senior oversight in service or the branch committee.
Hopefully you find this helpful.