If you cannot do this right now because it is too difficult, or if you will not do this because you are a man and will stand your ground, then arm yourself. Get a gun and learn to shoot it. You can get a good used 38SPL revolver for under $250.00. You don't need anything fancy. Learn to shoot it, trust it, love it. Have good quality hollowpoint bullets in it.
No one should be physically intimidated or violated--even those that are not physically strong--and the gun is the great equalizer. You don't need advanced martial arts skills to use a gun, and only basic aiming skills to use it to devastating effect at short ranges. It puts even a woman on equal footing with a man. If this man comes across onto your property to harm you again, shoot him. Possibly the mere sight of the weapon will make him fall back.
Am I the only lily livered wimp that feels ill reading this advice?