I think you need to do some research about Aspergers. This isn't personal to you - his response is how his brain is wired. If you love him, do some research and get some help to figure out what YOU need to do. This kind of interaction is one thing that people with Aspergers struggle with - but it doesn't mean that your relationship has to fail.
This has a lot to do with you being willing to accept him as he is, and see how to support him so that you both get what you want/need out of the relationship. It won't be a fast process, but if he is open and there is progress being made, then he deserves a chance to have this work.
http://www.asperger-training.com/as-and-relationships/
http://psychcentral.com/blog/archives/2012/05/17/5-tips-for-loving-someone-with-aspergers-syndrome/