Several things to keep in mind - most states and places of employment are at will. Meaning that you can be terminated for any reason and not just for cause. Next you need to determine if you have a job description and if w/i that job description there is anywhere it sets forth that you are to be available at all times. If there is no such language, then the next step is to do exactly what you did ... you need to be polite, but clear in your advising that you are not available 24/7/365. That just the same as you attempt to respect others when they are away from work, you would seek the same courtesy. Unless it was a true emergency (but you may have to define emergency ... loss of life or limb) you are not to be contacted. If this does not change then you should seek clarification with your HR Department and level a complaint. Of course that opens up a whole nother can of worms.
I disagree w/ the mentality that because someone is the boss that you are required to say "how high" when they say jump. We will soon become a nation where it is honorable to die at your desk because we are failing to not only take a vacation but respect others who do.