If you believe in alternate planes of existence, then heaven or hell could be literal and yet not physical. A 'ghost plane' of sorts. It would require that beings have a way of feeling joy or terror in this plane, and possibly comfort and pain. It's all possible when you are using your imagination.
This does force us to consider a being who would design such a universe, where eternal pain and suffering are possible in multiple planes of existence, for no other reason than whether or not he felt offended by the equivalent of a small hair on the back of a dust mite. The level of insanity, cruelty and plain old sadism required should make us thankful that it doesn't exist.