Poppers, you speak of believing that you are separate from God as the thing that keeps you trapped in the world of form and creates the notion of suffering and lack.
How then do you escape the world of form or even just the notion of suffering and lack?
You don't escape the world of form until the body dies. Instead you inquire into your own nature to discover that what you really are is not a form to begin with. You are the awareness that witnesses all form. One of those forms is the idea of "me" - when you look for the me it won't be found. All that will be found are changing ideas that are identified with. Sometimes even those ideas don't arise - so then, what are you then when there is no "idea of me" arising? You are the silent, still, and peaceful awareness that is always present that "sees" everything happening. This has to be seen directly for it to make a difference in how life is lived and experienced, otherwise it just becomes another concept to identify with.
When consciously resting in/as that awareness notions of suffering don't arise and there is no sense of lack. That resting is available at all times - this very moment is when it happens. So you don't have to wait for the end of suffering - it is always available in this moment. Right "here" right "now" is the entry point. Learn to live fully in this present moment and you will find peace and the end of suffering (it's the only time there actually is - think about it). Thoughts of past or future are only thoughts, but that is where mind usually dwells, in thoughts of past or thoughts of future. What gets missed is life as it actually is. People are living a life wrapped up in their minds, oblivious to life "as it is", and that's why they suffer. That is how one remains trapped in the world of form, by living life from the mind rather than through awareness.
Any notion of "suffering" is something that the "idea of me"/ego believes in. "Suffering" is a concept that ego takes as being true; it is the mental story ego tells itself when it doesn't get its way. So suffering is a mental phenomenon. Find what is actually real rather than what is imagined and suffering is extinguished. Yes, there may physical sensations or pain that can be attended to but they don't become "suffering".
I understand that all things are connected, but moments of transcendence are few and far between.
This is where self-inquiry comes into play. Look for the "me" - each failure to find it presents the opportunity for transcendence, to discover that "you" are the underlying awareness, that which is one with everything. This will help to break identification with the "me-thought", so that there is a shift away this false sense of self and on to awareness, your true nature. That is where peace and wholeness is found.
The idea that life would be less interesting without the bad stuff is not something I am so sure about. Does God really require such challenges to be amused?
Why would God put limits on how he expresses himself in the world of form? It makes for a more diverse experience. From the viewpoint of the ego that seems harsh, but find out if the ego is real. If it isn't real then what happens to "suffering"? Remember, it is God doing it all - part of that "all" is what appears to be "individuals". But upon investigation the individual is discovered to be illusory - it is a function of mind and memory only, but it isn't actually real.
Would life in general not be more pleasant without suffering?
For the ego, certainly - it sounds ideal. That's what appeals to JWs the most, a life without suffering. But even if their imagined scenario of armagedon played out and they survived they would still suffer. Why? Because they would continue to believe in the false sense of self that sees itself separate from everything else. Whenever there is a perception of separation fear and suffering are not far off, for what one sees as different becomes a source of suspicion - it poses a potential threat to "me".
Most egos are averse to suffering, but there are some that actually create an identity out of it and enjoy it in a twisted sort of way. In either case, the suffering isn't actually real - it is imagined. A simple question reveals this truth: What's wrong with right now unless I think about it? Right now is "what is" - it's what life has unfolded. To resist what is, is to wish it to be different - that is the beginning of suffering.
Could challenges not come from the endless possibilities of the intricacies of life itself?
Yes, of course. And one of those challenges is how suffering is faced. Suffering can be a tremendous avenue to awaken to reality, to the discovery of what is beyond suffering. In general, people who don't suffer don't look for "something else" to find happiness. In other words, they are content with things just as they are so there is no perceived need to change anything. The problem with looking for the end of suffering and the sense of lack is that people look in the wrong place. Instead of looking within they look outside themselves. They look to a relationship, to possessions, to knowledge, to power and status, and to "God" (who is "out there" somewhere. And which religion is the "right" one that leads to God?). Attention must be turned 180 degress so that awareness discovers itself. This is what self-enquiry does, it allows attention to turn inwardly instead of remaining fixed outwardly. Outward is where division and suffering exists, inward is where unity and peace is found. Once peace is found inwardly one begins to see it outwardly as well.