Maybe I didn't explain very clearly. Let me give it another go.
Our sun is roughly 93 million miles away. The speed of light is roughly 186,282 miles a second. Therefore when we observe our sun we are actually looking roughly 8 minutes into the past.
The farther away a source of light is, the farther back into the past we're looking.
The farthest galaxies from us are the ones that are moving away the fastest. (Hubble/Lemaître's law)
They're receding so fast that the wavelength of the light is "stretched" out of the visible spectrum and into the infrared.
That's red shift. Observing it is the mechanism of seeing some of the earliest objects in the history of our universe. The infrared light reaching us today was produced billions of years ago as visible light.
But it doesn't work without light. The JWST is not going to be able to see anything prior to earliest stars and galaxies. The earliest stars are theorized to be non- metallic and shorter lived and it would be cool if that could be observed, but they're not going to look much different than what we see today.
So the prediction in your OP is actually a pretty safe bet.