Key Takeaways
- Upscaling is necessary due to flat panel technology shifting from CRT to a fixed array of pixels on screens.
- UHD resolutions are computationally inexpensive but can cause color and motion artifacts in upscaled images.
- AI upscaling isn’t lazy but essential for optimization in modern computer graphics development to achieve the desired visual presentation.
You might have seen a certain kind of gaming enthusiast who insists that only pure, raw, native pixels pushed onto their screen count as computer graphics. Upscaling technology like DLSS or PSSR is “cheating” or a way for developers to avoid “optimizing” their games, because their requirements include use of these technologies. Is that really the case?
We Need Upscaling Because of Flat Panel Technology
When we moved from CRT (Cathode Ray Tube) displays to flat panels like LCDs, Plasma screens, and OLED technology, we fundamentally shifted how images are drawn on our screens.
With a CRT, the image you see is drawn with an electron beam that activates phosphors on the back of the screen. If you have a CRT monitor showing a digital image with a specific resolution (e.g. 1024×768), there’s nothing on the screen that physically corresponds precisely with the pixels in the image data. So, while there was an upper limit to the resolution that a CRT can create, all the resolutions within its supported range looked fine, they were just sharper or softer.
Flat panel displays, on the other hand, have a fixed array of physical pixels. For example, 1920×1080 pixels. If you display an image with that exact resolution on a flat panel, there’s no issue. However, if you want an image of any other resolution to fill the entire screen, you need to do some sort of math to invent values for pixels that don’t exist in the original image. This is the puzzle of image scaling to match the “native resolution” of a flat panel.
TV Makers Chose 4K Without Considering Computer Graphics
The UHD resolution (technically not 4K, but let’s pretend it is for now) is exactly four times the resolution of FHD. This makes it extremely simple, and computationally inexpensive, to scale video from FHD to UHD. If you want to scale a Blu-ray, for example, to fit a UHD TV then all you have to do is multiply the pixel values by two on each axis. In effect, every virtual pixel in the image, is represented by four physical pixels on the actual screen.
This works well, and the only real effect is that the FHD picture will look softer than a UHD image, but otherwise it will look the same as it would on a native FHD display. However, if your source image isn’t an exact multiple of the target resolution, you run into a problem where you can’t accurately represent every original pixel. Some of the pixels in the physical display will now have to be an averaged or combined result of several original pixels in the image. Which leads to color and motion artifacts in the upscaled image.
With computer graphics there are even more problems. Even with perfect integer scaling, computer graphics don’t come out nearly as well as video does. Issues like aliasing (the stair-stepping on thin edges) are amplified by the scaling process. In general, simple integer scaling of real-time 3D computer graphics has a raw, pixelated look, particularly in motion.
This is why, ideally, you need to match your rendering resolution to the native resolution of the monitor. Except, a UHD display has four times the pixels of an FHD display. Roughly two million pixels versus eight million pixels. This requires a massive increase in processing power. This is why consoles like the PlayStation 4 Pro exist, since the adoption of UHD over FHD TVs really started taking off during the PlayStation 4’s tenure.
UHD TVs all have scaling technology built in, but the quality of the output differs depending on how smart the algorithm in question is. So Sony had to bring out a console that took control of what the picture would look like on 4K TVs. They achieved this by giving the PlayStation 4 Pro a big chunk of extra performance, along with special hardware to perform a specialized upscaling method called “checkerboard” upscaling, which had a much more pleasing result at the time than the competition. Does this sound familiar? That’s because this is exactly what Sony later did with the PlayStation 5 Pro, except the scaling now uses artificial intelligence, rather than a (relatively) simple algorithm.
AI Upscaling Isn’t a Crutch for Lazy Developers, It’s Another Optimization
There’s an idea sometimes floated around gaming forums and social media that video games that include the use of upscaling technologies like NVIDIA’s DLSS in order to run on the most powerful hardware today is a sign that developers haven’t “optimized” their games.
However, these modern upscaling methods are probably one of the biggest optimizations in the history of computer graphics. They make it so that video game developers can divert GPU processing power from grinding out millions of pixels and instead focus on making a lower-resolution image with better effects and fidelity, using AI or at least sophisticated upscaling algorithms to make these images look sharp and attractive on high-resolution displays. It’s the technology developers use to achieve the final look they want with the hardware that’s currently possible, and there’s nothing “lazy” about that.
All Computer Graphics Are “Fake”
The other main criticism of games that rely on AI upscaling to present a good image on high-resolution displays is that the resulting image is “fake” since it’s been conjured up by artificial intelligence.
The thing is, all computer graphics today are a collection of tricks to draw pictures on a screen using as little computing power as possible. To make shadows, reflections, lighting, textures, and all the other elements of a rendered scene, there are numerous mathematical hacks that have been developed because the brute force method is out of reach, With the rise of real-time ray-tracing, that’s becoming less true, but even there we need the support of upscaling and denoising algorithms to get usable image quality.
Upscaling is exactly the same sort of thing as every other type of graphical method—a smart way to get as much from your resources as possible.
Only the Final Image Matters
With details of how games are getting improved on the PlayStation 5 Pro and its PSSR AI upscaling technology, there is some dismay that certain games still have “low” internal resolutions, but all of this is irrelevant to players and how much they will enjoy the game. It doesn’t matter how the cake is baked, what matters is whether the cake tastes good.
The only thing that matters is the final experience, and the final visual presentation you see on your screen. The technical details of how it’s made are interesting, especially to a geek like me, but it’s not at all relevant to me as a gamer, and it’s important to separate those two roles, or you’ll never be happy with video games ever again.
