Let’s Deep Dive Nintendo’s DLSS Patent And What It Means For The Switch 2

Nintendo has published a fascinating new patent that sheds light on what the company could have planned for the Switch 2 – and also on some things you may not know that Nvidia’s DLSS technology could do.

This patent was found and shared online by Laura Kate Dale on Bluesky. While the patent does not name the Switch 2, or even refer to their next console at all, the timing of the filing strongly suggests that this technology is to be used for that console. It does name Nvidia, and describe Nvidia specific technology like CUDA cores, but we’ll deep dive on another technology of their below.

Nintendo seems to have found their own ways of using upscaler technology that Nvidia may not have had to do for any of their other clients. We’ll share the original quotes from this patent in italics. You may choose to pass over them and read our summary. But if necessary, you can definitely cross reference the patent to confirm these details are as we described them below.

Most patents get too technical in their explanations. For this reason, many gamers and games press saw the patent for Nintendo’s Alarmo weeks in advance, but didn’t correctly figure out what it is (that includes us). But in this case, Nintendo has taken pains to explain very technical information as plainly as possible, perhaps because it needed such a clear explanation.

The patent describes one particular technology that Nvidia has developed for DLSS. On page 23, it describes an application that uses a neural network in the console to upscale an image, from 540p to 1080p. You can read that description below:

The game engine includes program structure for generating images that are to be output to the display. For example, the game engine may include program structure for managing and updating the position of an object(s) in a virtual space based on inputs provided from the input device. The provided data is the used to render an image of the virtual space by using, for example, a virtual camera. This image may be a source image that is generated in a first resolution (e.g., 540p). The source image is applied to the neural network that converts the source image into an upconverted image (e.g., an upconverted image is generated based on application of the source image to the neural network) that is at a higher resolution (e.g., 1080p) than the original source image. That upconverted image is then output to the display device for display thereon.

Now, you may wonder why the patent specifically described upscaling from 540p to 1080p. Most modern video games, even smaller games like Hades, do not natively output at 540p. So is Nintendo making those games run at lower resolutions for the Switch 2?

The answer is, surprisingly, yes! It seems that Nintendo has figured out you can get games running more efficiently if you let the neural network do most of the graphical performance tasks. And while it isn’t stated here, it sounds like Nintendo wanted games to run this specific way on the Switch 2, so that they could save on battery.

On page 33, Nintendo explains how games can get comparable performance with more efficiency by down-sampling games to this lower resolution:

With the techniques described above, the lower resolution input may be generated by down-sampling the high resolution input through point sampling (e.g., nearest neighbor). However, other down-sampling methods may be used in other example implementations.

In certain example embodiments, images that rendered fast (e.g., at 60 fps or the like) by a real-time game engine may naturally be similar to images resulting from point sampled data may be likely to better fit upscaling game engine outputs. It may also help game engines in certain example embodiments to run faster because additional costly anti-aliasing efforts may be skipped during the traditional rendering phase. Rather, such anti-aliasing may be handled more efficiently by the example neural network techniques discussed herein.

Now, if you were hoping that the Switch 2 could run games on 4K, there’s hope for you yet. Nintendo does state that these figures are merely given as examples. So we should not be surprised if the Switch 2 turns out to be capable of running some games, perhaps lower demanding games, at full 4K.

But the patent describes another scenario where a neural network could get even demanding games running at 4K. A cloud-based system could take that image that was upscaled on the Switch 2 from 540p to 1080p, and then upscale it even higher to 1440p or even full 4K. if that sounds familiar, it certainly sounds like what Treyarch implemented on Call of Duty: Black Ops 6. And we know that it is highly likely to come to the Switch 2 as part of Nintendo’s contract with Microsoft.

On pages 23 and 24, they share this description:

In certain embodiments, a cloud-based system may operate dynamically with respect to the target display that is being used by a user. Thus, for example, a video game may natively output images in 540p. A first user may use the cloud system to receive images that are 1080p (e.g., upconverted from 540p) and a second user may use the cloud system to receive a different resolution image (e.g., a 720p image, 4k image, or a 1440p image). Each instance of the video game application (and/or neural network) may be hosted within its own virtual machine or virtual container and thus allow for flexibly providing multiple different users with different options (e.g., outputting different resolution images) to “play” the same video game.

At this point, we’ll start describing things you may not know neural networks could do. This patent explains that neural networks don’t just come in the form of a standalone application. They can be part of the console’s operating system, or they can be packaged with the game itself. In fact, Nintendo’s patent says a neural network can fit in a Switch 2 cartridge.

On page 23, they share this description:

It will be appreciated that while a video game application program is used for the purposes of description, other applications that provide video output could be substituted. As well, while the neural network is shown as being part of a video game application program, it may be separately provided. For example, it may be part of an operating system service or the like that modifies or upconverts images as if they are being output by a video game application program.

And then on page 31, they provide further details:

In certain example embodiments, one or more trained neural networks may be delivered along with a game that is acquired by a user. For example, a user may download a game from an online store or the like and one of the components of the game may be a neural network for processing images produced by the game. Similarly, games that are provided on cartridges or other physical media may include one or more neural networks that can be used by the user to transform images produced by the game. In certain examples, multiple neural networks may be provided with the same instance of a game (e.g., an individual download or specific physical instance) to allow for the game to output to different types of displays (e.g., 1080p in one instance, 1440p in another, 4k in another, etc.)

As discussed herein different types of neural networks may be generated and distributed to the various game devices. Thus, for example, game device 1 may receive and use a neural network that is different than a neural network that is received and used by game devices 2, 3, 4, and 5. In certain example embodiments, each game (or more generally each application) may have a corresponding neural network (or multiple neural networks) that has been generated (e.g., by system 900) for that game. Accordingly, for example, game devices may store multiple different neural networks and use different such networks based on the game (or type of game) that is being played on the corresponding game device.

Finally, Nintendo’s patent describes the possibility of making different neural networks for different kinds of games, and having the console choose the appropriate neural network to run the game. For example, the Switch 2 could have a dedicated neural network for first person shooters, another one for strategy games, etc.

And so we’re left wondering, is this a new idea Nintendo developed with Nvidia? Or is this technology that the developers have known about all along, and gamers just weren’t aware existed?

On page 31, they provide this description:

In certain example embodiments, multiple games may share or use the same neural network. For example, one neural network may be generated for first person shooter type games, another for strategy games, etc. Accordingly, game may be group based on their “type.” Such type classifications may be based on the genre of game or may be based on another criteria, such as the type of rendering engine that the game used to generate images therein.

In certain example embodiments, the game engine (or other service that is providing converting functionality to the game engine) may dynamically decide to select one neural network among a selection of them depending on the remaining time available to “prepare the current video frame.” If the frame was rendered fast, it may have more time to be upscaled with such a high quality and slow neural network (e.g., one that includes additional layers), but if the frame used up more of the typically available 16 ms (for both rendering of the frame and subsequently upscaling images at 60 frames per second), the engine could select a faster neural network (e.g., one with less layers), although not providing as high image quality as the slower ones. Such a determination may be made through a “testing” phase of a video game application program (e.g., where the game engine produces a number of example images) and/or may be performed during normal gameplay.

Now, we will remind you, in the case of a patent, much like a verified leak, we know that the information is true and not hearsay. What we don’t know is if any of the technology described here will make it to the final product. And, of course, we can completely miss what the patent is, just like what happened with Alarmo.

But if this technology will be applied as described here, it will be a boon for portable handheld gaming for sure. Nintendo will leverage neural networks to make sure games can run for hours on the Switch 2, with graphics better than on the Switch. It will compete with the graphics the Steam Deck, ROG Ally, and Legion Go can put out, but with the efficiency to be a true portable console.

It’s a far cry from the days when they applied former Nintendo engineer Gunpei Yokoi’s philosophy of getting the most out of clearly obsolete technology on the Game Boy. Because this Switch 2 may not push polygons and triangles to as many astronomical numbers as the PlayStation 5 Pro will, but it will be cutting edge technology in the same way that Apple got us all used to having computers in our pockets.