Patents are always a rather intriguing thing to track in the technology industry – sometimes they represent ideas and products that’ll never see the light of day, but there are occasions when they give clues to future releases. A freshly released US patent application could be the latter, as it addresses technology that could allow Nintendo to boost visuals in its hardware.
Opened on 25th March 2020 and released publicly yesterday (30th September), the application is titled ‘systems and methods for machine learned image conversion’, and though the initial language can be a headscratcher it is essentially an idea similar to NVIDIA’s DLSS. That’s short for Deep Learning Super Sampling in NVIDIA’s case, which works on some of its GPUs to upscale image resolution and quality in real time, while also being remarkably efficient and ensuring that the graphics card is under less strain. It’s impressive technology, and it’s been front and centre in a lot of talk about how Nintendo could produce a new Switch-style portable device that outputs higher resolution visuals while still working with low power output. Digital Foundry has explored this in detail.
What makes this application intriguing is that Nintendo is clearly exploring this internally – a named party on the application is Alexandre Delattre, who is a co-founder of Nintendo European Research and Development. It’s also acknowledged in the ‘Introduction’ of the patent that this is an area being explored throughout the industry:
Machine learning can give computers the ability “learn” a specific task without expressly programming the computer for that task. One type of machine learning system is called convolutional neural networks (CNNs)—a class of deep learning neural networks. Such networks (and other forms of machine learning) can be used to, for example, help with automatically recognizing whether a cat is in a photograph. The learning takes places by using thousands or millions of photos to “train” the model to recognize when a cat is in a photograph. While this can be a powerful tool, the resulting processing of using a trained model (and training the model) can still be computationally expensive when deployed in a real-time environment.
Image up-conversion is a technique that allows for conversion of images produced in a first resolution (e.g., 540p resolution or 960×540 with 0.5 megapixels) to a higher resolution (e.g., 1080p resolution, 1920×1080, with 2.1 megapixels). This process can be used to show images of the first resolution on a higher resolution display. Thus, for example, a 540p image can be displayed on a 1080p television and (depending on the nature of the up-conversion process) may be shown with increased graphical fidelity as compared to if the 540p image were displayed directly with traditional (e.g., linear) upscaling on a 540 television. Different techniques for image up-conversion can present a tradeoff between speed (e.g., how long the process takes for converting a given image) and the quality of the up-converted image. For example, if a process for up-converting is performed in real-time (e.g., such as during a video game), then the image quality of the resulting up-converted image may suffer.
Accordingly, it will be appreciated that new and improved techniques, systems, and processes are continually sought after in these areas of technology.
Ultimately it shouldn’t be a surprise that Nintendo is researching upscaling through machine learning, as it is likely to be a vital factor should the company opt to retain a Switch-style form factor while offering greater graphical fidelity in the future. Whether Nintendo will still utilise NVIDIA technology in future devices is also interesting; if it develops its own solution it may not need NVIDIA’s DSSL tools. Of course, depending on what and who you believe, there are reports that ‘4K’ development units are already out in the wild.
Let us know what you think in the comments!
