Enhancing Photorealism Enhancement

What the heck does that mean?! Think of it as a real-time deepfake, for video games.

The deepfake in this instance, alters game engine imaging, and swaps it for real-world imaging, thus enhancing the photorealism of the videogame with AI enhancement, in real-time. Yes, real-time.

One can judge from the video themselves. Intel trained a convolutional neural network, using high-resolution, HDR cameras, and recorded tons of footage around Europe. This footage was processed by said CNN, and it can now recreate the videogame inagery, interactively (60 FPS+), and replace the imaging by the game engine, with the photoreal data the CNN has adapted to enhance the photorealism of the game.

ABSTRACT (from Intel ILS):

"We present an approach to enhancing the realism of synthetic images. The images are enhanced by a convolutional network that leverages intermediate representations produced by conventional rendering pipelines. The network is trained via a novel adversarial objective, which provides strong supervision at multiple perceptual levels. We analyze scene layout distributions in commonly used datasets and find that they differ in important ways. We hypothesize that this is one of the causes of strong artifacts that can be observed in the results of many prior methods. To address this we propose a new strategy for sampling image patches during training. We also introduce multiple architectural improvements in the deep network modules used for photorealism enhancement. We confirm the benefits of our contributions in controlled experiments and report substantial gains in stability and realism in comparison to recent image-to-image translation methods and a variety of other baselines."

CODE:

WHITEPAPER:

KEYNOTE: Eurographics (May 2021)

Vladlen Koltun: Further Towards Photorealism

An additional component related to these efforts is about "Stable View Synthesis," which is an adaptation of photogrammetry. Here is the whitepaper on the topic, including a video:

Grab all the goodies: code, whitepaper, videos, and over 2GB of imaging, from the Intel ILS webpage for this astonishing technological advancement:

Updated (sorry, cannot make it a proper link): https://github.com/isl-org/PhotorealismEnhancement