Facebook's DeepFovea tool reduces VR rendering resources by upto 99 percent

0 0

By Matthew Griffin Virtual World 28th November 2019

WHY THIS MATTERS IN BRIEF

Rendering high quality virtual worlds is resource intensive, DeepFovea eliminates up to 99 percent of those resources which will make AR and VR more accessible.

Interested in the Exponential Future? Connect, download a free E-Book, watch a keynote, or browse my blog.

One of the biggest issues with Augmented Reality (AR) and Virtual Reality (VR) worlds today is the huge amount of computing power and network bandwidth they need to suck up in order to provide users a high quality experience – something that hopefully cloud based rendering and 5G might help overcome in the future. In order to fix this issue Facebook just unveiled a new light weight Artificial Intelligence (AI) system called DeepFovea, a foveated rendering system, that’s specially designed to overcome both of these problems, and when you have a look at the quality of the VR images that it can render using amazing low amounts of compute and network bandwidth the results, it has to be said, are quite stunning.

An AI has learned to predict people's moods from the way they walk

This is also the first practical application of a Generative Adversarial Network (GAN), the same type of AI that helps generate everything from DeepFakes and Synthetic Content, to new helping innovate products like NASA’s interplanetary landers, and, because of the power of the technology it’s able to generate natural looking hi definition video based on nothing more than an incredibly sparse input – as you can see from the video. In tests, DeepFovea can decrease the amount of compute resources needed for rendering by as much as 10 to 14x while any image differences remain imperceptible to the human eye.

A high quality VR experience with just 10% of the data

When the human eye looks directly at an object, it sees it in great detail. Peripheral vision, on the other hand, is much lower quality, but because the brain infers the missing information, humans don’t notice. DeepFovea uses recent advances in GAN’s that can similarly “in-hallucinate” missing peripheral details by generating content that is perceptually consistent.

JPMorgan opens the world's first bank branch in the Metaverse

The system is trained by feeding a large number of video sequences with dramatically decreased pixel density as input. The input then simulates the peripheral image degradation and the target helps the network learn how to fill in the missing details based on statistics from all the videos it has seen. The result is a natural looking video generated out of a stream of sparse pixels that has been decreased in density by as much as 99 percent along the periphery of a 60×40 degree field of view. The system also manages the level of flicker, aliasing, and other video artefacts in the periphery to be below the threshold that can be detected by the human eye.

This VR gadget lets you feel spiders on your face

High quality AR and VR experiences require high image resolution, high frame rate, and multiple views, which can be extremely resource intensive, so in order to advance these systems and bring them to a wider range of audiences and devices, such as those with mobile chipsets and small, portable batteries, we’ll need to dramatically increase rendering efficiency, and that’s what DeepFovea helps achieves. DeepFovea also shows how deep learning can help accomplish this task via foveated reconstruction and it’s also hardware-agnostic which makes it a promising tool for potential use in next-gen head-mounted display technologies.

Matthew Griffin / About Author

Described by NASA as a "Walking Encyclopaedia of the Future" and one of the world’s most in demand keynote speakers Matthew Griffin, Founder and Futurist in Chief of the 311 Institute, a global Futures and Deep Futures advisory firm, is a multi-award winning Futurist and strategic foresight expert, author, lecturer, mentor, podcast and YouTube host.

With a distinguished career running global business units at Atos, EMC, and IBM today Matthew helps leaders and titans of industry from all over the world explore the impact of the inter sections of technology, culture, and leadership, then helps them create high performance organisations that can continuously disrupt, outpace, and outthink their competition.

15 times author of the hit series the Codex of the Future series Matthew's ability to identify, track, and explain the impacts of hundreds of emerging technologies and trends on global business, culture, and society has earned him a powerful reputation and a roster of clients that include royal households, world leaders across the G7, G20, and G77, and many of the world's most recognised and exceptional organisations including ABB, Accenture, ABN Amro, Adidas, AON, ARM, BCG, Centrica, Citi Group, Coca Cola, Deloitte, Dentons, Disney, Dow, EY, Google, KPMG, Lego, Microsoft, Legal & General, LinkedIn, Nokia, Paramount, PepsiCo, PWC, Qualcomm, RWE, Samsung, T-Mobile, UBS, Visa, WIRED, WPP and hundreds of others.

Matthew is regularly featured in the global media including the AP, BBC, Bloomberg, CNBC, Discovery, Forbes, Khaleej Times, Telegraph, TIME, ViacomCBS, WIRED, and the WSJ, and his mission is to help organisations create a fair and sustainable future whose benefits are shared by everyone irrespective of their ability, background, or circumstances.