Ben Mildenhall, a Google researcher based in London, released a snippet of their newly developed 3D reconstruction model called RawNeRF in June 2022. This new tool converts 2D images into well-lit 3D scenes. It is based on their open-source project, MultiNeRF.

Mildenhall teased a video of their most recent NeRF development, which combined their mip-NeRF 360, RawNeRF, and Ref-NeRF models. By synthesising and syncing 500 images, the combination was able to create a 3D space, allowing for a full 360-degree view with the camera moving across the space.

He also demonstrated the HDR view synthesis, which allows editing of the image’s exposure, lighting, tones, and depth of field. Because the photos 3D space or models are created using 2D raw images, the software can edit the images in the same way that Adobe Photoshop can.

This new Google Research tool, interestingly, recognizes light and ray patterns and then cancels out noise from images, generating 3D scenes from a set of single images.

NeRF’s Origins

Neural Radiance Field (NeRF), created in 2020 by Jon Barron, senior staff research scientist at Google Research, is a neural network capable of generating 3D scenes from 2D photos. In addition to recovering details and color from RAW captured images in a dark scene, the tool can process them to create a 3D space that allows the user to view the scene from various camera positions and angles.

3D Reconstruction Models Are Becoming More Popular

Meta recently announced the release of Implicitron, a PyTorch3D extension that is a 3D computer vision research tool for rendering prototypes of real-life objects. The new approach, which is still in the early stages of development, allows for the representation of objects as a continuous function and is intended for use in real-world AR and VR applications.

The Nvidia research team released Instant NeRF in March 2022, which can reconstruct a 3D scene from two 2D images taken at different angles in seconds. According to NVIDIA, using AI when processing images speeds up the rendering process.

Nvidia AI research also developed GANverse 3D in 2021, an extension to their Omniverse that uses deep learning to render 3D objects from 2D images. The model generates multiple views from a single image using StyleGANs.

Following Nvidia’s technique and innovation, Google’s research team led by Mildenhall was able to add the feature of removing noise from the scene created by 2D images and dramatically enhancing light. When combined with the 3D scene, the noise reduction method produces a high-resolution output that is seamless when transitioning between angles and positions.

Metaverse NeRF

There are several key technologies that are required to create an immersive Metaverse experience. AI, IoT, AR, Blockchain, and 3D reconstruction are all examples. While developers use frameworks such as Unreal Engine, Unity, and Cryengine to render 3D models into the Metaverse, leveraging 3D rendering technology can improve both the quality and the immersion.

According to Brad Quinton, founder of Perceptus Platform, the metaverse is heavily reliant on 3D recreation of scenes. The metaverse’s entire concept is to be able to see and interact with content within it. The Perceptus Platform tracks physical objects in 3D environments in real time.

The ability to create 3D objects and spaces by simply capturing multiple 2D images has the potential to significantly accelerate the construction of the metaverse. Furthermore, AR and VR technologies such as the Perceptus Platform have the potential to make the metaverse truly immersive.

There are numerous challenges in creating a perfect metaverse, such as physical properties of materials such as weight, fold, and so on. One of the challenges of lighting a model that represents the real-life quality of the object was recently resolved with the NeRF model. Developers were able to illuminate rendered objects in a variety of lighting scenarios.

NeRF models can also be converted to meshes by using marching cubes. Models can now be imported directly into the metaverse without the need for 3D rendering software. Vendors, artists, and other businesses will be able to create virtual representations of their products and render them accurately across the 3D world.