Rendering 3D worlds is a complicated technical challenge. One technique known as inverse rendering involves reconstructing 3D scenes from a handful of 2D images. It uses AI to approximate how light behaves in the real world. Nvidia recently showcased its work with neural radiance fields (NeRF), which uses AI to speed up the process of inverse rendering. The downside of NeRF models is that they leave you with images you can’t easily edit – that’s not great for a designer creating something new. Nvidia’s new research addresses that problem. The new Nvidia 3D MoMa pipeline reconstructs images with three features: a 3D mesh model, materials, and lighting. The 3D mesh model is built out of triangles. The materials are 2D textures that are overlaid on the 3D mesh like a skin. Lastly, 3D MoMa estimates how a scene is lit. The finished product is directly compatible with the graphics engines and modeling tools that creators already use, meaning creators can modify and experiment with their models – changing materials or lighting effects. Nvidia is presenting the inverse rendering pipeline this week at the Conference on Computer Vision and Pattern Recognition in New Orleans. To demo it, Nvidia’s research and creative teams have created 3D models of jazz band instruments. They started with 100 images each of five different instruments. After 3D MoMa created 3D mesh representations of each, the team took the models out of their original scenes and imported them into the Nvidia Omniverse 3D simulation platform to edit. They changed the materials on the instruments, replacing the trumpet’s original plastic with other materials like gold and wood. Then they demonstrated how the different materials reflect light differently.
