Neural graphics software development kits create metaverse content
To create the metaverse, a virtual reality, immersive world, internet developers will build 3D objects for building scenes for games. Engineers will also use 3D objects for product design or visual effects in gaming and film. When multiple objects and characters need to interact in a virtual world, simulating physics becomes critical. A robot in a virtual factory, for example, needs to look the same and have the same weight capacity and braking capability as its physical counterpart.
Nvidia has released software development kits (SDKs) to shorten the development process of 3D modelling for use in transportation, healthcare, telecommunications and entertainment, as well as product design.
The SDKs include NeuralVDB, an update to the OpenVDB library and Kaolin Wisp, a Pytorch library. For 3D creation, Kaolin Wisp is an addition to Kaolin, a PyTorch library. It enables faster 3D deep learning research by reducing the time needed to test and implement new techniques from weeks to days. Kaolin Wisp is a research-oriented library for neural fields, establishing a common suite of tools and a framework to accelerate new research in neural fields.
They also include instant neural graphics primitives. This captures real-world objects and inspired Nvidia’s Instant ReRF, an inverse rendering model that turns a collection of still images into a digital 3D scene. This technique and associated GitHub code accelerate the process by up to 1,000 times, said Nvidia.
A new inverse rendering pipeline is 3D MoMa. It allows users to quickly import a 2D object into a graphics engine to create a 3D object that can be modified with realistic materials, lighting and physics.
Progressing the GauGAN AI model, GauGAN360 turns rough doodles into “photorealistic masterpieces” said Nvidia. It generates 8K, 360 degree panoramas that can be ported into Omniverse (Nvidia’s version of the metaverse) scenes.
The Omniverse Avatar Cloud Engine (ACE) is a new collection of cloud APIs, micro services and tools to create, customise and deploy digital human applications. ACE is built on Nvidia’s Unified Compute Framework, allowing developers to seamlessly integrate core Nvidia AI technologies into their avatar applications.
In addition to the introduction of NeuralVDB, a version of OpenVDB for volumetric data storage and higher resolution 3D data, Nvidia has also introduced Omniverse Audio2Face, an AI technology that generates expressive facial animation from a single audio source. This is useful for interactive real-time applications and as a traditional facial animation authoring tool.
Further work on avatars is via ASE: Animation Skills Embedding. This uses deep learning learning to teach characters how to respond to new tasks and actions. The result is physically simulated characters are able to act in a more responsive and life-like manner in unfamiliar situations.
The TAO toolkit is a framework to enable users to create an accurate, high-performance pose estimation model, which can evaluate what a person might be doing in a scene using computer vision much more quickly than current methods.
For experiences representation, Nvidia has introduced image features eye tracking, a research model linking the quality of pixel rendering to a user’s reaction time. By predicting the best combination of rendering quality, display properties and viewing conditions for the least latency, it will allow for better performance in fast-paced, interactive computer graphics applications such as competitive gaming.
Following a collaboration with Stanford University, there are also holographic glasses for virtual reality. They are claimed to deliver full colour, 3D holographic images in a 2.5mm thick optical stack.
