At SIGGRAPH in Denver, NVIDIA Corporation introduced groundbreaking research and innovations in simulation, generative artificial intelligence, and robotics. The company announced a comprehensive suite of services, models, and computing platforms designed to empower robotics and AI developers to “develop, train, and build the next generation of humanoid robotics.”

“The next wave of AI is robotics, and one of the most exciting developments is humanoid robots,” stated Jensen Huang, founder and CEO of NVIDIA. “We’re advancing the entire NVIDIA robotics stack, opening access for worldwide humanoid robotics developers and companies to use the platforms, acceleration libraries, and AI models best suited for their needs.”

NVIDIA presented 20 research papers at SIGGRAPH. Rev Labaredian, vice president of Omniverse and virtualization strategy, highlighted NVIDIA’s long-standing work in graphics research since 2001, underscoring its established connection to graphics, simulation, and robotics.

NVIDIA NIMs: Developing Digital Twins and Beyond

NVIDIA’s NIM microservices, which are pre-built containers using NVIDIA inference software, are now available as a service. These microservices can significantly reduce model deployment times from weeks to minutes, aiding in the design and training of robots and their environments.

“The time to apply generative AI is now, but it can be daunting,” acknowledged Kari Briski, vice president of generative AI software product management at NVIDIA. “Enterprises need a fast path to production for return on investment. … This led to NVIDIA NIMs to standardize deployment of AI models, and it’s based on CUDA to run out of the box.”

Two new AI microservices, MimicGen NIM and Robocasa NIM, will enhance simulation workflows for generative physical AI in NVIDIA Isaac Sim, a reference application for robotics simulation built on the NVIDIA Omniverse platform. MimicGen NIM generates synthetic motion data based on recordings of teleoperation using spatial computing devices like Apple Vision Pro, while Robocasa NIM creates robot tasks and simulation-ready environments in NVIDIA’s OpenUSD framework.

NVIDIA also announced NIM microservices to aid in the development of digital twins and USD connectors for streaming massive NVIDIA RTX ray-traced data sets to Apple Vision Pro. Available now, NVIDIA OSMO is a managed cloud service that simplifies robot training and simulation workflows, reducing deployment and development cycle times from months to under a week.

“OSMO vastly simplifies robot training and simulation workflows, cutting deployment and development cycle times from months to under a week,” said NVIDIA. “Users can visualize and manage tasks such as generating synthetic data, training models, conducting reinforcement learning, and implementing software-in-the-loop testing at scale for humanoids, autonomous mobile robots [AMRs], and industrial manipulators.”

Training Foundation Models for Humanoid and Other Robots

NVIDIA’s new workflow leverages AI and Omniverse to enable developers to train robots with smaller datasets. Developers can capture teleoperated demonstrations using Apple Vision Pro, simulate recordings in Isaac Sim, and generate synthetic datasets using MimicGen NIM. This approach allows for the training of the Project GR00T humanoid foundation model with both real and synthetic data, saving time and costs.

Roboticists can use Robocasa NIM within the Isaac Lab framework to generate experiences for retraining robot models. Throughout the workflow, NVIDIA OSMO assigns computing jobs to different resources, eliminating weeks of administrative tasks.

“Developing humanoid robots is extremely complex — requiring an incredible amount of real data, tediously captured from the real world,” said Alex Gu, CEO of Fourier. “NVIDIA’s new simulation and generative AI developer tools will help bootstrap and accelerate our model-development workflows.”

NVIDIA’s new Humanoid Robot Developer Program offers early access to the NIMs and OSMO, as well as the latest releases of NVIDIA Isaac Sim on Omniverse, Isaac Lab, Jetson Thor compute, and Project GR00T foundation models. Early adopters of the program include 1X, Boston Dynamics, ByteDance Research, Field AI, Figure, Fourier, Galbot, LimX Dynamics, Mentee, Neura Robotics, RobotEra, and Skild AI.

“Boston Dynamics and NVIDIA have a long history of close collaboration to push the boundaries of what’s possible in robotics,” said Aaron Saunders, chief technology officer of Boston Dynamics. “We’re really excited to see the fruits of this work accelerating the industry at large, and the early-access program is a fantastic way to access best-in-class technology.”

NVIDIA’s events at SIGGRAPH include a fireside chat between Huang and Lauren Goode, senior writer at Wired, on the impact of robotics and AI in industrial digitalization, as well as a conversation with Meta’s Mark Zuckerberg.

By Impact Lab