{"id":3765,"date":"2024-10-22T08:43:56","date_gmt":"2024-10-22T08:43:56","guid":{"rendered":"https:\/\/salarydistribution.com\/machine-learning\/2024\/10\/22\/nvidia-brings-generative-ai-tools-simulation-and-perception-workflows-to-ros-developer-ecosystem\/"},"modified":"2024-10-22T08:43:56","modified_gmt":"2024-10-22T08:43:56","slug":"nvidia-brings-generative-ai-tools-simulation-and-perception-workflows-to-ros-developer-ecosystem","status":"publish","type":"post","link":"https:\/\/salarydistribution.com\/machine-learning\/2024\/10\/22\/nvidia-brings-generative-ai-tools-simulation-and-perception-workflows-to-ros-developer-ecosystem\/","title":{"rendered":"NVIDIA Brings Generative AI Tools, Simulation and Perception Workflows to ROS Developer Ecosystem"},"content":{"rendered":"
\n\t\t<\/span> <\/span> <\/span><\/span><\/p>\n

At ROSCon in Odense, one of Denmark\u2019s oldest cities and a hub of automation, NVIDIA and its robotics ecosystem partners announced generative AI tools ,simulation, and perception workflows for Robot Operating System (ROS) developers.<\/p>\n

Among the reveals were new generative AI nodes and workflows for ROS developers deploying to the NVIDIA Jetson platform<\/a> for edge AI and robotics. Generative AI<\/a> enables robots to perceive and understand the context of their surroundings, communicate naturally with humans and make adaptive decisions autonomously.<\/p>\n

Generative AI Comes to ROS Community<\/b><\/h2>\n

ReMEmbR<\/a>, built on ROS 2, uses generative AI to enhance robotic reasoning and action. It combines large language models<\/a> (LLMs), vision language models (VLMs) and retrieval-augmented generation<\/a> to allow robots to build and query long-term semantic memories and improve their ability to navigate and interact with their environments.<\/p>\n

The speech recognition capability is powered by the WhisperTRT ROS 2 node<\/a>. This node uses NVIDIA TensorRT<\/a> to optimize OpenAI\u2019s Whisper model<\/a> to enable low-latency inference on NVIDIA Jetson, resulting in responsive human-robot interaction.<\/p>\n

The ROS 2 robots with voice control<\/a> project uses the NVIDIA Riva<\/a> ASR-TTS service to make robots understand and respond to spoken commands. The NASA Jet Propulsion Laboratory independently demonstrated ROSA<\/a>, an AI-powered agent for ROS, operating on its Nebula-SPOT robot<\/a> and the NVIDIA Nova Carter<\/a> robot in NVIDIA Isaac Sim<\/a>.<\/p>\n

At ROSCon, Canonical<\/a> is demonstrating NanoOWL<\/a>, a zero-shot object detection model running on the NVIDIA Jetson Orin Nano<\/a> system-on-module. It allows robots to identify a broad range of objects in real time, without relying on predefined categories.<\/p>\n

Developers can get started today with ROS 2 Nodes for Generative AI<\/a>, which brings NVIDIA Jetson-optimized LLMs and VLMs to enhance robot capabilities.<\/p>\n

Enhancing ROS Workflows With a \u2018Sim-First\u2019 Approach<\/b><\/h2>\n

\"\"<\/p>\n

Simulation<\/a> is critical to safely test and validate AI-enabled robots before deployment. NVIDIA Isaac Sim, a robotics simulation platform built on OpenUSD<\/a>, provides ROS developers a virtual environment to test robots by easily connecting them to their ROS packages. A new Beginner\u2019s Guide to ROS 2 Workflows With Isaac Sim<\/a>, which illustrates the end-to-end workflow for robot simulation and testing, is now available.<\/p>\n

Foxglove<\/a>, a member of the NVIDIA Inception<\/a> program for startups, demonstrated an integration<\/a> that helps developers visualize and debug simulation data in real time using Foxglove\u2019s custom extension, built on Isaac Sim.<\/p>\n

New Capabilities for Isaac ROS 3.2<\/b><\/h2>\n

NVIDIA Isaac ROS<\/a>, built on the open-source ROS 2<\/a> software framework, is a suite of accelerated computing packages and AI models for robotics development. The upcoming 3.2 release enhances robot perception, manipulation and environment mapping.<\/p>\n

Key improvements to NVIDIA Isaac Manipulator<\/a> include new reference workflows that integrate FoundationPose and cuMotion to accelerate development of pick-and-place and object-following pipelines in robotics.<\/p>\n

Another is to NVIDIA Isaac Perceptor<\/a>, which features a new visual SLAM reference workflow, enhanced multi-camera detection and 3D reconstruction to improve an autonomous mobile robot\u2019s (AMR) environmental awareness and performance in dynamic settings like warehouses.<\/p>\n

Partners Adopting NVIDIA Isaac\u00a0<\/b><\/h2>\n

\"\"<\/p>\n

Robotics companies are integrating NVIDIA Isaac accelerated libraries and AI models into their platforms.<\/p>\n