How Rhoda AI uses video data to create generalized AI models for robots

Earlier this year, Rhoda AI emerged from 18 months of hiding with $450 million in Series A funding. The company is working with customers in the automotive, manufacturing, logistics and e-commerce industries to deploy its video direct action, or DVA, model for robotics.

“We’re focused on real-world implementation, and that means you have to deal with real-world confusion. Every run can be different,” said Jagdeep Singh, co-founder and CEO of Rhoda AI. The robot report. “Each configuration can be different and you need a really robust model. To create a robust model, you need a huge training dataset.”

A large training dataset is exactly what has allowed large language models (LLMs) like ChatGPT to advance rapidly in recent years. Physical AI has lagged behind these models because the same data set doesn’t exist for the physical world, Singh noted.

“Every word ever put on the internet, from Shakespeare to Reddit, has been used to train these AI models. In the physical world of AI, there was no internet-scale dataset,” he said. “People have tried to collect their own data using teleoperating robots, and this approach has resulted in some interesting lab demonstrations with robots that can fold t-shirts or make coffee and so on, but none of these demonstrations were robust enough to scale to the real world. The number of robots capable of autonomous manipulation that are solving real-world problems today is effectively zero.”

Rhoda AI is betting on using video as data to bridge this gap and create truly generalized robotics.

How does a DVA model work?

Many physical AI developers are creating vision-language-action (VLA) models, which allow robots to learn from data from images or videos of their surroundings and from text instructions. According to Rhoda AI, these models, however, still struggle to cope with real-world variability.

The company said its DVA model, recently extended with mobile DVA, combines manipulation and mobility in a single policy.

“The way it works is the model will look at a history of video frames, action tokens, and language tokens. From that history, or from that context, it will make a prediction in the video for the next 100 milliseconds, the next half second or so,” Singh explained. “It will predict the scene and what it thinks will happen in that scene. Once you have the prediction, you pass it to a second part of the model, which is an action converter. It takes the video scene and replays the robot’s actions.”

“Once you have the output in action space, you can perform those actions on a real robot. The robot then moves for that half second, then you make a new observation, because the state of the world has changed, and you make a new prediction for the next half second,” he continued. “It’s basically making an observation, a prediction of what should happen next, a conversion into action, an execution of the action, then you close the loop and make a new observation and a new prediction based on that new observation.”

For Rhoda AI, the amount of data is key

According to Singh, any AI model trained on a small dataset will be very fragile. These models might work if the inference time distribution of the data is very similar to the training time distribution, but as those distributions diverge, the model starts to fail.

“Hundreds of years of video were used to train the model, so the model saw an incredible variety of everything: videos of baseball, people throwing darts, waves crashing on a beach, or trees blowing in the wind,” Singh said. “Not just videos that contain humans. Every video has something to teach the model about how the world works. Essentially, the entire Internet becomes like a physics classroom.”

Rhoda AI uses video data for pre-training. “Once you have pre-training, you apply a small amount of task-specific training, perhaps 10 to 20 hours, similar to the way you would train a new worker on an assembly line,” Singh said. “They already come in with a certain set of skills, and you have to teach them the workflow of your particular company. With that little bit of post-training, the model is capable of doing things that traditional robots can’t do.”

The Palo Alto, California-based company aims to create a model that can understand the world at large. As such, it doesn’t limit the data it uses to just human-centric videos.

“We looked at more polished videos that contained only human and self-centered manipulation,” Singh said. “We wondered if maybe it was better to train on those videos. We found that the best-performing models were the ones actually trained on the largest datasets, including things that weren’t obviously relevant to the manipulation.”

This generalization also allows Rhoda AI models to generalize across different robot embodiments. The specific incarnation of the robot, in fact, only comes into play in the post-training phase.

How does Rhoda AI offer its models?

Singh said Rhoda AI is a full-stack company and offers its technology in a wide range of methods.

“We always own the AI ​​layer, that’s the core,” Singh said. “In terms of the body or hardware level, we can operate with our own body or with a third-party entity. In terms of skills, we can train the skills, or we can let a third party train them. That provides the broadest range of solutions to our customers, and that’s really our mission.”

Internally, Rhoda AI is focusing on developing expertise in manufacturing and logistics.

“We are pursuing those areas because there is an immediate problem there,” Singh added. “Many manufacturing and logistics companies have labor shortages. Many industries have been offshored for decades because it’s too expensive to build them in those economies. So this technology offers a path to repatriate some of those industries.”

Looking to the future, Rhoda AI aims to continue bringing its model into the real world. Singh said he believes robotics’ “ChatGPT moment” will arrive sooner than people expect.

“The kind of approach we’re taking really has the potential to get us there, because the only way to get that level of generality is to train on a dataset that has both the scale of the Internet and the diversity of the Internet,” Singh said. “If you don’t have both of those things, your models won’t be able to generalize well enough.”