Training Physical AI for Full Mission Variability

Advancing the Gideon Physical AI Platform with NVIDIA Cosmos 3 World Models

By Valentina Zadrija, Juraj Radić, Maja Magdalenić and Shruti Bhandary, Gideon AI

Gideon is teaching NVIDIA Cosmos 3 world models to detect anomalies, like damaged pallets, shifted stacks or overhanging freight in dock automation. Built on vast real-world and simulated data, the models give the TREY autonomous loading and unloading forklift an extra layer of slower, deliberate reasoning, known as System 2, that helps it handle unexpected situations and provide more uptime for critical supply chain operations.

Nearly everything people and businesses use moves through a supply chain, from food and medicine to vehicles, electronics and industrial components. Behind that movement is a vast network of people and equipment that does the physical work of moving goods – and it all starts at the loading dock, where transportation meets operations inside the warehouse or distribution center. Goods pass through the loading dock, moving out of semi-tractor trailers and into warehouses, and back again, safely, reliably and at a very fast pace. Much of that work relies on manual forklifts and their operators.

Gideon, a member of the NVIDIA Inception program, has developed TREY, an autonomous trailer loading and unloading forklift augmented by NVIDIA AI infrastructure. Powered by NVIDIA Jetson Orin and NVIDIA RTX GPUs, TREY automates the loading dock, helping supply chains become safer, more predictable and free from the constraints of manual labor.

Handling Pallets when Every Pallet Sits Different

NVIDIA Cosmos 3 models represent an opportunity to advance the Gideon Physical AI platform by helping TREY handle more of the anomalies that fall outside the patterns it normally encounters inside a trailer.

Trailers are inherently variable environments. They contain load carriers, pallets and steel racks that come in more than a hundred different types, sizes and weights. When the TREY autonomous forklift enters a trailer, it finds loads arranged in various layouts. Pallets and loads may also shift in transit, so the trailer can arrive with broken pallets, floor debris, or leaning loads.

One challenging scenario is a misaligned double stack, where the upper unit load is offset from the one beneath it, so that it overhangs the lower load’s front face or one of its sides. This creates an unstable situation, and if the automated forklift were to try to lift either pallet, it may fall, causing damage to the product and a work stoppage that would require manual cleanup.

In supply chain operations, every minute counts, as teams work to meet daily order deadlines. The same goes for manufacturing plants running just-in-time workflows that start the moment materials arrive at the dock door. Downtime is not an option.

Teaching NVIDIA Cosmos 3 to Safely Judge Anomalies

To identify and evaluate exceptions such as damaged pallets and loads, overhanging loads or non-palletized freight in a trailer of palletized goods, the Gideon development team has started to post-train NVIDIAs Cosmos 3 world foundation models. Even though only about 2–3% of situations fall into these categories, each one can stop work at the dock. This solution aims to improve dock performance and uptime, minimize product damage and loss, and maintain a safe working environment.

Currently, TREY already checks load stability, pallet condition, obstructions and wall clearance whenever it works inside a trailer, and its autonomy stack handles planning, navigation, actions and safety. Cosmos adds a second perspective focused on the unexpected. Inspired by the System 2 concept for Physical AI and robotics, Cosmos takes a slower, more deliberate look at the scene and reasons about what it sees. TREY stays in control, and Cosmos adds an extra layer of confidence to a system that is already built to be safe.

Out of the box, Cosmos has a broad understanding of the physical world, but it isn’t specialized for what happens inside a trailer. Getting it there takes real dock data.

Real Deployments, Turned Into a Data Flywheel by AI Agents

While much of physical AI is still being proven in the lab, Gideon’s is already at work on customer docks. TREY autonomous forklifts are loading and unloading trailers in commercial operation for enterprise customers in automotive, retail, food and beverage, and third-party logistics across the U.S. and Canada. Every shift on a live dock produces real-world data that feeds model development.

Across the robotics industry, deployment data is increasingly seen as the scarcest ingredient in physical AI, and edge cases are the hardest part of it to find. They are rare by nature and only show up in real production trailers, one dock door at a time. Through years of trusted work with customers, Gideon has access to what real operations reveal: regular operations, the moments where the robot fell short, and for each of those, a demonstration of how an experienced forklift driver handled the situation. This puts Gideon in a strong position to learn from the real world: access earned through deployments, and the work the team puts into turning that access into better models.

Gideon has put AI agents to work on operational data from its deployed fleet. The agents pinpoint where TREY struggled and why, and flag those moments as candidate edge cases. Engineers then review and curate the candidates, with agents assisting, into a physical AI dataset drawn from anonymized customer environments across the supply chain. For this project, the team used those scenarios to post-train three NVIDIA Cosmos 3 models: Edge, Nano and Super.

Fine-Tuning for Better Judgment and Fewer Surprises

Following NVIDIA’s post-training guidance, the team fine-tuned the reasoning tower of the model using LoRA, a post-training technique that adapts a small set of additional parameters while leaving the pre-trained model intact. Because so few parameters are trained, the whole process runs on a single NVIDIA RTX PRO 6000 GPU instead of a multi-GPU server. That keeps costs low and lets the team test new ideas quickly.

As the team trains the model, the autonomous forklift is able to identify and respond to more situations that make real-world unloading hard: a cracked pallet stacked on a second one, double-stacked loads bowing under their own weight or mixed freight with boxes overhanging their pallets and dunnage wedged between them. In these cases, there may be cues pointing in different directions, and the call to proceed is a close one. These are exactly the cases where better judgment adds the most value.

What’s Next for Gideon

Gideon is expanding TREY deployments with enterprise customers, and every new dock door feeds the models that make the whole fleet smarter. As work like this Cosmos 3 project matures, it reaches robots already in the field as software updates, so customers get more uptime from the TREYs they already run. For Gideon, the NVIDIA Inception program helps bring advanced physical AI out of the lab and onto the dock, and from there into more of the warehouse and factory.

To learn more about Gideon’s Physical AI platform visit Gideon.ai