How it works
Nav2 reports a blockage. A foundation model looks at the current view, the task, what the robot can do and what it has already tried, then names the control to press, in words only, never pixels or coordinates. An open-vocabulary detector (Grounding DINO) finds that description in the image and depth lifts it to a 3D point. The arm presses, the system checks whether the door or elevator changed state, and navigation resumes. If nothing changed, the attempt goes back into memory.
The two stages exchange only a language description and a 3D point. Nothing is trained; every model is used off the shelf.
Results in simulation
Isaac Sim, a three-floor office, 120 trials per method per task. Every method uses the same reasoning model, navigation, execution and verification.
| Task | GAIN | Direct Spatial VLM | Direct VLM-2Call |
|---|---|---|---|
| Door: success | 109/120 (90.8%) | 104/120 (86.7%) | 106/120 (88.3%) |
| Door: model calls per trial | 4.24 | 7.59 | 11.69 |
| Elevator: success | 104/120 (86.7%) | 50/120 (41.7%) | 60/120 (50.0%) |
| Elevator: model calls per trial | 8.59 | 15.74 | 16.99 |
The direct VLMs point near the control or box the structure around it; the detector boxes the control. That's most of the elevator gap.
Where it still fails
30 of the 120 door trials put a plausible wrong control, like a fire alarm, next to the right one. GAIN got 24/30 (80.0%) of those against 85/90 (94.4%) without one, and both baselines reached 23/30. With n = 30 the confidence intervals overlap, so that's a cost every method pays, not a separation.
On long missions (an elevator ride and a door, three interactions in order, no partial credit), GPT-5.4 mini completes 85/100, Grok 4.3 60/100, and a Gemma 4-family model 18/100. Gemma mostly fails at choosing the action. The stronger models get past that and fail later, at target acquisition, grounding and physical execution. A better reasoner moves the bottleneck downstream; it doesn't remove it.
On the real robot
The same loop runs on a Ranger Mini 3 with an xArm6 in a campus building. These results are qualitative: there are no counted real-robot trials and no success rate to report. Every number above is from simulation.
| Stage | Status |
|---|---|
| Navigate a recorded route leg | Working: 44 of 58 logged arrivals within 0.15 m |
| Detect the way is blocked and escalate | Working |
| Ground the door plate in the image | Working, preferred over a fire alarm 20 cm away |
| Refuse to press a fire alarm | Working: the strongest forbidden hit landed on the real alarm and was vetoed |
| Press the door plate and drive through | Demonstrated |
| Call the elevator | Demonstrated |
| Select the floor inside the car | Demonstrated, not reliable: one run pressed 5 instead of 1 in a dark car |
| Full floor 5 to floor 1 mission, one run | Not yet completed |
From the bring-up log, as of September 2026.
