Unlocking the Path
Most robots stop at the first closed door. This one works out how to get through it, then keeps going.
Robotics
VLA
Role
Paper
Status
Shipping
team
With Dr. Minghan Wei — FAU Center for Connected Autonomy & AI
platform
Isaac Sim · ROS2 · Nav2 · xArm6 · Ranger Mini 3
The Real Problem
Unlocking the Path is training-free robotic navigation. Most systems treat a closed door, an elevator, or a floor change as a failure and stop. This one identifies which physical intervention the obstacle needs, executes it, and continues — no retraining, no task-specific policy. It's built on NVIDIA Isaac Sim with ROS2, Nav2, and open-vocabulary detection from vision-language models, then deployed on real hardware with Dr. Minghan Wei at FAU's Center for Connected Autonomy & AI.
Here's what made it worse: a navigation stack has no concept of acting on the obstacle in front of it. A closed door, the call button on an elevator, a step up to another floor — each of those reads exactly like a wall. The robot plans a path, finds none, and reports the goal as unreachable. A situation a person would solve in two seconds looks identical to a genuine dead end.
The failure cases were blunt:
'Door detected. Path blocked. Goal unreachable — stopping.'
'Elevator required to change floors. No ground path exists.'
'Obstacle is a movable door, but no policy exists to open it.'
The underlying issue wasn't the planner — it was that every system treated the environment as fixed. Nothing connected 'the path is blocked' to 'the blockage is something the robot can physically change.' That link had never been built.
Finding the Fix
I started by breaking a blocked route into its parts — not just where the path stopped, but what was actually stopping it and whether the robot could change it. Looking across the failure cases, three problems stood out:
No sense of intervention. The stack could tell that a path was blocked, but never what kind of action would clear it.
No object understanding. A door, a wall, and a person all looked like the same generic obstacle.
No way to act. Even when something was recognized as openable, there was no policy to actually open it.
Moving through a building is one continuous story — go down the hall, open the door, take the elevator, cross to the next wing. But the robot could only ever read the first blocked step, and treated everything past it as impossible.
So the approach came down to three parts:
Open-vocabulary detection with vision-language models, so the robot can name what is blocking it — a door, an elevator, a step — even for objects it was never explicitly trained on.
A mapping from recognized obstacle to physical intervention — this is a door, so open it; this is an elevator, so call it and ride it — without a separate learned policy for each case.
Executing that intervention on real hardware through ROS2 and Nav2 — the xArm6 acting on the door or button while the Ranger Mini 3 repositions — then handing control back to the planner so navigation just continues.
All of it training-free — no new policy, no retraining, no task-specific data collection.
What Actually Happened
I built the pipeline in Isaac Sim first and ran it against doors, elevators, and floor changes I'd staged in simulation, alongside Dr. Minghan Wei at FAU's Center for Connected Autonomy & AI. That's where I caught my mistakes.
For example: I'd assumed detecting the object was the hard part and acting on it was easy. In practice it was the reverse — open-vocabulary detection handled unfamiliar doors fine, but lining the xArm6 up to actually operate a handle, from wherever the base happened to stop, was where things broke.
The hand-back logic also evolved. My first version treated the intervention as one-shot: act, then immediately resume navigation. When the action didn't fully clear the path, the robot charged ahead anyway. I changed it to re-check the route before returning control, so it only continues once the way is actually open. Small change, big difference in reliability.
Then it moved from simulation onto the real robot — the xArm6 and Ranger Mini 3. The behavior held up: the robot met a closed door, recognized it, opened it, and kept going, on hardware it had never rehearsed that exact door on. We tuned the sim-to-real gaps, then ran it end to end.
What Changed
Obstacles that used to end a run — a closed door, a floor change, an elevator — became things the robot handled on its own and drove past. The same navigation stack now finished routes it would previously have reported as impossible, without a single obstacle-specific policy added.
But the part that mattered most was what it demonstrated
It handled a door it was never trained on, zero examples.
Recognize the obstacle, choose the intervention, execute, continue — no retraining in the loop.
Same policy, new environment, and it still found a way through.
Not one obstacle in the test set forced a fall back to 'goal unreachable', which felt like the real result.
What I Had to Work With
No task-specific training data. The whole point was to handle situations the robot was never trained on, so there was no dataset of doors or elevators to learn from. I had NVIDIA Isaac Sim to generate and test scenarios, open-vocabulary detection to recognize objects it had never seen labelled, and ROS2 and Nav2 underneath. Not a training set, but enough to build against.
Real hardware, not just sim. Everything had to run on an xArm6 manipulator and a Ranger Mini 3 base, so a plan that only worked in simulation didn't count. The arm's reach and the base's footprint set hard limits on which interventions were even possible.
Zero-shot, by definition. The robot had to act on obstacles it had never encountered before, with no chance to fine-tune a policy first. Whatever it did had to work on the first try.
These constraints shaped everything. Instead of training a new policy for every obstacle, I focused on a single mechanism that could recognize an intervention and carry it out with what the robot already had.
What I'd Do Differently
I'd get onto real hardware earlier. Isaac Sim was invaluable, but a lot of what breaks — the base stopping just out of the arm's reach, a handle that needs more torque than the sim modeled — only shows up on the physical xArm6 and Ranger Mini 3. Even one more week on hardware up front would have saved time later.
I'd also test a wider range of obstacles more systematically. I leaned on a handful of doors, elevators, and floor changes, which proved the idea — but a broader, structured set of interventions would have shown more clearly where the training-free approach holds and where it strains.
What I Learned
Reusing beats retraining. The tempting version was a specialized policy for every obstacle. But given the goal — act on things it had never seen — that made no sense. Recognition, an obstacle-to-action mapping, and the existing Nav2 stack delivered the whole behavior without training anything new.
The physical world beats the simulator. What worked cleanly in Isaac Sim didn't always survive contact with the xArm6 and Ranger Mini 3. Validating on real hardware saved me from shipping something that was correct in sim but useless in a hallway.
Sample efficiency is the whole point. A robot that handles a situation it was never trained on, from zero examples, is the extreme case of learning less and doing more. That — not any single door — is what Unlocking the Path is really about.
Unlocking the Path — training-free robotic navigation, built with Dr. Minghan Wei at FAU's Center for Connected Autonomy & AI.
