01 Target-Object Masking
SuccessLIBERO-VPro: Benchmarking Closed-Loop
Visual Robustness of Robotic Foundation Models
1 Singapore Management University 2 Princeton University 3 Fudan University
† Corresponding author and project lead.
Abstract
Robotic foundation models achieve impressive performance on standard manipulation benchmarks, yet these evaluations typically assume clean, timely, and consistent visual observations throughout execution. We introduce LIBERO-VPro, a benchmark for systematically evaluating the closed-loop visual robustness of robotic foundation models by perturbing the visual evidence available during execution.
LIBERO-VPro covers four complementary dimensions, including Visual Evidence Degradation, Camera Staleness, Visual Source Consistency, and Task-Relevant Scene Variation, spanning 12 challenge categories, 96 experimental settings, and 3,296 task-condition cases. We evaluate three vision-language-action models and three world-action models over approximately 196,000 simulated episodes, complemented by 200 real-world rollouts on a Franka Research 3.
Our results reveal that strong nominal performance can mask substantial weaknesses in visual grounding and adaptation. Models often remain successful despite severe object-level occlusion, yet degrade sharply when local interaction cues are disrupted or familiar spatial priors are violated. They are also highly sensitive to stale or missing observations and struggle when changed task preconditions require behavioral adaptation.
Finally, VLAs and WAMs exhibit distinct robustness profiles, showing that visual robustness is multi-dimensional and architecture-dependent. LIBERO-VPro provides a systematic diagnostic framework for developing robotic foundation models that can more reliably ground and adapt their actions under challenging visual conditions.
Benchmark overview
LIBERO-VPro organizes 12 challenge categoriesinto four complementary dimensions of closed-loop visual robustness.Visual Evidence Degradation removes or corrupts task-relevant visual information; Camera Staleness introduces temporal mismatch between observations and the current physical state.Visual Source Consistency introduces contradictory evidence across views or between predicted and realized observations; Task-Relevant Scene Variation modifies visual properties that should require behavioral re-grounding.The benchmark covers 96 experimental settings and 3,296 task-condition cases across 40 LIBERO tasks. The representative simulation cases below instantiate these dimensions through controlled examples.
Dimension 01 · 5 categories
Visual Evidence Degradation
Removes or corrupts task-relevant visual information at different spatial and temporal scales.
Shared task
Put the bowl on the plate.
02 Target-Receptacle Masking
Success03 Interaction-Cue Masking
Failure04 Temporary Observation Corruption
Success05 View Unavailability
π0.5 · Wrist camera blacked outFailure
π0.5 · Third-person camera blacked outSuccess
Dimension 02 · 1 category
Camera Staleness
Introduces temporal mismatch between observations and the current physical state through delay, freezing, and replay.
Shared task
Put the bowl on the plate.
06 Camera Staleness
π0.5 · Delay-3 · third-personFailure
π0.5 · Freeze · third-personFailure
π0.5 · Replay · third-personFailure
Dimension 03 · 2 categories
Visual Source Consistency
Introduces contradictory evidence across views or between predicted and realized observations.
Shared task
Put the bowl on the plate.
07 Cross-View Contradiction
Success08 Prediction–Reality Divergence
Failure
Dimension 04 · 4 categories
Task-Relevant Scene Variation
Modifies visual properties that should require the policy to re-ground its behavior in the current scene.
Shared task
Pick up the alphabet soup and place it in the basket.
09 Task-Relevant Distractors
Failure10 Task-Precondition Variation
Failure11 Visual State Ambiguity
Success12 Target Geometry Variation
SuccessExperimental Results
Visual robustness is multi-dimensional. Table II summarizes performance across the 11 challenge categories shared by all six models.
No model consistently dominates across all categories, showing that robustness is multi-dimensional rather than a single capability that scales uniformly with model performance.
Visual grounding and adaptation
The results reveal that strong nominal performance can mask weaknesses in visual grounding and online adaptation.
What Visual Evidence Do Policies Rely On?
Interaction Cues Matter More Than Static Object Appearance.
Across all six models, gripper masking is the most disruptive intervention, while masking contact regions is generally less harmful.
How Do Policies Use Multi-View Visual Evidence?
Stale and Missing Views Reveal Strong Camera Dependence.
Five of the six models are more sensitive to wrist-view perturbations, while LaWAM shows stronger dependence on the third-person camera.
Can Policies Re-Ground When the Scene Changes?
Spatial Distractors Expose Reliance on Learned Positional Priors.
Position distractors increase distractor-only contact, whereas color and shape distractors more often preserve target-directed interaction.
Representative physical rollouts
Representative real-world rollouts under two LIBERO-VPro perturbations: Same-Position Distractor on the Pick-and-Place task and Third-Person Camera Staleness on the Move task.
Task-Relevant Scene Variation
Same-position distractor
The target object is moved nearby while a distractor occupies its original position.
“Pick up the yellow tape and place it in the purple box.”
Success: re-localizes the displaced target and places it in the purple box.
Failure mode: misses the displaced target after the distractor occupies its familiar original position.
Camera Staleness
Third-person delay by one replan
The third-person observation is delayed by one replanning cycle while the wrist view remains current.
“Move the white box to the right.”
Success: completes the Move task despite the delayed third-person observation.
Failure mode: follows a pushing trajectory without establishing contact with the white box.