ALIBERO-VPro × GPT-6 Astra

How does GPT-6 Astra respond when visual evidence becomes unreliable?

We evaluate GPT-6 Astra on three tasks across 14 LIBERO-VPro perturbation settings from six benchmark categories. These settings yield 41 applicable task–setting cases, together with three Normal cases. We compare Astra’s success rate with three vision-language-action models and three world-action models on the same task–setting subset.

Evaluation Tasks

NormalSuccessTurn on the stove and put the moka pot on it

NormalSuccessPut the black bowl in the bottom drawer and close it

NormalSuccessPut the yellow and white mug in the microwave and close it

Overall Results

Astra achieves the highest overall success rate. Across the evaluated task–setting cases, Astra reaches a 70.7% success rate. LingBot-VA records the highest rate among the six reference policies at 58.3%. The advantage is category-dependent: Astra is comparatively weak under complete object-level masking, but stronger under Cross-View Contradiction and task-relevant scene changes.

Overall success rate across the 41 applicable perturbation cases. Astra is evaluated once per case; each reference policy uses ten trials per case.

Success rates for the Normal condition and all 14 evaluated LIBERO-VPro perturbation settings. Bold marks the highest rate in each row.

Astra is more sensitive to missing task evidence than missing gripper cues

Among the three masking conditions, Astra’s success rate is lowest when the target object is fully hidden. Its success rate is higher when the target receptacle or gripper region is masked, indicating that removing target-object evidence is more disruptive than hiding the gripper cue. The reference policies generally retain higher success rates under object- and receptacle-level masking.

Success rate under complete target-object masking, complete target-receptacle masking, and gripper interaction-cue masking.

Same task · Turn on the stove and put the moka pot on it

AstraFailureTarget-Object Masking. Fully hiding the moka pot removes the task evidence Astra needs to locate and manipulate the target.

AstraSuccessInteraction-Cue Masking. With the gripper region hidden, Astra still locates the moka pot and completes the task.

Astra prioritizes task success over visual consistency

Astra achieves the highest success rate under Cross-View Contradiction, reflecting its strong focus on completing the instructed task even when the two views disagree. In both successful cases below, however, the gripper first closes on empty space where one view suggests the object should be. Astra continues replanning until the task succeeds rather than explicitly reconciling the inconsistent observations. Its strong final success can therefore conceal visually implausible interactions during closed-loop execution.

Success rate under existence, position, and state contradictions injected into the third-person or wrist observation. Overall combines all six settings.

AstraSuccessThird-person existence contradiction. The first grasp closes on empty space; Astra retries and ultimately completes the task.

AstraSuccessWrist-view existence contradiction. Astra initially grasps air, changes its approach, and succeeds on a later attempt.

Astra adapts to changed scene configurations more often than the reference policies

Astra substantially outperforms all six reference policies under Same-Position Distractors and Task-Precondition Variation. The successful cases show two complementary capabilities: re-localizing a target displaced from its expected position and adapting the action sequence when object pose or receptacle openness changes.

Success rate under Same-Position Distractors and Task-Precondition Variation.

Task-Precondition Variation · Receptacle 50% open

π0.5FailureThe policy does not adapt its execution to the changed receptacle state.

AstraSuccessAstra accounts for the half-open microwave and completes the task.

Task-Relevant Distractors · Same-Position Distractor

π0.5FailureThe policy grasps the distractor at the original location and ignores the displaced target object.

AstraSuccessAstra finds the displaced mug and ignores the distractor at the original location.

Setup

Benchmark	The evaluation covers three tasks and 14 LIBERO-VPro perturbation settings across six challenge categories.
Observations	Each trial provides synchronized third-person and wrist RGB observations. A perturbation is applied only to the scene component or visual stream specified by that setting.
Astra interface	Astra interacts with the simulator through RPent tools for scene inspection, image-to-world back-projection, Cartesian motion, and gripper control.
Astra configuration	Astra uses low reasoning effort, a 50-turn limit, environment seed 7, and a 10,000-step simulator horizon.
Reference models	The comparison includes π0, π0.5, OpenVLA-OFT, FastWAM, LingBot-VA, and LaWAM, each evaluated with its native inference runner.
Trials	Astra is run once for each applicable task–setting pair; each reference model is run ten times.
Outcome	The simulator’s native success predicate determines whether each trial succeeds or fails.

Limitations

Astra is evaluated once per applicable task–setting pair.
The study covers only part of the LIBERO-VPro perturbation taxonomy. We will extend Astra evaluation to the complete benchmark.