RADAR Closed-Loop Robotic Data Generation via Semantic Planning & Autonomous Causal Environment Reset

Yongzhong Wang1★, Keyu Zhu1★, Yong Zhong1★, Liqiong Wang1, Jinyu Yang2†, Feng Zheng1,3

¹Southern University of Science and Technology · ²Harbin Institute of Technology, Shenzhen · ³Spatialtemporal AI

IEEE/RSJ IROS 2026

★ Equal contribution † Corresponding author

Autonomous closed-loop data generation — plan, execute, verify, and reset with no human in the loop.

Abstract

The acquisition of large-scale physical interaction data is severely bottlenecked by the cost and scalability limits of human-in-the-loop collection. We introduce RADAR — Robust Autonomous Data Acquisition for Robotics — a fully autonomous, closed-loop data generation engine that removes human intervention from the collection cycle. Anchored by merely 2–5 3D human demonstrations as geometric priors, a Vision-Language Model first orchestrates scene-relevant task generation via semantic object grounding and skill retrieval. A Graph Neural Network policy then translates these subtasks into robust physical actions through in-context imitation learning. Following execution, the VLM performs automated success evaluation using a structured Visual Question Answering pipeline. Finally, a Finite State Machine orchestrates autonomous environment reset and asymmetric data routing — driven by simultaneous forward–reverse planning under a strict Last-In, First-Out causal sequence, seamlessly restoring workspaces and recovering from failures. This continuous brain–cerebellum synergy achieves up to 90% success on complex long-horizon tasks in simulation, and reliably executes contact-rich atomic skills — such as deformable object manipulation — on hardware via few-shot adaptation without domain-specific fine-tuning.

Method

RADAR separates high-level cognition from low-level control: the VLM reasons about what to do and whether it worked; the graph-diffusion policy handles sub-millimeter SE(3) execution grounded in real human demonstrations.

RADAR pipeline overview and decoupled Finite State Machine — **Pipeline & decoupled FSM.** Physical execution states (A Task Planning, B Forward Execution, C Reverse Execution) are separated from concurrent data-routing (D Dual Storage, E Single Storage). A fully successful cycle loops B→C→B and triggers Dual Storage; an asymmetric recovery loop B→C→A bypasses reset failure via Single Storage and re-plans on the altered workspace.

The Brain

Vision-Language Model

grounding · task gen · evaluation

Grounds objects, decomposes long-horizon goals into atomic subtasks, retrieves congruent 3D priors, and verifies outcomes through structured VQA — replacing fragile pixel generation and 3D coordinate hallucination.

The Cerebellum

Graph Neural Network Policy

in-context imitation · reverse-diffusion

Represents demonstrations, observations, and future actions as a heterogeneous graph, then denoises action nodes into valid end-effector transformations — high-frequency control from a handful of examples.

Scene-Relevant Task Generation

Ground objects and geometry, decompose goals into atomic subtasks, each matched to a library demonstration.

VLM

Execution via In-Context IL

A graph-diffusion policy maps retrieved demos and the point cloud to continuous trajectories under closed-loop control.

GNN policy

Automated Success Evaluation

A three-stage VQA pipeline — task-to-query, assessment, boolean decoding — yields a deterministic success signal.

VLM + LLM

Autonomous Environment Reset

The FSM computes the LIFO-causal inverse of the forward plan to restore the workspace — no hard-coded scripts.

FSM + VLM

Finite State Machine — operational loops

B → C → B Continuous success — forward and reset both succeed; harvest trajectory variations without re-planning. Dual Storage

B → C → A Asymmetric recovery — forward succeeds, reset fails; keep the valid trajectory, treat the scene as novel. Single Storage

B → A Forward abort — execution fails; discard the invalid trajectory, re-observe and re-plan. No Storage

Scene-Relevant Task Generation

Rather than back-projecting noisy 2D keypoints, the planner grounds objects semantically and retrieves the most geometrically congruent 3D prior — shifting the precision burden onto human-demonstrated trajectories.

Results

Evaluated in RLBench against two state-of-the-art vision-language manipulation baselines, MOKA and ReKep, over 10 rollouts per task with randomized initial states.

90%
Peak success on complex long-horizon tasks
10/10
Perfect on Push Block & Close Box skills
1-shot
A single demonstration as context

Task	ReKep	MOKA	Ours
Atomic
Large Container (Cup)	2/10	2/10	8/10
Large Container (Block)	0/10	3/10	8/10
Large Container (Laptop)	0/10	1/10	9/10
Push Block	4/10	4/10	10/10
Stack Block	4/10	1/10	8/10
Close Box	4/10	3/10	10/10
Open Box	2/10	2/10	7/10
Long-horizon
Put Laptop & Cup into Tray	1/10	0/10	8/10
Push & Stack Blocks	0/10	0/10	4/10
Close then Open Box	2/10	1/10	9/10

Table 1 · Success rates in RLBench over 10 rollouts per task.

Long-horizon task rollouts in RLBench — **Long-horizon rollouts (RLBench).** Multi-step sequences over distinct objects, and state-dependent articulated actions — opening a box that was previously closed.

Ablation · Point-Cloud Masking

Feeding raw scene point clouds into the execution policy causes catastrophic failure — VLM-driven selective attention is what makes execution robust in clutter.

Large Container (Cup)

w/o mask

0.10

with mask

0.80

Large Container (Block)

w/o mask

0.00

with mask

0.80

Push Block

w/o mask

0.00

with mask

1.00

Table 2 · Ablation on point-cloud masking.

Real-World Deployment

Under a 1-shot adaptation paradigm without domain-specific fine-tuning, RADAR executes contact-rich physical skills — deformable manipulation and precise grasping among distractors.

Real-world qualitative results: folding a towel, placing a strawberry, pushing and stacking a block — **Hardware rollouts.** (a) Folding a towel. (b) Selective attention isolates a strawberry among distractors for precise grasping. (c) A complex instruction decomposed into a push-then-stack skill chain.

Arm · Realman RM65-B Camera · RealSense D435i Segmentation · SAM + XMem++ Adaptation · 1-shot in-context

The Reset Challenge

Fully 100% reliable environment resetting remains open. Chaining a forward execution with a causal reverse reset inherently compounds failure rates:

p_total ≈ p_forward × p_reverse

The FSM currently acts as a robust proof-of-concept for simple-to-moderate scenes. Overcoming this compounding error in highly unstructured environments — via multi-modal tactile feedback or high-frequency visual servoing — is a frontier for future work, alongside using the generated data to train downstream visuomotor policies.

BibTeX

wang2026radar.bib

@inproceedings{wang2026radar,
  title     = {RADAR: Closed-Loop Robotic Data Generation via Semantic
               Planning and Autonomous Causal Environment Reset},
  author    = {Wang, Yongzhong and Zhu, Keyu and Zhong, Yong and
               Wang, Liqiong and Yang, Jinyu and Zheng, Feng},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots
               and Systems (IROS)},
  year      = {2026}
}