AIDA constructs demonstrations through the same high-level skill interface used by the deployed agent. Teacher decisions drive collection, making active visual behavior part of the supervision.
High-level labels are derived from motion trajectories.
Plan a motion
trajectory.
Split the path
into chunks.
Project path points
to image coordinates.
Search(k)Nav(u, v)Pick(u, v)Observations + skill labels
The teacher directly generates high-level actions in a closed loop.
Observe the environment.
Choose a skill + parameters.
Execute the selected skill.
Search(k)Nav(u, v)Pick(u, v)Observations + skill labels
The agent transfers an object to a target receptacle in a pre-mapped environment, using a list of scene key frames with known camera poses. A VLM predicts high-level skills and their parameters; low-level controllers execute the corresponding robot motions.
Task instruction“Pick banana from kitchen island to table in the living room.”
search_scene_frame(id)Select a scene key frame. Navigate to its associated pose.
turn(angle)Rotate in place to adjust the robot’s view.
nav_to_point(u, v)Select an image point. Depth and camera pose ground the navigation destination.
pick(u, v)Select a point on the object for the controller to grasp.
place(u, v)Select a destination point for the controller to place the held object.
AIDA builds training examples through the agent’s skill interface. The teacher provides skill and parameter targets; the offline annotator adds pre-action reasoning targets. The shared harness assembles input prompts and episode memory, with a consistent output schema for training and deployment.
The teacher selects and executes skill calls through the student’s interface, using simulator state during collection. It turns to acquire or recover targets, keeps them in view during approach, and moves within reach for picking or placing. Each executed skill and its parameters become action supervision.
After collection, deterministic templates use recorded simulator information to describe target visibility, relative direction, skill readiness, and task progress. These annotations form the reasoning targets that the student learns to produce before predicting a skill and its parameters.
The harness assembles prompts with the instruction, current image, skill definitions, and prior calls, parameters, and step summaries. Search prompts also include scene key frames. It pairs prompts with annotated outputs for training, then maintains the same format and updates memory during deployment.
We evaluate task completion, compare demonstration sources and teacher behaviors, and examine the role of the harness before testing real-robot deployment.
Full-task success
Object picked
Object-image retrieval
All three AIDA teachers outperform chunk-derived supervision. Active visual behaviors further improve success.
Shared model, training setup, four skills (no turn), reasoning labels, and harness. AIDA: mean of three evaluations.
| Key frames | History | Reasoning | Success |
|---|---|---|---|
| Every stage | None | No | 55.88% |
| Search | None | No | 60.00% |
| Search | Actions | No | 65.88% |
| Search | Actions | Yes | 71.76% |
Same AIDA demonstrations and five skills. The final configuration adds spatial reasoning and step summaries.
AIDA-OWMM runs on Atribot S1 without additional model training, using the same high-level skill interface.
