Imitation learning on the arm (tethered)
50 demos collected on the arm, trained on the 5090, policy runs on the workstation with the arm on a cable. P5 cuts the cord.
Weeks 5–9 · SO-ARM101 · optional second webcam ~$40 · 04_arm_imitation/






Before you start
Workstation means the physical RTX 5090 desktop, not a simulator. MuJoCo and Isaac are simulators that run on it.
- Data is collected on the arm. Teleoperation and cameras are saved locally by LeRobot over USB.
- Training runs on the RTX 5090.
- Inference runs locally with the arm tethered until P5 moves the policy to the Jetson.
Rule of thumb: train remote, infer local. Training is a batch job. The live control loop runs many times a second. An ocean-crossing round trip creates laggy motion and failed grasps.
- 5090 local: record, train, and run inference on the same desktop. Everything stays in one room.
- 5090 remote: record on the Mac, upload the dataset, train remotely, download the checkpoint, then run inference on the Mac beside the arm.
That is the entire exchange. ACT and Diffusion checkpoints can run locally on the Mac M4 Max. The 5090 is not needed for inference.
- Upload: the dataset, roughly 50 demonstrations containing joint angles and camera frames.
- Download: the trained checkpoint, a small file containing the policy weights.
A cloud model is too slow for motor commands and does not reason in torques. Keep the brain in the cloud and the reflexes local. Failed plans become new instructions or demonstrations. This is the same two-layer pattern used by the P6 rover.
Run both tracks and compare them: ACT the classic way, the cloud model the new way.
- Track A, classic imitation: ACT or Diffusion Policy learns from the 50 demonstrations. It is fast, deterministic, and runs locally.
- Track B, cloud planner: an Opus- or Gemini-class model observes the scene and emits a structured goal, never motor commands. Local motion primitives or the Track A policy execute it in real time.
Goal
This is the project people care about most for jobs. Contract first: record → load → replay 5 episodes before collecting 50. Validate schema, camera identities, timestamps, sync. Confirm every demo is visible from the policy’s cameras. Collect about 50 real pick-and-place demos (fixed object, modest start-pose randomization). Train ACT on the 5090. Deploy: 20+ trials with a failure taxonomy (approach, grasp slip, release miss, collision, timeout). Using the P3 model, collect sim demos for the same task, train the same architecture, deploy to the real arm, compare on identical cases. Run Diffusion Policy on the same data. Then write RL-from-scratch (P2) versus imitation-from-50-demos: compute cost, human time, reliability, generalization.
Why it matters
Data is collected on the arm through teleoperation and cameras. Training runs on the RTX 5090 workstation. Inference also stays on that workstation with the arm tethered until P5 moves the exact policy to the Jetson. Imitation learning and VLAs are where a lot of attention is right now. LeRobot is the open stack most people use, and you own the hardware it was built around. The experiment that makes this worth showing is the 2×2: ACT / Diffusion × real-data / sim-data, all four on the same 20 held-out cases on the real arm. LeRobot talks to cameras and servos directly. No ROS.
The work
- 01
Contract first: record → load → replay 5 episodes. Schema, cameras, timestamps, sync. Confirm the task is camera-observable.
- 02
Collect ~50 real demonstrations of one bounded pick-and-place. Version the dataset.
- 03
Train ACT on the 5090. Deploy. 20+ trials plus a failure taxonomy.
- 04
Sim-data arm: same task in the P3 model, same architecture, deploy to the real arm, compare.
- 05
Diffusion Policy on the same data and the same evaluation cases.
- 06
Look back at P2. Write the RL-vs-imitation one-pager from your own numbers.
Done looks like
- A versioned dataset of ~50 real demonstrations with a validated schema.
- A 2×2 results table (ACT / Diffusion × real-data / sim-data) on the same 20 held-out real-arm cases.
- A failure distribution histogram by category.
- Video: task success, plus a representative failure of each category.
- docs/gap_reports/GR02-sim-vs-real-demos.md
- A one-page RL-vs-imitation comparison referencing P2 numbers.
Stack card
| Simulator | MuJoCo (calibrated model from P3) for the sim-data arm |
|---|---|
| Trains on | RTX 5090 |
| Runs on | SO-ARM101 · inference on the workstation (P5 moves it) |
| Buy | $0 required · optional wrist cam ~$40 |
| ROS 2 | No |
| Gazebo | No |
| Compute / target | None yet |
| Install | lerobot (ACT + Diffusion Policy), torch ≥2.7 cu128, opencv-python, wandb or local logging, av |
Repos to study
Watch out
- Two identical USB webcams reassign paths and crash the recorder. Use two different cameras or pin USB paths.
- Do not watch the follower while recording. You will use information the cameras cannot see.
Workstreams
Contract, dataset, ACT train/deploy, sim-data arm, Diffusion train.
2×2 table, failure histogram, GR02, RL-vs-imitation one-pager.
Cross-check. Eval stream records at least 5 episodes; build stream runs one training end to end.
What this proves
You can run the full modern manipulation pipeline and design a controlled experiment inside it.
Run the full modern manipulation pipeline and design a controlled experiment inside it, rather than getting one policy to work.
Study videos
Gear / BOM


