Whole-body / VLA
Fine-tune a VLA on the arm you own, compare it to P4 ACT, and do not fake a humanoid.
Weeks 29+ · SO-ARM101 real · humanoid stays in sim · $0 · 10_wbc_vla/





Goal
Recommended route: SmolVLA (~450M, shortest public SO-101 path) or GR00T N1.7 on the LeRobot format. Compare against the P4 ACT baseline on the same held-out cases. Plot a data-scaling curve at 10 / 25 / 50 demos. Alternative A (TWIST2, honestly scoped): reproduce in sim (they publish MuJoCo models, including a $250 2-DoF neck) and deploy the manipulation half on the real SO-ARM101. Alternative B (cross-embodiment): one architecture and one dataset format across arm, M3 Pro, rover, quadruped, documenting where reuse works. Do not fake the humanoid. Restraint is the point.
Why it matters
Best skill-to-cost ratio in the plan. You get onto the current frontier using hardware you already have. By arrival you will have built the halves separately: tracker in P9, diffusion in P4, runtime in P5. What you will not have is a Unitree G1.
The work
- 01
Fine-tune SmolVLA or GR00T N1.7 on the P4 (and later) LeRobot datasets.
- 02
Compare vs P4 ACT on identical held-out cases: success, generalization, latency.
- 03
Data-scaling curve: 10 / 25 / 50 demonstrations.
- 04
Write what the VLA gave you that ACT did not, and what it cost.
- 05
Optional: TWIST2 in sim + real-arm manipulation half, or the cross-embodiment study.
Done looks like
- A fine-tuned VLA running on the real SO-ARM101.
- Comparison vs P4 ACT on identical held-out cases: success, unseen objects/positions, inference latency.
- Data-scaling curve at 10 / 25 / 50 demonstrations.
- Written account of what the VLA bought, and what it cost in compute and latency.
Stack card
| Simulator | MuJoCo (arm / neck) + Isaac Lab (humanoid locomotion baselines) |
|---|---|
| Trains on | RTX 5090 (SmolVLA fits; GR00T is tighter) |
| Runs on | SO-ARM101 real + sim |
| Buy | $0. No humanoid. No VR headset. |
| ROS 2 | No |
| Gazebo | No |
| Compute / target | SO-ARM101, the real arm. Everything humanoid stays in simulation. |
| Install | lerobot + SmolVLA, or NVIDIA/Isaac-GR00T; optionally GMR for retargeting study |
Repos to study
Watch out
- Don't fake a humanoid hardware demo.
- No ROS 2. LeRobot + MuJoCo + Isaac Lab.
Workstreams
Fine-tune, real-arm deploy, optional TWIST2-in-sim.
ACT comparison, scaling curve, writeup.
Cross-check. Same held-out cases as P4. No new evaluation set that flatters the VLA.
What this proves
Frontier skill on hardware you own, without overclaiming a humanoid.
The credible line is a cross-embodiment pipeline on hardware you own, not “we shipped a humanoid.”
Study videos
Gear / BOM
