Multiple humanoid robots training in NVIDIA Isaac Lab
// HOME LAB · 29 WEEKS · SIM-FIRST

Learning robotics at home, in public.

Path: MuJoCo → Isaac factory → sysid → imitation → edge → rover → collector → menagerie → legs → VLA. Most of the work stays in sim. The SO-ARM101 on the desk and the Yahboom M3 Pro rover are the exceptions.

10 PROJECTS29 WEEKSSIM-FIRSTWEEK 01 / 29
// THE ONE RULE

Sim is the default

If it can run in simulation, it runs in simulation. Hardware is the exception: the SO-ARM101 we already own, and the Yahboom M3 Pro (Orin NX 16GB) that carries both edge deployment and the rover skill loop. A humanoid stays in sim. Do not fake metal.

Anyone can clone a repo and get one of these demos running over a weekend. What we are collecting across all ten is the failure data: policies that worked in sim and broke on the desk. The gap report is the thing worth keeping.

KEY CONCEPTS
Teleoperation

you move the robot in real time, like puppeteering.

Whole-body control

arms, torso, and legs controlled together, the way a person moves (not each half on its own).

Retargeting

taking a human motion and mapping it onto a robot whose body is a different shape.

Imitation learning

the robot learns a task from demonstrations, not from hand-written rules.

Reinforcement learning (RL)

the robot learns by trial and error in a simulator, rewarded for getting closer to the goal.

Sim-to-real

train in simulation, then run the skill on a physical robot.

Domain randomization

randomize the simulator (masses, friction, delays) so the skill survives the messy real world.

Egocentric vision

the robot acts from its own onboard camera, not a fixed room camera.

Policy

the network that picks the next action from what the robot senses.

Diffusion Policy / ACT

two common ways to turn demonstrations into a policy.

Eval harness

shared logger across every project: task, metrics, failure type, sim-vs-real gap. An unlogged run did not happen.

VLA

vision-language-action model: images (and often language) in, robot actions out.

// HOW PROJECTS CONNECT

The eval harness

The eval harness is not an eleventh project. It is one shared logger used in every project: task, metrics, how it failed, and the sim-vs-real gap. We grow it a little each week. It is the last thing we show.

  • Locomotion often transfers from sim. Manipulation usually does not. Contact and camera realism are where sim lies. That is why P3 measures the arm before P4 teaches it, and why P2's Lift-Cube is the RL baseline that P4's 50-demo ACT has to beat.
  • Sim-first is a cost choice and a technical one. Get most of the way in sim so the real-world hours go to the last mile. Saying “we did it all in sim” as if that proves the real robot is an excuse. Keep the first honest version.
// A FEW TOOLS, USED WELL

The toolkit

// LEARNING PATH

The spine

MuJoCo → Isaac factory → sysid → imitation → edge → rover → collector → menagerie → legs → VLA

Robot learning curriculum from simulation to humanoid control
// STACK

Where the compute lives

Mac is the control station. The RTX 5090 is the engine. The M3 Pro Orin NX is the edge brain.

Tech stack: GPU, Isaac Sim, Isaac Lab, RL, teleop, VR
ToolWhat it isWe use it for
MuJoCoContact physics on the Mac (Apple Silicon)Foundations, sysid, arm dry-runs (P1, P3, P4)
Isaac LabGPU-parallel simulator + RL (thousands of envs)Factory lift, legged RL (P2, P9)
ROS 2 + GazeboRuntime + sim for robots that move through spaceRover and ball collector only (P6, P7)
LeRobotSO-101 teleop, ACT, Diffusion, SmolVLAImitation and VLA on owned hardware (P4, P8, P10)
  • Zenoh is not a ROS 2 replacement. It is a faster messaging layer you can swap inside ROS 2 (rmw_zenoh, supported from Jazzy). Ignore it until there is a real networking problem. In sim we will not hit one.
  • Newton (NVIDIA + Google DeepMind + Disney) aims to fix Isaac Sim's weaker contact physics by wrapping MuJoCo's solver at GPU scale. Watch it. Do not build on beta yet.
  • The arm, edge, and VLA path skip ROS 2 on purpose. One arm and two cameras do not need a distributed graph. ROS 2 is for the rover and collector.
// 26-WEEK CALENDAR · WORKING BACKWARDS

Work backwards from the demo. Log every failure.

Twenty-six weeks, ten projects. Core (P1–P7) is the path that has to stick. Advanced (P8–P10) is optional depth. Skip the advanced work before you thin out the core. If a project slips, cut the stretch goal. Never skip the harness log. Use the toggle to read the calendar from the end.

MuJoCo foundations
WEEKS 1–2 · P1 · CORE

MuJoCo foundations

Milestone: Fish CPG + PPO on the laptop; harness v0 with sim | real | paired.

Physics and control from scratch. Hand-written PD, a scripted fish gait, then a small PPO swimmer, all logged. Cartpole as the whole project is out.

View folder ↗
W1IN PROGRESSMJCF + PD + harness v0

Milestone: Viewer runs; pendulum PD holds a setpoint; harness v0 logs a run.

mjpython / make view on macOS. Freeze held-out targets before any training.

● YOU ARE HERE · WEEK 1
W2PLANNEDFish CPG vs PPODEMO 1

Milestone: Scripted vs learned on 20 held-out targets; one reward A/B.

The fish stays in sim. Do not waterproof a printed fish.

Isaac Lab factory
WEEKS 2–3 · P2 · CORE

Isaac Lab factory

Milestone: Lift-Cube at 4096 envs; headless-vs-headful table; reward A/B.

Blackwell sm_120 gate first. Cartpole is a five-minute smoke, not the project. No ROS 2 bridge.

View folder ↗
W2PLANNEDCapability gate + Reach

Milestone: get_device_capability() → (12, 0); Reach-Franka trains.

Python 3.11 only. PyTorch ≥2.7.0 cu128.

W3PLANNEDLift-Cube + throughputDEMO 2

Milestone: 4096-env Lift-Cube; headless vs headful penalty; reward A/B on 50 inits.

Train headless overnight; replay checkpoints headful for clips.

System ID on the arm
WEEKS 3–5 · P3 · CORE

System ID on the arm

Milestone: Calibrated MJCF; overlay plots; GR01; reusable common/sysid/.

Command the same trajectory in MuJoCo and on the real SO-ARM101. No ROS 2. Middleware corrupts the measurement.

View folder ↗
W3PLANNEDMJCF + trajectories

Milestone: Joint limits verified; fit vs held-out split frozen.

Steps, sinusoids 0.25–2 Hz, multi-joint reach.

W5PLANNEDFit + GR01DEMO 3

Milestone: Friction/damping/armature fitted; GR01 written.

STS3215 internal control is invisible. Do not model ideal torque sources.

Imitation learning on the real arm
WEEKS 5–9 · P4 · CORE

Imitation learning on the real arm

Milestone: 2×2 ACT/Diffusion × real/sim data; failure taxonomy; GR02; RL-vs-IL one-pager.

Contract first: record → load → replay 5 episodes. Highest-impact project for jobs. Two different webcams.

View folder ↗
W5PLANNEDContract + 50 demos

Milestone: Schema validated; ~50 real pick-and-place episodes versioned.

Do not watch the follower while recording.

W9PLANNED2×2 + look back at P2DEMO 4

Milestone: All four cells on the same 20 real-arm cases; RL-vs-IL one-pager.

This is the experiment that makes it a portfolio piece.

Edge deployment
WEEKS 8–10 · P5 · CORE

Edge deployment

Milestone: P4 task on Jetson, workstation unplugged; quantization table; watchdog clips.

PyTorch → ONNX → TensorRT. Measure p99, not average FPS. common/deploy/ is written once.

View folder ↗
W8PLANNEDExport + drift

Milestone: FP16/INT8 vs FP32 on identical inputs.

Do not buy Jetson Thor. The $249 Super is the point.

W10PLANNEDWatchdog + disconnected demoDEMO 5

Milestone: Stale camera / stale command / NaN clips; 20-case workstation vs Jetson.

Cutting torque is not universally safe.

Autonomous rover + cloud skill loop
WEEKS 10–16 · P6 · CORE

Autonomous rover + cloud skill loop

Milestone: M3 Pro navigates offline, follows grounded cloud plans, recalls AWS memory, and accepts only correction skills that shrink the sim-to-real gap.

The onboard Orin NX keeps it safe; Claude plans and authors readable skills; numerical fitting finds exact values; held-out metrics decide what survives.

View folder ↗
W10PLANNEDM3 Pro + offline edge

Milestone: Orin NX runs motors, watchdog, perception, SLAM, and Nav2 without cloud access.

Cloud never sends motor commands. Pull the network cable as an acceptance test.

W12PLANNEDShared contracts + cloud planner

Milestone: Validated action vocabulary; /plan returns grounded JSON goals; rover_orchestrator executes them.

Unknown actions and malformed model output must fail closed.

W14PLANNEDAWS memory + paired traces

Milestone: Facts and vector recall work across sessions; matching sim/real motions produce a gap metric.

Exact facts and fuzzy recall are separate stores.

W16PLANNEDCorrection skills + final demoDEMO 6

Milestone: Three accepted skills lower held-out gap; voice kitchen check plans, reports, and remembers.

LLM diagnoses and writes the skill; scipy fits values; the metric keeps or rejects it.

Tennis-ball collector
WEEKS 16–19 · P7 · CORE

Tennis-ball collector

Milestone: 20 balls cleared unattended; HSV vs YOLO table; jam rate; GR04.

Tennibot’s Rover is $2,275. Most of that robot is P6. Vision is the gift; the intake and the battery are the project.

View folder ↗
W16PLANNEDHSV + task loop

Milestone: Nearest-ball → Nav2 → intake → beam-break.

Keep P6 succeeding on its own terms.

W19PLANNEDYOLO + 20-ball demoDEMO 7

Milestone: Detector table; physical 20-ball clear; battery budget vs Tennibot.

Honest surprise: the intake jam rate is the number.

Robot menagerie
WEEKS 19–23 · P8 · ADVANCED

Robot menagerie

Milestone: M3 Pro mobile-manipulation; Bittle characterized; interop note.

Point common/ at robots you did not design. Skip TurtleBot 4. A Go2 decision waits until week 20.

View folder ↗
W19PLANNEDP6 M3 Pro + P4 pipeline

Milestone: Moving-base imitation running on the rover already built.

Document every SO-ARM101 assumption that broke.

W23PLANNEDBittle + interopDEMO 8

Milestone: Servo characterization; one harness schema; docs/interop-note.md.

Three embodiments, one file format, or write why not.

Legged sim-to-real
WEEKS 23–29 · P9 · ADVANCED

Legged sim-to-real

Milestone: Locomotion policy; domain-rand ablation; 2×2 transfer; GR05.

A large, attributed gap on Bittle is a result. An unmeasured clean Go2 clip is a claim. Buy a Go2 only if this becomes the specialization.

View folder ↗
W23PLANNEDIsaac Lab walk + ablation

Milestone: Velocity-tracking PPO; with/without rand × 3 seeds.

Start from unitree_rl_gym.

W29PLANNEDTransfer + GR05DEMO 9

Milestone: 2×2 sim vs real, randomized vs not; gap attributed to P8 servo numbers.

Line-by-line review. This deploy path is reused in P10.

Whole-body / VLA
WEEKS 29+ · P10 · ADVANCED

Whole-body / VLA

Milestone: SmolVLA or GR00T on the real SO-ARM101 vs P4 ACT; 10/25/50 scaling curve.

Highest skill-to-cost ratio. Do not fake a humanoid. Restraint reads as senior.

View folder ↗
◆ DESTINATION
W29PLANNEDFine-tune + compareDEMO 10

Milestone: VLA on the real arm; ACT baseline on identical held-out cases.

Same evaluation set as P4. No new set that flatters the VLA.

// HOW THE LAB WORKS

Repo, workstreams, weekly ritual

BUILD STREAM

Build

Environments, reward design, policy training, deployment, sim-vs-real measurement, and the writeups people will see.

EVAL STREAM

Eval + integration

Eval-harness design across all 29 weeks, plus integration. The harness gets touched in the second half of every project.

Every project ends in three things: a folder, a short clip, and one entry in the shared eval harness.

MONOREPO

Agree this on Day 1

  • README.md: living roadmap, update every week
  • common/eval_harness/: logging, metrics, failure taxonomy (grows all 29 weeks)
  • common/sysid/: trajectory generator, paired logger, gap-report generator
  • common/deploy/: ONNX / TensorRT / watchdog, reused from P5 onward
  • common/domain_rand/: reusable randomization configs
  • 01_mujoco_foundations/ … 10_wbc_vla/
  • docs/gap_reports/: GR01–GR05
  • docs/weekly/: WK01.md … WK29.md, honest notes
  • docs/site.json: live status the public site reads
  • demos/: clips and numbers for the showcase
  • ShippedWhat actually works now, with the clip or the number. Not the intention.
  • BrokeWhat failed and the current guess for why. Most important section. A week with no failures logged means we did not push hard enough.
  • LearnedThe one concept that clicked, in our own words.
  • Stuck / decidingWhere we need a unblock, a scope cut, or a call.

Branch per workstream per project: build/p2-record, eval/p2-train. Never push experiment work straight to main. One PR review per week, minimum. Reading the other stream's code is half the learning. Every experiment writes a row to the eval harness. No exceptions. An unlogged run did not happen. On the io.net box: every SSH session starts with conda activate env_isaaclab before any python/pip. Skip it and the shell falls back to system Python 3.10 and breaks the stack.

// COURSES

The right few, in order

TierCourseWhy
CoreHugging Face Deep RL Course ↗Free, hands-on on-ramp to deep RL (PPO, policy gradients) with Stable-Baselines3. Feeds P1 and P9.
CoreLycheeAI: Isaac Lab (YouTube) ↗The community go-to free Isaac Lab resource. Feeds P2 and P9.
CoreThe Construct: ROS 2 (Beginner → Advanced) ↗Browser-based, zero-setup ROS 2. Fastest literacy without dependency hell. Feeds P6 and P7.
CoreHugging Face LeRobot docs + tutorials ↗Imitation-learning / teleop pipeline for P3, P4, P10.
FoundationsModern Robotics: Mechanics, Planning & Control (Northwestern, Kevin Lynch) ↗Classic kinematics / dynamics / control. Audit free on Coursera.
FoundationsUC Berkeley CS285: Deep Reinforcement Learning (Sergey Levine) ↗Free lectures + assignments. Gold standard for RL depth.
AdvancedETH Zürich: Robot Dynamics / Legged Robotics ↗Free YouTube lectures. For P8 / P9.
AdvancedMIT Underactuated Robotics (Russ Tedrake) ↗Locomotion and control theory. Optional, deep.

Udacity Robotics Software Engineer (nd209) is solid and project-based on ROS + Gazebo + C++, but about $999, and it does not cover Isaac Lab RL, LeRobot imitation, or foundation models. Skip it as the backbone. ROS/Nav is better and free via The Construct. Revisit only if a specific role later needs the C++/ROS credential.

// HARDWARE & ENVIRONMENT

What we actually run

  • Local: MacBook M4 Max. MuJoCo, LeRobot recording, control station. Not the Isaac engine.
  • Train box: RTX 5090 workstation (Blackwell sm_120). Isaac Lab 4096-env jobs, ACT / Diffusion / VLA fine-tunes.
  • Owned robot: SO-ARM101 leader/follower (Feetech STS3215). Used in P3, P4, P8, P10.
  • Edge + rover: Yahboom ROSMASTER M3 Pro with Jetson Orin NX Super 16GB. Shared by P5 and P6; P7 rides the same base.
  • Print: Bambu H2C for the collector intake. The rover itself is an integrated ROS 2 robot, not a printed chassis.
// GUARDRAILS

Pin these

  • Public site and repos stay framed as personal learning and education.
  • Sim-only manipulation is weak evidence. That is why P3 measures the real arm and P4 ends on it. Never present a sim-only manipulation number as if it proves the real robot.
  • Don't fake a humanoid hardware demo. Restraint is the point.
  • P4 traps: two different webcam models; watch the camera feed only, not the follower arm.
  • Isaac Lab: Python 3.11 only. PyTorch ≥2.7.0 cu128 or Blackwell silently wastes the card.
  • Newton: watch it, don't build on beta.
  • If a project slips, cut scope inside it (drop the stretch goal), never skip failure logging. Core is P1–P7. Advanced is P8–P10. Seven deep beats ten shallow.
// DEMO REEL

What we want at the end

A set of clips and one shared log. Not ten disconnected tutorials.

PLANNED
HERO REEL · PLANNEDVideo lands here when this milestone ships.
Clips and numbers from all ten projects, plus the eval-harness dashboard. Added as each project ships.