SO-ARM101 kit
// GEARS

Hardware we own. Software we run.

The plan needs one real robot (the SO-ARM101 we already have) and a rented 4090. Everything else is sim. The stack lives here so we install the same things.

  • Local: MacBook M4 Max. MuJoCo, LeRobot recording, control station. Not the Isaac engine.
  • Train box: RTX 5090 workstation (Blackwell sm_120). Isaac Lab 4096-env jobs, ACT / Diffusion / VLA fine-tunes.
  • Owned robot: SO-ARM101 leader/follower (Feetech STS3215). Used in P3, P4, P8, P10.
  • Edge + rover: Yahboom ROSMASTER M3 Pro with Jetson Orin NX Super 16GB. Shared by P5 and P6; P7 rides the same base.
  • Print: Bambu H2C for the collector intake. The rover itself is an integrated ROS 2 robot, not a printed chassis.
ToolWhat it isUsed for
MuJoCoContact physics on the Mac (Apple Silicon)Foundations, sysid, arm dry-runs (P1, P3, P4)
Isaac LabGPU-parallel simulator + RL (thousands of envs)Factory lift, legged RL (P2, P9)
ROS 2 + GazeboRuntime + sim for robots that move through spaceRover and ball collector only (P6, P7)
LeRobotSO-101 teleop, ACT, Diffusion, SmolVLAImitation and VLA on owned hardware (P4, P8, P10)
P01 · Weeks 1–2

MuJoCo foundations

Sim only · MacBook · $0

View project →
MacBook M4 Max

MacBook M4 Max

IN PROGRESS

MuJoCo, LeRobot recording, control station. Not the Isaac engine

ownedWeeks 1–26
P02 · Weeks 2–3

Isaac Lab factory

RTX 5090 workstation · $0 extra

View project →
RTX 5090 workstation

RTX 5090 workstation

PLANNED

Isaac Lab 4096-env jobs, ACT / Diffusion / VLA fine-tunes. Blackwell sm_120

ownedWeeks 2–26
P03 · Weeks 3–5

System ID on the arm

SO-ARM101 you own · $0

View project →
SO-ARM101 (leader + follower)

SO-ARM101 (leader + follower)

IN PROGRESS

Owned arm: sysid, imitation, desk payload, VLA. Not the rover arm.

ownedWeeks 3–26
Where to buy ↗
P04 · Weeks 5–9

Imitation learning on the arm (tethered)

SO-ARM101 · optional second webcam ~$40

View project →
2× different USB cameras

2× different USB cameras

PLANNED

Gripper-cam + scene-cam. Two different models so the OS does not swap paths

~$50Week 5
Where to buy ↗
P06 · Weeks 10–16

Autonomous rover + cloud skill loop

Yahboom M3 Pro (Orin NX 16GB) · ~$2.3k

View project →
Yahboom M3 Pro (Orin NX 16GB)

Yahboom M3 Pro (Orin NX 16GB)

PLANNED

Integrated mobile manipulator for ROS 2 navigation, cloud planning, memory and the correction-skill loop. P5 and P6 run in parallel on this robot.

~$2.2k (verify)Weeks 8–16
Where to buy ↗

External SSD for full-disk clone

PLANNED

Clone the M3 Pro image before changing anything

~$60Week 10

Spare arm servos

PLANNED

M3 Pro 6-DOF arm replacements

TBDWeeks 10–16
DIY alternative (not used)

Replaced by an integrated ROS 2 robot so time goes into autonomy rather than base assembly.

P07 · Weeks 16–19

Tennis-ball collector

P6 M3 Pro + intake · ~$90

View project →
Collector intake kit

Collector intake kit

PLANNED

Roller, hopper, beam-break, tennis balls on the P6 M3 Pro

~$90Weeks 16–19
P08 · Weeks 19–23

Robot menagerie

M3 Pro from P6 + Bittle X · ~$300

View project →
Petoi Bittle X

Petoi Bittle X

PLANNED

Cheapest honest quadruped; reuse the P6 M3 Pro for mobile manipulation

~$300Weeks 19–23
P09 · Weeks 23–29

Legged sim-to-real

$0 on Bittle · or +$1,600 Go2

View project →
Unitree Go2 EDU (optional)

Unitree Go2 EDU (optional)

OPTIONAL

Buy only if P9 becomes the specialization. Default is Bittle.

$1,600+Week 20 decision
P10 · Weeks 29+

Whole-body / VLA

SO-ARM101 real · humanoid stays in sim · $0

View project →
Unitree G1 (not buying)

Unitree G1 (not buying)

OPTIONAL

Humanoid stays in sim. Do not fake a hardware demo.

not buyingNever in this plan
// SOFTWARE STACK

Where each piece runs

Mac is the control station. Heavy sim is remote. Filter by project or machine.

Environment

Homebrew

Mac package manager

P01 · P02 · P03 · P04 · P05 · P06 · P07 · P08 · P09 · P10🖥
Docs ↗

Miniconda (Apple Silicon)

Python environments

P01 · P02 · P03 · P04 · P05 · P06 · P07 · P08 · P09 · P10🖥
Docs ↗

Python 3.10 / 3.11

Runtime

P01 · P02 · P03 · P04 · P05 · P06 · P07 · P08 · P09 · P10🖥
Docs ↗

NVIDIA driver + CUDA Toolkit

GPU compute (remote only)

P02 · P04 · P05 · P09 · P10☁
Docs ↗

CH340 / CP210x USB-serial driver

so macOS detects the arm's servo adapter

P03 · P04🖥
Docs ↗

Docker

Optional containerized setups

P02 · P06 · P09☁
Docs ↗

Simulators & Physics

NVIDIA Isaac Sim

Robotics simulator (RT-core GPU)

P02 · P09 · P10☁
Docs ↗

NVIDIA Isaac Lab

RL framework on Isaac Sim

P02 · P09☁
Docs ↗

Isaac Gym (Preview)

Legacy sim, only if we fold old TWIST2 controller work into P5

P10☁
Docs ↗

Legged Gym

Locomotion RL envs, optional TWIST2 leftover

P09 · P10☁
Docs ↗

MuJoCo

Contact physics for the arm / P5 dry-runs; native Apple Silicon

P01 · P03 · P04 · P10🖥
Docs ↗

MuJoCo Menagerie

Open robot models (G1, Go2, arms) to render legally

P01 · P09 · P10🖥
Docs ↗

Gazebo

ROS 2-friendly rover sim for P3

P06 · P07☁
Docs ↗

Unitree RL Lab / unitree_rl_gym

Go2 and G1 locomotion training

P09 · P10☁
Docs ↗

Genesis

Optional newer sim

P02 · P09☁
Docs ↗

ROS 2

Runtime for moving robots only: rover and dog nav

P06 · P07☁
Docs ↗

RL & Training

PyTorch

Deep-learning backbone (MPS build on Mac; CUDA on remote)

P01 · P02 · P04 · P05 · P09 · P10🖥 ☁
Docs ↗

RSL-RL

PPO used by Isaac Lab / Unitree RL

P02 · P09☁
Docs ↗

Stable-Baselines3

Laptop-scale PPO for the P1 fish

P01🖥
Docs ↗

NVIDIA TensorRT

ONNX → TensorRT on the Jetson. FP16 / INT8.

P05🔩
Docs ↗

Imitation Learning & Manipulation

Ultralytics YOLO

Ball detector trained on own court footage (P7 vs HSV)

P07☁ 🔩
Docs ↗

LeRobot

SO-ARM101 + SmolVLA; teleop and data collection

P03 · P04 · P08 · P10🖥 ☁
Docs ↗

NVIDIA Isaac GR00T

GR00T N1.7 fine-tune + whole-body control path

P10☁
Docs ↗

ACT / ALOHA

Action-chunking transformer policy

P04☁
Docs ↗

Diffusion Policy

Visuomotor diffusion policy (P4 comparison vs ACT)

P04☁
Docs ↗

SmolVLA

~450M flow-matching VLA; shortest LeRobot fine-tune route (~50 episodes)

P10☁
Docs ↗

3D Diffusion Policy

Point-cloud visuomotor policy

P10☁
Docs ↗

iDP3 (Improved 3D Diffusion Policy)

TWIST2 high-level policy lineage. Fold in if we reuse that path

P10☁
Docs ↗

Teleop, Retargeting & VLA (P10)

TWIST2 (GitHub)

Folded into the P5 data engine (teleop + retarget)

P10☁
Docs ↗

GMR (General Motion Retargeting)

Real-time human→humanoid retargeting; CPU, so viz runs on Mac

P10🖥 ☁
Docs ↗

XRoboToolkit

Optional PICO→PC motion streaming if we run live VR teleop for P5

P10☁
Docs ↗

XRoboToolkit Teleop Sample (Python)

The streaming service + sample

P10☁
Docs ↗

Redis

Comms layer between teleop and controller

P10☁
Docs ↗

TWIST

The predecessor (reference)

P10☁
Docs ↗

AMASS

Human motion dataset for retargeting (free, registration)

P10🖥
Docs ↗

Datasets & Model Hubs

Hugging Face Hub

Host/share datasets + policies

P04 · P10🖥
Docs ↗

LeRobot datasets

Example demonstration datasets

P04🖥
Docs ↗

Compute & Remote Access

io.net

Rent RT-core GPUs (RTX 4090, x86 Linux)

P02 · P04 · P09 · P10☁
Docs ↗

Isaac Sim WebRTC streaming

View the remote sim viewport in the Mac browser

P02 · P09 · P10🖥
Docs ↗

Tailscale

Secure networking to the remote box

P02 · P04 · P05 · P09 · P10🖥
Docs ↗

Weights & Biases

Monitor remote training from the Mac browser

P02 · P04 · P09 · P10🖥
Docs ↗

TensorBoard

Metrics/curves in the browser

P01 · P02 · P04 · P09 · P10🖥
Docs ↗

Dev Tools

Cursor

AI code editor

P01 · P02 · P03 · P04 · P05 · P06 · P07 · P08 · P09 · P10🖥
Docs ↗

Git

Version control

P01 · P02 · P03 · P04 · P05 · P06 · P07 · P08 · P09 · P10🖥
Docs ↗

VS Code

Editor alternative

P01 · P02 · P03 · P04 · P05 · P06 · P07 · P08 · P09 · P10🖥
Docs ↗

Hardware (Product Pages)

Unitree G1 (sim)

Humanoid embodiment for P10. Sim only unless we already own one

P10🔩
Docs ↗

Unitree Go2 (sim)

Quadruped embodiment for P9. Sim-to-real-ready, not a purchase

P09🔩
Docs ↗

SO-ARM100/101 hardware repo

CAD, BOM, print files

P03 · P04 · P08🔩
Docs ↗

PICO 4 Ultra

Optional VR headset if we run live teleop for the P5 data engine

P10🔩
Docs ↗

PICO Motion Trackers

Optional ankle trackers for whole-body capture

P10🔩
Docs ↗

Local Linux + RTX 4090 box

Optional owned engine. Default plan uses rented io.net

P02 · P09 · P10🔩
Docs ↗

ZED Mini

Stereo egocentric camera, only if a physical G1 appears later

P10🔩
Docs ↗

Dynamixel / Robotis

Servos for a 3D-printed neck. Future hardware only

P10🔩
Docs ↗

P01 · MuJoCo foundations: Laptop only. $0. mjpython / make view on the M4 Max. Cartpole is not the project. The fish + harness v0 is.

P02 · Isaac Lab factory: RTX 5090. No ROS. Blackwell sm_120 gate first. Lift-Cube at 4096 envs. Train headless; replay headful.

P03 · System ID: Owned SO-ARM101. Direct serial. Middleware corrupts the measurement. common/sysid/ is written once here.

P04 · Real-arm IL: Highest-employability project. LeRobot on the Mac for record; train on the 5090. Two different webcams; watch the feed, not the follower.

P05 · Edge: M3 Pro Orin NX 16GB, shared with P06. P5 and P6 run in parallel on the same robot. Measure p99, not average FPS.

P06 · Rover: Yahboom M3 Pro ~$2.2k (verify). ROS 2 Humble + Gazebo Fortress. The project those tools exist for. Own the TF tree before you tune Nav2.

P07 · Ball collector: +$90 on the P6 rover. Vision is the gift. The intake jam rate and the battery are the project.

P08 · Menagerie: M3 Pro from P6 + Bittle ~$300 extra. Skip TurtleBot 4. Point common/ at robots you did not design. A Go2 decision waits until week 20.

P09 · Legged RL: $0 on Bittle, or +$1,600 Go2. A large attributed gap is a result. An unmeasured clean clip is a claim.

P10 · VLA / WBC: Real arm. Humanoid stays in sim. Compare against P4 ACT on the same cases. Do not fake metal.