Action
What the policy outputs each step: usually joint targets, torques, or gripper commands.
Short definitions for the jargon on the project pages. Skim by project, or jump A–Z. Written so you can follow along without a separate textbook open.
How people talk about looking inside neural nets. Useful background for VLAs and foundation models (P10), and for reading modern AI papers.
The numbers flowing through a layer while the model runs. Interpretability work often starts by asking what a particular activation pattern means.
Used in P10
Making sure the system’s behavior stays aimed at what we actually want, especially as models get more capable. Auditing tools matter because full decoding may never happen.
Used in P10
Systematic checks on a trained system: probes, scratchpad review, held-out tasks. The practical goal when you cannot fully decode every weight.
Used in P10
A model you can query but cannot easily explain. Modern nets are grown from data, so the insides are huge arrays of numbers, not hand-designed rules.
Used in P10
When a model writes intermediate reasoning before the final answer. Safety researchers watch it for cheating, test-awareness, or misaligned plans. Mostly an LLM idea; VLAs may expose less of this.
Used in P10
In mechanistic interpretability: a small subgraph of neurons and weights that implements one recognizable computation inside the larger net.
Used in P10
A vector inside the model’s hidden state that tends to line up with one concept. Steering or reading that direction is a common interpretability move.
Used in P10
Take a pretrained model and train it a bit more on your data so it adapts to your robot and task.
Research that tries to understand what is going on inside a trained model, instead of treating it only as input → output.
Used in P10
Train a simple linear classifier on frozen internal activations to test whether a concept (for example “gripper open”) is already encoded there.
Used in P10
A stricter flavor of interpretability: reverse-engineer specific circuits or computations inside the network, not just correlate inputs with outputs.
Used in P10
A tool used in interpretability work. It expands a model’s internal activations into many mostly-off features so individual concepts are easier to spot.
Used in P10
When a model seems to notice it is being evaluated and changes behavior (for example plays nice under tests). Chain-of-thought audits try to catch this.
Used in P10
Physics, control, and a first learned policy on the laptop.
Open project →What the policy outputs each step: usually joint targets, torques, or gripper commands.
A rhythmic pattern generator. For the fish: sinusoidal joint commands with a phase offset down the body, tracked by PD.
Used in P01
The swimming (or walking) motion produced by a CPG. In P1 it is the scripted baseline the PPO policy has to beat.
Used in P01
Our shared logger across projects. Every run gets a row: task, metrics, how it failed, and whether it was sim, real, or paired.
The standard Python API for RL environments (reset, step, observation, reward). Stable-Baselines3 talks to it.
Used in P01
Test cases you freeze before training. You do not tune on them. That is how you know the result is real.
The XML format that describes bodies, joints, actuators, geoms, and contacts for MuJoCo.
A fast physics engine for robots and contacts. Runs well on a Mac. Used for foundations, sysid, and arm dry-runs.
What the policy sees each step: joint angles, images, velocities, and so on. Write this down early. Wrong units here break everything later.
A simple controller: push harder when error is large (P), and damp the motion using velocity (D). We write this by hand in P1.
The neural network (or controller) that maps what the robot senses into the next action.
A popular reinforcement-learning algorithm. Stable, widely used, and what we run for the small fish policy and later locomotion.
The neural network trained with PPO. Same idea as “policy,” just naming the training method.
The agent learns by trial and error in an environment, getting rewarded for getting closer to a goal.
One piece of the reward function (for example lift height or action penalty). Change one at a time and keep before/after numbers.
A Python library of RL algorithms. Handy for laptop-scale PPO on the fish.
Used in P01
GPU-parallel RL and the Franka lift task.
Open project →The RTX 5090 GPU architecture. get_device_capability() should print (12, 0). Wrong PyTorch builds waste the card.
Used in P02
Run with the viewer on. Slower. Use it to record clips and check that the policy looks right.
Used in P02
Run the sim without opening a visual window. Faster for overnight training.
Used in P02
Test cases you freeze before training. You do not tune on them. That is how you know the result is real.
RL and robotics framework on top of Isaac Sim. Built for thousands of parallel environments.
NVIDIA’s GPU robotics simulator. Heavy, needs a serious GPU. The world Isaac Lab trains inside.
Thousands of copies of the same task running at once on the GPU. That is how Isaac Lab gets high sample throughput.
The neural network (or controller) that maps what the robot senses into the next action.
A popular reinforcement-learning algorithm. Stable, widely used, and what we run for the small fish policy and later locomotion.
The agent learns by trial and error in an environment, getting rewarded for getting closer to a goal.
One piece of the reward function (for example lift height or action penalty). Change one at a time and keep before/after numbers.
The RL library Isaac Lab often uses under the hood (PPO and friends), originally from Robotic Systems Lab.
The random-number starting point for a training run. Multiple seeds show whether a result is stable or lucky.
Trillions of floating-point operations per second. A rough measure of GPU compute. We log a GEMM number as a capability check.
Used in P02
How much memory the graphics card has. Limits batch size and number of parallel envs.
Making the sim match the real SO-ARM101.
Open project →Extra inertia from motors and gears that the sim must include or the arm will look too snappy.
Used in P03
Our shared logger across projects. Every run gets a row: task, metrics, how it failed, and whether it was sim, real, or paired.
The quantitative discrepancy between matching sim and real runs: position error, timing, overshoot, or another frozen measure. A correction survives only if this improves on held-out runs.
A short writeup of where sim and reality disagree, with numbers. The signature artifact of this lab.
The XML format that describes bodies, joints, actuators, geoms, and contacts for MuJoCo.
A fast physics engine for robots and contacts. Runs well on a Mac. Used for foundations, sysid, and arm dry-runs.
Use an optimizer to estimate precise values such as friction, damping, latency, or wheel radius. The model chooses what to fit; math finds the number.
A simple controller: push harder when error is large (P), and damp the motion using velocity (D). We write this by hand in P1.
How much the real joint lags behind the commanded or simulated motion, usually in degrees or milliseconds.
Used in P03
A single number for how far sim and real disagree on a joint trajectory. Lower after a good fit.
Used in P03
Train in simulation, then run the same skill on a physical robot. The gap between those two is what we measure.
The open leader/follower arm we own. Used for sysid, imitation, desk-side edge work, and VLA. Not the rover arm.
The Feetech smart servos inside the SO-ARM101. They run an internal controller you cannot see. Do not model them as ideal torque sources.
Measure the real robot, then tune the simulator (friction, damping, and so on) until held-out trajectories match.
Demos, ACT, Diffusion, and the 2×2 experiment.
Open project →In P4: ACT vs Diffusion, each trained on real-data vs sim-data, all scored on the same 20 real-arm cases.
Used in P04
Turn one ingredient off (for example domain rand) and compare. That is how you know what mattered.
An imitation policy that predicts a short sequence of actions at once. LeRobot’s usual first policy for the SO-101.
What the policy outputs each step: usually joint targets, torques, or gripper commands.
Plot performance at 10 / 25 / 50 demos (or similar). Shows whether more data still helps.
One recorded human (or teleop) example of the task. P4 aims for about 50 real pick-and-place demos.
An imitation method that generates actions by denoising, similar in spirit to image diffusion models. Our P4 comparison policy next to ACT.
Used in P04
One complete attempt at a task, from start until success, failure, or timeout.
Our shared logger across projects. Every run gets a row: task, metrics, how it failed, and whether it was sim, real, or paired.
A fixed list of failure types (grasp slip, collision, timeout, …). You count them instead of only saying “it failed.”
Used in P04
Take a pretrained model and train it a bit more on your data so it adapts to your robot and task.
A short writeup of where sim and reality disagree, with numbers. The signature artifact of this lab.
Test cases you freeze before training. You do not tune on them. That is how you know the result is real.
The robot learns a task by watching demonstrations, not by exploring with a hand-written reward.
Hugging Face’s open stack for teleop, datasets, ACT, Diffusion, and SmolVLA on hardware like the SO-101.
A fast physics engine for robots and contacts. Runs well on a Mac. Used for foundations, sysid, and arm dry-runs.
What the policy sees each step: joint angles, images, velocities, and so on. Write this down early. Wrong units here break everything later.
The neural network (or controller) that maps what the robot senses into the next action.
The agreed fields and types for every recorded episode (cameras, timestamps, joint names). Validate it before collecting 50 demos.
Used in P04
Train in simulation, then run the same skill on a physical robot. The gap between those two is what we measure.
The open leader/follower arm we own. Used for sysid, imitation, desk-side edge work, and VLA. Not the rover arm.
The Feetech smart servos inside the SO-ARM101. They run an internal controller you cannot see. Do not model them as ideal torque sources.
You move the robot in real time, like puppeteering. In LeRobot that usually means a leader arm driving a follower arm.
Moving the policy from the 5090 onto a Jetson.
Open project →Number formats: 32-bit float (full), 16-bit float (half), 8-bit integer. Lower precision is faster and riskier.
Used in P05
A repeated signal that says the planner or controller is still alive. If it goes stale, the edge watchdog stops or enters a defined safe state.
NVIDIA’s OS + library bundle for Jetson (CUDA, TensorRT, and friends).
Used in P05
The NVIDIA module on the Yahboom M3 Pro carrier. Shared by P5 and P6: it runs the exported policy, plus safety, perception, SLAM, and Nav2 offline. 16GB is shared — keep the LLM off-board.
Variation in loop timing from step to step. Bad jitter feels like a shaky or lunging robot.
Used in P05
Open format for exporting a trained model out of PyTorch so other runtimes can load it.
Used in P05
Latency percentiles. p99 is the slow tail. A policy that averages 20 Hz but hits 4 Hz at p99 will make the robot lurch.
Used in P05
The neural network (or controller) that maps what the robot senses into the next action.
Store weights with fewer bits so inference is faster and smaller. Measure accuracy drift against FP32 on identical inputs.
Used in P05
The open leader/follower arm we own. Used for sysid, imitation, desk-side edge work, and VLA. Not the rover arm.
NVIDIA’s inference engine. Takes an ONNX model and optimizes it for Jetson or discrete GPUs.
Used in P05
How much memory the graphics card has. Limits batch size and number of parallel envs.
Safety logic that watches for stale cameras, commands, heartbeats, NaNs, and missed deadlines, then runs a defined response such as a safe stop.
Integrated ROS 2 mobile manipulator: Orin NX Super 16GB, dual T-mini Plus ToF LiDAR, binocular depth camera, 6-DOF arm (~30 cm, 410 g), mecanum base. Sold as the no-display Orin NX 16GB kit. Replaced the DIY LeKiwi plan.
ROS 2 navigation on the edge, grounded cloud plans, persistent memory, and measured correction skills.
Open project →The small allowed set of rover actions, such as navigate, observe, manipulate, report, and stop. Cloud output must match this schema before the edge accepts it.
Used in P06
The FastAPI service around the Claude API. It turns a command plus relevant memory into validated goals and writes correction skills. It never drives motors.
Used in P06
A readable, versioned file describing the observed sim-to-real gap, a hypothesis, simulator edits, the fitting routine, and measured before/after metrics.
Used in P06
A grid of “how bad is it to drive here.” Obstacles inflate into lethal or expensive cells.
Used in P06
The safety line in P6: the M3 Pro’s Orin NX owns fast physical actions and safe stopping; the cloud may reason slowly, return structured artifacts, or disappear without making the rover unsafe.
Used in P06
The quantitative discrepancy between matching sim and real runs: position error, timing, overshoot, or another frozen measure. A correction survives only if this improves on held-out runs.
A short writeup of where sim and reality disagree, with numbers. The signature artifact of this lab.
Robot simulator that plays nicely with ROS 2 Humble. P6 uses Fortress with a calibrated M3 Pro model. Jazzy + Harmonic is a later migration.
Restrict a language model’s plan to actions the robot can actually execute and places that really exist on its map.
Used in P06
A repeated signal that says the planner or controller is still alive. If it goes stale, the edge watchdog stops or enters a defined safe state.
Sensor for acceleration and rotation. Helps odometry and balance estimates.
Used in P06
The NVIDIA module on the Yahboom M3 Pro carrier. Shared by P5 and P6: it runs the exported policy, plus safety, perception, SLAM, and Nav2 offline. 16GB is shared — keep the LLM off-board.
DIY mobile base that mounts an SO-101. Considered, then replaced by the M3 Pro so time goes into autonomy rather than base assembly. Not used.
Used in P06
Laser scanner that returns distances around the robot. Core sensor for SLAM on the rover.
Used in P06
ROS 2 navigation stack: go from pose A to pose B with costmaps and recovery behaviors.
Use an optimizer to estimate precise values such as friction, damping, latency, or wheel radius. The model chooses what to fit; math finds the number.
The robot’s estimate of how it has moved, usually from wheel encoders (and maybe IMU). Drift is normal.
Used in P06
Information saved outside the current task so the rover can use it tomorrow. P6 keeps exact facts in a normal database and fuzzy recall in a vector database.
Used in P06
Middleware for robots with many sensors and processes. Used here for the rover and collector, not for the arm stack.
ROS 2 framework for talking to motors and hardware interfaces in a standard way.
Used in P06
Train in simulation, then run the same skill on a physical robot. The gap between those two is what we measure.
Build a map while figuring out where you are inside it. We use SLAM Toolbox for live maps.
Used in P06
An ordered, schema-validated JSON list sent from cloud planning to the ROS 2 orchestrator. Goals name high-level actions and parameters, never wheel speeds.
Used in P06
The tree of coordinate frames (map → odom → base_link → sensors). Own this before you tune Nav2.
Used in P06
XML description of the robot’s links, joints, and sensor frames. Wrong frames break everything downstream.
Used in P06
A store searched by meaning rather than exact words. It retrieves relevant past observations before the cloud planner builds a new plan.
Used in P06
Safety logic that watches for stale cameras, commands, heartbeats, NaNs, and missed deadlines, then runs a defined response such as a safe stop.
Integrated ROS 2 mobile manipulator: Orin NX Super 16GB, dual T-mini Plus ToF LiDAR, binocular depth camera, 6-DOF arm (~30 cm, 410 g), mecanum base. Sold as the no-display Orin NX 16GB kit. Replaced the DIY LeKiwi plan.
Vision + intake on top of the rover.
Open project →A cheap sensor that fires when something crosses a light beam. We use it to confirm a ball entered the intake.
Used in P07
Find a connected patch of pixels (the ball) and track its centroid. Simple, brittle in hard lighting.
Used in P07
One complete attempt at a task, from start until success, failure, or timeout.
A short writeup of where sim and reality disagree, with numbers. The signature artifact of this lab.
Robot simulator that plays nicely with ROS 2 Humble. P6 uses Fortress with a calibrated M3 Pro model. Jazzy + Harmonic is a later migration.
A color space that often makes colored objects (tennis balls) easier to threshold than raw RGB.
Used in P07
How often the intake clogs per N balls. Often the real performance number, not the detector score.
Used in P07
The NVIDIA module on the Yahboom M3 Pro carrier. Shared by P5 and P6: it runs the exported policy, plus safety, perception, SLAM, and Nav2 offline. 16GB is shared — keep the LLM off-board.
ROS 2 navigation stack: go from pose A to pose B with costmaps and recovery behaviors.
Precision: of detections, how many were real balls. Recall: of real balls, how many we caught. Trade them off on purpose.
Used in P07
Middleware for robots with many sensors and processes. Used here for the rover and collector, not for the arm stack.
Integrated ROS 2 mobile manipulator: Orin NX Super 16GB, dual T-mini Plus ToF LiDAR, binocular depth camera, 6-DOF arm (~30 cm, 410 g), mecanum base. Sold as the no-display Orin NX 16GB kit. Replaced the DIY LeKiwi plan.
A fast object detector. We train a small one on our own court footage and compare it to HSV.
Used in P07
New robots, shared code, interoperability.
Open project →Petoi’s small quadruped. Cheap honest legs for characterization and later sim-to-real (P9).
One architecture or dataset format reused across arm, mobile base, rover, quadruped. Document where reuse actually works.
The physical body the policy controls: arm, rover, quadruped, and so on. Code that assumes one body often breaks on another.
Our shared logger across projects. Every run gets a row: task, metrics, how it failed, and whether it was sim, real, or paired.
The robot learns a task by watching demonstrations, not by exploring with a hand-written reward.
Whether the same tools (harness, deploy, sysid) work across robots you did not design. If not, write down why.
Used in P08
Hugging Face’s open stack for teleop, datasets, ACT, Diffusion, and SmolVLA on hardware like the SO-101.
The vendor’s libraries and docs for talking to a specific robot. Unfamiliar SDKs are the point of P8.
Used in P08
The open leader/follower arm we own. Used for sysid, imitation, desk-side edge work, and VLA. Not the rover arm.
Measure the real robot, then tune the simulator (friction, damping, and so on) until held-out trajectories match.
Locomotion RL and domain randomization.
Open project →Turn one ingredient off (for example domain rand) and compare. That is how you know what mattered.
Petoi’s small quadruped. Cheap honest legs for characterization and later sim-to-real (P9).
Randomize sim parameters (mass, friction, delays) during training so the policy survives messier reality. Measure whether it actually helps.
Used in P09
A short writeup of where sim and reality disagree, with numbers. The signature artifact of this lab.
Test cases you freeze before training. You do not tune on them. That is how you know the result is real.
RL and robotics framework on top of Isaac Sim. Built for thousands of parallel environments.
NVIDIA’s GPU robotics simulator. Heavy, needs a serious GPU. The world Isaac Lab trains inside.
Thousands of copies of the same task running at once on the GPU. That is how Isaac Lab gets high sample throughput.
The neural network (or controller) that maps what the robot senses into the next action.
A popular reinforcement-learning algorithm. Stable, widely used, and what we run for the small fish policy and later locomotion.
The neural network trained with PPO. Same idea as “policy,” just naming the training method.
The agent learns by trial and error in an environment, getting rewarded for getting closer to a goal.
One piece of the reward function (for example lift height or action penalty). Change one at a time and keep before/after numbers.
The RL library Isaac Lab often uses under the hood (PPO and friends), originally from Robotic Systems Lab.
The random-number starting point for a training run. Multiple seeds show whether a result is stable or lucky.
Train in simulation, then run the same skill on a physical robot. The gap between those two is what we measure.
Locomotion objective: make the robot follow commanded forward/turning speeds without falling.
Used in P09
Frontier policies on hardware you own.
Open project →An imitation policy that predicts a short sequence of actions at once. LeRobot’s usual first policy for the SO-101.
The numbers flowing through a layer while the model runs. Interpretability work often starts by asking what a particular activation pattern means.
Used in P10
Making sure the system’s behavior stays aimed at what we actually want, especially as models get more capable. Auditing tools matter because full decoding may never happen.
Used in P10
Systematic checks on a trained system: probes, scratchpad review, held-out tasks. The practical goal when you cannot fully decode every weight.
Used in P10
A model you can query but cannot easily explain. Modern nets are grown from data, so the insides are huge arrays of numbers, not hand-designed rules.
Used in P10
When a model writes intermediate reasoning before the final answer. Safety researchers watch it for cheating, test-awareness, or misaligned plans. Mostly an LLM idea; VLAs may expose less of this.
Used in P10
In mechanistic interpretability: a small subgraph of neurons and weights that implements one recognizable computation inside the larger net.
Used in P10
One architecture or dataset format reused across arm, mobile base, rover, quadruped. Document where reuse actually works.
Plot performance at 10 / 25 / 50 demos (or similar). Shows whether more data still helps.
One recorded human (or teleop) example of the task. P4 aims for about 50 real pick-and-place demos.
A vector inside the model’s hidden state that tends to line up with one concept. Steering or reading that direction is a common interpretability move.
Used in P10
The physical body the policy controls: arm, rover, quadruped, and so on. Code that assumes one body often breaks on another.
Take a pretrained model and train it a bit more on your data so it adapts to your robot and task.
NVIDIA’s humanoid / foundation-model stack. We may fine-tune on LeRobot-format data; humanoid hardware stays in sim.
Used in P10
The robot learns a task by watching demonstrations, not by exploring with a hand-written reward.
Research that tries to understand what is going on inside a trained model, instead of treating it only as input → output.
Used in P10
Hugging Face’s open stack for teleop, datasets, ACT, Diffusion, and SmolVLA on hardware like the SO-101.
Train a simple linear classifier on frozen internal activations to test whether a concept (for example “gripper open”) is already encoded there.
Used in P10
A stricter flavor of interpretability: reverse-engineer specific circuits or computations inside the network, not just correlate inputs with outputs.
Used in P10
The neural network (or controller) that maps what the robot senses into the next action.
Map a human motion onto a robot whose body proportions differ. Optional study path in P10.
Used in P10
A relatively small (~450M) VLA with a short public path onto the SO-101 via LeRobot.
Used in P10
The open leader/follower arm we own. Used for sysid, imitation, desk-side edge work, and VLA. Not the rover arm.
A tool used in interpretability work. It expands a model’s internal activations into many mostly-off features so individual concepts are easier to spot.
Used in P10
You move the robot in real time, like puppeteering. In LeRobot that usually means a leader arm driving a follower arm.
When a model seems to notice it is being evaluated and changes behavior (for example plays nice under tests). Chain-of-thought audits try to catch this.
Used in P10
A policy that takes images (and often language) and outputs robot actions. SmolVLA and GR00T are the routes we consider.
Used in P10
Controlling arms, torso, and legs together as one coordinated system, not each half on its own.
Used in P10
Every term once, alphabetically.