// PROJECTS · 29 WEEKS

Ten projects from a home lab.

Start on the laptop, then move to the arm on the desk, then to a rover in the garage. Projects 1–7 are the spine. Projects 8–10 are optional depth if the core is solid. Each one ends with a folder you can open, a short clip, and a row in the shared eval log.

PLANNEDP05 · Intermediate · core

Edge deployment

Unplug the 5090 (unplug the workstation). The arm keeps doing the task. The exact P4 policy now runs untethered on a $249 Jetson at 15 W.

Weeks 8–10 · Yahboom M3 Pro Orin NX 16GB (shared with P06) · 05_edge_runtime/

NoneTrain · RTX 5090 to exportRun · Yahboom M3 Pro Orin NX Super 16GBBuy · No extra board. P5 and P6 share the M3 Pro.ROS · NoGazebo · No
Edge deployment
PLANNED
DEMO 05 · PLANNED
DEMO 05 · PLANNEDVideo lands here when this milestone ships.
P4 task on the real arm with laptop and workstation disconnected. Watchdog clips.
Arm on the desk, the payload that must survive the move to Jetson
Arm on the desk, the payload that must survive the move to Jetson
JetsonTensorRTONNXWatchdogp99
01

Goal

Training on a 5090 and inferring on a 5090 is only half the job. Export PyTorch → ONNX → TensorRT. Measure accuracy drift at FP16 and INT8 against FP32 on identical inputs. Measure the loop properly: inference latency p50/p95/p99, camera-to-command latency, jitter. Not average FPS. Document the rate hierarchy: policy inference ≠ control loop ≠ servo update. Write the watchdog for stale frames, stale commands, NaN / out-of-range, and missed deadlines, with a defined response for each. Trigger every branch on purpose. Why not Jetson Thor ($3,499)? Being able to answer that in one sentence is judgment.

02

Why it matters

P5 inherits the exact policy from P4. Its whole job is to make that policy survive untethered on the Jetson. P5 and P6 run in parallel on the same M3 Pro. In the remote-5090 setup, this is where even the local Mac leaves the loop. The policy runs on the board bolted to the robot. A robot in the field has no 5090. It has a small ARM board, a hard deadline, and no one watching. Three problems only show up here: the policy shrinks under quantization, the loop misses its deadline and the robot lurches, and when a sensor drops out something has to decide the next move. common/deploy/ is written once and becomes the brain of P6–P9. ROS 2 here would add latency to the thing you are measuring.

03

The work

  1. 01

    PyTorch → ONNX → TensorRT. Measure FP16 and INT8 drift against FP32 on identical inputs.

  2. 02

    Latency: inference p50/p95/p99, end-to-end camera-to-command, jitter. Not average FPS.

  3. 03

    Write the rate hierarchy: policy rate ≠ control rate ≠ servo rate.

  4. 04

    Watchdog state machine. Trigger stale camera, stale command, and injected NaN. Three clips, three log rows.

  5. 05

    Same 20 evaluation cases: workstation vs Jetson. Attribute any drop to quantization or latency with evidence.

04

Done looks like

  • Demo: P4 pick-and-place on the real arm, laptop and workstation physically disconnected.
  • Quantization table: FP32 / FP16 / INT8 × output error × task success over 20 trials each.
  • Latency histogram with p50/p95/p99 marked, and the control rate you can actually guarantee.
  • Safety demo: unplug the camera mid-task and get a defined stop. Repeat for stale commands and injected NaN.
  • Same 20 cases, workstation vs Jetson, with an evidence-backed explanation of any drop.
05

Stack card

SimulatorNone
Trains onRTX 5090 to export
Runs onYahboom M3 Pro Orin NX Super 16GB
BuyNo extra board. P5 and P6 share the M3 Pro.
ROS 2No
GazeboNo
Compute / targetthe M3 Pro's Orin NX Super 16GB (shared with P06)
InstallJetPack 6.x, TensorRT, onnx, onnxruntime, pycuda; torch.onnx export on the workstation
06

Repos to study

07

Watch out

  • Cutting torque is not universally safe: the arm drops the payload; a balancer falls over.
  • Do not buy Jetson Thor for this. The $249 Super is the point.
08

Workstreams

BUILD STREAM

Export path, JetPack bring-up, watchdog, disconnected demo.

EVAL STREAM

Quantization table, latency histogram, workstation-vs-Jetson on 20 cases.

Cross-check. Eval stream can reproduce the p99 number from committed logs.

09

What this proves

Model export, ARM/embedded Linux, real-time scheduling, and failure-mode engineering. Measured, not claimed.

A policy that averages 20 Hz and drops to 4 Hz at p99 is a robot that lurches.

10

Study videos