Autonomous rover + cloud skill loop
A voice-commanded Yahboom M3 Pro that navigates locally, plans with Claude, remembers through AWS, and closes its sim-to-real gap with readable correction skills.
Weeks 10–16 · Yahboom M3 Pro (Orin NX 16GB) · ~$2.3k · 06_cloud_rover/












Goal
Build one robot with a strict three-way handoff. Edge: the M3 Pro's onboard Orin NX 16GB owns motor control, obstacle avoidance, camera/LiDAR processing, SLAM, Nav2, emergency stop, and a heartbeat watchdog; it must remain safe and navigable with the network unplugged. Cloud: a small FastAPI service uses the Claude API to turn speech plus retrieved memory into schema-validated goals from a fixed action vocabulary, never raw motor commands. Memory: AWS stores exact facts plus vector embeddings for semantic recall, and the planner retrieves relevant history before it acts. Sim-to-real: run the same motion in Gazebo and on the M3 Pro, compute a quantitative trajectory gap, ask the cloud model to diagnose it and author a versioned correction skill, use a numerical optimizer for exact parameters, apply the skill to the simulator, and keep it only if the held-out gap shrinks. The final demo is: “go check the kitchen,” then navigate, observe, report, and remember.
Why it matters
This project makes the modern robot boundary visible. Cloud reasoning can take seconds or disappear with Wi-Fi; collision avoidance cannot. ROS 2 keeps motors, sensors, mapping, localization, and planning in separate nodes without freezing one another. The model is useful as a director: ground a request into known actions, choose what to measure, interpret paired traces, and write an auditable correction. It is not a precision estimator or a motor controller. Numerical fitting supplies physical values, and a before/after gap metric rejects corrections that merely sound plausible. The result is a portable library of human-readable skills rather than another opaque trained policy.




The work
- 01
Hardware + edge gate: unbox the M3 Pro and clone the full SSD before changing anything. Inventory every node, topic and launch file, marking which are Yahboom-specific and which are standard ROS 2. Bring up /odom, TF, emergency stop and a heartbeat watchdog. Pull the network cable: manual stop, obstacle avoidance, SLAM and Nav2 must still work.
- 02
Navigation: spawn a calibrated M3 Pro model in Gazebo Fortress (start from Yahboom's URDF). Build and save a map with slam_toolbox using both LiDARs. Localize with LiDAR + IMU fusion (robot_localization), not wheel odometry alone; mecanum rollers slip on carpet. Feed the depth camera into the Nav2 costmap for what a 2D LiDAR misses (tabletops, cables, glass). Send 20 named Nav2 goals in sim and the same 20 on hardware; log success, collisions, path efficiency, localization loss and recovery count.
- 03
Scaffold the seams before features: edge/rover_orchestrator, cloud/cloud_planner, sim2real, memory_store, shared/schemas, and skills. Define one Pydantic action vocabulary (navigate, observe, manipulate, report, stop) and one correction-skill schema; import them everywhere and reject malformed model output.
- 04
Cloud planner: FastAPI /plan accepts command + retrieved memory and returns a validated ordered goal list. Ground the Claude prompt against current robot capabilities and map waypoints. rover_orchestrator executes goals against Nav2, perception, and arm nodes one at a time.
- 05
Persistent memory: store structured facts separately from vector embeddings on AWS. Log observations and actions, retrieve relevant memories by meaning at plan time, and demonstrate a fact learned on day one changing a plan on day two.
- 06
Correction loop: sim2real /correct ingests paired trajectories and a gap metric. The model writes {observation, gap_metric_before, hypothesis, simulator_changes, fitting_routine}; scipy fits precise friction, damping, latency, or wheel parameters. Mecanum wheel slip and lateral drift are the first expected correction-skill targets. Re-run held-out motions and fill gap_metric_after.
- 07
Skill gate: version accepted skills under skills/. Reject any skill that fails schema validation or does not lower the held-out metric. Build at least three accepted skills and one documented rejection.
- 08
Final demo: say “go check the kitchen.” Show speech → grounded JSON goals → Nav2 execution → observation/report → memory write, followed by the same request using recalled context.
- 09
Stretch, bridge to P07: floor pickup of a tennis ball with the M3 Pro arm, triggered through the “manipulate” action, plus a “what did you see?” query answered from memory tagged with map pose.
Done looks like
- M3 Pro SSD cloned and stack inventoried; the Orin NX runs safety, perception, SLAM and Nav2 with the network unplugged.
- One launch each for sim and real. Saved maps plus the same 20-goal Nav2 table in both worlds.
- shared/schemas defines the action vocabulary and correction skill once; invalid LLM output is rejected in tests.
- cloud_planner exposes /plan and /correct. The cloud emits goals and skills only, never velocity or motor commands.
- AWS facts + vector memory demonstrated across two sessions: yesterday’s fact is retrieved and changes today’s grounded plan.
- At least three versioned correction skills with before/after held-out gap metrics, plus one rejected skill whose metric did not improve.
- GR03 reports position error, timing, overshoot, path efficiency, and fitted parameters before and after the skill loop.
- Final uncut demo: voice command → grounded plan → autonomous kitchen check → report → memory write and later recall.
- Stretch: tennis ball picked off the floor via a grounded “manipulate” goal.
Stack card
| Simulator | Gazebo Fortress + calibrated M3 Pro model |
|---|---|
| Trains on | No learned control policy. Numerical fitting on MacBook / RTX workstation |
| Runs on | M3 Pro Orin NX 16GB edge + Claude API service + AWS memory |
| Buy | Yahboom M3 Pro Orin NX 16GB ~$2.2k (verify at checkout) · SSD for full-disk clone ~$60 · spare arm servos TBD |
| ROS 2 | Yes (mandatory on edge) |
| Gazebo | Yes (core) |
| Compute / target | Orin NX Super 16GB runs safety, perception, SLAM and Nav2 fully offline. Cloud never drives motors. |
| Install | ROS 2 Humble (preinstalled), Gazebo Fortress, ros_gz bridge, slam_toolbox, Nav2, robot_localization, ros2_control, RViz2, Foxglove Studio, FastAPI, Anthropic SDK, Pydantic, AWS SDK, PostgreSQL + pgvector (or OpenSearch Serverless), scipy; LeRobot for the stretch step only |
| Dev workflow | Mac → VS Code Remote SSH + Foxglove; robot runs headless |
Repos to study
Watch out
- The cloud never sends raw motor commands. If /plan or the network stalls, the edge completes a safe stop.
- An LLM is not a system-identification optimizer. Let it select measurements and write the skill; let scipy fit exact values.
- A correction that sounds good but does not lower the held-out gap is rejected.
- Keep facts and vector recall separate. Exact claims need structured storage; fuzzy memory needs embeddings.
- Verify the RobotShop checkout total. Price includes US tariffs and moves; do not treat ~$2.2k as a locked number.
- The TF tree is still where everyone suffers. Own it before tuning Nav2 or blaming the cloud.
- Humble support ends May 2027; plan the Jazzy + Gazebo Harmonic migration as a later milestone.
- JetPack updates depend on Yahboom's carrier-board support; confirm the shipped version before upgrading anything.
- Yahboom images vary, and some micro-ROS nodes expose no parameter service; do the day-one inventory before trusting tooling.
- The arm is floor/low-shelf only (~30 cm radius, 410 g, no force sensing). Design manipulation around that.
- 16GB is shared by Nav2, depth, detection and any policy; keep the LLM off-board.
- Camera and microphone roam the house: keep images local, send text to the cloud.
Workstreams
M3 Pro bring-up, SSD clone and stack inventory; offline Orin NX ROS 2 stack; shared schemas; rover_orchestrator; FastAPI planner; AWS memory; versioned skill application.
Network-loss safety test; paired 20-goal Nav2 table; cross-session recall; schema rejection tests; held-out gap before/after every correction skill; GR03.
Cross-check. Unplug the cloud and verify safety/navigation. Then disable the edge executor and verify the cloud cannot move hardware. A second person can regenerate every accepted-skill metric from committed paired logs.
What this proves
You can draw and enforce the edge-cloud boundary, ground language into a robot’s real capabilities, build memory that survives sessions, and improve a simulator with readable skills whose value is measured rather than asserted.
Cloud thinks slowly and writes verifiable artifacts. The Orin NX acts quickly and keeps the robot safe. Metrics, not confident prose, decide whether a correction survives.
Study videos
Gear / BOM
| Yahboom ROSMASTER M3 Pro, Orin NX Super 16GB, no display | ~$2.2k (verify) |
|---|---|
| External SSD for full-disk clone | ~$60 |
| Spare arm servos | TBD |
DIY alternative (not used)
Replaced by an integrated ROS 2 robot so time goes into autonomy rather than base assembly.
