A question every autonomous-vehicle team has to turn into numbers. This site does that for wing-av: it defines the metrics, backs each one with how Autoware, Apollo, openpilot, Waymo, nuPlan, Stanford/CMU and the regulators measure the same thing, and then computes them on the ten laps recorded on 2026-09-15 (five AV2.0, five AV2.5) — plans, commands, CAN frames and odometry included.
Split "executed" into four measurable questions, each with its own reference and its own measurement, all of which are already in the bags:
① Tracking — did the car go where the plan said? Cross-track and heading error between the model's predicted trajectory and the odometry 0.5 / 1 / 2 s later; curvature error (commanded vs yaw-rate/v); speed and acceleration error against the longitudinal plan. ② Actuation fidelity — did the car do what we commanded? Steering desired vs sent vs measured (with the limiter's clip and the saturation flag), command→response lag, and the CAN-level check that frame 482 asked what frame 287 reports. ③ Ride quality — was it done the way a passenger and a regulator accept? Lateral / longitudinal acceleration and jerk, steering rate and reversals, against UN R79, ISO 15622, ISO 2631 and nuPlan bands and against the human laps on the same road. ④ Supervision — did a human have to step in? Disengagements per engaged km, overrides, hands-on-wheel. Plus a prerequisite: ⓪ plan quality (plan jitter, MPC validity, plan age) because no controller can execute a plan that changes every frame.
→ Page 1 explains the chain and lets you play with a simulated controller; Page 2 is the full catalogue with formulas, bands and sources.
Yes — every 50 Hz sample of every lap is labelled by lateral situation (straight |κ| < 0.005 m⁻¹, curve, sharp |κ| ≥ 0.02 m⁻¹, i.e. R < 50 m) and longitudinal situation (cruise, decel a < −0.4, accel a > 0.4 m/s², stop = approach + standstill), then cut into episodes and scored. One honest caveat: …
→ Page 4: heatmap of drives × situations, 3-D replays of the representative episode of each situation, AV2.0 vs AV2.5 vs human.
As a post-drive batch job, the way Autoware's driving_log_replayer and comma's maneuver reports do it: every uploaded bag → one deterministic extractor (the one that produced these numbers, ~2.5 min per lap) → one JSON record per drive with the per-situation scorecard → a store keyed by vehicle / model / software version / route → a dashboard with trend lines and regression thresholds → an alert when a metric leaves its band. The online half is a guard node that publishes the same metrics on /diagnostics while driving.
→ Page 6 is that dashboard, built from the ten laps as a ten-vehicle-day "fleet".
The essentials bag already carries what the metrics on this site need: /control/predicted_trajectory, /control/lateral_plan, /control/longitudinal_plan, /control/actuators, /vehicle/actuators_output, /vehicle/state, /vehicle/engagement, /localization/odometry, /sensing/gnss/imu/data, /sensing/gnss/ekf_nav, /vehicle/can_tx, /vehicle/can_rx. What is missing is the controller's inside: a /control/debug message with the setpoints it actually used (lag-adjusted curvature, v_target_now, a_target), the PID terms, which limiter stage bound, and a per-drive software/parameter fingerprint — the list Apollo publishes every cycle.
→ Page 7: topic-by-topic table (have / partial / missing) and the proposed message.
Generated from the numbers, not written by hand — every figure links to the page that shows it. These are the "value for improvement" findings the metrics were chosen to surface.
Numbers on this site are computed by extract_metrics.py from the essentials bags (read-only) at 50 Hz and downsampled to 10 Hz for the charts. Every column is defined on Page 2; every threshold and its source on Page 8; every assumption on Page 9.
The command chain, the four questions, and a simulated controller you can detune to see each metric react.
Every metric with its formula, the topics it reads, its comfort/regulatory bands and the fleet values on the ten laps.
Where the trajectory starts on the car, what AV2.0 and AV2.5 publish, the car's own path in the same frame, and how the industry compares them (ADE / FDE).
Straight, curve, sharp turn, stop, decel/accel: how they are detected, a 3-D replay of each, and the scorecards.
Any lap, any second: front and cabin cameras, synchronized charts, situation bands, satellite map, lidar map and 3-D replay.
The post-drive pipeline and a dashboard over the ten laps with trend lines, thresholds and AV2.0 vs AV2.5.
Topics, rates and fields used by each metric; what is missing; the proposed /control/debug message.
Verbatim quotes from Autoware, Apollo, openpilot, nuPlan, Waymo, Stanford, CMU, UNECE, ISO, NIST and the CA DMV.
How every number was computed, the sign conventions, the thresholds, and what the data cannot tell you.
The "Control" section proposed for the Drive Live Tracker: one score per uploaded bag, mocked on the ten laps in the tracker's own layout, with the record the server would write.
Every parameter of the deployed control chain with its file, its value on the laps and the metric it moves; a ranked shortlist of what to turn; and the hardware ceilings — with what it would take to raise them.
A plan is a promise about the future: in one second the car will be here, heading there, at this speed. The controller's job is to turn that promise into steering and acceleration commands that the car then physically carries out. So "correctly executed" can only be checked by comparing three things that live at different places in the stack: what was planned, what was commanded, and what actually happened.
Reference and measurement. Every metric here is "reference minus measurement", and the trick is choosing the right pair. Tracking compares the plan with odometry. Actuation compares the command with the car's own sensors. Ride quality has no reference — it compares the measured motion with what people tolerate, which is why it needs external bands (regulations, comfort studies, the human laps). Supervision compares the state machine with the driver's hands.
Why not one number? Because the failure modes are different and need different fixes: a large cross-track error with a small steering error means the planner or the vehicle model is wrong (the controller did what it was told, the car did not end up on the plan); a large steering error with a small cross-track error means the limiter or the EPS is slow but the loop still closes; high jerk with perfect tracking means the plan itself is uncomfortable; a clean drive with three disengagements means the driver did not trust it. Apollo, Autoware and openpilot all log the layers separately for exactly this reason (Page 8).
A kinematic bicycle model of the Atto 3 (wheelbase 2.72 m, steering ratio 16.36) drives a closed course — straights, an 80 m curve, a 20 m junction, a 40 m curve, a 15 m hairpin — with a simple look-ahead steering law. The sliders detune it the way real controllers go wrong: actuator delay (the EPS and the 20 ms CAN cycle), rate limit (the vehicle interface's slew limiter, 125 °/s at low speed on the real car), gain (too timid or too aggressive), look-ahead and speed. The tyres deliver at most 8 m/s² of lateral acceleration, so a corner taken too fast pushes the car wide. Watch the metrics react: delay produces lag on curve entry and, with enough of it, a car that leaves the road at the junction; low gain produces steady-state cross-track error in curves; high gain tracks tighter but buys it with steering reversals and lateral jerk — the trade-off every tuning session is about.
Cross-track at a horizon mixes two things: how well the controller tracked and how much the plan itself moved in the meantime (the model re-plans every 100 ms). That is why the plan-jitter metric sits next to it: if consecutive plans disagree by 2 cm at 1 s and the car is 7 cm off, 5 cm is the controller's. Autoware scores exactly this in its planning evaluator — it "evaluates the trajectory stability by comparing the trajectory T(0) and previous trajectory T(-1)" — in a node separate from the control evaluator that judges the controller. Page 3 shows the trajectory itself, in the car's own frame.
Each card: what it measures, the exact formula and topics, the sign convention, the bands used to colour it — each edge tagged with its origin: a regulation or standard, a published benchmark, a constant of the deployed stack, or our choice — and its value on the ten laps for AV2.0 and AV2.5 while the corresponding loop was active. Colour is a guide, not a verdict — a band from a lane-keeping regulation is not a pass/fail for a test track.
Duration-weighted over the laps of each model — lateral metrics by lat_active seconds, longitudinal by long_active seconds; peak rows show the worst lap; disengagements are Σ/Σ over the engaged km. The "human" columns are the same road driven manually in the same laps (before engagement and after take-overs), so ride-quality metrics have a like-for-like baseline — the idea behind nuPlan's expert-derived thresholds and Waymo's log-likelihood metrics.
Every metric on this site compares a trajectory the model published with what the car did. This page shows that trajectory the way the car sees it: from its own origin, in its own frame, for both models, and next to the path the car actually drove in the following seconds.
Left and right show one automated episode each (pick any): the pink points are the model's trajectory exactly as published on /control/predicted_trajectory (x forward, y left, from the origin above), the cyan line is where the car actually went during the following seconds, transformed into the same frame at the same instant, the faint pink lines are the other plans of the episode (their spread is the plan jitter). Drag the scrubber to move through the episode; the point labels are seconds ahead.
Situations are labelled from the motion itself, so the same rule applies whether the driver or the controller is driving — that is what makes the human baseline comparable. Lateral and longitudinal labels are independent (a braking sharp turn is both), which gives the six labels the questions ask for.
Steering-wheel equivalents use δ ≈ 16.36 · atan(2.72 κ); the measured κ and the steering-implied κ agree with slope 1.01 on these laps (Page 9).
Episodes: gaps under 0.5 s are merged, runs under 1 s dropped. An episode counts as "AV" when the loop was active for ≥ 80 % of it and "human" when ≤ 20 %.
Duration-weighted means over all automated episodes of that situation (and over the manual ones for the human column). Colours follow the bands on Page 2. Click a situation to change the replay above.
Any lap, any second. The map is coloured by lateral situation, the timeline strip by longitudinal situation; engaged time is drawn brighter. Hover the charts to move the cursor everywhere; drag the scrubber or press play to drive the 3-D car along the recorded odometry with the recorded steering, tilted by the INS roll and pitch, over the lap's own relief. In the 3-D view, lidar map switches on the reflectivity-coloured point-cloud map built from this lap's lidar scans (the road reads dark, paint and kerbs bright; Page 9).
Source topics in the bag: odometry 50 Hz · /vehicle/state, /control/actuators, engagement 100 Hz · /vehicle/actuators_output 50 Hz (emitted cycles) · INS orientation and ekf_nav 50 Hz · plans and PredictedTrajectory 10 Hz nominal (one per model inference; 100 ms median interval on every lap, AV2.5's mean 9.4–9.6 Hz because some inferences run long) · CAN tx 50 Hz, rx 100 Hz · lidar 10 Hz · front camera 20 Hz, cabin 30 fps.
Analysis grid 50 Hz. What this page shows: charts, state card, HUD and map cursor at 10 Hz (every 5th grid sample); the 3-D car, its steer curves and the head of the executed path are re-posed every rendered frame while playing, interpolating x/y/yaw/steer/roll/pitch between the 10 Hz samples (the car moves < 1 m between two samples, so the interpolation error is centimetres); the fps button in the 3-D toolbar shows the render rate, and the view lowers its own resolution and far-field point density if the page cannot hold ~33 fps; the plan ribbons in the 3-D view and the vehicle-frame panel are every published plan (10 Hz nominal, the bag's own rate) with every published point (AV2.5's full ~5 s horizon, AV2.0's 4 points); the home and Page 4 replays also step through every plan but clip the ribbon to the 0–2 s the tracking metrics compare; the vehicle-frame trajectory panel shows the plan in force with all its points and its ±16 neighbours; the videos are 20 fps transcodes (the front camera's native rate); the lidar map is static (one build per lap, 0.2 m voxels on the road).
Click a row to jump the replay there. "AV" = loop active ≥ 80 % of the episode.
The bag is already uploaded after every drive and split into essentials, camera, lidar, cabin. The controller evaluation only needs essentials (48–123 MiB per lap), so it can run on the NAS as soon as _UPLOADED.json appears — no GPU, no replay of the stack, deterministic, ~3 minutes per lap on one core (2.6–3.3 min measured; 28 min for the ten laps run back to back).
Weighted by lat-active seconds. Lower is better for every row except engaged km. Δ is AV2.5 relative to AV2.0.
The real record for one lap, trimmed. This is what the store keeps and what the alert rule reads.
Each lap against the band of the other laps of the same model (median ± 2 × MAD) — the rule a nightly job would apply. Flags mark laps outside the band; they are prompts to look, not failures.
What the essentials bag already gives us, what each metric reads from it, and what is missing. Rates are measured on lap 08:50.
/control/debugActuatorCommand.curvature is the lag-adjusted curvature after the ±0.5/v² clip, and steering_angle_desired_deg already is the feed-forward angle: the lateral law is open loop and angle_offset_deg, steering_wheel_angle_bias and road_roll_rad are 0 on the car (while lateral is inactive it is the measured angle; a chained steering override — the joystick controller, off on these laps — would replace it). Not published: the pre-clip curvature and the clip centre, the longitudinal plan age the PID read at, v_target_now / v_target / v_target_1sec / a_target, and the PID P/I/D/F terms. The site re-derives v_target_now (the PID setpoint) the way LongControl does, but not v_target / v_target_1sec (the state-machine inputs) nor a_target (the feed-forward is a windowed value, not the plan's accel at that age, which is what the site plots); P and F could be re-derived from the gains, the integrator only as a residual where the clip did not bind.LongitudinalPlan.longitudinal_plan_source (2 CRUISE, 3 E2E) and VehicleState.cruise_speed (= max(SET_SPEED, 30 km/h)) are published and in the essentials bag but not in the metrics extract. In blended mode the planner takes the tighter of the model's plan and the cruise ray t·max(v_cruise, v_ego − 2 m/s), and the 30 km/h floor (8.33 m/s) sits right on the laps' 7.8–8.3 m/s mean speed: on the 15 Sep laps the source was CRUISE on 58–73 % of long-active plans on every AV2.5 lap and 0 % on every AV2.0 lap, with cruise_speed at 8.33 m/s on every message of all ten laps — so the AV2.5 e_v_rms is mostly tracking of the set-speed ray, not of the model, and the metric cannot tell. Extract the CRUISE share per lap and the set speed; any longitudinal A/B must log both (Page 11, lever 7). The .msg comment "always CRUISE" describes the ACC branch and is stale for the deployed blended mode.desired − sent shows that the limiter clipped, not which stage: in the deployed std-angle mode the speed-scheduled slew (2.5 / 1.5 / 0.4 ° per 20 ms frame up, 3.75 / 2.25 / 0.9 down in the x50 profile — 1.5× the opendbc port's down row), the ±200° clamp or the ±12° window around the measured angle; in std mode also the follow-the-hands path (while |DRIVER_EPS_TORQUE| exceeds driver_override_torque the request is replaced by the measured angle); in the legacy five-stage chain (not running on any of these laps) the speed-scheduled band around the measured angle (±10 / 7.5 / 5° winding up, ±9 / 6 / 4.5° down, at 0 / 5 / 15 m/s), the ±15° safety clip (inert behind that band), the 340→160° speed envelope (200° above 8 m/s), and the FIR and 10–50° low-speed clamp (both off on the car). Nothing in the chain records which stage bound; the follow counter exists in the transmit cycle but is not published.lateral_deviation_centerline and R79's "does not cross a lane marking" need one; the mosque loop has no map. A one-off surveyed centreline per route would enable lane-position metrics.run.yaml carries only route / kind / model. The model side is already structured and latched in the essentials bag — /perception/model_info (wing_msgs/ModelInfo, one message per lap: weights, input profile, waypoint grid), which the extractor should read instead of matching run.yaml's model string. The controller side exists only as free text in three /rosout start-up banners (steering limiter, vehicle model, delay compensation) plus the planner's MPC-horizon warning, which is printed only when the horizon is not the deployed 2 s grid — the interim fingerprint the extractor hashes and Page 10 groups by. Still missing, to be added as run.yaml keys and once on a latched /system/version topic (Page 10): the wing-av git describe, the profile name, a SHA-1 of the controller-node parameters plus the limiter block, a controller-type string, the lag delay actually used (the two schema keys are dead on the deployed host, Page 11) and, as a separate key, the planner's grid and cost weights. The panda firmware needs no new field: the 10 s panda tx: … verdict: verified line in /rosout asserts the board matched the profile's expected_firmware_hex.long_control.cpp:102-104), the actuator-delay bounds 0.3 / 0.4 s (:106-107), the blended-MPC weights x 0.1 / v 0.2 / a 5 / Δa 40 / j 1 (longitudinal_mpc.cpp:249-251), the accel box [−3.5, +2.0] (longitudinal_mpc.hpp:83-84) and the ±0.5/v² curvature band (kMaxLateralJerk 5.0, lateral_curvature.hpp:53) have no schema key, are printed by no banner and reach no file: the banners carry the limiter rows, the vehicle model and the profile clamp, never a compiled value, so a rebuild that changes any of them is indistinguishable in the bag from the previous build. Print them in the controller's start-up banner in the same commit that moves them (so the Page 10 fingerprint sees the change) and let the params_hash below cover them (Page 11, levers 4–8)./rosout carries panda tx: LIVE | verdict: verified | … errors 0, heartbeat-errors 0 | … board: mode 29, controls_allowed 1, safety-blocked 0 — zero on every line of all ten laps — but the extractor does not read it, and the board's own heartbeat_lost, fault_status and the health packet's rx / tx error counters are read over USB every 500 ms and never printed. CAN integrity on this site is therefore inferred from frame timing, counters and checksums; publishing the health packet on /diagnostics would make it a metric.replan: 8.95 Hz mean over 200 cycles against a 10.00 Hz target (5 skipped, the graph still busy) — which the extractor does not read; the AV2.0 path keeps no count at all. This site reconstructs stalls from LateralPlan inter-arrival gaps ("Longest plan gap", the Page 10 plan-feed badge) and Page 11 quotes the log line by hand. Publish the per-cycle graph time, the busy-skipped and stale-camera skip counts and the in-flight flag on /diagnostics, so a plan gap can be attributed to the GPU, to scheduling or to the camera set.EngagementStatus has alert_text1/2 fields that the controller never fills.wing_msgs/ControlDebug at 100 Hz, plus three fields on ActuatorsOutput at 50 Hz and one latched StackIdstd_msgs/Header header # stamped like ActuatorCommand (vehicle/can_stamp), same update() int64 frame_counter # same chain as ActuatorCommand — the join key # lateral — the controller's own terms (LagAdjustedCurvature + LatControlAngle) float64 curvature_plan_now # the clip centre: curvatures[0] while lateral_plan_age_compensation is off (every lap so far) float64 curvature_desired # lag-compensated, BEFORE the ±0.5/v² clip; the clipped value is ActuatorCommand.curvature bool curvature_jerk_clip_bound # |curvature_desired − curvature_plan_now| > 0.5 / max(1, speed_ema)² float64 speed_ema_mps # what the lag adjust and the band divide by (speed_ema_alpha 0.15), not v_ego float64 lag_delay_s # steer_actuator_delay + extra_delay actually used (0.21 s hard-coded on the deployed host) float64 steer_law_deg # LatControlAngle's output every cycle, even under a chained steering override bool steer_ref_overridden # a chained steering_angle_rad reference (joystick) replaced it in ActuatorCommand # longitudinal — locals of LongControl::update today; nothing publishes them float64 t_since_plan_s # the plan age the PID read at float64 v_target_now # interp(t_since_plan, knots, speeds) — the PID setpoint (≡ this site's v_tgt) float64 v_target # min(v at age + 0.3, v at age + 0.4) — the state machine's input, not the PID's float64 v_target_1sec # v at age + 1.4 s — decides planned_stop and the STOPPING → PID release float64 a_target # the feed-forward: min over the 0.3 / 0.4 s window of 2(v_end − v_now)/Δt − a_now float64 pid_p float64 pid_i float64 pid_d # always 0 (kd = 0 in the port) — carried so p + i + d + f is the unclipped sum float64 pid_f # a_target × kf (kf = 1) while long_control_state == PID, else 0 # ── on wing_msgs/ActuatorsOutput (50 Hz, one per emitted transmit): the limiter and the # CAN encode step only on even 100 Hz frames, so their result is known only there uint8 limiter_stage_bound # 0 none, 1 slew, 2 ±200° clamp, 3 ±12° window, 4 driver-override follow (std mode); 5–9 the legacy stages float64 accel_raw_can # clip(accel × 18, −50, +30) counts; 0 on brake-hold bool accel_capped # ── wing_msgs/StackId on /system/version — latched, once per run (Page 10's fingerprint) string software_commit # git describe of the deployed wing-av (nothing produces one today) string profile # x50 / sim / … string params_hash # sha over the parameter structs the nodes hold — not the yaml — plus the compiled constants above float64 lag_delay_s # the two schema keys are ignored by the deployed host, so the hash cannot prove the delay float64 pid_accel_min, pid_accel_max, angle_offset_deg, steering_wheel_angle_bias, curvature_scale bool std_angle_limiter # + std_rates_up/down_deg[3], std_max_angle_deg, std_max_angle_error_deg, driver_override_torque string panda_firmware_hex # the signature the arming check compares against expected_firmware_hex # planner: plan_horizon_s, plan_lat/long_intervals, knot spacing, path / motion / jerk / steering-rate costs
This is the Apollo SimpleLateralDebug / SimpleLongitudinalDebug idea sized for this stack. With it, the metrics on Page 2 need no re-derivation and the PID split becomes a metric too. Three rates, not one: the controller's terms go out at the ActuatorCommand rate and stamp; the limiter's verdict and the raw CAN accel exist only where the limiter steps (every second cycle — the bag shows /control/actuators at 10 ms and /vehicle/actuators_output at 20 ms with a 2:1 count on all ten laps, so a 100 Hz copy would repeat every odd frame), and ActuatorsOutput has no frame_counter — join the two on header.stamp, one cycle after the command it applied. Provenance is not a per-frame field: ActuatorCommand goes through a real-time publisher whose thread copies the whole message every tick, and two strings that never change do not belong there — they go once on the latched /system/version, the pattern /perception/model_info already uses. Fields that already exist were dropped from the proposal: steering_saturated, long_control_state and the feed-forward angle are on ActuatorCommand today.
● = the topic supplies a value the metric is computed from. Sample masks are not shown: lat_active / long_active (ActuatorCommand, /control/actuators; the lateral mask additionally drops samples where the intervention detector sees a hand on the wheel, MAIN_TORQUE on /vehicle/state) select the samples of every tracking, actuation, ride and plan metric, and v_ego < 0.3 m/s (/vehicle/state) opens the stop window. Disengagements per km keeps /control/actuators because lat_active defines its km denominator.
Every metric family on this site traces to at least one of these. The numeric bands on the ride-quality and stop cards come from the cited regulations and studies; the tracking and actuation bands are this site's working choices (openpilot's 2.5° saturation threshold, used as the watch bound of "Steering: desired vs measured", excepted) and are marked as such on the cards. Italic quotes are verbatim from the linked page (fetched 2026-09-19); lines marked summary condense a table or a list from it. The arrow line is what we take from it.
mcap-ros2-support: the thirteen topics on Page 7 that feed the extractor (twelve message topics plus the /rosout start-up banners; /diagnostics is recorded but not read).header.stamp — linear for continuous signals, zero-order hold for commands, flags and plans (plans are read at their age, exactly as the controller reads them).PredictedTrajectory in the map frame with the odometry pose at its img_tstamp (the points are absolute base_link displacements in that frame) and compare point τ with the odometry at header.stamp + τ — trajectory_node samples the fitted curve at image time + τ + (header.stamp − image time), so the first point is already "now" (Page 3). Horizons 0.5 / 1 / 1.5 / 2 s.interventions.py, a port of the team's wing_interventions detector, itself wing_sense's): steering from the signed column torque MAIN_TORQUE — clip(τ + 25, ±350) → 0.1 s moving average ÷ 2 → |·| → 0.3 s median → > 50 → 0.21 s dilation, i.e. trip points τ > +75 / τ < −125 — which sees a hand resting on the rim that openpilot's |DRIVER_EPS_TORQUE| > 6 flag misses (lap 08:50: 3 steering episodes, 18 s, against 0 steering_pressed events); gas > 1 % and brake > 1 % of pedal travel (brake not at a standstill); gaps under 0.2 s closed. The flags drive the driver rows of the timeline, the pills in the 3-D HUD and the state card; the episode list (0–30 s, outside the first/last 10 s) is in each lap's JSON. The detector runs at 100 Hz on a zero-order hold of /vehicle/state, where it is bit-identical to the team's package (a 50 Hz run halves every window's noise rejection). build_pcl_map.py, one map per lap, the "lidar map" picker in every 3-D view): each lap's ~3,100 Livox HAP scans (/sensing/lidar/points/soa_zstd, ~140 M points) are placed with the pose interpolated at each point's own timestamp through the URDF lidar mount — x, y, yaw from the (2-D) odometry, roll and pitch from the INS with the mount offsets removed, and the road height from the pitch integral pinned to the ekf_nav altitude where the GNSS was actually aiding it (39–55 % of each lap; 120 s low-pass) and loop-closed (a lap that ends where it started ends at the height it started; the closure error of up to 2.3 m is spread over the lap) — gated to 1.5–60 m range, −1.5…+7 m height and v > 0.5 m/s, then voxelised, each voxel keeping the mean reflectivity and the mean position of its returns (so a sloping road stays a continuous surface instead of 0.2 m terraces): 0.2 m within ±0.6 m of the terrain surface inside 10 m of the recorded tracks, 0.4 m for everything else (~2.9 M voxels, ~27 MB per lap). The terrain itself is the 15th percentile of the lowest solid voxel per column on a 2 m grid (a low quantile so overhanging foliage does not lift it), median-filtered and diffusion-filled beyond the lidar's reach; it is embedded for every lap and lifts the car, the ribbon, the plans and the draped satellite image. One map per lap because the laps' odometry frames are co-registered to GNSS only to 1–3 m — a lap shown on another lap's map sits beside the road. The lidar road comes out 0.3 m below base_link, i.e. the URDF lidar height is ~0.3 m short; the car is drawn with its wheels on the lidar road. Residual: the road is scanned from both directions only around the parking spot (first and last seconds), where the two passes disagree by up to ~1 m on three laps (08:56, 09:35, 09:41) — a doubled road surface for a few seconds at the lap start; elsewhere the car's own pass sits within 0.3 m of the terrain (p99) on every lap.Stops. The drivers disengaged before nearly every standstill; automated stop episodes exist in only . The stop metrics are implemented and shown on the manual stops as a baseline, but no fleet conclusion about automated stopping is possible from these laps.
One route, one day, one car. Ten laps of the mosque loop are enough to demonstrate the metrics and to compare AV2.0 with AV2.5 on that loop, not to set fleet thresholds. The regression rule on Page 6 uses ±2 MAD for illustration; a real band needs weeks of laps.
Plan-vs-execution ≠ pure tracking. The horizon cross-track includes plan changes. Read it with the plan-jitter column, and prefer the setpoint-level errors (κ, v, a) when attributing to the controller alone.
Human baseline is a different task. The drivers were driving and supervising; the manual segments are mostly the start of the lap and the recoveries after take-overs, which are not neutral samples of the road. Use the comparison for order-of-magnitude only.
Acceleration source. Longitudinal acceleration and jerk come from the wheel-speed v_ego, not from the IMU; jerk peaks above ~5 m/s³ should be read as "the speed signal moved fast", and the 3-D replay shows the IMU's own reading next to it.
No lane reference. Without a map or a surveyed centreline there is no "distance to lane marking"; all tracking here is relative to the model's own plan, which can itself be off the lane.
cd /mnt/w/wingfin/docs/wing-av/wing-av-control-metrics-src # one lap (read-only on the bags; needs a venv with mcap, mcap-ros2-support, numpy) python extract_metrics.py 085042 # all ten, logged ./run_all.sh 2>&1 | tee logs/extract.log # rebuild the site python build_data.py && python assemble.py ../wing-av-control-metrics.html
The tracker at data.wingfins.com scores every uploaded bag for the model (km per intervention, best run, S / B / G). This page proposes a fourth Control view next to it: one control scorecard per bag, computed post-drive from the essentials bag by the same job that made this site, grouped by controller build rather than by model, in the tracker's own layout — summary cards, "By controller build", "By planner configuration", "By month", "Runs", and a definitions footer. Below it is mocked on the ten 2026-09-15 laps, which are the tracker's 16:50–17:49 rows of that day.
By comparing three things that live in different places: what was planned (PredictedTrajectory placed in the map), what was commanded (ActuatorCommand / ActuatorsOutput) and what happened (odometry, VehicleState). Cross-track / heading error at 1–2 s, curvature / speed / acceleration error against the setpoint, sent-vs-measured steering and lag — Pages 1–2, 36 metrics with formulas and bands.
Yes: laps are labelled straight / curve / sharp from the curvature (the larger of commanded and measured while the lateral loop is active, measured otherwise) and cruise / decel / accel / stop from the speed profile, and every metric is aggregated per situation with a human baseline — Page 4. Stops are the gap: no lap has an automated stop yet, so that column is human-only.
Yes, as a post-drive batch job on the essentials bag (~3 min per lap on one core, no GPU, deterministic), keyed by vehicle / commit / model / route, with a rolling-median regression check — Page 6. This page is what that job feeds the tracker.
Thirteen recorded topics cover everything here (twelve message topics plus the /rosout banners that carry the controller fingerprint; /diagnostics is recorded but unused); the gaps are a /control/debug message (PID terms, limiter stage, raw CAN accel), a route reference for lane-centre metrics, and a controller fingerprint in the bag — Page 7 and the section below.
The tracker's view chips would gain a fourth entry — All monthlyDailyModelsControl — that answers a different question from the Models view: not which checkpoint drives best but how well does the control stack execute whatever plan it is given, across models and routes. So its grouping key is the controller build (the fingerprint below), the model becomes a column of the runs table, and the planner configuration gets its own breakdown. Route, Range and Class filters work as they do today.
Mocked on the ten laps. Local time (UTC+8), route and model from run.yaml; the controller build and planner config are read from the /rosout start-up banners because nothing structured for the controller is in the bag yet (the model's identity and knot schedule are, on the latched /perception/model_info — Page 7). Scores are 0–100 (rule below; A ≥ 90, B ≥ 75, C ≥ 60), raw cells are coloured by their catalogue band; clean / watch / degraded is the plan feed badge — the quality of the plans the controller was given, never a verdict on the drive (a lap can be clean and score D). Click a run to open it in the explorer.
One build drove all ten laps — the same std-angle limiter, vehicle model and delay compensation — so the ten scores are ten samples of one controller, not a comparison between two. The banners behind the fingerprint:
The planner did change between the two models: the AV2.5 laps ran a 10 s / 32-interval lateral-MPC horizon (the planner's own warning), the AV2.0 laps the deployed 2 s grid. This split is why the Control view keeps the planner separate from the controller: the same controller scored 89 behind one planner setting and 82 behind the other, and the Models view cannot show that.
The four letters are the layer scores, one per question along the control chain; the control score is their weighted mean and the grade is on that. Each is 0–100 and means the same thing everywhere on this page: 100 = every metric in the layer is inside its ok band; 60 = the metrics sit at their watch bound on average; 0 = beyond twice the watch bound. A layer is the plain mean of its metrics' scores, so one bad metric can pull a layer down while the others are fine — the card below each letter names the metrics, and the worked example opens on the lap with the lowest score of the ten, metric by metric — pick any other lap from its menu.
| layer score | reads as | colour |
|---|---|---|
| 90–100 | every metric inside, or nearly inside, its ok band | green |
| 75–89 | most metrics ok, one or two in the watch band | pale green |
| 60–74 | the layer sits around its watch bounds — look at which metric | amber |
| < 60 | at least one metric is well beyond its watch bound | red |
The same colours are used for the total and for the grade thresholds (A ≥ 90, B ≥ 75, C ≥ 60). A metric with too little exposure (< 60 s lat_active / < 30 s long_active) is left out of its layer rather than scored; a layer with nothing left is dropped and the remaining weights renormalised.
Definitions (the footer the tracker would carry). Every figure is over the engaged drive of the bag: lateral metrics while lat_active and no hand on the wheel (the intervention detector's steering flag is masked out — a driver correction is not the controller; the stack's own flag stays in lat_active_stack and hands_on_s in the record says how much engaged time was hands-on), longitudinal while long_active, no window, no end cuts (S / B / G excepted: the detector drops episodes ≥ 30 s and any inside the first / last 10 s of the bag). control score = 0.40 tracking + 0.20 actuation + 0.25 ride + 0.15 supervision, each layer the mean of its metrics' band scores (rule below); a metric whose loop had too little exposure (< 60 s lat_active for lateral metrics, < 30 s long_active for longitudinal ones) is left out of its layer's mean, not scored as bad; a layer with no metric left is dropped and the remaining layer weights renormalised; a run whose plan-feed metrics are all missing shows unscored. grade A ≥ 90, B ≥ 75, C ≥ 60, D below. Raw metric cells (ct@1s, e_v, lag, lat acc, diseng) are coloured by their Page 2 band — ok / watch / beyond — the same rule as the catalogue cards and the Page 4 heat map; diseng is coloured by its per-engaged-km rate. plan feed is the quality of the plans the controller was given — plan jitter @1 s, MPC-invalid share, plan age, longest plan gap — shown as a badge and kept out of the score: clean = all four inside their ok band, watch = one in its watch band, degraded = one beyond it. It is not a pass / fail of the drive: clean means the controller was given a fair plan, so a low score is the control chain's to own; a degraded run is one where the tracking numbers are partly the planner's. T / A / R / S are the layer scores. ct@1s is the plan-vs-execution cross-track RMS at 1 s, per lateral situation (straight / curve / sharp). e_v speed error RMS, lag sent→measured steering lag, lat acc the lap's peak lateral acceleration while lat_active (coloured by its band), diseng falling edges of enabled, S / B / G steering / brake / gas intervention episodes that begin while enabled, from the team's wing_interventions detector (MAIN_TORQUE filter, pedals > 1 %); the tracker's own S / B / G counts steer differently (a controller override with no pedal down), so its steer counts are lower — one rule should be adopted for both views. int% intervention time over engaged time, engaged m:ss, km the lap distance. In the group rows (controller build / planner configuration / month) the score is the mean of the runs' scores and ct@1s / e_v / lag are unweighted lap means (Page 6's by-model figures are exposure-weighted and can differ in the third decimal); controller build = the fingerprint of the limiter, vehicle-model and delay-compensation banners, planner = the MPC-horizon banner. The record also carries human — the same lap's manually-driven stretches (v > 2 m/s with lat_active off for the lateral ones, v > 1 m/s with long_active off for jerk), the driver baseline for the ride metrics and the one exception to the engaged-only rule — and stop_n, automated stop episodes (≥ 80 % long_active): none on these laps. loop · exposure (scoring table): the activity flag a metric is measured under and its exposure threshold.
Each metric is scored from its Page 2 bands: 100 inside the ok band, falling linearly to 60 at the watch bound and to 0 at twice the watch bound. Bands tagged on Page 2 as a regulation / benchmark (11 edges) or a constant of the deployed stack (7) are the only ones with an authority outside this site; the other 52 are working choices, so a score compares runs of the same route with each other — it is not a certification. Columns: the ten laps by time.
The metrics do not care which controller produced the command: they compare the plan with the executed path and the command with the measured response, so an LQR, a pure-pursuit or a learned controller would be scored on the same scale as today's chain (the lateral MPC lives in wing_planner; wing_controller turns its curvature plan into an angle request through the angle controller, and the vehicle interface's std-angle limiter bounds what reaches the wire). What the tracker cannot do today is tell two controllers apart: the bag records the model name and its knot schedule (structured, on the latched /perception/model_info) and, only as free text in the /rosout start-up banners, the limiter mode, the steer ratio and the MPC horizon — nothing structured for the controller, no commit, no parameter hash. It matters already: the five AV2.5 laps ran the 10 s / 32-interval lateral MPC horizon of av2.5.yaml (the wing_planner warning is in their /rosout), the five AV2.0 laps the 2 s grid, so the model comparison above is also a planner-configuration comparison. The proposal therefore adds a controller fingerprint to every run, and the section groups by it exactly as the Models section groups by checkpoint:
git describe of the deployed wing-av: 3.5.0 on a release commit, 3.4.0-3-gb62576ee past one) and the vehicle profile name (x50);angle+std-limiter, lqr) once more than one exists.Cheapest way to get them into the bag: the launch layer writes them into run.yaml next to route / kind / model, and publishes them once on a latched /system/version topic that the essentials recorder keeps. Then "By controller" is a third grouping next to "By model" and "By month", and a controller change shows up as a new row with its own mean score, not as a mystery jump in a model's curve. The only MPC-specific items on this page are two plan-feed metrics (mpc_solution_valid, plan age from LateralPlan); a planner without an MPC would report its own validity flag under the same name.
A score that drops tells you that something changed, not what. Three things make the cause readable:
With the fingerprint in every record, a change of controller commit, parameter hash or profile is a boundary in the run list. The section compares the runs after it with the runs before it on the same route and model (never across models — the AV2.0/AV2.5 comparison above already hides a planner-horizon change), using Page 6's rolling median ± MAD as the "before" band. A boundary with no metric moving is a harmless change; a metric moving with no boundary is not the controller — look at the model, the route, the driver or the day.
Protocol that makes this cheap: when a controller or setting changes, drive the old and the new interleaved on the same route and model, ≥ 3 laps each (the 15 Sep session did exactly this for two models). Per-situation columns then show whether the change is in curves, straights or stops.
The four layers are ordered along the chain, so the pattern of which moved is the diagnosis:
| what moved | what stayed | most likely |
|---|---|---|
| plan feed (plan jitter, plan age, plan gaps) | actuation | planner / model / compute load — not the controller |
| tracking (ct@1s, κ error) | actuation (sent ≈ measured, lag) | the control law or its tuning: the EPS did what it was told, the request was wrong |
| actuation (clip share, sent−measured, lag) | plan feed | limiter / vehicle-interface settings or the EPS itself (slew table, ±12° window, delay) |
| ride (jerk, reversals) with tracking flat | tracking | gains too aggressive — a tuning change traded comfort for error |
| supervision (S / B / G, disengagements) | everything else | driver confidence or route conditions, not the controller |
| everything, on every route | — | a stack-wide change (timing, CAN, sensor) — check Page 7's rates and the CAN integrity counters first |
The decisive test needs no new drive: the bag holds every plan the model published and every vehicle state, so the old and the new controller can be run offline against the same recorded inputs (open-loop: plan + VehicleState in, ActuatorCommand out) and their command traces compared second by second — the same plan, two commands. A difference there is the controller by construction; if the commands are identical, the on-road change was somewhere else. This is comma's maneuver-report logic applied to a whole lap, and it is the one comparison a scorecard cannot replace: on the road the model, the driver and the day always change with the controller.
_UPLOADED.json for a new <drive>/essentials bag (48–123 MiB per lap).extract_metrics.py <drive> (this site's extractor; ~3 min per lap on one core, read-only on the bag, no GPU, deterministic) → the per-lap JSON; the scorecard is about 1.5 kB derived from its overall, situations and intervention_summary blocks — the record on the right.control.json next to the bag and expose it as /api/control (runs) and /api/control/models (grouped), the shape the tracker already uses for /api/models.extractor, detector), as the tracker already prints Detector config 8f3620a861b6.stack is empty because the bag has no controller fingerprint yet)What the score is not. It is a within-route, within-fleet comparison: 52 of the 70 band edges are this site's working choices (Page 2 tags each one), the regulatory ones are applied to slightly different quantities than the regulations define, and a lap's tracking score is partly the planner's (hence the plan-feed badge). Read the layer scores and the raw columns, not only the total — and re-cut the bands once the tracker has a few weeks of runs, from the fleet's own distribution.
Page 10 says how a lap scores; this page says what in the software could move that score, where it lives, what it does today and which way each metric goes when it is turned — and where the car, a sensor or the computer, not a number in a yaml, is the limit. It is built from an inventory of … parameters and … hardware limits across the six subsystems of the chain, each entry checked against wing-av 3.5.0 @ 52d552e7 at the cited line and against the ten lap JSONs; … are worth tuning, … maybe, … are bounded by hardware. The shortlist first, then the metric → lever matrix, the full inventory and the hardware ceilings.
What "expect" means here. Every effect on this page is the direction and the mechanism, read from the code and from the lap statistics — not the result of an A/B on the car. The sim numbers quoted were measured on a ground-truth centreline rig, and the on-road AV2.0 / AV2.5 split changes the model and the planner grid together. The size of any effect is what the interleaved-lap protocol at the bottom has to establish; the Page 10 layers are how to read the result.
autonomy.launch.py (:244-268) stacks the yaml layers and hands the merged block to each node: schema default (config/schema/*.yaml) → common.yaml → network.yaml → vehicle byd_atto3.yaml → model yaml (av2.0.yaml / av2.5.yaml) → profile x50.yaml → the per-run test-plan overlay (only the keys listed under mapped: in plan_config_map.yaml; anything else is refused, nothing accumulates). A key that no layer sets runs at its schema default — which is how all four lateral MPC weights ran on 15 Sep.
Two things sit outside that stack. The ros2_control hardware component reads its values from the URDF (vehicle_interfaces.yaml, baked at build) with ros2_control.* profile keys substituted in where the parameter is declared substitutable — the limiter table, the clamps, the override torque. And compiled constants — the longitudinal PID gains, the blended MPC weights, the ±0.5/v² band, the accel box, the CAN encoding — change only with an image rebuild. Below them are the DBC and the panda firmware, which are the car's.
| stage | in force on 2026-09-15 |
|---|---|
| lateral controller | openpilot LatControlAngle port, open loop (no gain, no integrator), 100 Hz in the ros2_control host; look-ahead 0.01 + 0.2 s hard-coded; deviation band ±0.5/v²; vehicle model steer_ratio 16.36, slip −0.00255, roll 0 |
| std-angle limiter | up 2.5 / 1.5 / 0.4, down 3.75 / 2.25 / 0.9 °/msg at 0 / 5 / 25 m/s; clamp ±200°; windup ±12°; hands above 80 counts — the banner on every lap |
| longitudinal | LongControl PID kp 2.2 / 2.0 / 1.8, ki 0.45 / 0.4 / 0.32, deadzone 0, delay 0.3–0.4 s, clamp [−3.5, +2.0]; blended MPC (experimental_mode true), accel box [−3.5, +2.0], weights x 0.1 / v 0.2 / a 5 / Δa 40 / j 1 |
| wire | frame 482 STEER_ANGLE 0.1°/bit at 50 Hz; frame 814 ACCEL_CMD = clip(a × 18, −50, +30) = [−2.78, +1.67] m/s² at 50 Hz |
| planner, AV2.5 laps | 10 s / 32 lateral / 12 longitudinal intervals, quadratic knots, resample on; path 20 · motion 0.11 · accel 0 · jerk 0.04 · steering rate 350; B-spline fit, 1 s breakpoints |
| planner, AV2.0 laps | 2 s / 4 / 4, uniform, no resample (schema defaults on a 4-waypoint graph); same weights; cubic fit |
| model / timing | AV2.5 av33 on TensorRT fp32, every 2nd camera set (10 Hz), ~104 ms inference age; AV2.0 epoch49 on CUDA every frame, lidar-synchronised 10 Hz, 56–73 ms; plan age at the command ~50 ms; control loop FIFO 50, model container normal priority |
| sensing | SBG INS + ekf 50 Hz, single-antenna GNSS no RTK; wheel speed 0.019 m/s per count, no IMU fusion; cameras 20 Hz host-stamped; Livox HAP 10 Hz median-stamped |
Ranked by how much of the laps' out-of-band error each one owns, then by how cheap it is to try. Each card names the file and line, the value in force, which metrics it moves (chips coloured by the worst lap on the ten), what to expect when it is turned, the lap evidence (computed live from the embedded metrics) and the trade-off — every lever here moves at least one metric the other way, so score the whole card, not one number. Tags: profile edit and restart, per-run overlay through the test plan, compiled rebuild, hw-bound capped by the vehicle or a sensor.
| parameter | why to leave it |
|---|
The other direction: for each catalogue metric, every inventory entry that cites it, ordered worth-tuning → maybe → leave. The two dots are the worst lap of each model against the metric's Page 2 band.
Every entry: its name, where it is read (file:line at 52d552e7), the value in force on the laps and its source, the metrics it moves, and — click a row — what it is, what to expect either way and the caveat. The worth tuning / maybe / leave tag is the inventory's own verdict; the cost tag is read from the source of the value. hw-bound marks entries whose reach is set by the vehicle, a sensor or the computer.
Some metrics cannot be moved by any number in the tree because a component of the car, a sensor or the computer sets the floor. The table is the short version — what each ceiling is, what was measured on this car, which metrics it caps, what software can still do in front of it, and what it would take to raise it. The … inventory entries behind it follow, one per subsystem view.
At road speed the steering rate is software-bound: the slew table allows 67 °/s up at the laps' ~8 m/s against a rack that can do ~150 °/s. The longitudinal side is the reverse — the controller assumes a 300–400 ms actuator the bags measure at 140–180 ms, and asks for +2.0 m/s² on a wire that carries +1.67. The two cheapest wins on this page are therefore agreements between software and hardware (levers 4 and 6), not new hardware.
st_rate_cost, lateral_motion_cost, OP_steerRatio, OP_slipFactor, do_desired_curvature_clip) go through the test-plan overlay and vanish after the run; limiter rows, clamps and the override torque are x50.yaml lines; the grid, the fit and fp16 are model-yaml lines; the PID gains, the delay bounds, the MPC weights and the accel box need an image built to your own tag.run.yaml carries the controller fingerprint, the /rosout banners are the only record — the limiter banner prints its rows, so a limiter change is visible; a compiled gain is not. Print the values you change in the start-up banner in the same commit.The Page 10 score is 0.40 tracking + 0.20 actuation + 0.25 ride + 0.15 supervision, so the arithmetic favours levers that move tracking without spending ride — and every lever above trades between the two: path_cost and the slew rows buy cross-track with lateral jerk, kp buys speed error with longitudinal jerk. Two kinds of change escape the trade:
After those, the honest expectation is a different balance rather than a better total until the bands are re-cut from a few weeks of fleet runs — which is why the scorecard shows the layers and the raw columns, not the total alone.
The inventory is wing-av-control-metrics-src/research/parameters.json — six subsystem passes (lateral controller, longitudinal controller, planner, vehicle interface, model-to-trajectory timing, vehicle and sensor hardware), each written from the checkout at 52d552e7 and then re-checked entry by entry: every file re-opened at the cited line, every lap figure recomputed from data/metrics_<drive>.json, and the lateral chain re-run in Python against all ten bags (it reproduces the commanded curvature to ≤ 1.8e-6 m⁻¹, which pins the deployed constants beyond the banners). Corrections made in that pass are kept in the entries' text ("CORRECTED …"). The checked-out tree is the analysis copy, not the car's.