3.8 KiB
Route-only timing protocol v1
Parent execution order fixed before any outputs: preserve optimized binaries for released router source715036e, target-join J08656b18, and lanes L e8ec6db in a common harness. Run released, J, L, then L, J, released, with one complete corpus invocation per slot. Do not overlap compilation or other evaluator processes with timed execution. External timeout is90 seconds per invocation; retain partial output and explicitly mark missing cases if it expires. One separate untimed geometry invocation per variant uses auto_layout_probe --route-corpus MANIFEST. No new layout or router parameter tuning is part of this comparison.
Prepared example: crates/oa-core/examples/routing_cost_probe.rs. No execution, compilation or formatting by its author. Parent owns optimized builds, corpus and runs.
Invoke a preserved release/optimized binary as routing_cost_probe.exe <absolute-manifest.json>. No rounds override: every case uses one timed warmup followed by seven timed measured calls. External manifest is { "schema": 1, "cases": [{ "name": "unique-name", "project_file": "relative/project.json", "diagram": "UUID", "description": "fixture purpose" }] }. Project files are direct serde Project JSON, validated before routing; paths resolve within the manifest directory. Cases run in manifest order.
Projection, decoding, validation and Project clone happen once outside timing. Each Instant interval includes only route_diagram and result black_box; inputs pass through black_box before the clock. Result destruction, equality, endpoint checks, digest and printing are outside. Allocation performed by routing remains included. Each repeat's complete RoutedView is compared to warmup, outside the clock; any mismatch is emitted then exits unsuccessfully. Project equality is checked against its initial clone. Endpoint equality is relative to the warmup, not a substitute for the independent semantic endpoint/quality evaluator.
JSONL starts with protocol metadata, then one record per completed/censored case with all measured nanoseconds, warmup duration, median, observed maximum, counts/warnings, stable route-point digest and checks. FNV-1a64 is a diagnostic identity only; use external SHA256 receipts for corpus, source, executable and runtime binding. Full RoutedView equality covers more fields than the point digest. Metadata reports debug_assertions but does not prove compiler optimization settings; the build receipt must do so. No environment secrets/configuration are printed.
If warmup or any sample exceeds five seconds, finish that call, stop remaining repeats for that case and mark it censored. Median is null for censored/incomplete series; maximum is only of observed measured samples, possibly null. The threshold is not a timeout: a hung call still requires external process supervision. A warmup-only censored case has no repeated stability evidence. Process failure may leave only partial JSONL; never interpret missing cases as zero cost or successful completion.
Warmup is retained so cache-sensitive first-call behavior remains visible, but one warmup does not prove steady state. Retaining its RoutedView and the original Project adds constant memory pressure across variants; repeat comparisons may influence caches between calls. Measure the same frozen corpus with interleaved preserved binaries for base/J/L variants on one machine, using the same release toolchain/profile, runtime, machine load and manifest order; do not compare dev versus release or include compiler time. Repeat entire interleaved sessions when a difference is close to noise. Seven samples characterize this run, not a latency distribution or statistical guarantee. Pair timings with the parent's independently calculated geometry/quality vectors; faster routing alone is not acceptance.