Define frozen routing corpus and isolated timing protocol
This commit is contained in:
@@ -0,0 +1,64 @@
|
||||
# Independent routing corpus: pre-result adoption rubric
|
||||
|
||||
Prepared before the new corpus routing/timing outputs. This is a proposed engineering decision rule, not a replacement for user requirements or a production acceptance claim. Compare released715 router, J and L on identical frozen cases; route-only observation must not invoke the unsupported layered layout prototype.
|
||||
|
||||
## Existing requirements and precedent
|
||||
|
||||
`PROJECT_GOAL.md` provides no numeric routing-latency requirement in the scoped search. `harness/release-routing-verification.md:172–184` supplies an existing initial100ms settled-solve target, explicitly not a universal guarantee:100 nodes/150edges/eight pins per edge measured release119ms full solve and2.4ms maximum single-edge preview (debug863ms/16.4ms). The119ms miss remains a recorded limitation. `harness/allocation-arrange-planner-20260923.md` explicitly disclaims large-scene latency. No frame-rate or universal routing SLA is inferred from these records.
|
||||
|
||||
`docs/language.rst:1235–1256` requires semantic-port precedence, suspension of authored route data for promoted folded endpoints, restoration on expansion, exact stored route metadata and explicit route-edit lifecycle. These are hard constraints. Automatic unpinned non-port attachment sides may legitimately change with routing cost; classify their coordinate changes as geometry results, not invariant violations.
|
||||
|
||||
## Frozen cases and trace tasks
|
||||
|
||||
Before any result, freeze manifest, all complete Projects/diagram identities, input hashes, generation source and variant binaries. Include original and folded workshop SA Allocation; a named mixed8–12-node scene with shared/distinct physical semantic ports, authored anchors/pins, a cycle and an untouched other view; its fixed relation-ID permutation with explicit old/new semantic correspondence; and a sparse40-node/~55-edge scene. Preserve all frozen cases in every report, including failures and censored calls.
|
||||
|
||||
The corpus author should record these concrete trace tasks by actual source/target/edge names and IDs before outputs: (1) workshop cross-container exchange in original and folded projection, identifying promoted visible ends; (2) mixed shared-output fan-out, identify each of at least two named targets without switching routes; (3) two distinct ports on the same card, trace each to its correct receiver; (4) cycle return edge, trace its direction back to the intended source; (5) authored pinned edge, identify the exact endpoint and passage through each pin; (6) sparse first-to-last long-span exchange plus one neighboring distractor. Repeat mixed tasks under the frozen ID permutation, using its semantic mapping. Do not select whichever routes look best after observation.
|
||||
|
||||
For later native inspection, all target identifications must be correct without consulting the source model, route recoloring, temporary geometry edits or hiding the distractors. Normal selection/zoom/pan are allowed and recorded. Report actions, wrong turns, uncertainty and time; no invented universal human completion-time threshold. Inspect at the same declared readable zoom and one Fit view, acknowledging Fit may be unreadable for40 nodes. No pure geometry metric substitutes for this task gate.
|
||||
|
||||
## Hard gates and geometry evidence
|
||||
|
||||
- No mutation of Projects or stored routes; semantic endpoint/edge identity, authored port anchors/attachments/pins, folds, filters and other-view records exact. Physical semantic-port tips remain exact. Dormant attachments remain stored. Promoted authored routes correctly suspended rather than incorrectly enforced; compare visibility/projection eligibility explicitly.
|
||||
- No new missing route, nonfinite coordinate, invalid model, unrelated-node/header penetration or routing warning. Existing warning types must remain visible per edge; they cannot be averaged away. Authored unavoidable collision cases require exact predeclared exceptions, not blanket incident-edge exemption.
|
||||
- Report every raw overlap interval and unique proper/T/bend/endpoint contact with edge IDs and locations. Classify common physical terminals but retain raw counts. Labels/headers/port labels must not obscure required routes in native review. Automatic attachment changes and label shifts are explicit results.
|
||||
- No weighted aggregate quality pass. A new isolated crossing replacing coincidence is a reviewable tradeoff, not automatically failure or success. Adoption requires a predeclared trace task improvement or restored correctness and no newly ambiguous task elsewhere. ID sensitivity must be disclosed; a favorable naming/order cannot be chosen post hoc.
|
||||
|
||||
## Timing decision rule (proposed, not user-owned requirement)
|
||||
|
||||
Use optimized Windows variants with identical toolchain/profile and interleaved declared order. Parse/project outside the measured call; route_diagram alone is timed. One warmup plus seven measured samples per case; record each duration, median and maximum, not a claimed p95 from seven values. Verify identical output across repetitions outside the timed region. Exclude parsing/serialization/hash/drop work explicitly from the timer, while admitting that allocator/cache/OS state still affects observations.
|
||||
|
||||
For this modest frozen corpus, adopt the existing100ms target as the proposed per-case median ceiling and200ms maximum-sample ceiling. Also require candidate median <=1.25*released-baseline median +1ms per case; compare L against J as the simpler candidate too. These are deliberately declared engineering guardrails, subject to explicit future revision rather than post-result relaxation. If baseline itself fails the absolute ceiling, report a capacity limitation and do not treat relative improvement alone as a responsiveness pass. A meaningful correctness fix exceeding a guardrail needs a documented tradeoff and native response evidence; it is not an automatic pass.
|
||||
|
||||
Any completed warmup/measured call above5s censors the case and stops its remaining repetitions. Retain its actual elapsed time and failure status. This after-call cutoff cannot stop a hung individual call; parent supervision must provide a finite process timeout. Missing/censored samples never count as fast results. Seven interleaved samples are screening evidence, not a scaling or tail-latency theorem.
|
||||
|
||||
## Decision and stopping rule
|
||||
|
||||
Complete this fixed comparison once, investigate concrete correctness or measurement defects, and stop parameter tuning. Prefer the simpler J if L has no independently observable task benefit or exceeds the timing guardrails. Keep L experimental or discard it rather than add permanent settings to preserve sunk work. If none meets the rubric, report the failing constraint and choose the smallest conceptually different next experiment; do not quietly extend the corpus until a winner emerges. Production Command::Layout/reset integration, Undo/Redo/Save, desktop input behavior and native acceptance remain subsequent gates.
|
||||
|
||||
Driver/corpus source review is pending their readiness; this rubric makes no claim that they already implement these checks.
|
||||
|
||||
## Ready driver source review
|
||||
|
||||
Reviewed `crates/oa-core/examples/routing_cost_probe.rs` SHA256 `90ad27099a9486e9a7d45460bc9b66930e35d8336e3007fc30f90b48d88407d4` before compilation. No blocking source defect found for this protocol. The timer encloses routing plus result black_box; projection, parsing, cloned input, result equality/digest/serialization/destruction are outside. Seven full RoutedViews compare to warmup, with endpoint comparisons additionally emitted. Both warmup and samples can censor; median is absent for incomplete cases and stability_observed is false when warmup prevented all measured calls. Schema requires description in each case in addition to name/project_file/diagram. Paths are canonicalized under the manifest directory and duplicate case names reject.
|
||||
|
||||
The driver's censored case can still finish with process exit0 by design. External acceptance must require every expected case, complete=true, censored=false, seven measured samples, all equality flags true and project_unchanged=true; do not equate exit0 with responsiveness. A finite external process timeout is still required. The digest is correctly labeled noncryptographic; external corpus/source/binary hashes remain necessary. No execution occurred during this review. Corpus source review remains pending readiness.
|
||||
|
||||
## Final corpus and probe source review before execution
|
||||
|
||||
Read final prepared sources, with no builds/generation/routing calls by reviewer:
|
||||
|
||||
- `routing_eval_corpus.rs`: SHA256 `55ad185b3a72f5c1aa082732058604600cb016556c41aa5a3a21122766171bf6`.
|
||||
- `routing_cost_probe.rs`: SHA256 `6f67abae2f912ed27215f8823b6dbcd41ca880b198dc60d5a10e88c71b38b8c0`.
|
||||
- `auto_layout_probe.rs`: SHA256 `48e540f2176252d8ea1e110dd17fce37bd19f75f61fa6902356925bfe01540bc`.
|
||||
|
||||
No blocking source finding for the frozen five-case comparison. This completes the pending source review stated above; compilation/generation remain parent-owned verification.
|
||||
|
||||
Corpus constructs the saved12-placement SA Allocation workshop and its single collapse-flag change,10-function/14-exchange mixed case, reversed-edge-ID paired case, and40-function/55-exchange sparse case. No Layout/projection/router call constructs input geometry. Generated port IDs are v5-derived from fixed exchange IDs in ports::plan_functional_exchange; generated element/placement IDs are reidentified. The permutation uses temporary IDs and asserts inverse recovery of the entire Project, including authored-route references. Six fixed semantic-port anchors, an authored feedback pin/label, shared source/target ports and a separate-view sentinel provide meaningful independent constraints. The generator validates all cases before writing a fresh directory and refuses overwrite. Trace IDs, aliases, names and semantic endpoints are frozen in manifest metadata.
|
||||
|
||||
Coverage limits, not hidden successes: workshop contains no authored routes, so its folded case does not exercise suspension/restoration of a nonempty authored pin list. Mixed has semantic-port anchors and a pin/label but no explicit non-port EdgeAttachment. Sparse subsystem groups are flat named chains, not hierarchical containers. The frozen trace list selects representative branches; native review must inspect companion branches too to establish fan-out discrimination, using the manifest/model identities rather than selecting routes after seeing outputs. These are finite first-corpus limits, not blockers requiring the experiment to become comprehensive acceptance.
|
||||
|
||||
The geometry mode validates nonempty unique cases, Project validity and diagram existence; then calls metrics/roundtrip directly, without Layout/reset/candidate transformation. Analytic metric selfchecks still run before mode dispatch. Emitting full projected_edges, actual endpoint owners, effective pins, pin fractions and custom_suspended supports the required later semantic/geometry distinction. The output is observation, not an automatic oracle: later audit must compare authored/effective pins and port positions, account for folded promotion, and interpret legal automatic attachment changes. Full before_project and manifest remain present for this audit.
|
||||
|
||||
The updated cost schema intentionally removes deny_unknown_fields, retaining additional corpus/case metadata in flattened maps and emitting those maps. This now accepts generator trace_tasks/source_counts/metadata without losing them. Required typed fields remain required. Clock and censor logic agree with the earlier review. Neither probe must claim that a zero exit status alone establishes all semantic constraints or timing acceptance. External exact corpus/binary hashes and timeout remain necessary.
|
||||
|
||||
The geometry loader's relative-component check is less strict than the timing loader's canonical containment check (a symlink could resolve outside); for these parent-generated frozen ordinary files this is not a blocker. Both must read the same hashed files. No request is made for unrelated loader architecture or extra fixture families before the finite run.
|
||||
@@ -0,0 +1,21 @@
|
||||
# Frozen source-first routing corpus (#32)
|
||||
|
||||
Prepared from f72aa2ca without calling Layout, projection, routing, or reading new routing results. This defines inputs and invariants, not expected quality. Generator ownership: `crates/oa-core/examples/routing_eval_corpus.rs`; no production changes. Generation, validation, serialization, and disk I/O are outside routing timing.
|
||||
|
||||
## Cases fixed before generation
|
||||
|
||||
1. `workshop_sa_original`: decode the embedded existing `examples/routing-workshop.json` through the production persistence decoder. Select Allocation view `sa_system_boundary_and_functions`, UUID `cb21862d-79e1-5a9d-bdf0-ee344c2fb219`, with 12 authored placements. Preserve the complete loaded Project. The saved source currently has six views and no authored routes: preserving these empty route records is explicit, not claimed pin coverage.
|
||||
2. `workshop_sa_folded`: clone case 1 and set only the placement of `rescue_coordination_system` to collapsed. Semantic records, other views, all placement positions/sizes, routes and labels stay exact. This exercises hidden descendants and promoted endpoints through an actual allocation view. No generated route result is used to select the fold.
|
||||
3. `mixed_dispatch`: ten named Functions (Sensor A, Sensor B, Ingest, Validate, Archive, Correlate, Plan, Dispatch, Actuator, Operator) and 14 FunctionalExchanges. Edges in authored order: A→Ingest; B→Ingest; Ingest→Validate; Ingest→Archive; Validate→Correlate; Archive→Correlate; Correlate→Plan; Plan→Dispatch; Dispatch→Actuator; Dispatch→Operator; Operator→Actuator; Archive→Actuator; B→Archive; Plan→A. The final edge is explicit feedback; branching can create more than one graph-theoretic cycle. A/B→Ingest reuse one semantic input; Ingest fan-out and Dispatch fan-out each reuse one output. Other endpoints are explicitly distinct. Empty carried-item contracts are legal and avoid unrelated data typing. Six fixed anchor records include opposite-side feedback and vertical fan-out. Plan→A carries one authored waypoint and a label offset sentinel. A separate three-placement FunctionalDataflow view carries its own route/label sentinel. No Layout is used to construct either view.
|
||||
4. `mixed_dispatch_edge_ids_reversed`: exact clone of case 3 with the 14 relation IDs reversed across their original sorted set, using a collision-free temporary ID domain and production `reidentify` for every reference. Names, endpoint semantics, port identities, aliases, positions, and authored route geometry retain correspondence. The inverse permutation must recover the exact original Project. This diagnoses relation-ID order sensitivity; exact geometric equality is an observation, not assumed quality success.
|
||||
5. `subsystems_40_sparse`: five named subsystems, each with eight named processing stages, represented as 40 flat Function placements. Seven consecutive stage links in each subsystem (35), plus four cross-subsystem links per subsystem to its successor modulo five (20), totaling 55 distinct FunctionalExchanges. Cross links go stages 1/3/5/7 to successor stages 0/2/4/6. Positions are a predeclared 8-column × 5-row stage/subsystem grid. Every endpoint explicitly uses a distinct semantic port. This modest sparse cyclic workload is not the earlier diamond/bipartite corpus.
|
||||
|
||||
## Trace tasks and invariants
|
||||
|
||||
Predeclared trace tasks: workshop aliases `sa_report_to_relay`, `sa_direct_advice`, `sa_incident_decision`; mixed Sensor B→Ingest (shared target input), Ingest→Archive (shared source output), Dispatch→Operator, and feedback Plan→Sensor A; sparse subsystem 0 stage 1→subsystem 1 stage 0 and subsystem 4 stage 7→subsystem 0 stage 6. The manifest records trace edge identities and endpoints from the source model.
|
||||
|
||||
Every generated Project must pass `validate_candidate` before any output directory is created. JSON files serialize complete Projects directly. Manifest schema is 1; each case has name, relative project_file, diagram UUID, description, source counts, trace tasks, and relevant correspondence/sentinel metadata. Stable IDs and unique aliases are assigned independently of routing. Repeated generation should give byte-identical output; this is for the parent to verify, not asserted as already run.
|
||||
|
||||
Routing must preserve complete input Project bytes/model identity, semantic port assignments, authored routes/labels/attachments, fixed anchors, fold flags, and other views. Automatic non-port attachment-side choices are routing results, not preservation failures. The workshop contains no authored route sentinels; mixed cases supply that coverage. A promoted route may suspend authored geometry in rendered output while its stored metadata must remain intact. The generator does not prescribe which route topology, automatic side, contact count, length, or warning count is acceptable.
|
||||
|
||||
Usage (parent execution only): `cargo run --locked --offline -p oa-core --example routing_eval_corpus -- NEW_DIRECTORY`. The destination must not exist; manifest is written last. Existing files are never overwritten. No corpus generation, compiler, formatter, or router execution by this author before handoff.
|
||||
@@ -0,0 +1,15 @@
|
||||
# Route-only timing protocol v1
|
||||
|
||||
Parent execution order fixed before any outputs: preserve optimized binaries for released router source715036e, target-join J08656b18, and lanes L e8ec6db in a common harness. Run released, J, L, then L, J, released, with one complete corpus invocation per slot. Do not overlap compilation or other evaluator processes with timed execution. External timeout is90 seconds per invocation; retain partial output and explicitly mark missing cases if it expires. One separate untimed geometry invocation per variant uses `auto_layout_probe --route-corpus MANIFEST`. No new layout or router parameter tuning is part of this comparison.
|
||||
|
||||
Prepared example: `crates/oa-core/examples/routing_cost_probe.rs`. No execution, compilation or formatting by its author. Parent owns optimized builds, corpus and runs.
|
||||
|
||||
Invoke a preserved **release/optimized** binary as `routing_cost_probe.exe <absolute-manifest.json>`. No rounds override: every case uses one timed warmup followed by seven timed measured calls. External manifest is `{ "schema": 1, "cases": [{ "name": "unique-name", "project_file": "relative/project.json", "diagram": "UUID", "description": "fixture purpose" }] }`. Project files are direct serde Project JSON, validated before routing; paths resolve within the manifest directory. Cases run in manifest order.
|
||||
|
||||
Projection, decoding, validation and Project clone happen once outside timing. Each Instant interval includes only `route_diagram` and result black_box; inputs pass through black_box before the clock. Result destruction, equality, endpoint checks, digest and printing are outside. Allocation performed by routing remains included. Each repeat's complete RoutedView is compared to warmup, outside the clock; any mismatch is emitted then exits unsuccessfully. Project equality is checked against its initial clone. Endpoint equality is relative to the warmup, not a substitute for the independent semantic endpoint/quality evaluator.
|
||||
|
||||
JSONL starts with protocol metadata, then one record per completed/censored case with all measured nanoseconds, warmup duration, median, observed maximum, counts/warnings, stable route-point digest and checks. FNV-1a64 is a diagnostic identity only; use external SHA256 receipts for corpus, source, executable and runtime binding. Full RoutedView equality covers more fields than the point digest. Metadata reports debug_assertions but does not prove compiler optimization settings; the build receipt must do so. No environment secrets/configuration are printed.
|
||||
|
||||
If warmup or any sample exceeds five seconds, finish that call, stop remaining repeats for that case and mark it censored. Median is null for censored/incomplete series; maximum is only of observed measured samples, possibly null. The threshold is not a timeout: a hung call still requires external process supervision. A warmup-only censored case has no repeated stability evidence. Process failure may leave only partial JSONL; never interpret missing cases as zero cost or successful completion.
|
||||
|
||||
Warmup is retained so cache-sensitive first-call behavior remains visible, but one warmup does not prove steady state. Retaining its RoutedView and the original Project adds constant memory pressure across variants; repeat comparisons may influence caches between calls. Measure the same frozen corpus with interleaved preserved binaries for base/J/L variants on one machine, using the same release toolchain/profile, runtime, machine load and manifest order; do not compare dev versus release or include compiler time. Repeat entire interleaved sessions when a difference is close to noise. Seven samples characterize this run, not a latency distribution or statistical guarantee. Pair timings with the parent's independently calculated geometry/quality vectors; faster routing alone is not acceptance.
|
||||
Reference in New Issue
Block a user