38 lines
7.8 KiB
Markdown
38 lines
7.8 KiB
Markdown
# Independent accepted-output test review checklist
|
|
|
|
Prepared before implementation/test handoff from root commitb86c8dbc's `harness/analysis-mixed-acceptance-contract-20260925.md`, baseline2941713, ACCEPTANCE-ORACLE and WARNING-ORACLE. No compiler/source edits or execution. This is a review rubric, not a new contract or passing result. The selected contract supersedes the older proposal's bundle-only destination: accepted success now also moves actual Project/warnings.
|
|
|
|
## Essential positive and negative controls
|
|
|
|
- Minimal real p→next path must use one unchanged inherited entries17/bytes152 policy from original parse through verification. Compare the entire literal bundle and entire independently expected Project, empty warnings, final Slots(1,0,105), exact Limits/ProjectLimits and verification Usage. Expected verification work is bytes81, raw tokens19, entries1, references0, deepest_frame0, numeric0; original work remains separately bytes78. Merely checking aliasnext or equality with a Project obtained by reparsing the emitted output is circular.
|
|
- The corresponding original bytes151 policy must demonstrably reach genuine emitted PendingRename before verify refuses at main37..38, line2column27, exact byte-limit error. It must not mutate the pending policy. Entries16 rejects emission first and cannot be called a verification entry boundary.
|
|
- Warning-bearing real path uses entries18/bytes161, complete literal returned Project and exactly the independently expected unnamed warning, including object ID, fieldname, message, path, relatedempty, severity,11..69 span and line2column1. Final inventory must be(2,0,157), not O alone or live syntax+W. Original warning11..66 must not be returned as output evidence. Same-path bytes160 must reach verification then refuse mapping at full11..69; a generic is_err or parser one-byte span is inadequate.
|
|
- Reuse real OA/SA/all-file Unicode/CRLF/swap fixtures with previously independent expected bytes and complete desired models. Compare all files including untouched files, entry/key set, returned whole model and warnings. Tests must exercise the actual consuming verification method, not repeat emitter-only assertions after bypassing it.
|
|
|
|
## Equality tests that actually discriminate
|
|
|
|
- On a fresh real pending result, equal-byte aliasnext→nope and nameProject→Changed preserve O and remain admissible; verifier must reject at the exact semantic-mismatch0..0 diagnostic. If tests cannot establish actual admission, a generic failure may only be a parser/model failure. Add a narrowly observable stage or an independent admitted fixture control, not a second permissive production verifier.
|
|
- Malformed version-byte corruption must refuse before equality, separately from model-invalid but parseable output. A model-invalid fixture needs the specific admission diagnostic, not the semantic mismatch message. Keep these cases distinct.
|
|
- ActualE1→3 corruption `10000`→`1 + 2` uses equal-length literal bytes and original policy4096. Assert the *actual reservation amount* or narrow test observation is3, not merely that verification rejects. Both old-E and actual-E implementations can reject the changed root variant; mismatch alone cannot prove provenance of the bound. The fixture is an injected corruption, never claimed authentic emitter output.
|
|
- Wide/deep positives and target-only shape mismatch must reach equality. Arrange a common admitted actual left model and independently authored unequal desired target; identify list/call-size checks versus child traversal. Root-type inequality alone does not prove a large live frontier. Never infer a required equality cap-minus-one from E if earlier lowering dominates that same unchanged cap.
|
|
- Signed-zero checks must reach whole-Project comparison while semantic fields otherwise agree, and expect the established PartialEq behavior. A no-op returning before verify or comparing a model to itself is vacuous evidence.
|
|
- Any length-changing corruption must have an explicit current-inventory rejection or independently constructed actual-O fault provenance. Do not silently update retained scalars after mutating bytes, then call that a real inherited owner. Test helper mutability must remain cfg(test), with no arbitrary production bundle/model/policy/Slots constructor.
|
|
|
|
## Ownership, transfer and warning accounting
|
|
|
|
- Return AcceptedRename outside a lexical scope that drops original bundle, original source witness and desired Project, then inspect bundle/model/warnings. A success variable merely used before those inputs drop does not prove owning lifetime. Compile-time borrow restrictions should prevent pending move/mutation while its carried guard exists; runtime counters alone cannot prove this.
|
|
- Verify actual resolved Project is moved into the result rather than desired cloned into it; source review is necessary because equality makes successful values observationally equivalent. Likewise warning/model cloning cannot be reliably excluded by value-equality tests alone.
|
|
- Warning inventory controls cover checked sums, non-Error severity and empty related; unexpected-shape/overflow tests are labeled internal faults. Do not manufacture a real late finalO+W refusal when the larger earlier resolver/equality peak already prevents reaching it.
|
|
- Full resolver live inventory must include warnings throughout equality; finalO+W is checked as a subset without resetting live storage before actual drops. Tests may use narrow transition observations, but cannot substitute artificial counter rewrites for source-backed lifecycle review.
|
|
- Failed verify must return only diagnostics, no accepted/usable partial output, and preserve original source/desired values. Small static mismatch diagnostics must not recursively format large models. Returning a borrowed accepted type or separately mutable model/bundle access violates the selected contract despite passing value assertions.
|
|
- Compare exact inherited policy fields, including model/traversal and numeric controls, not just entries/bytes. New snapshot Usage is independent; no original+verification work sum is claimed. Accepted retained slots exclude temporary syntax/equality arenas and include all mapped warnings.
|
|
|
|
## Stack and evaluator boundaries
|
|
|
|
- Scoped128KiB verify/drop tests prepare legitimate predecessors outside the worker. Worker still performs actual verify, assertions and accepted/result destruction for meaningful deep/wide inputs and a late refusal. Do not replace it with an already accepted value move, discard the returned value outside the worker, enlarge the stack, or silently route a failure around the intended phase.
|
|
- Preserve old test bodies/limits and source-pin all new results. Inspect added representation/prologue cost; a few scalar fields or nonrecursive code do not prove harmless stack use. Ordinary-harness success alone is not128KiB evidence.
|
|
- Keep exact-cap syntax checks separate from existing inventory-walker scratch. The prior entry review caught that a full-cap syntax witness cannot necessarily run a later scratch-charging recount. A roomy separate same-literal recount is valid; raising the original boundary to accommodate the evaluator changes the claimed test.
|
|
- Initial compilation/runtime failures must remain visible. Build gates and source reviews establish different facts; read-only precompile review is not a compiler pass. Final readback must preserve prior281 outcomes,16 failure bodies normalized only for numeric thread identifiers, format bytes and artifact/source concordance unless a separately justified explicit change is selected.
|
|
|
|
At handoff, bind TEST-REVIEW to final test/source hashes, identify which assertions implement each discriminating boundary above, and report concrete omissions rather than expanding into universal parser-scratch or physical-memory testing. New accepted-output tests do not by themselves establish public source8, installation/history, repeated carried editing or GUI integration.
|