From f130a365d576901ea0d23c169427e6415b71b842 Mon Sep 17 00:00:00 2001 From: "sorokin.oleg.edu" Date: Thu, 24 Sep 2026 17:51:19 -0700 Subject: [PATCH] Specify private rename output ownership before verification --- ...alysis-mixed-emission-contract-20260925.md | 128 ++++++++++++++++++ .../.gitattributes | 1 + .../EMISSION-CONTRACT-REVIEW.md | 43 ++++++ .../EMISSION-TRANSITION-PROPOSAL.md | 105 ++++++++++++++ .../NEXT-PRODUCER-CRITIQUE.md | 60 ++++++++ .../NEXT-PRODUCER-HOOKS.md | 70 ++++++++++ 6 files changed, 407 insertions(+) create mode 100644 harness/analysis-mixed-emission-contract-20260925.md create mode 100644 harness/analysis-mixed-emission-design-20260925/.gitattributes create mode 100644 harness/analysis-mixed-emission-design-20260925/EMISSION-CONTRACT-REVIEW.md create mode 100644 harness/analysis-mixed-emission-design-20260925/EMISSION-TRANSITION-PROPOSAL.md create mode 100644 harness/analysis-mixed-emission-design-20260925/NEXT-PRODUCER-CRITIQUE.md create mode 100644 harness/analysis-mixed-emission-design-20260925/NEXT-PRODUCER-HOOKS.md diff --git a/harness/analysis-mixed-emission-contract-20260925.md b/harness/analysis-mixed-emission-contract-20260925.md new file mode 100644 index 00000000..bb68c2c4 --- /dev/null +++ b/harness/analysis-mixed-emission-contract-20260925.md @@ -0,0 +1,128 @@ +# Private mixed rename emission prerequisite + +2026-09-25. Selected private prerequisite following bounded design and independent +ownership/oracle review; implementation and runtime acceptance pending. Source baseline: +`fa7a980418ef99d772e67c66c28dfa7b5181f875`. This implements the output construction +and old-owner lifetime portion of the [rename successor](analysis-mixed-rename-plan-contract-20260924.md#mandatory-successor-accepted-rewritten-output). +It does not implement accepted output, public source8, persistence or history. +The [carried parser prerequisite](analysis-carried-phase-contract-20260925.md) +and its remaining before-growth limitations retain their stated scope. +The [producer inventory](analysis-mixed-emission-design-20260925/NEXT-PRODUCER-HOOKS.md), +[independent critique](analysis-mixed-emission-design-20260925/NEXT-PRODUCER-CRITIQUE.md), +[transition proposal](analysis-mixed-emission-design-20260925/EMISSION-TRANSITION-PROPOSAL.md) +and [contract review with ownership correction](analysis-mixed-emission-design-20260925/EMISSION-CONTRACT-REVIEW.md) +record candidates and rationale. This selected contract supersedes conflicting +earlier proposal choices, including the owning enum and runtime byte guard. + +## Ownership and API + +Consume the opaque `RenamePlan` and borrow a complete current `SourceBundle`. +Revalidate current entry, complete file set, paths and all bytes against the +original snapshot before any copying, including for no-op. Reuse the planner's +complete physical edit validation; no external edit list, Project or policy may +replace its authority. + +If no names changed, return the original `AuthoredMixedProject` unchanged, with +its actual bundle, policy, work usage, warnings and storage. Allocate no output. +The changed result owns a complete pending bundle, the exact inherited `Limits` +and `ProjectLimits`, its verified output storage total O, the desired Project +borrow and scalar original expression-site count E. It retains no old syntax, +catalog, warning, index or plan map. Original caller-owned source and desired +Project remain untouched. This does not claim those external borrows are freed. + +Keep pending fields private. Expose no bundle extraction, mutation, installation, +history, save or accepted-result conversion. Tests may inspect fields. No general +constructor accepting arbitrary bundle/Slots/policy is authorized. The only +construction path consumes the actual plan after checked emission. Reuse or add +a sealed carried-snapshot connection only if it accepts this actual pending +owner, never loose parts; invoking parser verification remains outside this slice. + +For changed emission in this prerequisite require original legacy syntax +(`carried == None`). Reject an explicit carried source before output reservation, +even its borrowed-zero mode. Dropping a source witness only drops its guard; an +external RetainedSnapshot would remain alive, so claiming only O survived would +hide C. Future repeated owned editing needs an explicitly consuming owner +transition or surviving C guard. No-op may preserve any source and its guard +unchanged, because no old reservation is discarded. This narrow provenance +restriction is intentional, not a numeric-zero shortcut or a final editing API. + +Select two explicit consuming methods: +`into_unchanged(self, current) -> Result, Failure>` and +`emit_changed_unverified(self, current) -> Result, Failure>`. +Both validate current snapshot and complete edits and enforce their branch; +calling the wrong branch refuses. Callers can select using the existing immutable +`changed_names().is_empty()` observation. A branch cannot change after that read. +This separates source and desired lifetimes without boxing, a large common +owning result enum, generic callback machinery or lint exemptions. Do not +duplicate full policy inside old syntax owners. Keep all existing stack controls. + +## Storage and construction + +Let B be the existing plan owner's live ledger, which already includes both the +R changed-name entries and M physical-edit entries. Do not add R+M twice. +Compute the complete final file byte sum S using the existing checked original +coordinate sizing walk, including untouched files. Then compute + +```text +O.entries = original complete file count F +O.occurrences = 0 +O.text = S + sum(canonical key UTF-8 byte lengths) + entry UTF-8 byte length +peak during emission = B + O +``` + +Every addition/subtraction must be checked. Reserve all O against the original +ledger and exact inherited policy before constructing any output String, copied +key, entry or map element. Reserving the complete monotone output up front covers +its partial construction; capacity/allocator overhead remain outside the logical +metric. Original sources and desired replacement aliases are borrowed during +copying. No lengths map, copied edit list, temporary complete file or second +output buffer is needed. Any additional owned container requires explicit scope +and reservation before growth. Source-size shrinkage does not release old B. + +Walk canonical files and the sorted edit stream. For each file, append original +untouched ranges and the exact qualified target's desired alias in original byte +coordinates, then the tail. Produce one String per file, including untouched +files; preserve complete paths and entry. Avoid reparsing, decoding, UUID creation, +normalization and global textual replacement. Validate original UTF-8 boundaries +and nonoverlap through the existing complete edit validator before construction. +Check the constructed length against independently computed size before ownership +transition. Check final complete file count and byte total without a second copy. + +Capture policy and E as scalars; destroy old plan maps and the complete old source +owner explicitly before returning the pending owner. O is then the only retained +in-scope owner from this operation. Failure yields existing located diagnostics +and drops partial output; no partial owner escapes. Preserve failure ordering: +stale snapshot, invalid edit map/size, output storage reservation, construction +consistency; changed carried-source refusal precedes output reservation. +Arithmetic fault injection must be labeled as such, not successful +real-world owner provenance. + +## Verification and remaining boundary + +Use independent literal original-coordinate maps and expected full bytes for OA +and SA, simultaneous global/scoped swaps, repeated/reordered labels, cross-file +references, ordinary categories, Unicode/CRLF and comments. Verify copied untouched +files and exact entry/keys. No-op must preserve original owner data/pointers, not +an equivalent reconstructed owner. Corrupt current touched/untouched data, paths, +entry and file set, and corrupt plan edit locations/targets through test-only +access; require no result. Reuse valid planner tests where they cover the same +validator rather than duplicate all cases. + +Independently hand-count a small real fixture's B and O. Choose exact entry/text +caps that admit all predecessor phases and refuse cap-minus-one at emission. +Do not mutate an admitted live ledger or widen policy to manufacture this test. +Assert final O, original exact policies, desired borrow and E. Include combined +grow/shrink and unchanged-file sizing. Small-stack emission/drop checks prepare +parser prerequisites on the normal thread, then move the complete consumed plan +into the worker; preserve all prior full-pipeline controls and limitations. + +Independent byte fixtures test the emitter; the pending type does not assert +runtime semantic authentication. A duplicate streaming emitter/comparator is not +required just to return this unverified internal state. Final acceptance still +requires actual-output parse/resolve/admission under the exact policy and carried +O, strict whole-Project equality with E scratch reserved, a resolved temporal +parser accounting contract, and an owning accepted result after temporary syntax +drops. Those requirements cannot be substituted by length checks or byte tests. +Original E is provenance metadata, not a demonstrated bound on arbitrary faulty +emitted syntax. The later verifier must justify its equality bound against both +actual operands and cannot rely on an unperformed runtime topology check. diff --git a/harness/analysis-mixed-emission-design-20260925/.gitattributes b/harness/analysis-mixed-emission-design-20260925/.gitattributes new file mode 100644 index 00000000..631ce65f --- /dev/null +++ b/harness/analysis-mixed-emission-design-20260925/.gitattributes @@ -0,0 +1 @@ +*.md -text diff --git a/harness/analysis-mixed-emission-design-20260925/EMISSION-CONTRACT-REVIEW.md b/harness/analysis-mixed-emission-design-20260925/EMISSION-CONTRACT-REVIEW.md new file mode 100644 index 00000000..a64f885c --- /dev/null +++ b/harness/analysis-mixed-emission-design-20260925/EMISSION-CONTRACT-REVIEW.md @@ -0,0 +1,43 @@ +# Independent emission contract review + +Reviewed `harness/analysis-mixed-emission-contract-20260925.md` against the current `fa7a980418ef99d772e67c66c28dfa7b5181f875` planner/guard shapes, accepted rename successor and carried-phase scope. Read-only; no contract/source edits, builds or network activity. + +Disposition: **no blocking design finding; suitable for the bounded private implementation described.** This review accepts its unverified prerequisite boundary, not final edited-output acceptance. + +## Runtime byte guard decision + +The selected omission is reasonable. My earlier NEXT-PRODUCER-CRITIQUE recommended a streaming guard partly to establish authenticated topology for a potential later parser envelope. This draft does not grant that authority. An opaque unverified pending owner does not need a second runtime comparator merely to prove the same emission operation again. Independently authored complete byte fixtures discriminate emitter defects without installing another production algorithm. + +A runtime guard becomes worthwhile if a later phase relies on unchanged token/AST/include topology **before** parsing actual bytes, or promises runtime preservation of all non-name bytes independently of emitter correctness. In that case it should consume actual output by matching borrowed original/replacement chunks and prefixes, rather than reconstructing output through the same append loop/helper; comparison must separately cover entry, file set, keys and untouched files. Whole-Project equality alone does not establish lexical preservation of comments/formatting. None of those stronger claims is made by this pending type, so the guard can remain deferred. + +## Ownership and sizing + +The contract correctly requires stale complete-snapshot and complete-edit validation before any output copies, including no-op. No-op returns the original authored owner and allocates no output. Changed output derives solely from the consumed sealed plan, borrows aliases while copying, retains exact policy and desired borrow, and offers no extraction/mutation/save/adoption escape. + +O is correctly F entries plus every final source byte, copied canonical key and the distinct entry-path bytes, with zero lower occurrences. B already contains R+M. Reserving B+O before construction covers monotone partial output without double-charging plan maps or prematurely crediting shrinkage. Repeating a scalar sizing walk is preferable to an extra lengths map. Per-file length, complete count and total consistency checks detect size-construction disagreement without implying byte authentication. + +The named old-owner drop is essential: capturing output/policy/desired/E does not itself release old syntax/index/warnings/maps. Implementer must make that destruction precede return and preserve the original external source/desired borrows. Failure diagnostics retain the existing bounded rejection exception; no partial pending owner escapes. + +The proposed callback or nested-result shape is appropriately left for actual type review. It should not evade accounting with an otherwise unnecessary Box, duplicate the full policy inside old syntax or change expectations merely to satisfy enum-size lint. Compiler/Clippy and the inherited stack controls remain necessary evidence. + +## Useful acceptance boundaries + +The independent full-byte fixtures, combined growth/shrink, copied untouched files and no-op ownership assertions test the actual new behavior. Reusing the existing complete edit validator and its corruption evidence avoids duplicating the entire planner test matrix. New B+O exact-cap tests must admit predecessor phases under the same profile; final O assertions must follow actual old-owner destruction rather than scalar subtraction alone. + +Normal-thread setup followed by scoped small-stack emission/drop isolates the new operation without falsely promising that upstream parsing fits every worker stack. Existing full-pipeline controls and prior abort limitations remain separate. + +## One explicit future-use caution + +Original E is valid provenance metadata, but without runtime byte authentication it is **not automatically a bound on arbitrary actual output topology**. A faulty emitter may produce admissible-but-wrong bytes with a different structure. A future topology-based parser envelope or whole-Project comparison must prove its scratch bound for that case, derive a safe bound from admitted actual output/intent, or reject structural mismatch before exceeding the reserved bound. Merely retaining E from the original owner is not that proof. + +This is not a blocker to pending emission: the draft defers parser verification, temporal accounting and strict whole-intent equality. Preserve that deferral in code/API/docs; do not wire a future accepted-result constructor while relying only on this review, output lengths or test fixtures. No broader parser scratch expansion is required to implement the selected private milestone. + +## Qualification after carried-lifetime finding + +The original no-blocker disposition missed a material ownership edge identified by the other oracle reviewer. A source parsed through a retained carried guard borrows an external RetainedSnapshot. Destroying its syntax/index/plan drops only the guard, not that external owner or C. Returning a changed pending result with only O would therefore discard live-storage accounting authority without ending the corresponding ownership. My original disposition is insufficient without the following amendment. + +Reread the amended contract: **changed emission requires `syntax.carried == None`; any Some variant, including borrowed-zero mode, is refused before O reservation.** This is an explicit provenance boundary, not `slots == 0`. No-op remains permitted for any source only because it returns the same authored owner and guard without releasing the original baseline. Complete-current-snapshot and edit validation still precede copying/no-op as specified. + +With this restriction, the pending operation can honestly retain only O after the consumed legacy syntax/index/plan owner dies. Future repeated edits of owned output require consuming the actual external output owner or retaining its guard and counting surviving C; merely borrowing and later dropping a source witness is inadequate. Runtime tests should distinguish changed legacy success, changed borrowed-zero refusal, changed retained refusal and guarded no-op preservation, with adequate caps so unrelated resource failures do not hide this rule. + +The amendment resolves the identified edge for this prerequisite; the qualified disposition is suitable for bounded implementation. I also verified the contract now explicitly states that original E does not bound arbitrary faulty emitted syntax and requires a justified bound against both future equality operands. diff --git a/harness/analysis-mixed-emission-design-20260925/EMISSION-TRANSITION-PROPOSAL.md b/harness/analysis-mixed-emission-design-20260925/EMISSION-TRANSITION-PROPOSAL.md new file mode 100644 index 00000000..02974201 --- /dev/null +++ b/harness/analysis-mixed-emission-design-20260925/EMISSION-TRANSITION-PROPOSAL.md @@ -0,0 +1,105 @@ +# Private emitted-unverified rename transition — proposal + +2026-09-25. **Design only; not an accepted contract, implementation or edited-output acceptance.** Source pin `fa7a980418ef99d772e67c66c28dfa7b5181f875`. Paths below are relative to `crates/oa-core/src/`. No source changes, builds or Gitea operations. + +## Minimal outcome and privacy + +Parent selection supersedes the earlier enum/callback candidates: use two named consuming methods, with callers selecting from immutable changed_names().is_empty() before consumption. Both methods call validate_current and the existing validate internally, then reject the wrong branch. No common result enum, callback adapter, heap indirection or lint suppression is needed. This is accepted design direction, pending implementation/runtime gates. + +Use a narrow `rename::emission` child owning the transition and an opaque pending owner: + +```rust +// Fields private to rename::emission. Not Clone/Copy/Serialize/AsRef/Deref. +struct PendingRename<'t> { + bundle: SourceBundle, + limits: Limits, + project_limits: ProjectLimits, + retained: Slots, + expression_nodes: usize, // E, verified original count for later equality + desired: &'t Project, +} + +impl<'s, 't> RenamePlan<'s, 't> { + fn into_unchanged( + self, current: &SourceBundle, + ) -> Result, Failure>; + + fn emit_changed_unverified( + self, current: &SourceBundle, + ) -> Result, Failure>; +} +``` + +Use the repository's existing restricted visibility (`pub(in crate::language::analysis_source)`) only where the private caller must name types/methods. Implementation in a child of `rename` can read its parent's private plan fields; no mutable plan/model/token authority is exposed. `current` is only a freshness comparison input; emission reads the retained original bundle. It cannot substitute the source to emit. + +For **this bounded step**, PendingRename owns the bundle/policy/Slots/E directly. Do not widen `carried::RetainedSnapshot` with a sibling-callable `new(bundle, slots, policy)`. The future sole consuming transition can take an actual sealed PendingRename and transfer its verified fields into RetainedSnapshot after its execution contract is accepted. Until then the pending type exposes no production bundle getter/extractor, no borrowed carried input, no verification/adoption method and no conversion to SourceBundle/Project/session state. `cfg(test)` read-only inspection suffices to verify emitted bytes/counts/intent/policy. Existing RetainedSnapshot fixture constructors and carried parse APIs remain unchanged. + +This is an explicit staging choice: the current result is an **emitted-unverified owner**, not yet a RetainedSnapshot usable by production carried parsing. Its retained formula is exactly the future snapshot formula; eventual transfer can move fields without copying. If parent instead requires constructing RetainedSnapshot in this slice, the alternative is a carried-owned factory accepting the **sealed RenamePlan itself** and performing the emission transition internally, with minimal read-only/consuming plan access. An arbitrary-parts constructor is not justified. The pending-only option changes fewer existing privacy boundaries and creates no premature verification entry. + +An owning pending result with inspection only in fixtures is sufficient for an independently testable producer: consumers can hold/drop it but cannot save, install or treat its raw bytes as accepted model state through its API. The private name and absence of extraction are intentional authority restrictions, not runtime cryptographic authentication. A future trusted private implementation can add its sole verifier; no hash/token duplication is needed now. + +## No-op and unchanged authority + +First run existing `validate_current(current)` (`rename.rs:161–169`) so even no-op execution refuses a stale complete snapshot. If the sealed plan's names/edits are empty, consume its empty maps and return the original AuthoredMixedProject unchanged. No output String/key/entry copies, new snapshot parsing or new storage reservation occur. The source's original exact bundle borrow, maps, warnings, Project and policy survive. + +The unchanged owner remains an already-admitted original owner, not a newly emitted result. The whole-target gate already established alias-only equality; signed-zero-equivalent desired representation does not authorize rewriting scalar bytes. The no-op branch can end the desired borrow because it does not need later output equality. + +Do not implement a no-op by allocating an identical owned SourceBundle and calling that free. Do not run a second whole-model comparator merely to reconfirm the sealed plan. + +## Validation and emission sequence + +1. Consume the opaque plan. Complete-current-bundle validation compares entry, all keys, exact file set and every byte, including untouched files. +2. Rely on plan construction's `validate()` proof for whole token membership, original spelling, UTF-8 ranges and nonoverlap (`rename.rs:178–239`). No API mutates the maps afterward. Reuse its sizing logic and source authority; do **not** re-lex/lower/admit or build another seen/token/map inventory. Calling the same private validator once at emission can support internal fault-injection tests, but is redundant for sealed production authority and is not required as a new authentication layer. Parent should choose deliberately; prefer the sealed proof and cheap checked emission invariants below. +3. If changed, reject every explicit carried source (syntax.carried.is_some()), including borrowed-zero mode, before output reservation. Its external retained owner would survive dropping the source guard; O alone would then omit live C. No-op preserves any carried guard without rebasing. For this prerequisite changed emission therefore starts only from legacy syntax. Then compute checked complete output length S using the existing subtract-removed-before-add-replacements formula (`rename.rs:261–308`). Include every original file. Sum copied canonical key byte lengths K and entry byte length J using checked arithmetic; let F be the existing file count. Capture exact Budget::limits, syntax ProjectLimits, desired borrow and E as scalar metadata, without cloning the model or names. +4. Reserve the entire output increment `O=(F,0,S+K+J)` against the **still-live plan ledger** before allocating/copying any output String, map key or entry. The current ledger already includes old B plus R+M; do not add those again. Failure maps through existing source diagnostics while the old witness is alive and returns no pending owner. +5. Emit each file exactly once into one final String from original coordinates, traversing canonical files and the sorted physical edit map. Repeated sizing iteration is allowed; a new lengths map is unnecessary. For each file compute its final length with a scalar pass over that file's edits, allocate that String capacity, then append unchanged original slices and replacement names in order, followed by the untouched suffix. Clone that canonical path once into the final map. Files with no edits still get one exact full copy. Clone entry once. Markers/comments/includes/prose/UUIDs remain in unchanged slices. +6. Every cursor/range calculation is checked; exact token boundaries are inherited from the plan. At each file require produced logical length equal its computed length, final cursor within source and all file edits consumed. Require complete file count and total S after assembly. These are inexpensive producer invariants, not a new independent syntax authenticator or guarantee of intended semantic validity. +7. No partial map escapes on any failure. After complete assembly, snapshot the surviving O/policy/E/desired scalars, destructure and explicitly drop the old names/edit maps and AuthoredMixedProject **before constructing and returning PendingRename**. The output bundle moves once into pending. Drop the old ledger with its owner; pending retained state is O, not B+R+M+O. + +Rust's return/drop order should be visible in code: use a narrow local owning output, capture scalar/reference metadata, `drop(names); drop(edits); drop(source);` then construct pending. Borrowed original SourceBundle remains caller-owned after the source auxiliary owner drops, exactly as before; this transition does not claim to free or charge that external borrowed input. The external desired Project also remains borrowed and unchanged. + +## Counts and lifetimes + +Let B be the admitted authored owner's existing live auxiliary Slots, R changed targets and M changed physical edits. The plan already retains `L=B+(R+M,0,0)`. During emission the output reservation establishes peak **L+O** in each existing dimension: + +``` +O.entries = F +O.occurrences = 0 +O.text = S + sum(canonical_path_utf8_bytes) + entry_utf8_bytes +``` + +The entry is a separately owned String even when it equals a map key. Unchanged files still contribute their complete output bytes to S. Old source Strings are externally borrowed, not copied into B; all newly owned output bytes are counted once. Existing copied old syntax/name/warning text remains live in L throughout production and cannot be released early merely because a replacement shrinks a file. + +A whole-output upfront reservation avoids per-file partial reservation bookkeeping and an extra lengths map. Final file capacity uses its exact precomputed length, but accounting is logical bytes, not allocator capacity/BTree node overhead/RSS. Temporary String then map insertion transfers the same reservation, not two copies. Checked reserve before creation does not promise recovery from operating-system allocator failure; existing logical resource-failure behavior remains the scope. + +Once the old owner/maps drop, pending stores O and the exact inherited policy, with E retained only as original expression-count provenance for a future whole-Project equality reservation; it is not a proven bound on arbitrary faulty emitted syntax. Future carried verification must still account O plus parser/resolver allocations and decide required producer timing; the successful emission reservation proves none of those later checks. It also does not prove output is syntactically admissible or semantically equal to desired. + +## Avoiding another stack-layout regression + +Do not add Box/Rc/Arc or clone the old owner/model to simplify the outcome. Such allocation would require its own design/accounting and is unnecessary. Enum size is the maximum variant payload plus tag/alignment, not the sum of both payloads; nevertheless debug return temporaries can amplify a large unchanged owner. Keep helper layering shallow and do not retain duplicate inline policy in witnesses that already own it. Pending needs its one policy because the old owner actually drops. + +The selected split consuming API avoids a common large enum and requires no callback framework. Each method enforces its branch; the two explicit wrong-branch errors are acceptable bounded API checks. Ordinary Result layouts and generated stack usage still require parent gates; this source-level choice is not a measured stack guarantee. + +## Independent acceptance cases + +- **No-op:** pointer identity of original bundle and retained owner state, identical Slots/Usage/numeric/warnings, no output allocation/copy; stale unchanged-file or entry mismatch still refuses. No desired mutation. +- **Whole bytes:** rich OA/SA global+scoped swaps, nested/reordered/repeated labels, canonical multi-file coincident spans, Unicode/CRLF and category fixture with line-comment-separated references. Independently authored entire expected file strings/maps, not reconstruction through an emitter-like expected helper. Every non-token byte must match original at its corresponding unchanged segments; historical UUIDs/prose untouched. +- **Combined sizing:** the existing31-byte node shrink plus typed f growth example emits924 bytes from930, despite a hypothetical typed-only954; the933-byte negative refuses the inherited930-byte output-size cap before copying. Owned-text overlap may legitimately refuse under that same cap even when final S fits: use sufficient inherited text allowance for a successful emission test and distinguish planning-size versus live-owned-text caps. +- **Exact overlap:** independently hand-count F/S/K/J and old plan L. A cap fitting L and O separately but below L+O refuses before the first output copy. Exact combined cap succeeds if all actual predecessors fit; test entries and text independently. Assert final pending retained exactly O, not L+O or O twice. Do not tighten a live owner's Budget after admission to manufacture a boundary. +- **Complete stale input:** touched and untouched source bytes, file removal/addition, path rename and entry changes reject before allocation. Equal independent current bundle content succeeds; emitted authority still comes from the retained original. +- **Failure lifecycle:** malformed internal fixtures must enter existing plan-validation tests before a sealed plan exists; do not fabricate externally installable edited maps. Where emission has its own checked-arithmetic/range invariants, targeted private fault injection can assert no partial pending owner escapes. Overflow beyond valid profile ceilings is explicitly labeled helper/fault-injection evidence. +- **Provenance limitation:** changed sources from both real retained C and explicit borrowed-zero carried mode refuse before output allocation; no-op preserves each original guard. No numerical zero bypass. A future consuming carried-owner transition is required before repeated owned edits. +- **Ownership/API:** desired remains externally borrowed, old owner is dropped before Pending creation, output is independently owned and immutable through available API; only cfg(test) inspections reveal bytes/policy/O/E. No production extraction, parse/admission, session adoption or persistence path is added. +- **Regressions:** preserve all existing planner/carried outcomes and unchanged small-stack controls, inherited failure dispositions and formatting exclusions. Emitted-unverified success is reported separately from future output acceptance. + +## Minimal files and next decisions + +- `mixed/project/ordinary_index/rename/emission.rs`: emission outcome, pending owner, sole consumed-plan producer and its tests as a child module. Access existing private plan/source fields through lexical ancestry, not public mutable accessors. +- `rename.rs`: module wiring and, if needed, narrowly factor existing per-file checked-length iteration for use by sizing/emission without allocating a lengths table. Preserve the sealed constructor validation and existing inspection APIs. +- Scoped implementation note/tests. `mixed/carried.rs` need not change for pending-only delivery; no arbitrary RetainedSnapshot constructor is opened. + +Read against the parent draft harness/analysis-mixed-emission-contract-20260925.md: parent selected pending-only staging, split consuming methods, changed legacy-only provenance, one upfront O reservation and no production extraction. The draft requests reuse of complete edit validation and malformed-plan fixture coverage; invoking the existing validate once before construction is compatible and avoids inventing a second authenticator. No duplicate streaming byte comparator is proposed. Parent retains API selection and contract acceptance authority. A later accepted contract must specify the sole Pending-to-RetainedSnapshot/verifier transition and cannot mistake this owner for accepted rewritten output. + +Minimal independent oracle supplied by reviewer: empty project p renamed next has authored A=(14,0,9), plan L=(16,0,9), output O=(1,0,105), peak L+O=(17,0,114). Exact entries 17/text 114 and predecessor-compatible minus-one caps are the preferred first actual-path acceptance cases. These are independent hand counts, not new runtime results. + +Superseding parent decision: both split methods rerun the existing complete validator after stale-current validation, enforce the correct no-op/changed branch, and refuse changed carried input before O allocation. Earlier proposal preferences for relying only on construction proof or callback delivery are not the implementation selection. Await parent claim/contract publication before source edits. diff --git a/harness/analysis-mixed-emission-design-20260925/NEXT-PRODUCER-CRITIQUE.md b/harness/analysis-mixed-emission-design-20260925/NEXT-PRODUCER-CRITIQUE.md new file mode 100644 index 00000000..abe684b1 --- /dev/null +++ b/harness/analysis-mixed-emission-design-20260925/NEXT-PRODUCER-CRITIQUE.md @@ -0,0 +1,60 @@ +# Independent critique of the next producer proposal + +Read-only review against source `fa7a980418ef99d772e67c66c28dfa7b5181f875`, `NEXT-PRODUCER-HOOKS.md`, the accepted carried-phase contract, the rename-plan successor requirements, Project resource contract and execution domain audit. No source changes, Cargo, network or runtime execution. + +## Recommendation + +Do not make the proposed complete U/M/T producer ledger plus selected parser scratch the next single implementation milestone. The inventory is useful and identifies real timing gaps, but spans the shared scanner, ordinary parser/projection, typed cursor/declarations/expressions, traversal and resolver handoff. Calling that a small set of hooks understates its policy, lifetime and regression surface. + +The smallest useful next rename implementation is **private emission and authenticated output ownership**, stopping before accepted-output authority. Reserve exact output ownership while the old plan is live, emit from original coordinates, independently authenticate actual bytes against the authorized edit stream, and establish a constructor-controlled output owner with exact policy and a documented old-owner drop. Preserve an explicit unchanged outcome for no-op intent. This is necessary work under every plausible future parser accounting decision; it makes actual rename progress without silently settling that decision. + +Do not expose installation/save/history adoption or label this result accepted edited source. The proposed milestone should be explicitly approved as a private prerequisite, just as carried-phase accounting was. A complete accepted-output API still requires actual-byte parse/resolve/admission, strict whole-Project equality with reserved E scratch and a resolved temporal accounting contract. + +## What the contracts actually require + +Three categories should remain separate: + +1. **Necessary new output ownership.** The rename-plan successor explicitly names F output-map entries, complete strings, canonical copied keys and entry; output overlaps the old B during emission. Old syntax/index/plan allocations must actually drop before their reservations disappear, or both owners remain charged. These requirements are unambiguous and cannot be deferred behind a generic parser exclusion. +2. **Existing retained inventory.** Project §3 requires before-allocation aggregate checks for in-scope representations. Existing retained syntax aliases, field keys, AST text and projected copies are counted today, but some only at resolution. The domain audit and carried contract explicitly preserve this timing limitation, without waiving the stronger obligation for final execution. Producer hooks for those exact existing families are a defensible way to close it; a proven conservative reservation is another. The successful phase prerequisite does not itself discharge it. +3. **Explicit temporaries versus inferred universal scratch.** New emitter containers and already named accounting/equality work must be admitted. The original successor does not separately list every inherited traversal/path/scanner/duplicate-set allocation. Project §3's broad wording and parser proposal's temporary wording deserve an explicit finite scope interpretation; they do not by themselves establish that all of those structures already belong to one exact slot inventory. Automatically adding all of them is a metric expansion, not merely repairing the timing of existing T. Automatically exempting all of them would also be unjustified. + +Accordingly, broad parser scratch expansion is **not established as a prerequisite to implementing private emission**, and should not be imposed implicitly. It may become necessary for the final selected guarantee if parent explicitly includes those structures. That decision should identify each family and its logical unit, rather than using either “everything is already bounded” or “all parser allocation must now be metered.” Allocator capacity, model/serde-conversion heaps, raw validator Issues, BigInt limbs and native stack remain distinct exclusions/limits as documented. + +## Concrete bounded next implementation + +Use the existing opaque RenamePlan and its complete snapshot/edit validation. Add one private emission module beside planning; do not change shared parser producers in this milestone. + +1. Revalidate complete original bundle authority and all physical edits before copying. Preserve canonical paths/file set/entry and both occurrence families. No caller-supplied partial file list or replacement model authority. +2. Repeat the existing checked per-file sizing walk, without a lengths map. Compute exact O: F entries and `sum(final source lengths) + sum(copied canonical key byte lengths) + entry byte length`, zero lower occurrences. The existing resolved ledger already includes retained R+M plan maps: reserve O against that live total before constructing any output buffer/key/map. Reserving all O up front is simple and sufficient for a once-built complete bundle whose logical live output grows monotonically toward O. Do not debit old storage for shrinkage. +3. Construct each final file once by appending original untouched slices and borrowed desired aliases in increasing original coordinates. Copy untouched files too when delivering the specified complete owned output. Charge every implementation-added temporary if one is introduced; the preferred walk requires only iterator/scalar state and the currently growing output String. Container capacity is not a new logical-byte charge. +4. While the original owner still lives, use a separate streaming comparison of the **actual** output against the original-slice/replacement sequence. Check entry, exact canonical file set and every byte, including untouched files. No second output buffer, lexer or output hash as a substitute. This catches admissible-but-wrong non-name damage and creates useful authenticated topology evidence for any later envelope option. +5. Capture exact immutable policy and E scalar while the old witness still authorizes them. Consume/drop old syntax, indexes, warnings and plan maps at a named transition; retain only the complete output owner, desired borrow, policy and documented scalar metadata. Install O through the sole private output transition, never a general `new(bundle, arbitrary_slots)` constructor. The stable owner may later lend bytes through the carried guard. Do not return a self-borrowing bundle-plus-syntax pair. +6. For no-op intent, return the original owner in an explicit unchanged branch without constructing/copying a complete output. No history/install behavior follows from this private result. + +The output handle should be opaque and have no public extraction/adoption route. Test-only inspection can examine bytes and provenance. A later internal scoped verifier can borrow it and drop temporary syntax before returning owning acceptance. Failure drops partial output and consumes no install authority; the immutable original caller-owned source bytes and desired model remain unchanged. + +This is a proposal requiring a small successor contract, not authorization to add a production constructor under the already-finished phase-only contract. + +## Acceptance for that bounded milestone + +Reuse the independent complete rich OA/SA, swaps, scoped repeated/reordered call labels, cross-file Unicode/CRLF and ordinary-category maps. Author exact expected bytes from the original fixture and literal replacement coordinates, without invoking the emitter as its own oracle. Include untouched files and the previously agreed combined grow/shrink sizing case. Verify original bytes/intent unchanged and no-op preserves ownership. + +Hand-count emission cap/cap-minus-one at **old live B including R+M plus O**, ensuring the same profile first admits parsing, indexing and planning. Assert exact O after old ownership ends. Corrupt emitted touched/untouched bytes, a non-name byte, entry/key/file set and a replacement belonging to the wrong qualified target; streaming authentication must reject each. These are emitter/ownership tests, not substitute proof of actual output semantic admission. Parent should retain the later whole-Project equality requirement explicitly. + +## Quoted-text bound and future hooks + +The proposal's `decoded UTF-8 bytes D <= raw payload UTF-8 bytes P` is sound for valid JSON strings, including escapes and surrogate pairs. P must exclude exactly the two ASCII quotes. Reserving P before decode and releasing P-D is a straightforward conservative logical-text bound. It can reject escape-heavy source whose decoded retained text would fit; that rejection is a real selected policy change for the opt-in path, not exact minimal acceptance. An escaped/unescaped pair with identical decoded text should document this tradeoff. No second decoder or Unicode normalization claim is warranted. + +The scanner already decodes once for validation, before the grammar decoder. Therefore grammar-only hooks cannot claim preadmission of that earlier temporary. If it enters the selected domain, its reservation must occur inside the scanner after token extent discovery and before validation, then release before the later grammar decode. A post-next callback is too late. Adding this hook changes a shared API/error-transport seam; it should have independent invalid-string and legacy precedence controls, not be hidden in a retained-text patch. + +If parent instead chooses a parser producer milestone next, narrow it first to **retained syntax text T and its once-only transfer**. List every counted String producer and temporary UUID/declaration-key disposition, keep U/M phase checks explicitly unchanged, and do not claim complete before-growth execution acceptance. The most consequential tests are copied ordinary reference/key overlap, escaped strings at P versus D caps, temporary UUID release and no double charge in resolver. This is still a meaningful prerequisite, but should not silently grow into traversal and all scratch in the same change. + +## Compatibility, maintainability and stack + +The accepted carried route is opt-in even when C=0; legacy constructors must keep their original admission and diagnostic timing. Any new hooks need an explicit mode boundary, not numeric-zero detection. Avoid cloning Budget, resetting live Usage or rebuilding policy from defaults. Resolver reconciliation must verify/transfer admitted retained counts, not reserve them twice. + +A ledger moving from parser to resolver introduces subtleties: U is monotone grammar work, value-end/token-end metadata contains deliberate conservative retention, and scratch is genuinely released. Treating all three as interchangeable live Vec sizes risks weaker caps or double counting. Keep a short authoritative ownership table and tests for those distinct rules; a universal allocation abstraction is unnecessary. + +The recent stack regression demonstrates that even an inline policy field can enlarge many ancestor frames. It does not prove producer hooks are unsafe, nor that more hooks solve stack usage. Keep policy in existing Budget/syntax, retain the slim owner guard, use scoped borrowed accounting state, and avoid large inline callback/guard enums on recursive paths. Preserve the nine existing stack cases and the earlier abort evidence. Neither emission nor producer accounting earns a universal stack guarantee. + +Unresolved before accepted execution: the exact selected pre-growth retained/scratch domain; actual-output parser/resolver temporal coverage; E-reserved strict whole-intent equality; owning accepted-result construction and install/history lifecycle. The bounded emission step deliberately leaves these visible rather than letting a larger parser refactor substitute for actual rename output. diff --git a/harness/analysis-mixed-emission-design-20260925/NEXT-PRODUCER-HOOKS.md b/harness/analysis-mixed-emission-design-20260925/NEXT-PRODUCER-HOOKS.md new file mode 100644 index 00000000..1b4828a5 --- /dev/null +++ b/harness/analysis-mixed-emission-design-20260925/NEXT-PRODUCER-HOOKS.md @@ -0,0 +1,70 @@ +# Next parser producer hooks — bounded proposal + +2026-09-25. **Proposal, not accepted requirements or implemented behavior.** Read-only source pin `fa7a980418ef99d772e67c66c28dfa7b5181f875`. Paths below start at `crates/oa-core/src/`. No source edits, builds, runtime execution or Gitea activity. Scope follows the accepted carried-phase contract and domain audit: logical auxiliary entries/occurrences/text, not universal heap/RSS/stack accounting. + +## Recommendation + +The already-counted retained syntax representations can be admitted before their growth with a small set of named producer hooks and one carried ledger. Preserve current work Usage separately, transfer existing reservations rather than count them again at resolve, and use exact lengths for copies. Quoted-string payload length is a safe conservative reservation before the existing decoder, with exact release afterward. + +This closes **named retained-representation timing gaps**, not every current parser allocation. Traversal, scanner-validation Strings and certain parser scratch tables require explicit scope decisions below. The accepted broad before-growth obligation cannot be called completely discharged while those questions are unresolved. No universal allocator interception, second numeric parse, new decoder, or large whole-source multiplier is necessary. + +## Existing domain and minimum producer inventory + +C is the surviving owned-bundle baseline; U remains cumulative grammar-entry work; M is existing logical metadata. New live text reservations T and named scratch W must participate in aggregate checks with C. Do not globally reduce Limits to account for C: source-byte/token/reference/Exact work is a separate domain. + +| Counted representation | Current producer / minimum pre-growth hook | Transfer, release and reconciliation | +|---|---|---| +| Syntax entry slots U: project/declaration/include/description, field/value/list items, typed declaration/parameter/expression events | Existing `Parser::syntax_entry` (`language.rs:1168`), `State::entry` (`mixed.rs:98`), typed `Budget::entry` calls in declarations/syntax. Preserve exact charge events; additionally check carried live entries plus prospective U/M/named scratch before the event permits its representation to grow. | Do not increment U twice or reinterpret every Vec element as another grammar event. U's established monotone/conservative syntax baseline transfers once into resolver storage. | +| Borrowed file records and per-file bookkeeping | `mixed.rs:296–306` initial file count; parsed-file insertion and final collection `:384–410`. Existing F is reserved before traversal. | Preserve F once; moving the map's file values into canonical Vec does not create another retained F. If implementation temporarily retains two actual coordination containers, name/reserve that overlap rather than hiding it. | +| Ordinary value-end entries | `Parser::value`, `language.rs:1298–1307`, immediately before `value_ends.insert`. | Check one new logical entry before insertion (existing-key replacement must not invent a new slot). Existing final metadata may conservatively retain dropped value-end subtotal; do not both carry that subtotal and re-add table length. | +| Lexical-owner entries | `language.rs:1627–1630`, before push. | Transfer scalar count to M once. Keep correspondence with declaration occurrences, not unique IDs. | +| Token-end metadata (one token-count component) | `edit_positions_from_tokens`, `language.rs:1022`, before collecting `ends` of length significant-token count; currently `mixed.rs:339–355` admits this component at the phase boundary. | Preserve the current **single** tokens.len() metadata component. It is conservative after projection scratch drops. Do not add another tokens.len() merely to label retained Token storage; see distinction below. | +| Ordinary EditDeclaration/header/field/value/list slots | Existing `ordinary_position_slots` (`mixed.rs:415+`) and `edit_positions_from_tokens` (`language.rs:1014–1140`). Reserve the existing exact/conservative projected slot total before construction; keep the current seven-per-declaration/header envelope and field/value table semantics unless independently revised. | Positions transfer to retained syntax. Header role keys are borrowed static strings (`language.rs:1114–1127`), **not copied text**. Reserve counting-walk work before its own push, or derive the projection count incrementally from successful grammar events without another traversal. | +| Typed retained names, references, display names/descriptions/symbols | `Cursor::name` exact borrowed spelling before `to_owned` (`cursor.rs:67–79`); `Cursor::quoted` before existing decode (`:81–95`). | Name/reference strings move into typed declarations once. Reserve decoded-string upper bound, reconcile to exact length, and transfer surviving text. UUID spelling temporaries consumed by identity parsing must release when dropped; they do not appear in the retained syntax text oracle. | +| Ordinary retained aliases/names/includes/descriptions/reference/word strings | `Parser::identifier/quoted` (`language.rs:1194–1210`), Ast word copy (`:1370+`), project/declaration/include consumers. | Exact spelling length before copy; quoted rule below. Distinguish retained use from consumed UUID spelling and temporary declaration-kind word. The latter must not become retained T accidentally. On an aborted parse the entire phase unwinds; no reusable ledger/partial result escapes. | +| Ordinary field keys and their distinct copies | `Parser::field` (`language.rs:1250+`); declaration fields/key-span map insertion (`:1515–1529`); nested `fields` (`:1230–1246`). | Initial key ownership transfers to one retained map; reserve additional key.len() before each clone. Top-level declaration field_spans owns a second key. Nested-field `key.clone()` coexists with the original local key until scope end: either account/release that temporary or move the key after a duplicate check, preserving error order. | +| Projected copied field keys/reference spellings | `Positions::fields/value`, `language.rs:1047–1090`. | Reserve exact existing key/name lengths before cloning, aggregate with still-live AST copies. They remain in T; Atom projection does **not** copy decoded literal text. Existing T45 ordinary-description test is a useful discriminating oracle. | +| Resolve syntax inventory / counting work | `mixed/project/accounting.rs:233–352`. | Consume parser's checked U/M/T reservations once; don't seed C then add already-carried syntax totals again. Keep the existing bounded walk for independent reconciliation/LowerStorage calculation if useful, but change its text action from re-reservation to verification. Any walk work still reserves/release entries before growth. Require equality with the established final inventory (including conservative metadata), not silent subtraction after over-allocation. | + +### Tokens versus metadata + +`lex::tokens` charges source bytes once and every **raw** scanner token, including trivia, before scanner advance (`lex.rs:19–45`). It stores only significant tokens (`:47–52`). Current mixed module documentation says tokens themselves use the token bound, while its one tokens.len() entry component represents token-end metadata. That is the declared existing metric even though both token Vec and projection ends can coexist physically. + +Do not silently add retained significant tokens as a second auxiliary entry family, nor claim the old single metadata component separately counts both physical Vecs. The minimal proposal preserves raw-token prechecks and token-bound-only retained storage, and reserves the one established token-end component before its producer. Any change to count significant-token Vec entries separately is an explicit metric revision with new oracles, not a timing repair. + +## Quoted text: raw payload bound is valid + +For a syntactically valid JSON string token, let P be the **UTF-8 byte length between its surrounding ASCII quotes** (checked token.len()-2), and D the decoded Rust String.len(). Then **D <= P**: + +- Unescaped valid UTF-8 scalar text copies its bytes unchanged. +- Each simple escape (`\"`, `\\`, `\/`, `\b`, `\f`, `\n`, `\r`, `\t`) occupies two ASCII source bytes and decodes to one byte. +- One nonsurrogate `\uXXXX` occupies six ASCII bytes and decodes to at most three UTF-8 bytes. +- A valid surrogate pair occupies twelve ASCII bytes and decodes to one scalar of four UTF-8 bytes. +- Empty payload has P=D=0. JSON decoding does not Unicode-normalize text or expand entities; invalid escapes/unpaired surrogates are rejected, not larger valid strings. + +Therefore reserve P owned-text bytes against the live ledger **before the existing `serde_json::from_str` call**, then on success assert/check D<=P and release P-D. Move the D reservation with the returned String, or release D when a temporary UUID spelling is consumed and dropped. Use byte lengths, not character counts. Overflow/subtraction failures refuse safely; do not use saturating arithmetic to accept malformed bounds. The envelope may transiently reject an escaped string whose final decoded bytes would fit; that is an explicit conservative pre-growth tradeoff, not a claim of exact minimal admission. No second decoder, escaped-length pass or Exact conversion is required. + +This bound covers logical String bytes only. It does not count String capacity, serde scratch allocation or allocator overhead. A decode failure may have transient partial internal allocation bounded by the raw string extent, but that is not an exact internal-heap claim; the attempt fails and returns no witness. + +### Important earlier existing decode + +`raw_tokens::Scanner::next` scans a quoted token and already calls `serde_json::from_str::` to validate it (`raw_tokens.rs:107–130`, allocation at126). The grammar later decodes it again. Merely hooking Cursor::quoted/Parser::quoted therefore cannot claim **every** decoded String was pre-reserved: scanner validation has already created and dropped one. + +Label that validation-only decoded String as a distinct parser temporary; it is not retained T. It is not automatically covered by the accepted ordinary model/serde-conversion exclusion merely because serde_json produces it. Those exclusions concern model conversion/raw validation domains, not a generic exemption for every parser String. If parent requires this named parser temporary within the before-growth envelope, add an optional private reservation/release hook **inside the scanner immediately before its existing validation call**, after its span is known. Reserve P, drop validation value, release P; later grammar decode receives its separate nonoverlapping reservation. A callback after `scanner.next()` is too late. Do not add a third validation/decoding pass. Public scanning behavior and raw-token work semantics must remain unchanged. + +## Named scratch and traversal decisions — not an invitation to all-heap accounting + +The current inventory does not separately Ledger-meter every structure below. They cannot be declared fully covered merely because another bound exists. Choose their status explicitly before final verification acceptance: + +- Projection counting Vec (`mixed.rs:418+`) and delimiter stack (`language.rs:1023–1030`): they are narrow borrowed work frontiers. A reserve-before-push/release-after-pop hook fits the existing auxiliary-work model, but adds explicitly named scratch peaks; it must not double count token-end metadata. Prefer borrowing/taking existing parsed value ends where possible without a second parse, only if semantic coverage remains identical. +- Typed duplicate-name sets clone parameter/override names (`declarations.rs:293,610`). These are temporary syntax coordination copies absent from retained T. Reserve their live map entries and copied name bytes before insertion, release when the set drops, or use borrowed names where lifetimes permit. Do not count them as new retained names. Whether these already-existing temporary sets join this patch's declared domain needs parent approval. +- Ordinary open-declaration stack and local String clones: distinguish the existing grammar U reservation from additional simultaneously live coordination. Avoid adding hidden copied strings; any newly introduced work structures require explicit reservations. +- Shared traversal owns folded-path table, visited/active/children/stack and normalized/copied path/location strings (`source_project/traversal.rs:9–16,59–87,128,144–180`). Current file/include/depth prechecks bound workload, not every overlapping map slot/owned path byte. If these are included now, use a localized optional private accounting hook at actual insert/push/copy/drop points, retaining public behavior through a no-op path. `path.to_lowercase()` can expand UTF-8 spelling; original path.len() is **not** automatically its bound. Prefer a checked chars-based length sum followed by the existing conversion, or avoid its owned copy with a proved equivalent borrowed comparator; no general huge multiplier. Do not assume all visitation structures share one F slot. + +Raw model construction/serde conversion and architectural validation heaps retain the accepted exclusions; retained mapped warnings are already charged separately. BigInt limbs remain under numeric policy, not this text ledger. Native stack size and allocator/RSS are outside the logical metric. None of these exclusions permits newly owned output strings/map/path keys or actual retained syntax copies to disappear. + +## Bounded next decision + +Yes: pre-growth admission for the **named existing retained auxiliary inventory** is achievable with producer hooks, decoded-text reconciliation and once-only transfer. No: that statement alone does not prove every declared parser-temporary obligation has been met. Parent must pin the scratch/scanner/traversal rows it intends to include or leave explicitly unresolved; doing so is a finite representation-table decision, not universal allocation metering. + +Recommended next design is a producer ledger for U/M/T with per-copy hooks and raw-payload quoted reservations, plus explicitly selected borrowed-work scratch. Keep policy exact and legacy behavior isolated. Prove constructor/ownership transfer and independent cap oracles before integrating actual output verification. Stop here: no requirement is amended, no stronger guarantee accepted, and no code is proposed as already tested.