Select retained entry producer timing contract and independent critique

2026-09-24 19:15:30 -07:00
parent d99e1a369d
commit 7a02333c4a
3 changed files with 215 additions and 0 deletions
@@ -0,0 +1,102 @@
 # Private retained-entry producer prerequisite
 Baseline `97a8703adc75467a7f8e75f5c6cb00ca542b707c`. This selected contract
 strengthens the timing of the existing logical entry metric. It does not grant
 accepted-output, public source8 or save/history authority. Source/evidence remain
 local until the Gitea storage publication blocker is resolved.
 ## Metric and authority
 Add one explicit fixture route enabling both text and entry producers, using the
 existing opaque CarriedInput and exact inherited policy. Preserve legacy,
 carried-phase and text-only routes and their diagnostic timing. C comes from the
 actual carried guard, including an explicitly selected zero guard.
 U remains exactly Budget.usage.entries, the existing cumulative syntax events.
 M remains F supplied file records + one significant-token-end subtotal + unique
 ordinary value-end keys + ordinary lexical-owner occurrences + projected slots.
 Projected slots remain seven per ordinary declaration (including its synthetic
 body and conservative header envelope), one per field, one per actual Ast value.
 Node headers also retain the seven envelope. Dropped token/value-end structures
 keep their conservative subtotal through syntax return. Usage never includes C,
 M or quota consumption, and significant tokens do not gain a second M charge.
 Before selected growth check checked `C.entries + U + M + delta` against the
 exact entry ceiling. A failed check changes no U/M total. Use only small scalar
 optional Budget state; do not duplicate policy or create another owning wrapper.
 Preserve current byte/guard checks, then atomically validate/install C and M=F
 before Traversal construction. F is transferred, not charged again, when file
 records move between containers.
 ## Producer and reconciliation rules
 - Budget::entry preserves its existing syntax-work check first; in the new mode
  check the prospective aggregate before incrementing U or returning to callers.
  Existing typed/ordinary event sites remain unchanged.
 - Encapsulate value-end key-presence check, reservation and insertion in one State
  helper. Charge only a new key. Encapsulate lexical-owner reservation and push
  likewise; charge occurrences, not distinct UUIDs. Hide unchecked mutable
  metadata access from the ordinary parser.
 - Count pending projection with checked scalar grammar hooks: seven per ordinary
  declaration, one per successfully parsed field and one per successful value.
  Nested values/fields naturally contribute. Project/include/description do not
  call these hooks; duplicate fields fail the entire parse. The synthetic body
  is already in seven, and adds no value event.
 - At projection boundary atomically reserve token count plus pending projection
  into M before ends/header/field/value construction. The stronger route must
  not call the old unmetered borrowed-work Vec counting pass. Other routes keep
  that existing path.
 - A scalar prepaid quota is consumed before projector growth: seven before a
  declaration/header envelope, one before each field, one before each actual
  value (before recursive children). It neither recharges nor refunds M. Require
  zero after projection. Under/over quota is an internal invariant failure,
  never a successful witness with corrected/subtracted counts.
 - After each file compare reserved map/owner counts with actual lengths and
  M delta with checked token/owner/map/projected inventory. At syntax completion
  compare total M with the independently accumulated metadata field. Keep the
  existing once-only C then U+M resolver ledger transfer; text reconciliation
  remains unchanged. Never seed a combined total and add syntax a second time.
 ## Rejection behavior
 New aggregate ceiling refusal uses `Source8 lowering entry limit exceeded.`
 Overflow uses `Source8 producer entry size overflow.` Both latch in the new
 entry mode, leave totals unchanged and propagate immediately. Legacy phase
 checks keep their existing nonlatching semantics. U-only failure retains its
 existing precedence and message. Invariant mismatches return an explicit error;
 they cannot silently repair a count or produce an accepted result.
 Attribute U to its existing event span; owner to the original declaration start;
 value-end to the completed Ast's start/end; file and complete projection
 reservation to Span::default in the current file. Grammar counter overflow uses
 its corresponding declaration/field/value span. Ordinary diagnostic transport
 continues to use the existing one-byte start representation. Tests distinguish
 these actual seam diagnostics from later phase or resolver refusal.
 ## Acceptance and exclusions
 Real E: C1/U1/M11 gives syntax13/12; full resolver17/16 is a separate boundary.
 Real three-file chain: C3/U3/M19 gives syntax25/24, separate resolver29/28.
 Use sufficiently large unchanged text policies (at least E146 and chain358),
 so earlier text limits cannot preempt the entry assertion. First ordinary owner
 refuses at prefix cap4; first description value-end refuses at prefix cap7.
 Neither next integer is claimed to admit the complete document. Review the
 reserve-plus-mutate helpers and test unchanged collections on refusal directly.
 Independently author a nested record/array/reference/historical UUID fixture and
 node/element envelopes, with literal counts and spans fixed before runtime.
 Cover repeated map keys, quota underflow/leftover, overflow and latched refusal
 as labeled internal controls. Verify actual-owner and explicit C0 routes,
 whole OA/SA and multi-file Project/Usage/numeric parity against text-only, source
 conservation, once-only transfer and unchanged old caps/errors. Run new focused
 cases, full existing disposition, strict core Clippy and format comparison;
 inspect changed Budget/State/projector prologues and all old small-stack controls.
 Do not enlarge stacks, loosen tests or add a heap wrapper to hide regressions.
 This retained metric does not bound allocator capacity/node overhead, file-table
 physical transfer overlap, significant-token storage beyond its existing token
 bound, traversal/open-declaration/delimiter/Pratt work stacks, duplicate-name
 scratch, numeric/decoder internals, model/validator heap or native stack. Those
 named scratch domains need a later explicit finite decision. Actual emitted-byte
 verification, justified whole-Project equality work and accepted bundle ownership
 also remain open. No claim of before-every-allocation or all-heap accounting.
@@ -0,0 +1,54 @@
 # Independent critique of retained entry producer prerequisite
 Reviewed ENTRY-PROPOSAL.md against local immutable HEAD `97a8703adc75467a7f8e75f5c6cb00ca542b707c`. No code, compiler, runtime, network or publication action. References below are repository-relative at that source. This is a design review, not implementation acceptance.
 ## Disposition
 The proposed scalar grammar count plus prepaid projection quota is a reasonable bounded next prerequisite. It preserves the established logical U/M domain without pretending to bound every allocation. I found no inherent accounting contradiction or need for a parser rewrite. Before implementation, settle the new rejection contract and explicitly distinguish prepaid quota consumption from live accounting. Broader output verification and owner transition remain open.
 ## Required precision before implementation
 1. Specify the new aggregate error message, attribution and poison behavior. Existing `Budget::entry` first checks `active`, checks the syntax-only bound, latches a failure, then increments U (`budget.rs:170–206`). Existing metadata phase checks are nonlatching (`mixed.rs:1–17`, `budget.rs:238`). The new mode must run the existing syntax-work check first, then checked aggregate arithmetic, and commit U only after both succeed. Define whether aggregate failure latches, and require immediate propagation consistently; do not inadvertently change legacy error precedence. An overflow must not mutate U/M or masquerade as a limit comparison.
 2. Initialize and validate `C.entries + F` before installing the entry state or constructing Traversal. C comes from the actual guard, including an explicitly present zero guard. The existing source also checks source bytes before the file reservation (`mixed.rs:300–329`): retain that ordering unless the new route's contract intentionally says otherwise. Policy caps in entry tests need generous text/byte/token limits so this earlier validation cannot preempt the claimed entry seam.
 3. Define projection quota as consumption of an already reserved logical subtotal, not another live charge. At the boundary reserve `tokens.len + pending_projection` atomically once; every producer consumes quota before its representation growth, and completion requires zero. Consumption must never reduce Budget M or charge M again. Seven per declaration covers the synthetic body and conservative header envelope, even though node headers have only four actual roles (`language.rs:1121–1168`). No release of M follows temporary ends/value-map destruction: the established metric conservatively retains those subtotals through the witness (`mixed.rs:42–47`, `368–429`).
 4. Make the attribution of new owner/value-map/pending overflow checks explicit. The owner seam occurs after consuming the opening brace (`language.rs:1693–1702`), so a generic current-token error would point into the body, not the declaration start. Pass the selected declaration span deliberately. The value-end seam has the completed Ast offset and end (`1357–1375`); choose and test that source span, including the ordinary one-byte diagnostic transport. The proposal currently promises the original declaration span but does not fully define all new diagnostics.
 ## Coverage and lifecycle review
 U is cumulative syntax work, not a count of simultaneously live allocated objects. Keep exactly the old calls: ordinary project/declaration/include/description/field/value events; typed declaration/field/type/dimension/override and expression/operator/call-argument events. Existing `syntax.rs:60–63` excludes grouping from a new entry, and `169–176` charges binary operators before their deferred construction. The new check belongs in the common Budget seam, not additional Vec pushes. Existing Pratt/open-declaration work stacks stay outside this selected metric.
 Successful grammar-derived projection counts match the old walk's shape: seven per ordinary declaration, one per successful field and one per successful value. `language.rs:1294–1343` recursively routes nested object fields through field(), and `1357–1375` routes every child Ast through value(); `mixed.rs:440–465` counts precisely those same retained fields/values. Duplicate fields cause whole-parse failure after insertion (`language.rs:1307`, `1589`) and cannot create a successful replacement that needs a projection refund. Project description and includes bypass field/value and correctly do not contribute projected slots. Incrementing only after successful grammar completion avoids presenting a partial failed parse as a reconciled witness.
 The value-end helper should check key presence, reserve only a new key, then perform insertion in the same private operation. Real successful source generally supplies distinct Ast offsets; a repeated key is a useful labeled internal replacement control, not a claim about ordinary grammar provenance. The owner helper counts declaration occurrences, including repeated IDs, before the Vec push. Final comparisons against actual lengths are needed because merely carrying counters alongside unchecked public fields would not establish producer coverage.
 F is initialized once for every supplied file. The later BTreeMap-to-Vec transfer (`mixed.rs:410–436`) moves those same logical records. It is not another F, and this policy does not claim to bound transient physical container overlap. Similarly, the resolver reconstructs a distinct ledger from C and existing U+M; it must not seed the combined subtotal and add syntax again. Keep existing file/path/model validation and resolver equality checks, rather than inventing a second accepted witness authority.
 ## Finite oracle assessment
 The existing independently authored E and three-file fixtures support the proposed totals:
 | Fixture | C | U | M | Syntax total | Separate resolver peak |
 | --- | ---: | ---: | ---: | ---: | ---: |
 | E | 1 | 1 | 11 = F1 + token ends10 | 13 | 17 |
 | Three-file chain | 3 | 3 | 19 = F3 + token ends16 | 25 | 29 |
 Use syntax-only 13/12 and25/24 boundaries, then separate full-resolution17/16 and29/28 boundaries. The text-producer route must have a sufficient actual inherited byte cap; E146 remains the known complete text control. Scanner raw token work31 in the chain must not become M16. Exact chain refusal file/span requires the literal traversal fixture, not aggregate arithmetic alone.
 The owner cap4 control is sound as a prefix: C1 + F1 + project U1 + declaration U1 =4 immediately before the first owner M reservation. The first ordinary description-value cap7 is also a prefix: C1 + F1 + U4 + owner1 =7 before its value-end insertion. Neither cap5 nor cap8 proves a complete successful syntax; the following token/projection subtotal is larger. Timing must be shown by a finite observer or directly reviewed reserve-plus-mutate operation, with a roomy positive instance of the same real fixture. A final error alone cannot prove that no map/Vec insertion happened first.
 For the additional nested fixture, author the literal source and expected U, distinct value offsets, owner count, significant tokens and projection subtotal before writing test expectations. Include both a nested Ast Record and the synthetic declaration body so they cannot be conflated; retain node's seven-envelope distinction. Quota underflow/leftover and arithmetic overflow controls are internal fault injections, not complete-source acceptance evidence. Whole OA/SA/project and Usage parity at ample limits provides the positive semantic control; no generic output-derived counting oracle is needed.
 ## Simpler alternatives and boundaries
 Keeping the old borrowed-work Vec count and adding a live scratch allowance would touch fewer grammar sites, but introduces a new peak metric and policy, precisely the expansion this slice avoids. A recursion-based counting pass adds another depth-sensitive walk and is worse after the existing stack investigation. The proposed scalar counter is the simplest option that both eliminates that extra counting frontier on the new path and keeps U/M unchanged.
 Quota could technically be omitted after trusting the grammar count, but then a future projector branch could silently escape the prepaid inventory. A single narrow scalar consumption callback is worthwhile as an internal coverage invariant; do not turn it into a generic visitor or a second retained ledger. Ensure value quota is consumed before recursive children and header envelope before header collection; field quota can precede its recursive value construction. Text and quota callbacks may require a carefully scoped common borrower, not a second Budget copy.
 The existing delimiter Vec at `language.rs:1033–1040`, Pratt stacks, traversal structures and declaration open stack remain named excluded scratch. Significant tokens remain covered by their existing token bound; token-end M is reserved before ends construction, not falsely billed as two N-sized containers. Those exclusions are compatible with a retained-inventory prerequisite but forbid all-heap, physical peak or complete before-allocation claims. Measure actual Budget/State/projector frames and retain all existing small-stack controls without changes; a few scalar fields are not evidence of harmless stack cost.
 Recommended next step: implement only the explicit fixture route, common prospective U check, private metadata helpers and grammar counter/prepaid quota with these finite cases. Retain legacy, carried-phase and text-only behavior verbatim. This produces earlier enforcement of the existing entry metric, not accepted output or closure of broader scratch obligations.
 Parent's subsequent selected clarification resolves the diagnostic ambiguity if adopted explicitly: aggregate refusals reuse the lowering-entry-limit message at the actual producer span and latch only in the new mode; checked overflow has a distinct source-producer-entry-size-overflow error. Nested ordinary declarations do not undermine the successful-count argument: `language.rs:1540–1600,1693–1723` keeps declarations as separate flat occurrences with their own fields and seven-slot envelopes. The open stack supplies lexical ownership, not a second declaration or projected synthetic body.
@@ -0,0 +1,59 @@
 # Existing retained U/M producer timing — proposal
 Source `97a8703adc75467a7f8e75f5c6cb00ca542b707c`; 2026-09-25. Original read-only proposal, selected by the sibling entry-producer contract. Source references are repository-relative at the pinned commit; original local worktree links are transcribed as plain paths for relocation. No build or runtime result is inferred. This is a next prerequisite for the existing private output-verification path, not an accepted-output API or all-heap accounting.
 ## Selected-domain invariant
 U remains exactly Budget.usage.entries, the existing cumulative syntax-event charge; do not relabel every allocated object or Vec push as a new U event. M remains exactly the existing conservative subtotal: F complete-file slots + significant-token-end count + unique ordinary value_ends entries + lexical-owner entries + ordinary projection slots. Its dropped value-end/token-end portions remain represented through the syntax witness as before.
 C is the actual retained guard's immutable Slots, never Usage. In a new explicit entry-producing mode, every selected U/M growth must first check checked `C.entries + U + M + delta <= exact Limits.entries`. A refused reservation installs no new U/M total. Existing occurrence and numeric work semantics do not change; text-producing checks continue independently against the same policy.
 The final U/M inventory must equal the existing inventory for successful input. This strengthens timing, not the metric. Token storage still uses the token bound: M has ONE token-end component, not separate charges for significant tokens and ends. Capacity, container node overhead and native stack do not become entries.
 ## Compatibility and concrete owner
 Keep legacy, carried-phase and text-only routes unchanged, including their phase timing/diagnostics. Add a fixture-only route enabling text plus entry producers from the existing opaque CarriedInput; no arbitrary baseline/profile constructor and no numeric-zero bypass. Existing text-only E146/145 and phase-only tests retain their meanings.
 A small optional Budget entry state can hold only carried-entry baseline and current M; U is read from existing Usage, never duplicated. Derive C directly from the guard at initialization. Initialize M=F before Traversal::new using the existing file check. No full policy copy, owner enum, new heap container or String wrapper is needed. Measure generated frames after implementation rather than spending the repaired stack margin by assumption.
 Use Budget::entry as the common prospective-U seam: preserve the old syntax-work check/error behavior, then perform the new aggregate check before incrementing Usage or letting the caller create its associated representation. These checks must not reset work, reinterpret references as occurrences, or add U again at final handoff.
 A per-file scalar pending_projection count belongs in mixed State, where ordinary grammar callbacks already meet the shared Budget. It counts future projection slots without reserving them prematurely. Keeping this scalar distinct from live M avoids charging output that has not been produced yet.
 ## Concrete seams and ownership
 | Existing component | Producer seam | Required action / reconciliation |
 | --- | --- | --- |
 | U ordinary events | `crates/oa-core/src/language.rs`:1192 syntax_entry; field1321, value1357, project1513, include1613, description1637, declaration1657 | All route through State::entry/Budget::entry before their current grammar event. Preserve exact event counts and caller order. No charge for every open-stack push or every Rust enum. |
 | U typed events | `crates/oa-core/src/language/analysis_source/declarations.rs`:261,290,476,542,611; `crates/oa-core/src/language/analysis_source/syntax.rs`:61,169,242 | Existing pre-event calls cover declaration/field/type/dimension/override and Pratt expression/operator/argument events. Grouping mutates an existing expression and does not invent an entry. Deferred node construction uses its already reserved event. Keep the repaired iterative driver unchanged. |
 | F file records | `crates/oa-core/src/language/analysis_source/mixed.rs`:314–329 | Initial complete-file reservation stays before traversal. Map-to-canonical-Vec consumption transfers the same logical file records; do not add another F merely for the new container. |
 | Ordinary value-end map | language.rs:1373, after successful Ast construction but before map insertion | A State::remember_value_end helper checks whether the offset is new, reserves one M only for a new key, then inserts. Ast already has its U event. Replacement updates do not add a slot. Per-file reserved count must equal actual map.len before its conservative subtotal is retained/dropped. |
 | Lexical owners | language.rs:1700, before push and ordinary declaration insertion | A State::push_lexical_owner helper reserves one M before its current Vec push. Count occurrences, not unique IDs. Per-file delta must equal Vec.len. Typed roots/project header do not gain fabricated ordinary-owner entries. |
 | Token-end metadata | language.rs:1032 ends collection; mixed.rs:368+ preprojection boundary | Reserve exactly tokens.len once before ends creation, alongside projection below. No need to charge M during lex::tokens significant-token push: that producer belongs to the existing token-bound-only domain, and ends does not yet exist. |
 | Ordinary projected slots | language.rs:1042–1168 Positions and per-declaration construction | Pre-reserve pending_projection once before any projected declaration/header/field/value allocation. Preserve seven logical slots per ordinary declaration/body/header envelope, one per field key, one per Ast-projected value; keep the node-header envelope conservative rather than revising it. |
 Direct map/Vec growth should remain inside the same narrow State helper as its reserve; expose no independent unchecked mutable metadata operation for the new path. The shared ordinary Parser invokes these helpers only through its existing optional mixed State. Public grammar need not acquire a new state owner.
 ## Avoid the current counting-work Vec on the new path
 The current `crates/oa-core/src/language/analysis_source/mixed.rs`:440+ is a postparse saturating tally with an unmetered borrowed-work Vec. Calling it before a projection reservation does not establish before-growth accounting for that Vec. Do not silently charge its scratch under the retained projection subtotal or call saturation checked arithmetic.
 Recommended replacement ONLY for the new route: increment checked pending_projection by7 when an ordinary declaration is created, by1 for every successful ordinary field(), and by1 for every successful ordinary value(). Nested objects/arrays are visited naturally by those existing calls; each child value and nested key contributes exactly once. Project/include/description grammar does not call field/value in that path and must not enter this projection counter. Failed whole parses need no successful-witness reconciliation.
 At the projection boundary, reserve pending_projection + tokens.len into M once. Maintain a scalar projection quota consumed by the actual projected producer:7 per declaration,1 per field,1 per EditValue. Check quota subtraction before the corresponding production and require zero at completion. This compares the independent grammar-derived inventory with actual projection consumption without a second AST walk or an extra collection. Static header roles are included in the existing7 envelope; do not also subtract their individual count.
 Legacy/carried-phase/text-only retain ordinary_position_slots and their prior behavior. The stronger path's quota hook can be an additional narrow optional callback/scalar in the existing shared projector; do not add a generic visitor framework or clone ASTs. The source review must compare grammar counting order and projection consumption coverage, including synthetic body Record handling.
 The synthetic per-declaration body Record is already part of7 and is NOT a second value() contribution. Ast::Object below a field is an actual value and consumes1 plus its field entries; Ast::Array consumes1 plus items. Atom/Reference/HistoricalUuid each consume1. This distinction needs explicit tests.
 ## Transfer and final checks
 After each file, compare its reserved value-end/owner counts with actual lengths and its consumed projection quota with zero. Compute the expected M increment from those lengths, tokens.len and the reconciled projection count; compare against the producer M delta instead of adding it again. Initial F is checked separately once.
 At final syntax return assert Budget's M equals ParsedMixedSyntax.metadata and the file inventory's checked total. Keep Usage.entries as the original U. The existing resolver still seeds C once and reserves U+M once in its distinct ledger; this is transfer/reconstruction, not an additional simultaneously retained allocation. Do not seed C+U+M and then call the unchanged additive syntax path again. Text T transfer/reconciliation remains as implemented.
 No new occurrence storage is allocated in this slice; C.occurrences continues its inherited checks. If a future producer introduces an auxiliary container, it is not covered by these retained counters unless explicitly inventoried.
 ## Finite acceptance oracles and limits
 1. Empty E: C.entries1, U1, M11 (F1 + ten token ends), final syntax13; cap13 parses,12 refuses before token-end/projection creation, full resolver peak17 remains distinct. Real inherited text policy must be sufficiently large (producer E146) so text cannot preempt the entry control.
 2. Existing three-file chain: C.entries3, U3, M19 (F3 + sixteen token ends), final25; syntax25/24 and resolver29/28 remain separate cases. Verify actual file/checkpoint attribution and no double-add at transfer. Scanner raw-token work is31, not M's sixteen significant tokens.
 3. Minimal ordinary empty activity: before its first lexical-owner push C1+F1+U2=4; entry cap4 can distinguish the new owner reservation refusal. It is a negative timing control, NOT a claim cap5 admits the complete syntax. Use actual guard ownership, generous text/token limits and original declaration span.
 4. First ordinary `description ""` value: C1+F1+U4+one lexical owner=7 immediately before its new value-end entry; cap7 refuses before map insertion. Again cap8 only fits that prefix, not the later complete metadata. A bounded test observer or separately reviewed reserve-plus-insert helper must establish ordering; a final retained count alone cannot.
 5. Independently hand-count flat fields, nested record/array/reference/HistoricalUuid and node versus element headers: exact grammar projection counter equals old retained inventory, and actual projection quota ends at0. Invalid under/over quota and repeated map-key controls are labeled internal fault injection, not real source provenance. No AST cloning or output-derived expected model.
 6. Whole real OA/SA/multi-file outcomes and Usage/numeric parity against text-only at ample limits; explicit C0 new mode; old route cap/error behavior unchanged. Work Usage counts must not absorb C, M or projected quota events.
 7. Preserve all old 128 KiB/deep/wide controls and measure changed Budget/State/projector frames. No stack/test-size relaxation or RUST_MIN_STACK. Repeated failure must not leak a partial witness or new source authority.
 ## Finite domain decision still required
 The projector also uses a delimiter Vec at language.rs:1033–1040; traversal and ordinary open-declaration coordination likewise remain existing scratch, not retained M. The proposed contract can honestly cover U/M while leaving those named scratch producers open. It cannot claim the single N token-end reservation accounts two simultaneous N-sized containers. If the parent requires these scratch entries now, choose an explicit additional live W, reserve before each push/release on pop, and derive new peak oracles; that is a separate finite metric addition.
 Comparison: keep the old count walk and add bounded W for its work frontier (smaller initial grammar edits, but expands scratch policy and peak caps), versus grammar scalar + producer quota (more named seams, no new frontier/domain). Recommend the latter for this narrowly retained-inventory prerequisite. Broader scratch, final actual-output equality and accepted-owner transition remain out of scope. No additional task is proposed or started here.