ARCADIA-Studio/harness/analysis-mixed-emission-design-20260925/NEXT-PRODUCER-HOOKS.md

Next parser producer hooks — bounded proposal

2026-09-25. Proposal, not accepted requirements or implemented behavior. Read-only source pin fa7a980418ef99d772e67c66c28dfa7b5181f875. Paths below start at crates/oa-core/src/. No source edits, builds, runtime execution or Gitea activity. Scope follows the accepted carried-phase contract and domain audit: logical auxiliary entries/occurrences/text, not universal heap/RSS/stack accounting.

Recommendation

The already-counted retained syntax representations can be admitted before their growth with a small set of named producer hooks and one carried ledger. Preserve current work Usage separately, transfer existing reservations rather than count them again at resolve, and use exact lengths for copies. Quoted-string payload length is a safe conservative reservation before the existing decoder, with exact release afterward.

This closes named retained-representation timing gaps, not every current parser allocation. Traversal, scanner-validation Strings and certain parser scratch tables require explicit scope decisions below. The accepted broad before-growth obligation cannot be called completely discharged while those questions are unresolved. No universal allocator interception, second numeric parse, new decoder, or large whole-source multiplier is necessary.

Existing domain and minimum producer inventory

C is the surviving owned-bundle baseline; U remains cumulative grammar-entry work; M is existing logical metadata. New live text reservations T and named scratch W must participate in aggregate checks with C. Do not globally reduce Limits to account for C: source-byte/token/reference/Exact work is a separate domain.

Counted representation	Current producer / minimum pre-growth hook	Transfer, release and reconciliation
Syntax entry slots U: project/declaration/include/description, field/value/list items, typed declaration/parameter/expression events	Existing Parser::syntax_entry (language.rs:1168), State::entry (mixed.rs:98), typed Budget::entry calls in declarations/syntax. Preserve exact charge events; additionally check carried live entries plus prospective U/M/named scratch before the event permits its representation to grow.	Do not increment U twice or reinterpret every Vec element as another grammar event. U's established monotone/conservative syntax baseline transfers once into resolver storage.
Borrowed file records and per-file bookkeeping	mixed.rs:296–306 initial file count; parsed-file insertion and final collection :384–410. Existing F is reserved before traversal.	Preserve F once; moving the map's file values into canonical Vec does not create another retained F. If implementation temporarily retains two actual coordination containers, name/reserve that overlap rather than hiding it.
Ordinary value-end entries	Parser::value, language.rs:1298–1307, immediately before value_ends.insert.	Check one new logical entry before insertion (existing-key replacement must not invent a new slot). Existing final metadata may conservatively retain dropped value-end subtotal; do not both carry that subtotal and re-add table length.
Lexical-owner entries	language.rs:1627–1630, before push.	Transfer scalar count to M once. Keep correspondence with declaration occurrences, not unique IDs.
Token-end metadata (one token-count component)	edit_positions_from_tokens, language.rs:1022, before collecting ends of length significant-token count; currently mixed.rs:339–355 admits this component at the phase boundary.	Preserve the current single tokens.len() metadata component. It is conservative after projection scratch drops. Do not add another tokens.len() merely to label retained Token storage; see distinction below.
Ordinary EditDeclaration/header/field/value/list slots	Existing ordinary_position_slots (mixed.rs:415+) and edit_positions_from_tokens (language.rs:1014–1140). Reserve the existing exact/conservative projected slot total before construction; keep the current seven-per-declaration/header envelope and field/value table semantics unless independently revised.	Positions transfer to retained syntax. Header role keys are borrowed static strings (language.rs:1114–1127), not copied text. Reserve counting-walk work before its own push, or derive the projection count incrementally from successful grammar events without another traversal.
Typed retained names, references, display names/descriptions/symbols	Cursor::name exact borrowed spelling before to_owned (cursor.rs:67–79); Cursor::quoted before existing decode (:81–95).	Name/reference strings move into typed declarations once. Reserve decoded-string upper bound, reconcile to exact length, and transfer surviving text. UUID spelling temporaries consumed by identity parsing must release when dropped; they do not appear in the retained syntax text oracle.
Ordinary retained aliases/names/includes/descriptions/reference/word strings	Parser::identifier/quoted (language.rs:1194–1210), Ast word copy (:1370+), project/declaration/include consumers.	Exact spelling length before copy; quoted rule below. Distinguish retained use from consumed UUID spelling and temporary declaration-kind word. The latter must not become retained T accidentally. On an aborted parse the entire phase unwinds; no reusable ledger/partial result escapes.
Ordinary field keys and their distinct copies	Parser::field (language.rs:1250+); declaration fields/key-span map insertion (:1515–1529); nested fields (:1230–1246).	Initial key ownership transfers to one retained map; reserve additional key.len() before each clone. Top-level declaration field_spans owns a second key. Nested-field key.clone() coexists with the original local key until scope end: either account/release that temporary or move the key after a duplicate check, preserving error order.
Projected copied field keys/reference spellings	Positions::fields/value, language.rs:1047–1090.	Reserve exact existing key/name lengths before cloning, aggregate with still-live AST copies. They remain in T; Atom projection does not copy decoded literal text. Existing T45 ordinary-description test is a useful discriminating oracle.
Resolve syntax inventory / counting work	mixed/project/accounting.rs:233–352.	Consume parser's checked U/M/T reservations once; don't seed C then add already-carried syntax totals again. Keep the existing bounded walk for independent reconciliation/LowerStorage calculation if useful, but change its text action from re-reservation to verification. Any walk work still reserves/release entries before growth. Require equality with the established final inventory (including conservative metadata), not silent subtraction after over-allocation.

Tokens versus metadata

lex::tokens charges source bytes once and every raw scanner token, including trivia, before scanner advance (lex.rs:19–45). It stores only significant tokens (:47–52). Current mixed module documentation says tokens themselves use the token bound, while its one tokens.len() entry component represents token-end metadata. That is the declared existing metric even though both token Vec and projection ends can coexist physically.

Do not silently add retained significant tokens as a second auxiliary entry family, nor claim the old single metadata component separately counts both physical Vecs. The minimal proposal preserves raw-token prechecks and token-bound-only retained storage, and reserves the one established token-end component before its producer. Any change to count significant-token Vec entries separately is an explicit metric revision with new oracles, not a timing repair.

Quoted text: raw payload bound is valid

For a syntactically valid JSON string token, let P be the UTF-8 byte length between its surrounding ASCII quotes (checked token.len()-2), and D the decoded Rust String.len(). Then D <= P:

• Unescaped valid UTF-8 scalar text copies its bytes unchanged.
• Each simple escape (\", \\, \/, \b, \f, \n, \r, \t) occupies two ASCII source bytes and decodes to one byte.
• One nonsurrogate \uXXXX occupies six ASCII bytes and decodes to at most three UTF-8 bytes.
• A valid surrogate pair occupies twelve ASCII bytes and decodes to one scalar of four UTF-8 bytes.
• Empty payload has P=D=0. JSON decoding does not Unicode-normalize text or expand entities; invalid escapes/unpaired surrogates are rejected, not larger valid strings.

Therefore reserve P owned-text bytes against the live ledger before the existing serde_json::from_str call, then on success assert/check D<=P and release P-D. Move the D reservation with the returned String, or release D when a temporary UUID spelling is consumed and dropped. Use byte lengths, not character counts. Overflow/subtraction failures refuse safely; do not use saturating arithmetic to accept malformed bounds. The envelope may transiently reject an escaped string whose final decoded bytes would fit; that is an explicit conservative pre-growth tradeoff, not a claim of exact minimal admission. No second decoder, escaped-length pass or Exact conversion is required.

This bound covers logical String bytes only. It does not count String capacity, serde scratch allocation or allocator overhead. A decode failure may have transient partial internal allocation bounded by the raw string extent, but that is not an exact internal-heap claim; the attempt fails and returns no witness.

Important earlier existing decode

raw_tokens::Scanner::next scans a quoted token and already calls serde_json::from_str::<String> to validate it (raw_tokens.rs:107–130, allocation at126). The grammar later decodes it again. Merely hooking Cursor::quoted/Parser::quoted therefore cannot claim every decoded String was pre-reserved: scanner validation has already created and dropped one.

Label that validation-only decoded String as a distinct parser temporary; it is not retained T. It is not automatically covered by the accepted ordinary model/serde-conversion exclusion merely because serde_json produces it. Those exclusions concern model conversion/raw validation domains, not a generic exemption for every parser String. If parent requires this named parser temporary within the before-growth envelope, add an optional private reservation/release hook inside the scanner immediately before its existing validation call, after its span is known. Reserve P, drop validation value, release P; later grammar decode receives its separate nonoverlapping reservation. A callback after scanner.next() is too late. Do not add a third validation/decoding pass. Public scanning behavior and raw-token work semantics must remain unchanged.

Named scratch and traversal decisions — not an invitation to all-heap accounting

The current inventory does not separately Ledger-meter every structure below. They cannot be declared fully covered merely because another bound exists. Choose their status explicitly before final verification acceptance:

• Projection counting Vec (mixed.rs:418+) and delimiter stack (language.rs:1023–1030): they are narrow borrowed work frontiers. A reserve-before-push/release-after-pop hook fits the existing auxiliary-work model, but adds explicitly named scratch peaks; it must not double count token-end metadata. Prefer borrowing/taking existing parsed value ends where possible without a second parse, only if semantic coverage remains identical.
• Typed duplicate-name sets clone parameter/override names (declarations.rs:293,610). These are temporary syntax coordination copies absent from retained T. Reserve their live map entries and copied name bytes before insertion, release when the set drops, or use borrowed names where lifetimes permit. Do not count them as new retained names. Whether these already-existing temporary sets join this patch's declared domain needs parent approval.
• Ordinary open-declaration stack and local String clones: distinguish the existing grammar U reservation from additional simultaneously live coordination. Avoid adding hidden copied strings; any newly introduced work structures require explicit reservations.
• Shared traversal owns folded-path table, visited/active/children/stack and normalized/copied path/location strings (source_project/traversal.rs:9–16,59–87,128,144–180). Current file/include/depth prechecks bound workload, not every overlapping map slot/owned path byte. If these are included now, use a localized optional private accounting hook at actual insert/push/copy/drop points, retaining public behavior through a no-op path. path.to_lowercase() can expand UTF-8 spelling; original path.len() is not automatically its bound. Prefer a checked chars-based length sum followed by the existing conversion, or avoid its owned copy with a proved equivalent borrowed comparator; no general huge multiplier. Do not assume all visitation structures share one F slot.

Raw model construction/serde conversion and architectural validation heaps retain the accepted exclusions; retained mapped warnings are already charged separately. BigInt limbs remain under numeric policy, not this text ledger. Native stack size and allocator/RSS are outside the logical metric. None of these exclusions permits newly owned output strings/map/path keys or actual retained syntax copies to disappear.

Bounded next decision

Yes: pre-growth admission for the named existing retained auxiliary inventory is achievable with producer hooks, decoded-text reconciliation and once-only transfer. No: that statement alone does not prove every declared parser-temporary obligation has been met. Parent must pin the scratch/scanner/traversal rows it intends to include or leave explicitly unresolved; doing so is a finite representation-table decision, not universal allocation metering.

Recommended next design is a producer ledger for U/M/T with per-copy hooks and raw-payload quoted reservations, plus explicitly selected borrowed-work scratch. Keep policy exact and legacy behavior isolated. Prove constructor/ownership transfer and independent cap oracles before integrating actual output verification. Stop here: no requirement is amended, no stronger guarantee accepted, and no code is proposed as already tested.