Carried storage domain audit

2026-09-24. Read-only inspection of private source 18073a6f2eabce83fc83997f094e331eec835a16 in .local/as3batch. This is a design audit, not an accepted contract or implemented carried parser. No source changes, Cargo, commits or Gitea actions. Parent owns scope and acceptance decisions. Code paths below are relative to crates/oa-core/src/.

Finding and recommended boundary

Carry the existing three-dimensional logical auxiliary metric, not a newly inferred whole-heap metric. Preserve (entry_slots, lower_occurrence_slots, owned_in_scope_text_bytes) and the exact inherited policy. A successor must not hide emitted output behind a fresh zero-baseline parser/resolver ledger. Conversely, neither user editing requirements nor accepted contracts require accounting allocator capacity, every serde allocation, BigInt limbs, compiler stack frames, RSS or raw validator heap.

The narrow next dependency can introduce a private, constructor-controlled surviving baseline plus aggregate checks at the current parser and resolver phase boundaries. It must say precisely whether it preserves the existing phase-check guarantee or additionally strengthens selected parser producers to pre-growth reservation. Existing source does not establish preallocation accounting for every parser transient or retained syntax String. Therefore “all parser allocations already counted; just seed resolve” and “must instrument all heap allocation before proceeding” are both unsupported conclusions.

The smallest honest proposal has two distinct decisions:

Adopt carried ownership/aggregate within the existing declared domain, preserving the existing metric and explicit exclusions. Keep all current syntax/work counters separate from the carried live auxiliary baseline.
If the execution contract requires aggregate refusal before each counted parser representation grows, add narrowly named hooks for those representations, or a proven conservative reservation before their production. This is an explicit temporal strengthening of current parser checks, not permission to broaden the domain to model/serde/validator heap. Do not claim it is already implemented.

Authority and the actual user need

The user-owned application goal requires graphical/text/API editing parity and reliable persistence; it does not specify a physical heap meter. Accepted original-span/mixed contracts derive resource, provenance and failure invariants supporting that need. Their precise logical metric is an accepted engineering policy, not a paraphrase to silently replace with a larger or smaller one.

Relevant accepted documents:

harness/analysis-mixed-parser-proposal-20260923.md, “Budget and parser invariants”: one complete-parse Budget; original bytes/raw tokens charged once; ordinary declaration/field/value/list storage bounded separately from final model records; aggregate metadata; no rescanning/reset. Its review §3 explicitly calls lowering/index functions noncharging phase checks and distinguishes syntax versus model counts.
harness/analysis-mixed-project-contract-20260923.md §3: checked aggregate live auxiliary triples, moves transfer reservations, copies coexist, borrowed text zero copied bytes, logical slots not allocator capacity or heap/RSS. Explicitly excludes ordinary serde/model allocation preadmission and architectural validator/raw Issue allocation. Retained mapped diagnostic data must become charged.
harness/analysis-mixed-index-contract-20260924.md, “Live storage and failure policy”: preserve previous domain; inventory the specific new maps/path/work structures; no raw model/validator/RSS claim.
harness/analysis-ordinary-index-contract-20260924.md, “Storage and depth”: retained increment 1+D+A, scratch reverse globals/typed physical keys/live work, no new frame interpretation or recharging parse/Exact work.
harness/analysis-mixed-rename-plan-contract-20260924.md, “Storage and combined sizing” and “Mandatory successor”: two plan maps R+M, E equality scratch, output F map entries and owned bytes/path keys/entry, exact inherited per-snapshot policy and explicit old-owner transition; surviving output baseline participates in verification. It does not repeal prior heap exclusions.

The last document deliberately requires an aggregate surviving-storage boundary. A fresh parser/resolver pass independently fitting the same numeric cap is insufficient when output or old auxiliary state remains live. It does not require cumulative input+output parse-work counters: exact policy is reused for each deliberate snapshot, while each snapshot reports its own work Usage.

Existing dimensions and scope

Quantity	Actual meaning	Evidence
`Budget::usage.bytes/tokens/entries/references`	Cumulative work/grammar counters for one snapshot; bytes once per scanned file, every raw token including discarded trivia, declared syntax entry events, typed/active ordinary references. Not live memory.	`analysis_source/budget.rs:6–14,27–35,84–125`; `lex.rs:19–45`; `mixed.rs:94–97`; `project.rs:403–434`.
Exact numeric work / deepest frame	Separate measured numeric operations and maximum grammar depth. Neither is auxiliary slots nor heap bytes.	`budget.rs:127–141`; `syntax.rs:46,150,223`; typed/ordinary existing grammar policies.
Auxiliary entries	Logical syntax metadata + syntax-entry baseline, catalogs/projections/locations, coordination, raw expression sites, charged maps/sets/vectors/path nodes and bounded work frontiers.	`mixed/project/accounting.rs:1–70,233–249`; `project.rs:437–543`; accepted contracts above.
Auxiliary occurrences	Lowered occurrence records; budget ceiling entries+references. Later index/ordinary/edit maps are entries, not another occurrence charge.	`budget.rs:160–166`; `accounting.rs:94–103`; `project.rs:528–543`; index contract formula.
Auxiliary text	Separately owned in-scope strings, paths, names and retained diagnostics; borrowed original source bytes not copied auxiliary text. Output becomes separately owned carried text for the next parse.	`accounting.rs:124–128,259–352`; `project.rs:593–627`; project contract §3; rename successor contract.
Output complete byte limit	Sum of all final file bytes, including untouched files, checked under inherited source-byte policy. This is not itself a live-storage reservation.	`ordinary_index/rename.rs:267–308`; rename contract.
Static/model admission	Separate caller record/graph/step limits and static work policy. Not relabeled auxiliary entries.	`project.rs:565–583`; `mixed.rs:228–234,98–118`; project contract §3/5.

Ledger::reserve computes checked live+delta and checks all three ceilings before installing the new totals; release uses checked subtraction (accounting.rs:6–70). It counts logical entries rather than Rust object sizes. Vec spare capacity, BTree node layout and inline enum byte size do not add logical slots. Copying an in-scope String adds its length, not capacity; an Arc share adds its enclosing map entry but not another shared Path node.

Explicit exclusions, not accounting defects

Complete model maps/Expr/value payload allocations and ordinary serde conversion/model construction are outside preallocation auxiliary accounting (accounting.rs:72–79; project contract §3). Final model limits still apply.
Architectural validation indexes and produced raw Vec<Issue>/owned issue strings are excluded until mapping/retention. project.rs:583–588 calls validation; thereafter reservations precede retained destination slot/path/text production. Never call raw validation preallocation bounded.
Exact numeric internals/BigInt limb allocation are controlled by their own numeric policy, not represented as auxiliary string bytes or slots. No accepted inventory assigns every arithmetic heap object a slot.
Allocator capacity/overhead/RSS and stack-frame bytes are not this metric. The observed environment-sensitive upstream small-stack parse limit must remain a separate empirical limitation; it is not repaired by a carried Ledger.
Transient failure diagnostics have the documented escape needed to report resource refusal (project.rs:160–177). They are not silently retained in a successful witness.

These exclusions do not excuse a new in-scope output buffer, map, copied path, planning list or retained diagnostic merely because some other allocation is excluded.

Parser implementation: what is checked and when

ParsedMixedSyntax retains Budget, exact ProjectLimits, bundle borrow, canonical files and scalar metadata (mixed.rs:39–58). parse_with_profile creates one Budget and rejects any widening against caller-derived maximum (:244–268). The current checks are:

Complete input file bytes checked through index_bytes; file count initializes metadata and is checked before Traversal::new (:269–284). Scanning later cumulatively charges bytes and raw tokens (lex.rs:19–45). The precheck is not another work charge.
Shared ordinary State::entry delegates grammar event charging (mixed.rs:94–97); typed declaration/expression parser uses the same Budget (declarations.rs:261,290,473,539,608; syntax.rs:46,150,223). Ordinary parser hooks charge its declared field/value/list/declaration events.
After each unit parse, metadata adds retained significant tokens, value-end map count, projected ordinary-position slots and lexical-owner count, then checks before materializing the positions (mixed.rs:319–340). ordinary_position_slots uses seven conservative declaration/header slots, field slots and walked value/container slots (:379–400). Dropped value-end entries can remain conservatively represented in metadata.
Only on resolving does accounting::syntax reserve metadata + usage.entries together, then walk retained structures to reserve their owned text and counting-work slots (accounting.rs:233–352). Resolver currently begins with Ledger::default() (project.rs:437–442). Catalog/lowering/index phases continue that ledger.

Consequences that must not be obscured:

Existing parse checks syntax-entry usage and metadata separately; the full sum is checked by resolution. Adding a baseline only to resolve does not make the earlier parser checks aggregate.
Much retained syntax text exists before the bounded accounting walk reserves it. Typed name .to_owned() and quoted decoding are at cursor.rs:67–95; ordinary AST fields/names and position key/reference clones are other producers. Calling the later accounting walk “before all syntax String allocation” would be false.
There are actual parser temporaries not separately enumerated by the retained scalar metadata: traversal folded-path/visited/active/visit structures (source_project/traversal.rs:9–16,59–87,97–180), ordinary_position_slots counting Vec (mixed.rs:382–396), projection ends Vec and delimiter stack (language.rs:1022–1030). They have surrounding file/token/grammar/depth controls, but that is not a proved independent live Ledger reservation for every container.
The position projection clones field keys and reference strings (language.rs:1048–1090). Those retained copies are included by the later syntax text walk; this distinguishes delayed accounting of in-scope retained data from unproven universal transient accounting.

The parser proposal's general “bound temporary containers before allocation” wording is stronger than a claim that every temporary is independently represented in the current metric. The accepted Project contract concretizes the retained baseline and specific work accounting without asserting full parser heap enumeration. Parent should record any temporal/domain clarification explicitly rather than retroactively labeling every unenumerated traversal allocation either fully covered or forbidden.

Smallest carried-baseline design proposal

Use a narrow private continuation carrying scalar Slots and exact Limits/ProjectLimits from a consumed owning transition. No caller-supplied reset, profile widening or arbitrary baseline fabrication. Existing ordinary entry points use zero carried slots and keep behavior. Do not add a generic allocator framework or alter model/public APIs.

For an execution successor, let O be the live output bundle's F entries plus UTF-8 strings, copied canonical key strings and entry path. Let B be old auxiliary storage while it remains live. Reserve output against B during emission. After the documented drop of old syntax/index/plan owner, carry O into verification; if old data is retained, carry B+O instead. Model/intent borrows retain their declared exclusion, not a second hidden auxiliary baseline.

For verification parser checks, maintain disjoint scalars:

C = surviving carried entries/text/occurrences, never parse-work Usage;
U = current snapshot syntax-entry events;
M = established parser metadata subtotal (file references, significant tokens, value-end bookkeeping/position/lexical-owner slots, with documented conservative retained overcount);
T = current snapshot counted owned syntax text, at whichever producer/phase boundary the refined contract actually guarantees;
W = only explicitly inventoried auxiliary scratch reservations, not an invented “all runtime allocations” proxy.

Required aggregate checks are C.entries + U + M + W.entries, C.text + T + W.text and C.occurrences + W.occurrences. Checked arithmetic and inherited ceilings apply; new snapshot work counters still start at zero and are never used to debit C. Avoid charging significant tokens twice by treating token work and metadata storage as distinct counters, and avoid charging value-end conservative metadata twice when actual tables drop.

Narrow growth seams if parent selects pre-growth aggregate guarantees:

Seam	Minimal responsibility / caution
New snapshot budget construction and grammar entry events	Preserve exact private Limits and ProjectLimits; carry C separately; entry-event checks include retained M and selected scratch. Existing numeric/byte/token work semantics unchanged.
Lexer retained significant-token push and file metadata	Admit counted token/file slots against C+U+M before retained growth; raw trivia work remains separate and does not allocate a retained token.
Mixed value-end insertion and lexical-owner push	Count newly owned logical records before growth or prove a conservative pre-reservation; release or retain documented overcount consistently.
Position preprojection/count walk	Add C+U to the existing projection precheck; explicitly reserve any new counting scratch. If adding bounds for existing `ends`/delimiter scratch, state the new named logical reservation rather than claim it was previously exact.
Counted syntax text producers	Either reserve exact decoded/copied bytes before the in-scope copy with a bounded decoder/count seam, or reserve a proven conservative whole phase bound. Raw-token spelling length is not automatically the exact decoded-string length; name/field/projection copies coexist and each counts. Do not reparse/convert Exact to estimate them.
Resolve handoff	Replace the internal zero start with the carried ledger. Transfer/reconcile already reserved U/M/T once; do not run old `syntax()` admission as an additive double charge. Its bounded walk can verify counts and derive LowerStorage under the same carried baseline.
Mapped warnings, final equality and output-owning result	Retain current diagnostic reservations; reserve verified E equality scratch against surviving storage; drop temporary syntax/model-borrowing environments at an explicit transition. Returning owned output is not returning a self-referential syntax witness.

An incremental phase-check-only carried seam is smaller: add C to each existing parser phase check and syntax reconstruction, and seed resolver with C. It preserves the old timing limitations and must be named accordingly. It is not sufficient evidence for a newly promised before-every-growth parser invariant. If that stronger temporal invariant is essential for the imminent execution contract, implement the selected counted producer hooks before calling emission/re-admission accepted; this is a bounded prerequisite, not grounds for whole-model allocator interception.

Acceptance and explicit decisions to revisit

Parent should name the exact retained/temporary syntax charge table and temporal guarantee. Are traversal/path and projection scratch to become newly enumerated auxiliary representations now, or do their current independent file/token/depth bounds remain an explicit limitation? Either answer needs a written scope decision; do not quietly infer an all-heap requirement.
Does carried parsing promise aggregate before-growth on counted syntax Strings, or only at the established syntax-to-resolver phase? If the former, decoded string and projection-copy hooks are necessary; a fresh Ledger plus postparse recount cannot prove it.
Preserve exact prior Limits, caller ProjectLimits and snapshot provenance without exposing arbitrary profile/baseline constructors. A dedicated new-snapshot continuation should report new Usage separately, not reset or subtract the original witness's counters.
Test nonzero carried baselines at actual growth/phase boundaries, comparing exact cap/cap-minus-one through the real predecessor path; include output text plus copied path/entry bytes. Carry zero must preserve established behavior. Hand-count named containers, not allocator capacities.
Test dropped-old-owner versus simultaneous-old-owner transitions, string moves/copies and no double charge; verify no failed output/admission result escapes. Include all-file sizing and complete actual-output model equality. Do not conflate planned size with accepted bytes.
Keep legacy/model/raw-validator exclusions and existing stack limitations visible. Whole-RSS or full parser/model allocation containment would be a distinct future requirement with its own design, evidence and cost; it is not automatically the next smallest editing dependency.

No stronger guarantee is proven by this audit. The recommended decision is to carry the declared logical domain faithfully, strengthen only the named parser timing gaps needed by the successor, and document remaining exclusions and phase limits rather than alternate between zero-baseline acceptance and universal heap accounting.

18 KiB Raw Blame History Unescape Escape