ARCADIA-Studio/harness/analysis-mixed-parser-proposal-20260923.md

# Next private mixed-parser slice

**Original proposal, subsequently narrowed by the [independent review](analysis-mixed-parser-review-20260923.md).** The parent accepted the private syntax-only direction with that review's corrections; implementation and runtime acceptance remain open. In particular, the review resolves the base to A and corrects this proposal's inference of ordinary grammar drift from whole-file statistics. The following inspection is retained as its original reasoning, not a public source8 claim.

Read-only inspection, 2026-09-23. Private source pin A=`10d7f3340d5cd61a656dc0bcf7c143f8897c5551`; current ordinary/public pin R=`195ddbbba5b352af09bca385a6c3e4dd1c77f8c8`. Both pins checked before and after inspection. No compiler, source edits, GUI or Gitea operations. Parent reports A passed Windows 173 listing/65 typed/strict; those gates do not exercise mixed parsing.

Read the published `harness/analysis-original-span-proposal-20260922.md` and `harness/analysis-original-span-review-20260923.md` at R, plus A's `harness/analysis-source8-20260913/analysis-source8-repair-design-r2-20260913.md`, especially “Source pipeline and exact editing” and the resource/older-version contracts. This note narrows their next parser dependency; it does not replace them.

## Recommendation and exact deliverable

Implement **one private complete-bundle syntax parse**, producing an opaque `ParsedMixedSyntax` witness. It retains ordinary unresolved syntax and ordinary edit positions alongside typed unresolved declarations, from one significant-token stream per original file and one inherited budget for the complete pass. Start with explicit stable IDs only, source8 entry, existing ordinary grammar, all existing typed declaration grammar, and normal include/path rules. It returns no Project, Index, rename authority, output text or installed document. A complete semantic/session witness is a dependent slice, not an alternative name for this syntax result.

This resolves actual mixed-file dispatch, full-file coordinates, lexical ownership, aggregate scanning and same-parse ordinary edit provenance before undertaking cross-domain lowering or production composition. Root public dispatch remains closed; a private constructor selects source8 policy before scanning and verifies that the actual entry says 8. No schema constant, public parser acceptance, save route or model change belongs in this patch.

## Existing source seams

Paths below are repository-relative; line numbers belong to their indicated pin.

| Existing site | Consequence for the slice |
| --- | --- |
| R `language.rs:1066` `parse_unit`; `:1272` Parser; `:1489` unit | `strict_tokens` owns a separate token Vec; unit mixes entry/header/includes, iterative ordinary declaration nesting and fields. Add a private mixed entry through factored grammar operations, retaining the iterative declaration-owner stack. Public versions currently stop at 7. A has the corresponding unit at `:1332`. |
| R `language.rs:1155–1270` `edit_declarations`/`edit_positions` | `edit_declarations` reparses and `edit_positions` rescans, derives delimiter ends and assumes token shapes. A mixed caller cannot invoke these helpers on the original mixed file. Factor a positions projection over the already parsed ordinary AST and retained token/span metadata. Keep the legacy wrappers' observable behavior. |
| R `language.rs:1392` ordinary `value_inner` | Ordinary numbers remain finite JSON values; they are not Exact. Raw `-` + number must be consumed as one ordinary signed value with its complete span; typed `1-2` is subtraction. Preserve ordinary rejects such as `+1`, `1e`, `1_0`, `0x1` and nonfinite JSON numbers. |
| A `language/raw_tokens.rs` Scanner and legacy adapter; `analysis_source/lex.rs:23` tokens | Scanner already preserves raw full-buffer spans and separates typed operators. Legacy adaptation greedily groups adjacent numeric parts and performs JSON classification. Do not use that whole-file legacy adapter for mixed source; typed lex currently charges bytes/raw tokens, including trivia, before retaining significant tokens. |
| A `analysis_source/cursor.rs:5` Cursor; `declarations.rs:243` parse | Typed parse already consumes a borrowed significant cursor, checks inherited version=8, rejects lexical owners, and returns unresolved full-buffer declaration/field/reference/literal spans. This is the useful dispatch seam; `analysis_source::parse` is an expression-only whole-input wrapper. |
| R `source_project.rs:303–605` compile_once | Existing iterative include visitor owns path normalization, cycle/depth/edge/visited checks, aggregate ordinary counts, unreachable-file rejection and located errors. Reuse/factor its traversal policy for private syntax collection; do not call compile_once and discard its model. Its `:429` parse_unit call is the later production substitution point. |
| A `analysis_source/index.rs:147–195` parse/build | Current parse scans typed-only numeric-keyed files and accepts a separately supplied ordinary Project. Calling it on mixed buffers fails; calling it on extracted typed text invents provenance. Keep build private. This slice does not manufacture an Index. |
| A `analysis_source/lower.rs:269` lower | Accepts ordinary skeleton plus all typed units; collects all typed globals/signatures before bodies; deliberately does not admit the skeleton. Needed in the next semantic slice after the shared syntax witness exists. |
| R `source_project.rs:895`, `:918`, `:984`, `:1050–1071` update path | Actual-file ordinary edit ASTs, formatted comparison ASTs and local project-header spans have different roles. Later production editing uses retained actual-file positions; retain CompleteForEdit/Compact formatting and their comparison semantics. Scalar application, structural second pass and full candidate equality remain outside this parser patch. |

R and A's ordinary files are materially different (427 insertions/81 deletions across language.rs/source_project.rs in the inspected comparison). R also contains current CompleteForEdit/Compact work. Do not copy A's whole language/source_project files over R or overwrite the parent's Scenario integration. Pin the combined implementation base and agree ownership of the shared grammar hooks before coding.

## Minimal private interfaces and ownership

Names are illustrative, not a request for a public framework:

```rust
parse_mixed_syntax(bundle: &SourceBundle, limits: ProjectLimits,
                   test_profile: /* optional tighter private profile */)
    -> Result<ParsedMixedSyntax<'_>, Vec<SourceDiagnostic>>;
```

The result privately borrows the **actual complete SourceBundle**, owns the one finished Budget, canonical file table, parsed units and retained significant tokens/ordinary position metadata. It retains entry, every canonical path and original bytes through that borrow. File keys are minted once from canonical bundle keys before traversal; the include visitor visits each reachable file once. A FileKey is meaningful only through its owning result, never an independently accepted `(usize, path)` mapping. Store paths/source references into the borrowed bundle rather than a self-referential owned Index-plus-source-map structure. Constructors are private; expose read-only inspection only. This establishes syntax provenance, not semantic validity or mutable-current freshness.

Each file result contains its source reference, entry/fragment role, include spans, ordinary declarations with lexical owners, typed declarations, and ordinary edit positions. Positions remain half-open **full original file UTF-8 byte offsets**, even after Unicode comments, the project header, includes or preceding declarations. No header/include offset is added to typed spans. Nested ordinary declaration spans legitimately contain descendant declarations; declaration ranges are descriptive ownership, not automatically disjoint replacement authority.

Use one shared significant-token cursor state. Ordinary grammar methods may have a domain-specific borrowed view; they must advance the same token position used by typed `declarations::parse`, without copying/rescanning a declaration or replaying a speculative parse. Preserve legacy grammar through factoring, not by cloning a second independently maintained ordinary parser. Keep token access/classification adapters small and local; no general parser trait framework is required.

At document scope, dispatch exact typed declaration words before generic ordinary `declaration_kind`. Inside an ordinary element's open body, recognize a typed header only to deliver the existing “must be roots” error; do not reinterpret it as an ordinary field. Contextual words such as `unit`/`calculation` must remain legal aliases where already allowed. Entry descriptions/includes keep current placement rules. Generic derived long analysis kind names must not become an alternative accepted syntax.

For ordinary edit positions, a raw numeric value may cover several significant tokens (`-12.5`). The current token-end heuristic would cover only `-`. Record ordinary value end spans as grammar consumes values (a small per-file side table keyed by the existing AST start offset is sufficient), then project EditValue from that AST plus spans. Keep the shared legacy/migration Ast representation unless a demonstrated need justifies changing it. Record ordinary header token roles from actual parsing as well; do not obtain them with another `edit_header` scan. Typed declaration spans/expressions never enter ordinary EditKind::Record recursion.

**Stable-ID scope:** reject omitted project, ordinary global, typed global or parameter IDs in this first private entry before returning a witness. The ordinary `optional_id` path currently creates UUIDs, so the private mode must reject before that generation; typed IdentitySyntax already retains missing identity/insertion information. This intentionally postpones stabilization instead of assigning hidden IDs or granting edits against pre-insertion offsets. Later accept missing IDs only through bounded insertion into the returned full documents and a new complete parse of those actual bytes.

## Budget and parser invariants

Create one Budget at the private complete-parse boundary from source8 hard ceilings intersected with caller restrictions and any tighter test profile. Own it in the returned witness; lower/index/compose later obtain it from that witness, with no ambient replacement/reset API. Creating a Budget for each include or switching token policy after seeing local fragment syntax is invalid. Selecting explicit private source8 mode before lexing avoids a public version-probe implementation in this slice; eventual public dispatch needs a bounded header read with unambiguous accounting.

Precheck aggregate file/UTF-8 byte size before broad token/String allocations. Charge each file's original bytes once in scanning; a prior noncharging aggregate limit check is not another charge. Charge every raw token including trivia once, before storing; significant filtering and ordinary legacy-compatible numeric consumption do not recharge tokens. EOF carries no token charge. Do not scan or convert typed exact atoms again to derive positions. JSON ordinary values keep their finite numeric domain, and exact numeric WorkMeter use belongs only to typed numeric consumption.

The mixed parser must also bound ordinary temporary containers/references before allocation; existing per-unit graph counts and final `check_model` alone are insufficient. Specify the charge table in the patch: ordinary declaration/field/value/list-entry allocations join the private syntax-storage ceiling; retain caller graph/record accounting separately; ordinary reference storage is bounded before copying, while caller ordered-step/reference semantics remain explicit rather than silently relabeling every scalar reference a model step. Typed existing entry/reference charges stay intact. Count aggregate phase metadata (file table, positions, catalogs later) before reserving; the existing typed budget phase checks are reusable but do not magically bound ordinary additions. Preserve ordinary value-depth64 and separate typed frame256/logical-depth128 semantics and safe failure cleanup. Tests must discriminate which bound fails.

## What this does not yet solve

The dependent **ResolvedMixedBundle** must consume this syntax witness, collect one global alias/identity/category catalog including typed declarations, build the ordinary skeleton against it without premature complete-model admission, lower typed units once, merge them, admit the whole Project, and construct Index internally from the same tokens/units/lowerer result. Only that result can bind complete Project, Index, file table, original bundle and inherited Budget for composition. Current A Index borrows `BTreeMap<usize,String>`; adapting its private source accessor to witness-owned file references is justified there. Exposing `build(Lowered, spans)` to arbitrary callers is not.

R `language::resolve_declarations` combines ordinary construction, version migration and final limits; A additionally migrates source7 through `migrate_v7`. Neither is an appropriate “parse ordinary and admit first” shortcut: forward ordinary references may target typed records, and typed subjects/types/calls may target later files. The semantic slice needs a narrowly extracted non-admitting ordinary construction step, complete catalog, and one final model admission. Syntax witness tests cannot prove these relationships or satisfy intended-Project equality.

Coordinated version7→8 blockers for production are: reconcile root schema7 with private schema8 definitions/analysis and Scenario payload; preserve frozen older DTO/source gates and the source3 old-envelope boundary; choose the one public dispatch/migration change; stabilize missing global/parameter IDs and reindex actual returned bytes; retain required categories and ordinary complete/compact edit behavior; supply complete mixed lowering/index provenance and supported-edit diff gate; compose original edits once and recompile/admit/compare the entire actual output; pass migration/history/checked-save and resource calibration gates. None requires a new metamodel or blocks this private explicit-ID syntax slice. Root should coordinate them, not accept `arcadia 8` early to make a fixture pass.

## Rejected shortcuts and cost comparison

| Approach | Decision |
| --- | --- |
| Shared scanner/cursor + existing grammar extraction + same-parse ordinary positions | Recommended: moderate localized factoring; directly tests mixed boundaries and retains ownership. Limit deliverable to syntax until semantic construction is ready. |
| Parse ordinary and typed projections independently, then combine models/indexes | Reject: masking preserves numeric offsets at best but loses actual-source token/ownership proof; substring/concatenation creates local offsets; separate parses reset or double-charge resources and cannot establish cross-domain global resolution. A clever remap does not repair these invariants. |

Also reject whole-file legacy tokenization (typed punctuation fails/typed number domain changes), silent retries with larger budgets, counter subtraction after rescanning, caller-supplied numeric file remapping, and treating equal bytes/shared lifetimes as proof of the same semantic parse. A full replacement edit AST is unnecessary for this step.

## Six observable first-slice fixtures

All are proposed, unrun. Use independently authored source literals and expected span/substring tables rather than expectations extracted from the result itself.

1. **Interleaved CRLF/Unicode:** entry description/comment, ordinary nested element, typed calculation with `1-2` and `1e-2`, ordinary record with negative coordinates after it. Verify exact original substring at every selected ordinary/typed/header/value span, lexical owners, and byte-identical complete input. Explicit `-12.5` ordinary span must include the minus and full atom. No output Project claim.
2. **Complete include binding:** root + nested-path module + shared dependency reached twice; mixed declarations after include prefixes in two files with identical local offsets/spellings. Check unique scan count, fixed FileKey/path mapping, exact include spans; cycle, missing, escaping and unreachable-file negatives return no complete witness. Entry/path/map replacement is not an accepted constructor operation.
3. **Dispatch and grammar separation:** `unit`/`calculation` used as allowed ordinary aliases, all short typed declaration kinds, typed roots between nested ordinary blocks. Reject typed lexical nesting, long derived analysis names, fragment version headers and source7 at this private source8 entry. Existing public source3–7 and public source8 rejection remain unchanged.
4. **Number/position compatibility:** ordinary signed/exponent JSON values and typed exact ratio/scientific expressions, with comments between legal typed tokens. Ordinary `+1`, split-sign forms not admitted by the old grammar, malformed/nonfinite JSON values reject. Typed values beyond JSON precision retain exact values/literal spans. Existing ordinary token compatibility tests plus independently stated old outcomes prevent broadening by accident.
5. **Aggregate budget and failure cleanup:** one-file and split-file variants with exact byte/token/ordinary-storage/reference/numeric ceilings and plus-one failures; include trivia; add an earlier ordinary allocation so a later typed declaration crosses the shared ceiling, and reverse producer order. Previously latched failure returns no witness. Test-only scanner visitation/Exact conversion counters demonstrate one scan/one conversion; unchanged counters alone do not. Run retained 128-KiB typed depth/rejection cleanup controls when parser ownership changes warrant them.
6. **Opaque syntax scope:** explicit-ID positive across all declaration/parameter kinds; remove one ID in each domain and require refusal with original bundle unchanged. Read-only ordinary EditValue and typed occurrence syntax inspect the same witness; no external AST/Lowered/FileKey-map constructor exists. Unresolved forward ordinary/typed references can be retained syntactically, but do not assert they resolve. A following semantic slice must add reordered cross-file forward references, repeated/reordered calls, reused parameter UUIDs, catalog collisions, wrong-session batch rejection and independent whole-Project equality.

First patch completion means these syntax/ownership/resource fixtures and unchanged ordinary compatibility tests pass on an immutable Windows source pin with fresh review. It does not mean mixed renames, source8 save, full S3/S4, public schema activation, or all semantic references work.