Source8 S2a immutable source handoff

2026-09-13. Codex-956atf /root/source8_lowering_plan. Worktree .local/as2-20260913, branch codex/analysis-s2-956atf-20260913, latest clean commit b2c970b4673e062e47eadd12906d2316c4b1a7b3. Initial S2a checkpoint 021bdc5a921885d1f5ce5b3a4b9cdbc03a73ab68 was based on 6216cff4cfeb1502b8453b40547aacb74941f2b2; the latest commit contains the narrowly reviewed corrections below.

This is the authorized first declaration/cursor/span checkpoint, not S2 lowering/writer completion. No Cargo, compiler, executable tests, native UI or Gitea actions were run by this worker. Parent owns immutable review, compiler lease, integration and publication. The original .local/analysis-core-worktree-20260911 was not edited during S2a.

Parent-reported baseline evidence: strict passed on 9ff6e7c; 6216cff4 has 118 listed core library tests and actual four bridge + eleven syntax + four save-guard passes, with 520 unchanged inputs, followed by all-targets strict PASS. None of that proves this new commit passes. Retain the old public source8 failure evidence until the complete public adapter exists.

Concrete result and ownership boundary

Initial checkpoint: eight source files changed, 1,719 insertions / 325 deletions. The totals include moved lexical/cursor code and 531 lines of declaration tests plus 120 lines of lexical fixtures. Correction b2c970b adds 90 lines/removes2 in three files (two new frame tests account for most additions). All tests remain unrun on the latest source by this worker.

File	Change
`language.rs`	Add `mod raw_tokens`; replace existing `tokenize` body with `raw_tokens::legacy(source)`. No other source changes.
New `language/raw_tokens.rs`	One streaming token iterator over raw source; legacy strict/tolerant adapter reproduces existing token classes/shapes, greedy signed JSON numeric classification, and individual-character errors for new expression punctuation. Authored compatibility fixtures. JSON string validation uses temporary allocation inherited from the old tokenizer; the iterator is not allocation-free.
`language/analysis_source.rs`	Add private cursor/declarations modules. Existing expression-only wrapper scans once, invokes borrowed binding parser, then requires cursor EOF.
`language/analysis_source/lex.rs`	Replace separate scanner with source8 significant-token adapter over the shared raw scanner. It charges source bytes once and every token, including trivia, before scanning/storing.
New `language/analysis_source/cursor.rs`	Move shared cursor, token/span/name/literal/frame helpers out of expression parser; add quoted strings and exact-only literal consumption for declarations.
`language/analysis_source/syntax.rs`	Existing iterative Expr machine consumes the shared cursor; caller now owns EOF/field delimiter. Reuse shared exact literal helper. Entry checks the latched budget before even `unbound`.
New `language/analysis_source/declarations.rs`	Required typed header/body/field/type/input/override grammar, identity insertion metadata and duplicate checks. No resolution/model construction.
New `language/analysis_source/declarations/tests.rs`	Eight authored declaration/cursor/identity/budget contract tests.

The existing #[cfg(test)] mod analysis_source boundary remains. No public8 header admission, new public API, new production DeclKind, writer, model/semantic mutation, source_project integration or schema/codec switch. The nine short words are recognized only by the staged declaration function, and do not enter reserved() or the public declaration map. The public legacy token contract remains intentionally unchanged, including negative numbers as one Number token and +1 as Error. This adapter choice was explicitly confirmed by the parent during implementation.

source3_bridge.rs and both root-owned hooks are unchanged. File SHA256 remains e3a76cfdacf6d85ce3086dc9031aeab5115496786ddf197510939ac0e0c5310c; the production language diff contains only the raw module declaration and tokenizer body replacement. No source_project/model/persistence files changed.

Exact private interfaces

raw_tokens::Scanner::new(source) implements Iterator of raw Token { class, span, legacy_number_part }. Numeric tokens are never converted through JSON here. String validity uses the same JSON-string admission as the former public tokenizer. The legacy adapter uses bounded token lookahead to join the old contiguous numeric-token alphabet, then applies the existing JSON Number classification. This intentionally preserves old numeric validation at that legacy adapter boundary. The source8 adapter bypasses that number conversion and uses Exact at literal consumption. Operators do not become accepted old grammar just because the shared scanner recognizes them.

analysis_source::lex::tokens(source, &mut Budget) -> Result<Vec<Token>, Error> remains the charged source8 adapter. Tokens are significant-only and keep full-file byte spans. Scanner trivia still consumes token charges. A compile owner must pass the same Budget across its reachable files; no implicit session or per-field budget was added.

Cursor::new(source, &tokens, &mut Budget) borrows the full source/token stream. Its public-to-sibling fields are source, tokens, cursor, frames, budget. syntax::binding(&mut Cursor) -> Result<Binding, Error> consumes one binding and leaves its following token untouched. It does not search ahead for field endings or rescan source substrings. Nested commas and closes still belong to the existing explicit-task Expr machine. Cursor::end() enforces EOF when the caller needs it; all spans including numeric trivia remain original full-file offsets.

declarations::parse(&mut Cursor, version: u32, lexical_owner: Option<Id>) -> Result<Declaration, Error> consumes exactly one typed declaration from that cursor. version is the caller's inherited entry policy, not a fragment-selected mode. It rejects any version other than8, any lexical owner and unsupported/long declaration word before allocating a declaration. The outer parser/include owner is deliberately not implemented here.

Typed syntax:

DeclarationKind: QuantityKind, Unit, Calculation, Property(Layer), Check(Layer), Case(Layer). declaration_kind(word) recognizes precisely the nine short neutral/OA/SA words.
Declaration: Header, whole span, brace-excluding body span, optional description/explicit root-parent field, typed Body.
Header: kind/token span, alias token+name, display name and its string token span, identity metadata and header span.
IdentitySyntax: optional UUID value, optional id keyword and literal spans, optional insertion byte offset. No UUIDs are generated. Missing global/parameter IDs stay unresolved and repeat parsing yields identical syntax. The later stabilization owner assigns IDs, inserts them once and reparses exact bytes. This follows r2 line196 without pretending syntax parsing already performs bundle stabilization.
Field<T>: complete field span including keyword/semicolon, separate keyword span, original value span, typed value. The value-span meaning matches existing top-level field consumers.
TypeSyntax: whole type span, ordered scalar syntax/span and List boolean; exactly six forms. Nested List rejects. Quantity references remain unresolved.
ParameterSyntax: whole input span, name/token, identity metadata and type. Authored order stays a Vec. Duplicate names and explicit IDs reject within one calculation; IDs are not inserted into any global registry.
DimensionSyntax: fixed seven exponent slots plus authored ordered entry/axis/exponent spans. Unknown/duplicate axes and values outside -32..32 reject.
ExactSyntax: Exact value plus full/component literal syntax. Scale/offset accept literal-only syntax, using the same exact helper as Expr literals.
OverrideSyntax: full entry, target alias token, binding and its span; an ordered Vec until future typed resolution. Duplicate names reject before map construction.
Body variants have required non-Option typed fields: QuantityKind(dimension/algebra), Unit(kind/scale/offset/symbol), Calculation(ordered parameters/result/body), Property(subject/type/base), Check(predicate), Case(ModelBaseline/explicit overrides). Only documented optional fields remain Option. Description can be absent; the three analysis roots may explicitly state parent none; definitions reject parent fields.

Expr syntax remains all thirteen variants. Repeated call argument tokens have distinct spans under separate actual Expr nodes; the authored declaration test verifies all four labels in the repeated two-call discriminator. S2b/S3 must attach owner/field/path and resolved callee/ParameterId identities. This checkpoint does not claim those qualified semantic index keys already exist.

Authored tests, all unrun on this commit

Existing eleven language::analysis_source::tests::* tests were retained byte-identical. Existing public token control expectations were not weakened or replaced. The new tests are:

language::raw_tokens::tests::legacy_adapter_retains_authored_numeric_operator_and_trivia_tokens: exact expected classes/text/contiguous spans for adjacent/spaced minus, exponent signs, plus, malformed atoms, combined operator legacy errors, comments, Unicode, invalid/incomplete strings, control characters and contextual words.
language::analysis_source::declarations::tests::required_declarations_have_independent_typed_body_and_header_oracles: independently expected required bodies/headers for all nine declaration words, dimensional/algebra/unit numeric values, calculation parameter order/Expr topology, property/check/case fields.
...::empty_unbound_and_all_scalar_list_types_are_explicit_valid_syntax: six types, zero parameters, unbound, empty dimensions/overrides and optional links.
...::missing_required_fields_and_unknown_or_duplicate_entries_reject_locally: remove every required field, duplicate singleton fields/axes/parameter names+IDs/call arguments/overrides, unknown fields, literal-only violations, invalid types/dimensions, required subject/base violations and exact duplicate target span.
...::borrowed_cursor_preserves_full_file_spans_and_does_not_rescan_fields: two declarations on one charged stream with Unicode/CRLF/ratio comments, cursor continuation, exact body/input/type/literal/component spans, four distinct repeated-call key spans and unchanged lexical byte/token counters. Independently expected nine reference occurrences.
...::missing_id_metadata_is_stable_and_parameter_ids_are_owner_scoped: repeat parse exact syntax equality, header/parameter insertion offsets including comments, explicit parameter/global UUID collision and separate calculation reuse.
...::inherited_version_root_and_short_word_contracts_do_not_enable_public8: inherited version/root failures, long words excluded, existing contextual aliases still valid on public source7, public8 still rejects and old versions still reject short analysis declarations.
...::declaration_budgets_share_literals_types_references_and_failure_state: deterministic numeric/entry near-limit success and limit-1 failure, shared cross-unit type/subject/unit refs, latched first failure with no later literal work, and borrowed unbound cannot escape an exhausted budget.
...::typed_frame_budget_and_truncated_declaration_cleanup_have_nearby_successes: declaration + nested List/Quantity frame3 success/frame2 reject after correction b2c970b, truncation across declaration/override/call/ratio boundaries and fresh subsequent syntax parse.
...::typed_brace_frames_have_exact_zero_one_two_boundaries (added in b2c970b): standalone unbound succeeds with zero frames; the smallest calculation needs one; empty dimension and override blocks need two including their declaration. Each 0/1/2 ceiling has exact success/failure and located-brace assertions; successful declarations restore zero and report the independently expected deepest frame.
...::declaration_frames_restore_the_callers_inherited_context (added in b2c970b): a caller holding one outer frame causes a scalar calculation to reject at ceiling1 and succeed at ceiling2. Two sequential declarations restore the same caller frame rather than accumulating depth or resetting it.

Reviewed correction history

Parent/fresh critic source review of 021bdc5 identified two concrete corrections before executable gating. b2c970b4673e062e47eadd12906d2316c4b1a7b3 addresses them without changing unrelated production behavior:

The authored @alias lexical fixture now expects Keyword, matching both the frozen reserved() behavior and the new legacy adapter. Only the test expectation changed; the production tokenizer did not.
declarations::parse, Cursor::dimension and Cursor::overrides now reserve an active syntax frame before their opening brace and close it after the matching closing brace. The existing Expr/type/algebra/ratio delimiter accounting is unchanged. This makes typed-brace overhead explicit and consistent; it is not a claim that the previous fixed brace undercount permitted unbounded recursive syntax. Standalone S1 expression tests remain byte-identical and their frame counts remain unchanged.

The current private cursor starts at zero only when its caller creates a standalone parse context. Future S4 project/include integration owns the inherited outer syntax context and must pass/preserve that context consistently; declaration parsing adds its own braces and restores the caller's depth on success. A failed cursor remains discarded as documented below. The new inherited-context test exercises the interface without prematurely implementing or deciding the outer Project/include parser.

The handoff's former description of the iterator as allocation-free was inaccurate: raw string classification JSON-validates strings using a temporary String, as the old tokenizer did. It has been corrected to streaming; no allocation/RSS result is claimed.

Correction verification performed: rustfmt edition2024/skip_children on exactly the three corrected files exited0; git whitespace check passed; source diff was inspected and committed. No Cargo/compiler, test executable, GUI or Gitea action was performed. Latest source is paused for fresh critic verification and the parent's serialized executable gate.

Worker verification: rustfmt edition2024 with skip_children=true exited0 for the seven implementation/test paths beneath language/ that changed. language.rs contains only the hand-formatted module/function hunks; its unrelated pre-existing formatting was preserved. Git staged whitespace check passed. Source inspection checked tokenizer compatibility, shared budget/cursor use, required-field paths and unchanged bridge/header hooks. Rustfmt parses Rust syntax but does not establish typechecking, test correctness, runtime safety or resource performance.

Review focus and remaining work

The shared production scanner is the consequential surface to review first. Compare its legacy adapter against the frozen tokenizer, especially malformed greedy numeric runs, lookahead across trivia, UTF-8 invalid/incomplete string spans, and punctuation splitting. Root should run the authored legacy test, unchanged S1 tests, affected ordinary source/nesting/coordinate/message controls and strict all-targets checks after immutable review. The compiler has not yet validated this commit; repair findings on a new pin and retain failed evidence.

The typed stage intentionally does not resolve refs, check purity/layers/type compatibility, admit whole Projects or write source. Examples such as a property reading a wrong-domain @alias remain unresolved syntax and must reject in later lowering/admission. It also does not select a bounded entry header policy in production, thread budgets through includes, combine ordinary references/graph counts, map latched spans to file paths, stabilize IDs, or expose static admission usage. Those are explicit S2b/S3/root integration obligations, not accomplished behavior.

Caller contract on error: abandon the failed cursor/candidate and retain the source path that produced the first error. Budget exhaustion latches its original span; continuing in a new file and reattaching that span to the new file would be incorrect. No broad generic parse-session abstraction was introduced to hide this responsibility.

S2b can now build global categories/signatures and lower these typed bodies; S3 can design owner/field/Expr-path descriptors against these original spans. Before adding public DeclKind or replacing generic Declaration payloads, coordinate exact dependent match hooks with the source_project/root owner. Continue to preserve root's source3 bridge and public version gate. Canonical writer, full worked OA/SA whole-Project tests, proper qualified rename/swaps, fixed include/stabilization/migration pass counts and measured source profile adoption remain subsequent gates.

Worker is paused after this immutable checkpoint; no broad lowering or writer work began.

17 KiB Raw Blame History