Preserve source comments and layout through model token edits

harness/release-source-project.md

@@ -10,7 +10,7 @@ Status: pure source3 subsystem implemented and tested. This is not native projec
 - Portable relative includes, root containment, canonical map keys, case collisions, repeated includes, cycles, missing/unreachable files, duplicate JSON file keys and aggregate admission checks.
 - Generated identities are inserted into their original headers. Returned diagnostics/index address the resulting stabilized buffers. Caller drafts are never mutated.
 - Canonical modules follow layers, work roots, behavior contexts and diagrams. Ports follow their owners; placements follow diagrams; relations follow their sources. Existing ownership is retained during model updates. Huge sets of independent roots consolidate by category rather than produce thousands of files.
-- Prototype model updates preserve untouched files/text, rewrite changed declarations and alias uses, add new owner/root declarations and includes, and retain empty files after deletion. Every update recompiles and checks exact semantic equality before return.
+- Model updates preserve authored comments, whitespace and surviving fields, change header/value tokens and alias uses, add new owner/root declarations and includes, and retain empty files after deletion. Every update recompiles and checks exact semantic equality before return.
 
 ## Consumer interface
 
@@ -34,23 +34,27 @@ For native discovery, `inspect_includes(path, source, is_entry, limits)` uses th
 
 `CompiledSourceProject.decoded.migration` is `None` for source3 bundles. Legacy import remains `language::parse_with_report` / `persistence::decode_with_report`; retain that report in the importing UI, then explicitly export the migrated model into source3 modules. Do not independently migrate fragments or fabricate a migration report during subsequent source3 compilation.
 
-## Explicit authoring limitation and next seam
+## Comment-preserving update policy
 
-`update_bundle` currently canonicalizes an entire changed declaration. It preserves comments outside that declaration and all untouched document bytes, but **drops inline comments/custom layout inside a changed declaration**, including for a scalar-only change. This is a tested core candidate, not the adopted final project-save policy. `format_bundle` is an explicit canonical export and intentionally does not reproduce authored trivia.
+The earlier whole-declaration canonicalization limitation is fixed. `update_bundle` derives language-specific editing positions from the existing validated AST and tokenizer. The shared V2 AST and migration behavior are unchanged; no alternate grammar, persistent syntax model or generic CST framework was added.
 
-The bounded next improvement is a lossless update representation, not a second semantic model:
+- Old/new canonical typed values identify semantic changes; surviving authored fields retain their exact text, field order, comments, omitted defaults and escape/number spelling. Header and scalar changes replace only the affected tokens. Reference aliases change without removing comments between `@` and the alias.
+- New fields insert at the owning record using existing indentation/line endings. Omitted defaults stay omitted unless their semantic value changes. Constructor tag changes preserve shared nested fields.
+- Removing a field, constructor or declaration erases its significant syntax tokens, retaining comments/whitespace as annotations in the parent record or file. Complete value-shape replacements retain the old value's comments next to the replacement. The updater does not guess how orphaned engineering notes should be reworded.
+- Lists match unique referenced UUIDs, message-value `field` references, binding `referenced` identities, or literal values. Retained entries move with their inter-separator trivia; removed-entry comments remain in the enclosing list. Other/duplicate-key lists use positions. Trivia after a comma belongs to the next entry; trivia before a comma belongs to the previous entry. Separators are emitted before trailing line comments to keep syntax valid.
+- Declaration identity is UUID-based; no alias/name heuristic silently treats a new UUID as an old record. Explicit UUID replacement removes/creates records while retaining removed-record comments as file annotations. Existing UUIDs and file ownership remain unchanged for retained records.
+- `format_bundle` remains an explicit canonical export, not the update/save path. Keep compiled workspace snapshots for undo/redo; model-only undo cannot recover prior omissions or authored layout choices.
 
-1. Extend parser-derived locations to header alias/name/identity tokens, nested record keys/values, list items, and closing-brace insertion points. Existing top-level field value spans already distinguish unchanged fields.
-2. Diff old/new typed values using field paths, then replace only changed literal/reference tokens. Preserve field order, spelling/trivia of unaffected fields, and comments between tokens. Adding a field inserts at the owning record; an omitted default needs insertion only when its semantic value changes.
-3. For constructor/list structural edits, use keyed record/list changes with explicit trivia retention. Do not silently fall back to replacing the whole record and call it lossless. Keep explicit canonical export separate.
-4. Add exact-byte regressions for guard/name/message edits with comments before, within and after the changed record, reordered authored fields, omitted defaults, nested scenario message values, alias uses and CRLF source. Verify model equality and the unchanged bytes in every case.
+This production core correction does not itself establish complete human-authoring, project-editor or native save acceptance. Experimental nested grammar remains separate and unintegrated.
 
 ## Verification
 
-- `cargo test -p oa-core`: **150 tests passed**, including 20 new source-project tests.
+- `cargo test -p oa-core`: **160 tests passed**, including 30 source-project tests.
 - `cargo clippy -p oa-core --all-targets -- -D warnings`: passed.
 - Owned Rust files formatted with `rustfmt --edition 2024 --config skip_children=true`.
 
 The source-project tests cover cross-file typed-port/exchange references, Unicode syntax/structural diagnostics, duplicate declarations with related locations, include diamonds/cycles/missing/unreachable files, portable paths, aggregate byte/object/record/step/file/depth/edge budgets, JSON duplicate keys, UUID stabilization and adjacent tokens, declaration relocation, alias propagation, owner inheritance, new module includes, retained empty files, stale/invalid update rejection, canonical case collisions and bounded large-root output. Full demo equality covers behavior, artifacts, trace baselines and placements. Moving a placement changes only its owning view document.
 
+Ten additional regressions verify exact CRLF/name/guard/project-metadata updates, Unicode escape spelling and comments after `@`, omitted defaults and field insertion, kind/constructor changes with removed-field comments, scalar/record value-shape replacements, reference and literal list changes, deleted-record comments, and nested scenario message-value edits/reordering by field identity. These tests check exact unchanged bytes or the retained comment set as appropriate, plus complete model equality and source ownership.
+
 No native CUA, editor, server, project-directory loader/save journal, port metamodel or Session changes were performed in this subsystem. Those integration milestones remain open. `docs/language.rst` sections 6.1 and 7 distinguish implemented core behavior from these pending authoring/integration requirements; the obsolete source2 grammar production is corrected to source3.