ARCADIA-Studio/harness/release-source-project.md

Modular source core candidate

Status: pure source3 subsystem implemented and tested. This is not native project-editor, filesystem-save, REST-bundle, or final human-authoring acceptance. Includes retain the current declaration syntax; a separate usability review is evaluating more concise nested authoring.

Delivered

• oa_core::source_project: serde SourceBundle { entry, files }, ProjectLimits, CompiledSourceProject, located diagnostics and derived SourceIndex.
• Shared source3 declaration parser/resolver and typed declaration writer in language.rs. Flat APIs and V1/V2 migration remain supported. Flat parsing explicitly rejects includes instead of reading a filesystem.
• Global forward references and duplicate alias/UUID detection across files, including project and placement identities. Errors refer to real file offsets; duplicates include the previous location. Engineering warning positions index each file once.
• Portable relative includes, root containment, canonical map keys, case collisions, repeated includes, cycles, missing/unreachable files, duplicate JSON file keys and aggregate admission checks.
• Generated identities are inserted into their original headers. Returned diagnostics/index address the resulting stabilized buffers. Caller drafts are never mutated.
• Canonical modules follow layers, work roots, behavior contexts and diagrams. Ports follow their owners; placements follow diagrams; relations follow their sources. Existing ownership is retained during model updates. Huge sets of independent roots consolidate by category rather than produce thousands of files.
• Model updates preserve authored comments, whitespace and surviving fields, change header/value tokens and alias uses, add new owner/root declarations and includes, and retain empty files after deletion. Every update recompiles and checks exact semantic equality before return.

Consumer interface

use oa_core::source_project::{compile_bundle, format_bundle, update_bundle, ProjectLimits};

let compiled = compile_bundle(&draft_bundle, ProjectLimits::default())?;
// Adopt compiled.decoded.project as one Session change only after success.
// Retain compiled.bundle, not the original draft with omitted UUID clauses.
let updated = update_bundle(&compiled, &edited_project, ProjectLimits::default())?;
let canonical_export = format_bundle(&edited_project);

ProjectLimits contains file/depth/edge budgets and the existing language::ParseLimits as model. Defaults: 256 files, depth 32, 1,024 include edges, 64 MiB combined UTF-8 text; other semantic counts are caller policy. A server should substitute its current aggregate model budgets and retain its encoded request cap. Input and stabilized output must both fit the supplied limits.

The entry must be a root-level filename, normally project.arcadia; fragments inherit source3 and have no project/version wrapper. Paths are relative to the including file, normalized within the virtual root. resolve_include(from, include) exposes the same pure path rule. It checks the containing path too. All actual includes must be supplied explicitly in files; no host filesystem or network is consulted.

For native discovery, inspect_includes(path, source, is_entry, limits) uses the shared strict parser before referenced files are available. It returns normalized targets and exact directive spans, excluding comments and strings. A syntax failure returns located diagnostics and no partial include list; retain the draft and surface that error rather than scanning arbitrary strings. Cross-file resolution/engineering validation remain in compile_bundle. Inspection checks one document's syntax/bytes/records/include edges; the loader must enforce aggregate admission and physical ownership while traversing.

SourceIndex.declarations maps UUID to file and complete declaration span, including the project header under its UUID. fields maps (UUID, top_level_field) to its complete value span. includes holds each directive location and normalized target. Project header, body interior, and optional complete description-field spans support targeted metadata/include edits. Errors use original draft coordinates; successful compilation returns coordinates in stabilized text.

CompiledSourceProject.decoded.migration is None for source3 bundles. Legacy import remains language::parse_with_report / persistence::decode_with_report; retain that report in the importing UI, then explicitly export the migrated model into source3 modules. Do not independently migrate fragments or fabricate a migration report during subsequent source3 compilation.

Comment-preserving update policy

The earlier whole-declaration canonicalization limitation is fixed. update_bundle derives language-specific editing positions from the existing validated AST and tokenizer. The shared V2 AST and migration behavior are unchanged; no alternate grammar, persistent syntax model or generic CST framework was added.

• Old/new canonical typed values identify semantic changes; surviving authored fields retain their exact text, field order, comments, omitted defaults and escape/number spelling. Header and scalar changes replace only the affected tokens. Reference aliases change without removing comments between @ and the alias.
• New fields insert at the owning record using existing indentation/line endings. Omitted defaults stay omitted unless their semantic value changes. Constructor tag changes preserve shared nested fields.
• Removing a field, constructor or declaration erases its significant syntax tokens, retaining comments/whitespace as annotations in the parent record or file. Complete value-shape replacements retain the old value's comments next to the replacement. The updater does not guess how orphaned engineering notes should be reworded.
• Lists match unique referenced UUIDs, message-value field references, binding referenced identities, or literal values. Retained entries move with their inter-separator trivia; removed-entry comments remain in the enclosing list. Other/duplicate-key lists use positions. Trivia after a comma belongs to the next entry; trivia before a comma belongs to the previous entry. Separators are emitted before trailing line comments to keep syntax valid.
• Declaration identity is UUID-based; no alias/name heuristic silently treats a new UUID as an old record. Explicit UUID replacement removes/creates records while retaining removed-record comments as file annotations. Existing UUIDs and file ownership remain unchanged for retained records.
• format_bundle remains an explicit canonical export, not the update/save path. Keep compiled workspace snapshots for undo/redo; model-only undo cannot recover prior omissions or authored layout choices.

This production core correction does not itself establish complete human-authoring, project-editor or native save acceptance. Experimental nested grammar remains separate and unintegrated.

Verification

• cargo test -p oa-core: 160 tests passed, including 30 source-project tests.
• cargo clippy -p oa-core --all-targets -- -D warnings: passed.
• Owned Rust files formatted with rustfmt --edition 2024 --config skip_children=true.

The source-project tests cover cross-file typed-port/exchange references, Unicode syntax/structural diagnostics, duplicate declarations with related locations, include diamonds/cycles/missing/unreachable files, portable paths, aggregate byte/object/record/step/file/depth/edge budgets, JSON duplicate keys, UUID stabilization and adjacent tokens, declaration relocation, alias propagation, owner inheritance, new module includes, retained empty files, stale/invalid update rejection, canonical case collisions and bounded large-root output. Full demo equality covers behavior, artifacts, trace baselines and placements. Moving a placement changes only its owning view document.

Ten additional regressions verify exact CRLF/name/guard/project-metadata updates, Unicode escape spelling and comments after @, omitted defaults and field insertion, kind/constructor changes with removed-field comments, scalar/record value-shape replacements, reference and literal list changes, deleted-record comments, and nested scenario message-value edits/reordering by field identity. These tests check exact unchanged bytes or the retained comment set as appropriate, plus complete model equality and source ownership.

No native CUA, editor, server, project-directory loader/save journal, port metamodel or Session changes were performed in this subsystem. Those integration milestones remain open. docs/language.rst sections 6.1 and 7 distinguish implemented core behavior from these pending authoring/integration requirements; the obsolete source2 grammar production is corrected to source3.