feat: preserve nested source projects across desktop and REST

docs/server.md

@@ -1,6 +1,6 @@
 # Arcadia Studio server
 
-`arcadia-server` is a headless Rust REST service for the same schema-2 model, commands, source language, validation, artifacts and OA/SA traces used by the desktop. It stores each current project in an atomic JSON envelope. The service provides a deployment artifact and configuration examples; this repository does not assert that a remote instance has been deployed.
+`arcadia-server` is a headless Rust REST service for the same schema-3 model, commands, source language, validation, artifacts and OA/SA traces used by the desktop. It stores each model and its authored source project together in an atomic JSON envelope. The service provides a deployment artifact and configuration examples; this repository does not assert that a remote instance has been deployed.
 
 The complete HTTP contract is [api.rst](api.rst); machine-readable schemas are in [openapi.json](openapi.json), also served at `/api/v1/openapi.json`. The original [service research](research-v2-service.md) discusses broader alternatives; retained history, SQLite, idempotency receipts and stored transition plans from that research are **not** the implemented storage contract.
 
@@ -34,17 +34,19 @@ curl --fail-with-body -i -X POST http://127.0.0.1:8787/api/v1/projects \
   -H 'Content-Type: application/json' -d '{"name":"Harbor assistance"}'
 ```
 
-The create response returns a service `resource_id`, revision and `ETag`. Fetch `/api/v1/projects/{resource_id}` to obtain the full `{resource_id, revision, model}` envelope. Use the exact ETag as `If-Match` on updates. With authentication enabled, add `Authorization: Bearer <your-token>`; avoid putting the token in URLs, shell history or shared example files.
+The create response returns a service `resource_id`, revision and `ETag`. Fetch `/api/v1/projects/{resource_id}` to obtain the full `{resource_id, revision, model, source_project}` envelope. Use the exact ETag as `If-Match` on updates. With authentication enabled, add `Authorization: Bearer <your-token>`; avoid putting the token in URLs, shell history or shared example files.
 
 The service resource ID is distinct from the editable model ID. Model IDs, aliases, diagrams, artifact URIs/revisions and trace records round-trip through source/JSON. Imported copies receive new resource IDs. An artifact's `revision` is an opaque external-artifact revision string; it is not a retained server model revision.
 
 ## Storage and concurrency
 
-The data directory contains `.arcadia.lock` and one `<resource-id>.json` file per hosted project. Each envelope has exactly `resource_id`, positive `revision` and `model`. The process keeps an exclusive OS file lock on the directory lock file for the entire store lifetime. A second process using the same directory fails at startup. Do not directly edit, synchronize or replace files while the service is running.
+The data directory contains `.arcadia.lock` and one `<resource-id>.json` file per hosted project. Each envelope has exactly `resource_id`, positive `revision`, `model` and `source_project`. The source bundle contains its entry filename and a map of file text; compiling it must reproduce the stored model exactly. The process keeps an exclusive OS file lock on the directory lock file for the entire store lifetime. A second process using the same directory fails at startup. Do not directly edit, synchronize or replace files while the service is running.
 
 Startup validates every committed envelope, its filename/UUID, revision, size and complete model. Any corrupt or unsupported envelope stops startup; the service does not silently skip bad data or invent an empty replacement. Sibling `.arcadia-write-*` files from an interrupted pre-commit write are not committed models and are ignored on startup.
 
-Parsing, validation and blocking disk operations run on blocking workers. A shared store mutex captures snapshots and serializes final revision checks and writes. Parsing, semantic validation and query processing use the captured snapshot outside that lock; an expensive candidate does not block the catalog or another model's preparation. Commit rechecks the exact ETag, so a candidate prepared against a superseded snapshot returns `412 revision_conflict`. A successful changed update increments revision once; an unchanged model returns `changed: false` and retains its revision. Each write serializes the whole envelope to a sibling temporary file, syncs it, and atomically replaces the destination. Unix also syncs the directory. The in-memory snapshot changes only after the replacement succeeds. A directory-sync error after replacement is reported as an uncertain write outcome while the in-memory head still matches the installed file.
+Earlier model-only envelopes are accepted through the existing model migrations and receive deterministic source files in memory. Opening, reading or an exact no-op neither rewrites those envelopes nor advances their revision. The next changed edit persists the coupled representation. Already-coupled envelopes must contain stable source IDs and matching model/source; inconsistent pairs are rejected at startup. Capacity accounting includes source generated in memory for old envelopes, so an over-capacity old store fails explicitly without rewriting or pruning it.
+
+Parsing, validation and blocking disk operations run on blocking workers. A shared store mutex captures snapshots and serializes final revision checks and writes. Parsing, semantic validation, source writeback and query processing use the captured snapshot outside that lock; an expensive candidate does not block the catalog or another model's preparation. Commit rechecks the exact ETag, so a candidate prepared against a superseded snapshot returns `412 revision_conflict`. A successful changed update increments revision once. Both model and source bundle must be unchanged for `changed: false`; comment-only edits advance the same revision. Each write serializes the whole envelope to a sibling temporary file, syncs it, and atomically replaces the destination. Unix also syncs the directory. The in-memory snapshot changes only after the replacement succeeds. A directory-sync error after replacement is reported as an uncertain write outcome while the in-memory head still matches the installed file.
 
 Atomic replacement protects against partially written envelopes. It does not promise survival of every device/filesystem power-loss failure, protect against out-of-band edits, or coordinate OneDrive/network filesystems. The storage directory must be local to this one process. DELETE permanently removes the envelope; there is no recycle bin, tombstone or server undo history.
 
@@ -54,14 +56,15 @@ Back up by stopping the server cleanly and copying committed JSON envelopes to a
 
 ## Limits and errors
 
-- Raw request bodies: 8 MiB, including requests without `Content-Length`. Encoded/compressed request bodies are rejected. The same limit applies to serialized model JSON; the storage envelope permits small framing overhead.
+- Raw request bodies: 8 MiB, including requests without `Content-Length`. Encoded/compressed request bodies are rejected. Serialized model JSON and serialized source-bundle JSON each have an 8 MiB cap. The latter is checked again after omitted IDs are inserted. The storage envelope allows both representations plus small framing overhead.
+- Source bundles contain at most 256 files, with at most 32 include edges along a path and 1,024 include edges overall. Includes resolve only against submitted logical `.arcadia` paths; no server filesystem or network access is involved. All files must be reachable, and cycles, duplicate keys, case collisions and escaping paths are rejected.
 - At most eight admitted model requests and eight active/waiting blocking workers. Overload returns `503` with `Retry-After: 1`. A 30-second request deadline bounds body receipt and response waiting. A started blocking write can finish after that deadline; `504 outcome_unknown` means reload before retrying.
 - At most 5,000 elements plus relations, 20,000 total persistent model records, and 50,000 ordered step references per model. Graph requests allow 1–100 roots, depth 0–4, and at most 1,000 returned objects. List pages contain 1–1,000 records, default 100.
 - Source declarations and decoded JSON records are counted before structural validation. V1 import includes the occurrences and placements migration will generate in its early budget check. These limits apply to import, source/model replacement, candidate validation and full-model transaction commands. Over-budget candidates cannot monopolize the catalog while being rejected.
-- Transactions accept at most 200 commands, including nested batch wrappers. All commands apply to a candidate, then shared validation accepts or rejects the batch atomically. Dry run returns the candidate and diagnostics without writing.
+- Transactions accept at most 200 commands, including nested batch wrappers. All commands apply to a candidate, then shared validation and authored-source writeback accept or reject the batch atomically. Dry run returns the proposed model, source project and diagnostics without writing.
 - The fixed store cap is 128 hosted models and 256 MiB of serialized envelopes. New/imported/replaced models that exceed it return `507 store_capacity`. Startup enforces the same limits. Delete/export unneeded projects before adding more data; there are no hidden eviction or pruning rules. The server stores all current models in memory; memory includes Rust object overhead and in-flight candidates in addition to serialized bytes.
 
-Errors use `application/problem+json` with stable `code`, HTTP `status`, human `detail`, and contextual `issues` when available. Responses carry `X-Request-ID`; configured bearer credentials and source/model bodies are not logged. The core currently supplies human command errors; the API maps them to `422 command_rejected` without parsing English messages into invented machine codes. Source errors preserve line/column diagnostics.
+Errors use `application/problem+json` with stable `code`, HTTP `status`, human `detail`, and contextual `issues` when available. Responses carry `X-Request-ID`; configured bearer credentials and source/model bodies are not logged. The core currently supplies human command errors; the API maps them to `422 command_rejected` without parsing English messages into invented machine codes. Source-project errors preserve logical file paths, UTF-8 byte spans, line/column and related-file locations.
 
 ## Linux deployment
 
@@ -86,6 +89,10 @@ Ctrl+C/SIGTERM stops admission and drains HTTP requests. Already-started blockin
 
 A desktop or automation client can query current JSON/source, edit with shared typed commands, validate/dry-run, and commit using the original ETag. For hosted editing, keep the user's draft until the service acknowledges the new revision. On `412`, preserve the draft and reload server state for review; a blind replacement retry would discard another client's work.
 
+Use `POST /api/v1/projects/import-source-project` to import `{"entry":"model.arcadia","files":{"model.arcadia":"..."}}`. Use `GET` and conditional `PUT /api/v1/projects/{id}/source-project` to export or edit that complete authored workspace. These routes preserve comments, line endings, entry names and module layout. Missing declaration IDs are inserted once; keep the accepted bundle returned by PUT or subsequent GET. Candidate validation accepts `{"format":"source-project","source_project":{...}}` and reports diagnostics without adoption.
+
+Typed commands, renames and transitions update the same authored files using command identity correspondence, including UUID swaps and delete/recreate operations. Model JSON and canonical single-file source replacements preserve the existing authored layout using common UUIDs. Canonical `/source` remains an interchange view; use `/source-project` to retain or edit file layout and comments. Source writeback failure rejects the whole candidate instead of discarding the authored files. Comment-only edits share the same ETag as semantic edits, preventing one client from overwriting another client's source work.
+
 External tools can use the artifact-link and trace query routes, and create/edit those records in the same atomic transaction as model changes. URI and external revision fields preserve the author's values; the service supplies no fetching, credentials, artifact-content verification or AI execution runtime. OpenAPI is the integration contract. An optional future adapter can call this API without introducing a second mutation path.
 
 `POST /projects/{id}/transitions/preview` produces a shared core transition plan and `base_revision` without storing a plan or changing the model. Send that response to `/transitions/apply` with the original ETag. Apply checks the server revision and the core's model-change fingerprint, preserves planned IDs, validates the candidate, and commits once. Plans with unresolved conflicts cannot apply. A submitted plan is a client-owned command proposal, not a signed server authorization; a client able to apply a plan already has the same write permission as the transaction endpoint.
@@ -102,6 +109,7 @@ The HTTP integration suite starts real loopback listeners over isolated temporar
 
 Verified on 2026-09-05:
 
+- Windows authored-source service extension: nine new real HTTP tests and one OpenAPI contract test pass, covering exact multi-file/comment round trips, comment-only ETags, ID insertion, source/model parity, command provenance through UUID swaps and equal-model delete/recreate, dry runs, transitions, invalid dependencies, concurrent writes, restart migration and forced source-write failure. The source-byte accounting unit passes; the focused `source_` regression selection passes all 14 HTTP/schema tests. Strict server all-targets Clippy passes. These are local Windows results; Linux and remote deployment remain separate checks.
 - Windows R1/R2/R4 source/API integration: all 16 current HTTP/configuration tests plus the deterministic commit test pass; strict server all-targets Clippy passes. New cases cover mission maps, branching/reused chains, typed call value edits, original V2 compatibility, source/JSON equality and persistence after restart. See `harness/release-language-integration.md`; native UI and remaining release semantics are separate gates.
 - Windows R2 release checkpoint: all 14 server integration/configuration tests plus one deterministic commit test pass; formatting and strict all-targets Clippy pass. A separate optimized executable probe rejects the previously stalling 12,000-activity request in 0.050 seconds; concurrent JSON import and accepted-source preparation leave catalog reads responsive. See `harness/release-text-service-audit.md` for exact measurements and scope.
 - Ubuntu 24.04 under WSL, prior V2 baseline: all 12 then-existing tests passed using native Linux processes and temporary filesystem stores; strict all-targets Clippy and the optimized build passed with the locked, vendored dependency set. A separate release-executable smoke run verified bearer authentication, non-loopback token enforcement, exclusive directory locking, conditional create/update/delete, update and deletion persistence across restarts, served OpenAPI, and clean SIGTERM shutdown. The new first-release changes still require native Linux re-verification.