Verify compact source editing in the native application
This commit is contained in:
@@ -0,0 +1,86 @@
|
||||
# Compact source native workflow evidence
|
||||
|
||||
This package preserves two completed Windows native runs of the default-omission
|
||||
followup, including the original screenshots, window/accessibility metadata,
|
||||
process receipts, producers, input fixture, and both saved source bundles.
|
||||
Run the read-only archive check from any checkout with Python 3:
|
||||
|
||||
```text
|
||||
python harness/experiments/default-native-20260913/verify.py
|
||||
```
|
||||
|
||||
The verifier reads ZIP members in memory. It does not extract files, run the
|
||||
archived programs, build code, or modify the checkout. `files.json` pins the other
|
||||
three outer files; `capture-files.json` pins every original payload inside the ZIP.
|
||||
These hashes establish internal consistency, not cryptographic attestation.
|
||||
|
||||
## Source and executable provenance
|
||||
|
||||
Both native runs record the same 115,056,640-byte `arcadia-studio.exe`, SHA-256
|
||||
`4070927256414a770c5fbcedfaab922810cea33d9f07610b81e9ab8c0465da44`.
|
||||
Their build receipt and the original build stdout/stderr, producer, and complete
|
||||
613-file executed-input snapshot are included under `build/`. These are copied
|
||||
unchanged from the
|
||||
[published consumer package](../default-consumers-20260913/README.md) at commit
|
||||
`94383107153cccb32f3fdcbd8111b6381f3e15aa`.
|
||||
The actual build recorded HEAD `df2e1cd65ca5e56f47669e04a9d4bcb426900fcd`
|
||||
plus the HTTP test subsequently published in `9438310`; the captured source bytes,
|
||||
rather than HEAD alone, identify the tested input. Production code is unchanged
|
||||
from `7bb366f66a8c6f8b6f66fc9b04c9e3b39f84b78e`.
|
||||
No executable is distributed in this evidence package.
|
||||
|
||||
Both native process receipts close with exit code zero and empty stdout/stderr.
|
||||
Trial 2 additionally pins its producer at launch and its immutable baseline ZIP.
|
||||
Trial 1's runner and the screenshot helper are preserved as original local
|
||||
producers but were not contemporaneously hashed by that trial's receipt.
|
||||
Do not mistake a packaging-time hash for stronger historical provenance.
|
||||
The helper initially selects `trial-1`; the interactive session changed its
|
||||
output directory to `trial-2` for the continuation. Original producer bytes are
|
||||
preserved, including their historical machine-specific paths.
|
||||
|
||||
## Workflow and evidence boundaries
|
||||
|
||||
Trial 1 opened the minimal source fixture, created one activity through Model
|
||||
Explorer, inspected its compact source, and saved the resulting two-file bundle.
|
||||
The important frames are `13-created-activity`, `17-compact-new-activity`, and
|
||||
`27-cleanup-save-submitted`. Intermediate dialog focus and retry captures remain
|
||||
in the archive; the trial was closed before the cleanup interruption.
|
||||
|
||||
Trial 2 reopened that baseline and dragged Work. Its Text view showed x=190,
|
||||
model Undo showed x=120, and model Redo showed x=190 again. It then used Save As,
|
||||
cleared the current model, reopened the saved result, and inspected both source
|
||||
files. Review these frames with their adjacent JSON metadata:
|
||||
|
||||
- `03-baseline-loaded`, `04-drag-work`, `05-text-after-move`.
|
||||
- `06-model-actions`, `07-undo-model`, `08-redo-menu`, `09-redo-model`.
|
||||
- `17-save-complete`, `19-clear-current-model`, `22-reopened-root`,
|
||||
`25-reopened-compact-activity`.
|
||||
|
||||
Screenshot interpretation and the intermediate undo/redo observations require
|
||||
human review. The archive verifier checks their inventory, metadata, timing and
|
||||
byte hashes; it does not infer visible behavior from frame names or run OCR.
|
||||
The machine-checkable before/after oracle independently requires exact bundle
|
||||
membership and bytes: one compact new activity with stable UUID
|
||||
`1e1da408-da79-42ab-a990-9d00b7dc2631`, one root include, and the sole subsequent
|
||||
change `position { x 120.0; y 80.0; };` to
|
||||
`position { x 190.0; y 80.0; };`. The explicit
|
||||
`port_anchors [] ; // Keep explicit automatic placement.` survives exactly.
|
||||
|
||||
`native/bundle-checks/` preserves both closed CLI checks and their original
|
||||
`native/check-saved-bundles.py` producer. Both full bundles parsed and validated,
|
||||
with four expected draft warnings (each activity is unallocated and lacks behavior
|
||||
context), no errors, no IDs requiring stabilization, and no source changes. These
|
||||
checks do not compare the entire parsed model. After explicit extraction to a
|
||||
chosen directory, a rebuilt CLI can check `native/saved-before/project.arcadia`
|
||||
and `native/saved-after/project.arcadia`; the verification script itself never
|
||||
extracts or executes them.
|
||||
|
||||
On reopen the diagram fits/recenters, so screen coordinates cannot prove the
|
||||
persisted x value; the reopened Text frame supplies that observation. Save dialog
|
||||
frames 14–16 retain a filename-selection automation quirk: text first appended,
|
||||
then was visibly corrected before submission. No incorrect path was submitted.
|
||||
|
||||
This is evidence for one ordinary source/graphical workflow on Windows. It is
|
||||
not an exhaustive usability assessment, Linux result, performance experiment,
|
||||
packaged application verification, or full release acceptance. See the linked
|
||||
consumer and core evidence for broader automated coverage.
|
||||
Binary file not shown.
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"evidence.zip": {
|
||||
"bytes": 5251658,
|
||||
"sha256": "0eb6f2a566deff1275b19b8c5718ebb56f9afde0518ac65c09d885a1d6e84d68"
|
||||
},
|
||||
"README.md": {
|
||||
"bytes": 4964,
|
||||
"sha256": "713c7ccddcffba87c38327ee39e4a82cdff42fe8adb9ce91d3affa26a01d75e4"
|
||||
},
|
||||
"verify.py": {
|
||||
"bytes": 13770,
|
||||
"sha256": "4ea03bbd274e731f4e3856a5965be29a93011c31dc26c62ae09784e7f7b75b27"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,172 @@
|
||||
"""Read-only verification of frozen native receipts and exact saved source bytes."""
|
||||
from pathlib import Path, PurePosixPath
|
||||
from datetime import datetime
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import zipfile
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
HEAD = "df2e1cd65ca5e56f47669e04a9d4bcb426900fcd"
|
||||
GUI = {"bytes": 115056640, "sha256": "4070927256414a770c5fbcedfaab922810cea33d9f07610b81e9ab8c0465da44"}
|
||||
CLI = {"bytes": 20608512, "sha256": "d89aba9f10b598926dfdc5c644474ae891b753e8583a184860e941a359ed0eea"}
|
||||
BUILD = {"bytes": 5538, "sha256": "25c7878f56395ed1e8e6b3532fd9135e0b313fd2b781a4c982f9f2b01a4ec64b"}
|
||||
INPUTS = {"bytes": 102567, "sha256": "6bf74c1f68431a034681b754617c39d387783f71354c9106649aec9480e4f29f"}
|
||||
SOURCES = {"bytes": 2483742, "sha256": "e20f33dc363850a9289103a7be28ad613c3fffb2039061957a0186cac72d541b"}
|
||||
ACTIVITY_PATH = "oa/activities/activity_new_activity.arcadia"
|
||||
ACTIVITY = b'\noa_activity new_activity "New activity" id "1e1da408-da79-42ab-a990-9d00b7dc2631" {\n}\n\n\n'
|
||||
FIXTURE = b'''arcadia 7;
|
||||
project p "Compact source workflow" id "00000000-0000-4000-8000-000000000001" {
|
||||
oa_activity work "Work" id "00000000-0000-4000-8000-000000000002" {}
|
||||
diagram view "Overview" id "00000000-0000-4000-8000-000000000003" {}
|
||||
node placement id "00000000-0000-4000-8000-000000000004" {
|
||||
diagram @view; element @work;
|
||||
position { x 120.0; y 80.0; };
|
||||
port_anchors [] ; // Keep explicit automatic placement.
|
||||
}
|
||||
}
|
||||
'''
|
||||
LABELS = {
|
||||
"trial-1": "01-home 02-open-dialog 03-open-file-field 05-open-after-uia-error 06-open-focused 07-open-path-typed 08-project-opened 09-project-settled 10-open-keyboard-ready 11-open-submitted 12-model-explorer 13-created-activity 14-generated-source 15-source-folder 16-activity-source-row 17-compact-new-activity 18-save-before-request 19-save-before-dialog 20-cleanup-save-dialog 21-cleanup-filename-focus 22-cleanup-filename-caret 23-cleanup-save-path 24-cleanup-save-path-settled 25-cleanup-filename-refocus 26-cleanup-save-path-retry 27-cleanup-save-submitted".split(),
|
||||
"trial-2": "01-home 02-open-baseline 03-baseline-loaded 04-drag-work 05-text-after-move 06-model-actions 07-undo-model 08-redo-menu 09-redo-model 10-save-as-request 11-save-dialog 12-save-filename-focus 13-select-filename 14-enter-save-path 15-correct-path-selection 16-save-path-corrected 17-save-complete 18-home-reopen 19-clear-current-model 20-empty-home 21-reopen-request 22-reopened-root 23-open-source-folder 24-activity-source-list 25-reopened-compact-activity".split(),
|
||||
}
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise SystemExit(message)
|
||||
|
||||
|
||||
def digest(data):
|
||||
return {"bytes": len(data), "sha256": hashlib.sha256(data).hexdigest()}
|
||||
|
||||
|
||||
def membership(archive, expected, label):
|
||||
names = archive.namelist()
|
||||
require(len(names) == len(set(names)) and set(names) == set(expected), label + ": ZIP membership mismatch")
|
||||
require(all(not PurePosixPath(n).is_absolute() and ".." not in PurePosixPath(n).parts and "\\" not in n and ":" not in n for n in names), label + ": unsafe ZIP member")
|
||||
|
||||
|
||||
def time(value):
|
||||
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
require(parsed.tzinfo is not None, "Timestamp missing timezone")
|
||||
return parsed
|
||||
|
||||
|
||||
outer = json.loads((ROOT / "files.json").read_text(encoding="utf-8"))
|
||||
require(set(outer) == {"README.md", "verify.py", "evidence.zip"}, "Outer manifest membership mismatch")
|
||||
require({p.name for p in ROOT.iterdir()} == set(outer) | {"files.json"}, "Outer directory membership mismatch")
|
||||
for name, pin in outer.items():
|
||||
require(digest((ROOT / name).read_bytes()) == pin, "Outer hash mismatch: " + name)
|
||||
|
||||
with zipfile.ZipFile(ROOT / "evidence.zip") as z:
|
||||
def read(name):
|
||||
return z.read(name)
|
||||
|
||||
def obj(name):
|
||||
return json.loads(read(name))
|
||||
|
||||
pins = obj("capture-files.json")
|
||||
membership(z, set(pins) | {"capture-files.json"}, "Evidence")
|
||||
for name, pin in pins.items():
|
||||
require(digest(read(name)) == pin, "Payload hash mismatch: " + name)
|
||||
|
||||
build = obj("build/result.json")
|
||||
require(digest(read("build/result.json")) == BUILD, "Original build receipt mismatch")
|
||||
require(build["head"] == HEAD and build["exitCode"] == 0 and build["inputsUnchanged"] is True, "Build closure mismatch")
|
||||
require(build["inputs"] == INPUTS and build["archive"] == SOURCES, "Build source identity mismatch")
|
||||
require(all(build.get(k) == v for k, v in obj("build/started.json").items()), "Build start/result mismatch")
|
||||
require(time(build["start"]) <= time(build["finish"]), "Build chronology mismatch")
|
||||
require(build["command"] == ["cargo", "build", "--locked", "--offline", "--workspace", "--bins", "--message-format=json"], "Build command mismatch")
|
||||
for key, name in [("inputs", "inputs.json"), ("archive", "inputs.zip"), ("helper", "gate-policy.py"), ("stdout", "stdout.log"), ("stderr", "stderr.log")]:
|
||||
require(digest(read("build/" + name)) == build[key], "Build payload mismatch: " + name)
|
||||
source_pins = obj("build/inputs.json")
|
||||
require(len(source_pins) == build["inputCount"] == 613, "Source count mismatch")
|
||||
with zipfile.ZipFile(io.BytesIO(read("build/inputs.zip"))) as sources:
|
||||
membership(sources, source_pins, "Source")
|
||||
for name, pin in source_pins.items():
|
||||
require(digest(sources.read(name)) == pin, "Source blob mismatch: " + name)
|
||||
build_events = [json.loads(line) for line in read("build/stdout.log").decode().splitlines() if line.startswith("{")]
|
||||
require([e.get("success") for e in build_events if e.get("reason") == "build-finished"] == [True], "Build success event mismatch")
|
||||
require([e for e in build_events if e.get("reason") == "compiler-artifact" and e.get("executable")] == [a["cargo"] for a in build["artifacts"]], "Build artifact log mismatch")
|
||||
binaries = {a["cargo"]["target"]["name"]: a for a in build["artifacts"]}
|
||||
require(binaries["arcadia-studio"]["readback"] == GUI and binaries["arcadia"]["readback"] == CLI, "Native binary build identity mismatch")
|
||||
|
||||
expected = {"build/" + n for n in ["result.json", "started.json", "stdout.log", "stderr.log", "inputs.json", "inputs.zip", "gate-policy.py"]}
|
||||
expected |= {"native/" + n for n in ["run-native.py", "run-native-followup.py", "capture-snippet.js", "check-fixture.py", "verify-creation-checkpoint.py", "fixture-check.result.json", "fixture-check.stdout.json", "fixture-check.stderr.log", "input/project.arcadia"]}
|
||||
before = FIXTURE[:-2] + b'\n include "oa/activities/activity_new_activity.arcadia";\n}\n'
|
||||
after = before.replace(b"position { x 120.0; y 80.0; };", b"position { x 190.0; y 80.0; };")
|
||||
require(read("native/input/project.arcadia") == FIXTURE, "Input fixture bytes mismatch")
|
||||
for label, root in [("saved-before", before), ("saved-after", after)]:
|
||||
expected |= {f"native/{label}/project.arcadia", f"native/{label}/{ACTIVITY_PATH}"}
|
||||
require(read(f"native/{label}/project.arcadia") == root and read(f"native/{label}/{ACTIVITY_PATH}") == ACTIVITY, label + ": exact source construction mismatch")
|
||||
|
||||
previous_finish = time(build["finish"])
|
||||
for trial, labels in LABELS.items():
|
||||
prefix = "native/" + trial + "/"
|
||||
expected |= {prefix + n for n in ["started.json", "result.json", "process.json", "stdout.log", "stderr.log"]}
|
||||
result, started, process = (obj(prefix + n + ".json") for n in ["result", "started", "process"])
|
||||
require(all(result.get(k) == v for k, v in started.items()), trial + ": start/result mismatch")
|
||||
require(result["exitCode"] == 0 and result["binary"] == result["binaryAfter"] == GUI, trial + ": process/binary closure mismatch")
|
||||
require(result["buildReceipt"] == BUILD and result["buildHead"] == HEAD and result["buildInputs"] == INPUTS, trial + ": build provenance mismatch")
|
||||
require(result["command"] == [binaries["arcadia-studio"]["cargo"]["executable"]], trial + ": launch command mismatch")
|
||||
require(previous_finish <= time(result["start"]) <= time(process["observed"]) <= time(result["finish"]), trial + ": chronology mismatch")
|
||||
require(isinstance(process["pid"], int) and process["pid"] > 0, trial + ": PID mismatch")
|
||||
for stream in ["stdout", "stderr"]:
|
||||
require(result[stream] == digest(read(prefix + stream + ".log")) == digest(b""), trial + ": unexpected process output")
|
||||
previous_capture = time(result["start"])
|
||||
for label in labels:
|
||||
expected.add(prefix + label + ".json")
|
||||
frame = obj(prefix + label + ".json")
|
||||
require(previous_capture <= time(frame["time"]) <= time(result["finish"]), trial + ": capture chronology mismatch")
|
||||
previous_capture = time(frame["time"])
|
||||
require(frame["window"]["id"] and (isinstance(frame["accessibility"], dict) or (trial, label, frame["accessibility"]) == ("trial-1", "08-project-opened", None)), trial + ": capture metadata mismatch")
|
||||
require(len(frame["images"]) in [1, 2], trial + ": image inventory mismatch")
|
||||
for i, image in enumerate(frame["images"]):
|
||||
name = f"{label}-{i}.jpg"
|
||||
require(image["file"] == name and image["width"] > 0 and image["height"] > 0, trial + ": image metadata mismatch")
|
||||
require(read(prefix + name).startswith(b"\xff\xd8"), trial + ": JPEG bytes mismatch")
|
||||
expected.add(prefix + name)
|
||||
previous_finish = time(result["finish"])
|
||||
if trial == "trial-2":
|
||||
expected.add(prefix + "baseline.zip")
|
||||
require(result["helper"] == digest(read("native/run-native-followup.py")), "Trial 2 producer pin mismatch")
|
||||
require(result["baselineUnchanged"] is True and result["baselineArchive"] == digest(read(prefix + "baseline.zip")), "Baseline receipt mismatch")
|
||||
require(result["baseline"] == {"project.arcadia": digest(before), ACTIVITY_PATH: digest(ACTIVITY)}, "Baseline source pins mismatch")
|
||||
with zipfile.ZipFile(io.BytesIO(read(prefix + "baseline.zip"))) as baseline:
|
||||
membership(baseline, result["baseline"], "Baseline")
|
||||
require(baseline.read("project.arcadia") == before and baseline.read(ACTIVITY_PATH) == ACTIVITY, "Launch baseline bytes mismatch")
|
||||
|
||||
fixture = obj("native/fixture-check.result.json")
|
||||
require(fixture["exitCode"] == 0 and fixture["binary"] == CLI, "Fixture CLI receipt mismatch")
|
||||
require(fixture["fixtureBefore"] == fixture["fixtureAfter"] == digest(FIXTURE), "Fixture CLI source mismatch")
|
||||
for key, suffix in [("stdout", "stdout.json"), ("stderr", "stderr.log")]:
|
||||
require(fixture[key] == digest(read("native/fixture-check." + suffix)), "Fixture CLI stream mismatch")
|
||||
require(read("native/fixture-check.stderr.log") == b"", "Fixture CLI stderr was not empty")
|
||||
expected.add("native/check-saved-bundles.py")
|
||||
expected.add("native/bundle-checks/summary.json")
|
||||
producer = digest(read("native/check-saved-bundles.py"))
|
||||
require(producer == {"bytes": 4238, "sha256": "b1116c3c5227fad2d99d0d4ab421e0840a09120b1aedfa1fd4364b25787450cc"}, "Saved-bundle producer pin mismatch")
|
||||
summary = obj("native/bundle-checks/summary.json")
|
||||
require(summary["status"] == "PASS" and summary["runs"] == ["saved-before", "saved-after"] and summary["producer"] == producer, "Bundle summary mismatch")
|
||||
diagnostics = [{"severity": "warning", "code": code, "object_id": identity}
|
||||
for code in ["unallocated_leaf", "work_without_behavior_context"]
|
||||
for identity in ["00000000-0000-4000-8000-000000000002", "1e1da408-da79-42ab-a990-9d00b7dc2631"]]
|
||||
for label, source in [("saved-before", before), ("saved-after", after)]:
|
||||
prefix = "native/bundle-checks/" + label
|
||||
expected |= {prefix + suffix for suffix in [".result.json", ".stdout.json", ".stderr.log"]}
|
||||
result, report = obj(prefix + ".result.json"), obj(prefix + ".stdout.json")
|
||||
entry = "C:\\ar-default-9e8r6t\\.local\\default-omission-native\\" + label + "\\project.arcadia"
|
||||
require(result["command"] == [binaries["arcadia"]["cargo"]["executable"], "check", entry, "--json"], label + ": CLI command mismatch")
|
||||
require(result["exitCode"] == 0 and result["binary"] == CLI and result["buildReceipt"] == BUILD and result["producer"] == producer, label + ": CLI provenance mismatch")
|
||||
require(result["filesUnchanged"] is True and result["files"] == {"project.arcadia": digest(source), ACTIVITY_PATH: digest(ACTIVITY)}, label + ": CLI input mismatch")
|
||||
require(previous_finish <= time(result["start"]) <= time(result["finish"]), label + ": CLI chronology mismatch")
|
||||
previous_finish = time(result["finish"])
|
||||
require(result["stdout"] == digest(read(prefix + ".stdout.json")) and result["stderr"] == digest(read(prefix + ".stderr.log")) == digest(b""), label + ": CLI streams mismatch")
|
||||
require(report["entry"] == entry and report["operation"] == "check" and report["status"] == "checked" and report["exit_code"] == 0, label + ": CLI status mismatch")
|
||||
require(report["written"] is False and report["changes"] == [] and report["identities_to_stabilize"] == 0, label + ": CLI changed source or identities")
|
||||
require(result["diagnostics"] == [{k: d[k] for k in ["severity", "code", "object_id"]} for d in report["diagnostics"]] == diagnostics, label + ": expected draft diagnostics mismatch")
|
||||
require(set(pins) == expected, "Exact payload inventory mismatch")
|
||||
|
||||
print("Verified closed native trials, unchanged verified GUI, original captures, 613 build inputs, and exact saved-bundle bytes.")
|
||||
print("Intermediate GUI behavior requires screenshot review; no Linux, performance, packaging, or full-release acceptance claim.")
|
||||
Reference in New Issue
Block a user