Compare commits

..
17 changed files with 755 additions and 2549 deletions
+10 -10
View File
@@ -35,26 +35,26 @@ Use Gitea for live coordination and the relevant code version for implementation
## Coordination and integration
All independent agents coordinating through Gitea run on completely separate machines, each with its own clone, filesystem, tools, processes, and runtime. Gitea and pushed Git history are the shared communication and delivery channels. Local files (including `harness`), unpushed commits, worktrees, test results, and localhost services are not accessible to peers. Matching branch names or paths do not imply synchronized contents.
1. On starting or resuming, including after compaction, inspect the working branch, local changes, applicable instructions, relevant shared records, and existing implementation. Recover context from authoritative records. Check for newer shared instructions on the actual remote default branch without overwriting local work. Read what is relevant; avoid repeated full-history scans and constant polling.
2. Search before creating tasks. Claim a meaningful outcome in its Gitea issue with a stable session identity, scope, remote repository/branch, and acceptance criteria. Keep small steps within that task. Recheck after claiming; comments are not atomic locks. Respect existing ownership and resolve overlapping claims before overlapping work.
3. Independent sessions are peers. Each owns its contribution and delegated work on its own machine. Record one integration/release owner when a milestone needs shared coordination. Use distinct remote work branches, partition files/interfaces, and coordinate changes to shared specifications and documents. Worktrees provide local isolation only; they do not share work between machines.
4. At meaningful checkpoints or handoffs, record changed results, commit/PR links, verification commands and environment, blockers, and next actions. Identify the tested code version and label uncommitted or unpublished work as local-only. Before asking peers to consume results, push the required code and versioned documentation, including relevant `harness` updates; publish other required artifacts in Gitea and link them. Use repository-relative paths with commit references, reproducible commands, and necessary environment details; local paths and localhost URLs alone are not handoffs. Do not assume another agent is running or has read a message; silence alone does not establish abandonment. Record evidence and preserve existing work when reassigning a task.
5. Follow project review and merge conventions. Fetch the relevant remote branches and inspect the identified commits before reviewing or integrating another machine's work. Resolve conflicts while preserving others' edits, and verify the combined result locally; report material environment differences. Local instruction-file edits reach peers only after publication and retrieval. Do not overwrite others' work or force-push shared history to resolve coordination problems.
2. Search before creating tasks. Claim a meaningful outcome in its Gitea issue with a stable session identity, scope, branch/worktree, and acceptance criteria. Keep small steps within that task. Recheck after claiming; comments are not atomic locks. Respect existing ownership and resolve overlapping claims before overlapping work.
3. Independent sessions are peers. Each owns its contribution and delegated work. Record one integration/release owner when a milestone needs shared coordination. Use separate worktrees or checkouts, partition files/interfaces, and coordinate changes to shared specifications and documents.
4. At meaningful checkpoints or handoffs, record changed results, commit/PR links, verification commands and environment, blockers, and next actions. Identify the tested code version and uncommitted or unpublished work; make required artifacts accessible to other sessions. Do not assume another agent is running or has read a message; silence alone does not establish abandonment. Record evidence and preserve existing work when reassigning a task.
5. Follow project review and merge conventions. Inspect upstream changes, resolve conflicts while preserving others' edits, and verify the combined result. Do not overwrite others' work or force-push shared history to resolve coordination problems.
6. Close work when its acceptance criteria are met and the result is integrated or delivered as appropriate. Mark duplicates and superseded tasks explicitly with replacement links. Then choose further useful unowned work while the requested objective remains incomplete.
Use Gitea MCP for shared coordination. If unavailable, report the failure, buffer necessary handoff information locally, and continue already-owned or clearly non-overlapping work. Reconcile with current Gitea state when access returns; do not claim an ownership update, review, or publication succeeded without confirmation.
## Delegation and reasoning
Use GPT-6 Astra exclusively. Gitea peers are independent remote sessions, not subagents controlled by this session. Spawned subagents use the execution environment actually provided by the agent tools; do not assume they share files, and publish their results before Gitea peers need them. Delegate bounded independent research, design, implementation, or review when it improves progress, judgment, or preserves the parent's context. Give subagents the relevant context, ownership boundaries, result format, and evidence requirements. Use fresh critics when practical. The parent owns synthesis and integration and normally consolidates results into Gitea. Choose concurrency according to useful independent work and available resources, including other sessions; there is no agent-count quota.
Use GPT-6 Astra exclusively. Delegate bounded independent research, design, implementation, or review when it improves progress, judgment, or preserves the parent's context. Give subagents the relevant context, ownership boundaries, result format, and evidence requirements. Use fresh critics when practical. The parent owns synthesis and integration and normally consolidates results into Gitea. Choose concurrency according to useful independent work and available resources, including other sessions; there is no agent-count quota.
Optimize reasoning effort for verified outcomes per cost, including review, retries, and repair:
- Main agent: default to high for planning, synthesis, integration, and acceptance decisions; medium for dispatching a settled plan and routine coordination.
- Subagents: default to medium for bounded implementation and targeted research; low for straightforward retrieval, mechanical edits, and running established checks; high for complex debugging, correctness review, adversarial analysis, or ambiguous design.
- Use extra-high selectively for difficult foundational semantic, methodological, formal, or cross-domain decisions. Reserve max for exceptional unresolved problems where additional reasoning is likely to justify its cost. Retain higher settings when evidence supports their value.
Main agent: default to high for planning, synthesis, integration, and acceptance decisions; medium for dispatching a settled plan and routine coordination.
Subagents: default to medium for bounded implementation and targeted research; low for straightforward retrieval, mechanical edits, and running established checks; high for complex debugging, correctness review, adversarial analysis, or ambiguous design.
Use extra-high selectively for difficult foundational semantic, methodological, formal, or cross-domain decisions. Reserve max for exceptional unresolved problems where additional reasoning is likely to justify its cost. Retain higher settings when evidence supports their value.
Allocate by task difficulty and consequences, not role or length alone. Start higher when a clearly difficult decision has expensive downstream consequences. Escalate for concrete reasoning difficulties; resolve missing evidence or unclear requirements at their source. Set subagent effort explicitly through supported controls to avoid unintended inheritance. Do not claim an unapplied change or alter global configuration merely to enforce these preferences.
+28 -171
View File
@@ -1,8 +1,8 @@
//! Typed OA/SA language, tolerant source analysis, and explicit V1/V2/V3/V4 migration.
mod raw_tokens;
#[path = "language/semantic_source.rs"]
mod semantic_source;
mod source3_bridge;
mod raw_tokens;
// Staged syntax foundation only. Public source8 admission waits for complete
// declaration lowering, source updates and the root-owned resource gate.
#[cfg(test)]
@@ -480,8 +480,6 @@ fn parse_document(source: &str, limits: ParseLimits) -> Result<ParsedDocument, V
cursor: 0,
value_depth: 0,
limits,
#[cfg(test)]
mixed: None,
}
.document()
.map_err(|e| vec![diagnostic(source, e)])
@@ -916,8 +914,6 @@ pub(crate) fn parse_unit(
cursor: 0,
value_depth: 0,
limits,
#[cfg(test)]
mixed: None,
}
.unit(entry, true)
.map_err(|e| Box::new(diagnostic(source, e)))
@@ -1009,52 +1005,31 @@ fn edit_positions(
unit: SourceUnit,
) -> Result<Vec<(Id, EditDeclaration)>, Box<Diagnostic>> {
let tokens = strict_tokens(source).map_err(|e| Box::new(diagnostic(source, e)))?;
edit_positions_from_tokens(source, &unit, |at| tokens[at], tokens.len(), None)
}
fn edit_positions_from_tokens(
source: &str,
unit: &SourceUnit,
token: impl Fn(usize) -> Lexeme,
count: usize,
value_ends: Option<&BTreeMap<usize, usize>>,
) -> Result<Vec<(Id, EditDeclaration)>, Box<Diagnostic>> {
let mut ends: Vec<_> = (0..count).map(|at| token(at).span.end).collect();
let mut ends: Vec<_> = tokens.iter().map(|token| token.span.end).collect();
let mut stack = Vec::new();
for index in 0..count {
let token = token(index);
for (index, token) in tokens.iter().enumerate() {
match token.text(source) {
"{" | "[" => stack.push(index),
"}" | "]" => ends[stack.pop().expect("parser checked delimiters")] = token.span.end,
_ => {}
}
}
struct Positions<'a, F> {
token_at: F,
struct Positions<'a> {
tokens: &'a [Lexeme],
ends: Vec<usize>,
value_ends: Option<&'a BTreeMap<usize, usize>>,
}
impl<F: Fn(usize) -> Lexeme> Positions<'_, F> {
impl Positions<'_> {
fn token(&self, start: usize) -> usize {
let mut low = 0;
let mut high = self.ends.len();
while low < high {
let middle = low + (high - low) / 2;
if (self.token_at)(middle).span.start < start {
low = middle + 1;
} else {
high = middle;
}
}
low
self.tokens
.partition_point(|token| token.span.start < start)
}
fn fields(&self, fields: &BTreeMap<String, Ast>) -> BTreeMap<String, EditField> {
fields
.iter()
.map(|(key, ast)| {
let value = self.value(ast);
let start = (self.token_at)(self.token(value.span.start) - 1).span.start;
let end = (self.token_at)(self.token(value.span.end)).span.end;
let start = self.tokens[self.token(value.span.start) - 1].span.start;
let end = self.tokens[self.token(value.span.end)].span.end;
(
key.clone(),
EditField {
@@ -1069,15 +1044,10 @@ fn edit_positions_from_tokens(
let index = self.token(ast.offset());
let span = Span {
start: ast.offset(),
end: self
.value_ends
.and_then(|ends| ends.get(&ast.offset()).copied())
.unwrap_or_else(|| match ast {
Ast::Reference(..) | Ast::HistoricalUuid(..) => {
(self.token_at)(index + 1).span.end
}
_ => self.ends[index],
}),
end: match ast {
Ast::Reference(..) | Ast::HistoricalUuid(..) => self.tokens[index + 1].span.end,
_ => self.ends[index],
},
};
let kind = match ast {
Ast::Object(fields, _) => EditKind::Record(self.fields(fields)),
@@ -1092,12 +1062,11 @@ fn edit_positions_from_tokens(
}
}
let positions = Positions {
token_at: &token,
tokens: &tokens,
ends,
value_ends,
};
unit.declarations
.iter()
.into_iter()
.map(|declaration| {
if declaration.id_insertion.is_some() {
return Err(Box::new(diagnostic(
@@ -1117,8 +1086,8 @@ fn edit_positions_from_tokens(
let start = positions.token(declaration.span.start);
let header = roles
.iter()
.zip(start..count)
.map(|(role, at)| (*role, token(at).span))
.zip(&tokens[start..])
.map(|(role, token)| (*role, token.span))
.collect();
Ok((
declaration.id,
@@ -1145,35 +1114,10 @@ struct Parser<'a> {
cursor: usize,
value_depth: usize,
limits: ParseLimits,
#[cfg(test)]
mixed: Option<analysis_source::mixed::State<'a>>,
}
impl Parser<'_> {
fn token(&self, index: usize) -> Option<Lexeme> {
#[cfg(test)]
if let Some(mixed) = &self.mixed {
return mixed.token(index);
}
self.tokens.get(index).copied()
}
fn current(&self) -> Lexeme {
self.token(self.cursor).unwrap_or(Lexeme {
class: TokenClass::Punctuation,
span: Span {
start: self.source.len(),
end: self.source.len(),
},
})
}
fn syntax_entry(&mut self) -> Result<(), Error> {
#[cfg(test)]
{
let span = self.current().span;
if let Some(mixed) = &mut self.mixed {
mixed.entry(span)?;
}
}
Ok(())
self.tokens[self.cursor.min(self.tokens.len() - 1)]
}
fn text(&self) -> &str {
self.current().text(self.source)
@@ -1210,10 +1154,6 @@ impl Parser<'_> {
}
fn optional_id(&mut self) -> Result<Id, Error> {
if self.text() != "id" {
#[cfg(test)]
if self.mixed.is_some() {
return Err(self.error("Private mixed syntax requires explicit IDs."));
}
return Ok(Id::new_v4());
}
self.cursor += 1;
@@ -1254,7 +1194,6 @@ impl Parser<'_> {
Ok(fields)
}
fn field(&mut self) -> Result<(String, Ast, Span), Error> {
self.syntax_entry()?;
let key = self.text().to_owned();
if !matches!(
self.current().class,
@@ -1265,19 +1204,15 @@ impl Parser<'_> {
self.cursor += 1;
let start = self.current().span.start;
let value = self.value()?;
let end = self
.token(self.cursor - 1)
.expect("consumed token")
.span
.end;
let end = self.tokens[self.cursor - 1].span.end;
self.take(";")?;
Ok((key, value, Span { start, end }))
}
fn nested_header_kind(&self) -> Option<DeclKind> {
let kind = declaration_kind(self.text())?;
let alias = self.token(self.cursor + 1)?;
let following = self.token(self.cursor + 2)?;
let alias = self.tokens.get(self.cursor + 1)?;
let following = self.tokens.get(self.cursor + 2)?;
// A node's ordinary `diagram @view;` field is not a declaration.
if !crate::valid_alias(alias.text(self.source)) {
return None;
@@ -1292,33 +1227,14 @@ impl Parser<'_> {
if self.value_depth >= 64 {
return Err(self.error("Typed values may not nest more than 64 levels."));
}
self.syntax_entry()?;
self.value_depth += 1;
let result = self.value_inner();
self.value_depth -= 1;
#[cfg(test)]
if let Ok(ast) = &result {
let end = self
.token(self.cursor - 1)
.expect("consumed value")
.span
.end;
if let Some(mixed) = &mut self.mixed {
mixed.value_ends.insert(ast.offset(), end);
}
}
result
}
fn value_inner(&mut self) -> Result<Ast, Error> {
let token = self.current();
let offset = token.span.start;
#[cfg(test)]
if let Some(mixed) = &self.mixed
&& (token.class == TokenClass::Number || matches!(self.text(), "-" | "+"))
{
let value = mixed.ordinary_number(&mut self.cursor)?;
return Ok(Ast::Literal(value, offset));
}
match self.text() {
"uuid" => {
self.cursor += 1;
@@ -1416,31 +1332,16 @@ impl Parser<'_> {
fn unit(&mut self, entry: bool, includes_allowed: bool) -> Result<SourceUnit, Error> {
let mut header = if entry {
self.take("arcadia")?;
#[cfg(test)]
let mixed_mode = self.mixed.is_some();
#[cfg(not(test))]
let mixed_mode = false;
if if mixed_mode {
self.text() != "8"
} else {
!matches!(self.text(), "3" | "4" | "5" | "6" | "7")
} {
return Err(self.error(if mixed_mode {
"Private mixed syntax requires arcadia 8."
} else {
"Unsupported schema version: expected arcadia 3, 4, 5, 6 or 7."
}));
if !matches!(self.text(), "3" | "4" | "5" | "6" | "7") {
return Err(
self.error("Unsupported schema version: expected arcadia 3, 4, 5, 6 or 7.")
);
}
let schema_version = self.text().parse().expect("checked source version");
self.cursor += 1;
self.take(";")?;
let start = self.current().span.start;
self.take("project")?;
#[cfg(test)]
if let Some(mixed) = &mut self.mixed {
mixed.record(false, Span { start, end: start })?;
}
self.syntax_entry()?;
let alias = self.identifier()?;
let name = self.quoted()?;
let id_insertion = (self.text() != "id").then_some(self.current().span.start);
@@ -1481,14 +1382,6 @@ impl Parser<'_> {
let mut open: Vec<usize> = Vec::new();
loop {
let offset = self.current().span.start;
#[cfg(test)]
if let Some(mixed) = &mut self.mixed
&& mixed.typed_header(self.cursor, open.is_empty())
{
let owner = open.last().map(|&at| declarations[at].id);
mixed.parse_typed(&mut self.cursor, owner)?;
continue;
}
if let Some(&owner) = open.last() {
if self.text() == "}" {
declarations[owner].body.end = offset;
@@ -1532,14 +1425,6 @@ impl Parser<'_> {
if !includes_allowed {
return Err(self.error("An include requires a source bundle; single-string parsing never reads files."));
}
#[cfg(test)]
if let Some(mixed) = &mut self.mixed {
mixed.include(Span {
start: offset,
end: offset,
})?;
}
self.syntax_entry()?;
self.cursor += 1;
let path = self.quoted()?;
self.take(";")?;
@@ -1547,11 +1432,7 @@ impl Parser<'_> {
path,
Span {
start: offset,
end: self
.token(self.cursor - 1)
.expect("consumed token")
.span
.end,
end: self.tokens[self.cursor - 1].span.end,
},
));
continue;
@@ -1563,40 +1444,20 @@ impl Parser<'_> {
if entry_header.description.is_some() {
return Err(self.error("Duplicate project description."));
}
self.syntax_entry()?;
self.cursor += 1;
let description = self.quoted()?;
self.take(";")?;
entry_header.project.description = description;
entry_header.description = Some(Span {
start: offset,
end: self
.token(self.cursor - 1)
.expect("consumed token")
.span
.end,
end: self.tokens[self.cursor - 1].span.end,
});
header = Some(entry_header);
continue;
}
#[cfg(test)]
if self.mixed.is_some() && analysis_source::mixed::long_analysis_word(self.text()) {
return Err(self.error("Use the short source8 analysis declaration word."));
}
self.syntax_entry()?;
let word = self.text().to_owned();
let kind = declaration_kind(&word)
.ok_or_else(|| self.error(format!("Unknown declaration '{word}'.")))?;
#[cfg(test)]
if let Some(mixed) = &mut self.mixed {
mixed.record(
matches!(kind, DeclKind::Element(..) | DeclKind::Relation(..)),
Span {
start: offset,
end: offset,
},
)?;
}
graph_count += usize::from(matches!(
kind,
DeclKind::Element(..) | DeclKind::Relation(..)
@@ -1624,10 +1485,6 @@ impl Parser<'_> {
let body_start = self.current().span.end;
self.take("{")?;
let lexical_owner = open.last().map(|&owner| declarations[owner].id);
#[cfg(test)]
if let Some(mixed) = &mut self.mixed {
mixed.lexical_owners.push(open.last().copied());
}
declarations.push(Declaration {
kind,
alias,
@@ -6,7 +6,6 @@ mod declarations;
mod index;
mod lex;
mod lower;
pub(super) mod mixed;
mod syntax;
mod update;
mod write;
@@ -1,400 +0,0 @@
//! Private syntax-only complete-bundle witness. No model admission or edit authority.
//!
//! One scan charges original bytes and raw tokens (including trivia). Typed
//! syntax keeps its existing entry/reference/numeric charges. Ordinary syntax
//! charges one entry per project/declaration/include/description, field, and
//! value before allocation; array values already account for their list slots.
//! These private storage ceilings are independent of caller model record counts.
//! Ordinary references consume value storage, not ordered-model-step allowance.
//! Typed reference occurrences retain the caller max_step_references ceiling.
//! Only declaration graph/record counts are known here; semantic nested-model
//! records and ordinary ordered steps still require the later complete admission.
//! Metadata phase slots aggregate across all files: retained token-end indices,
//! value-end entries, projected value/field/header/declaration positions, lexical
//! owner slots and the file table. Checks do not recharge syntax or latch errors;
//! every failure must return immediately. Tokens themselves use the token bound.
//! This deliberately bounded staging policy is not calibrated public source8.
use super::{
budget::{Budget, Limits},
cursor::Cursor,
declarations, lex,
};
use crate::Id;
use crate::language::{
self, Ast, EditDeclaration, Error, Lexeme, ParseLimits, Parser, SourceUnit, Span, TokenClass,
};
use crate::source_project::{ProjectLimits, SourceBundle, SourceDiagnostic, traversal::Traversal};
use std::collections::BTreeMap;
struct ModelCounts {
limits: ProjectLimits,
graph: usize,
records: usize,
includes: usize,
}
pub(super) struct ParsedMixedSyntax<'s> {
bundle: &'s SourceBundle,
// Preserve caller policy for the later semantic pass. Syntax storage uses
// separate ceilings and cannot stand in for these graph/record/step limits.
limits: ProjectLimits,
files: Vec<FileSyntax<'s>>,
budget: Budget,
}
struct FileSyntax<'s> {
path: &'s str,
source: &'s str,
tokens: Vec<lex::Token>,
ordinary: SourceUnit,
typed: Vec<declarations::Declaration>,
positions: Vec<(Id, EditDeclaration)>,
// A lexical owner is an occurrence index, not only a possibly repeated UUID.
lexical_owners: Vec<Option<usize>>,
}
// Only Parser borrows this state; the stored witness has no self-reference.
pub(in crate::language) struct State<'a> {
source: &'a str,
tokens: &'a [lex::Token],
budget: &'a mut Budget,
model: &'a mut ModelCounts,
typed: Vec<declarations::Declaration>,
pub(in crate::language) value_ends: BTreeMap<usize, usize>,
pub(in crate::language) lexical_owners: Vec<Option<usize>>,
}
fn ordinary_error(error: super::Error) -> Error {
Error {
code: "invalid_source",
offset: error.span.start,
message: error.message,
}
}
fn lexeme(token: lex::Token) -> Lexeme {
Lexeme {
class: match token.class {
lex::Class::Word => TokenClass::Identifier,
lex::Class::Number => TokenClass::Number,
lex::Class::String => TokenClass::String,
lex::Class::Punctuation => TokenClass::Punctuation,
},
span: token.span,
}
}
impl State<'_> {
pub(in crate::language) fn token(&self, at: usize) -> Option<Lexeme> {
self.tokens.get(at).copied().map(lexeme)
}
pub(in crate::language) fn entry(&mut self, at: Span) -> Result<(), Error> {
self.budget.entry(at).map_err(ordinary_error)
}
pub(in crate::language) fn record(&mut self, graph: bool, at: Span) -> Result<(), Error> {
let records = self.model.records.checked_add(1);
let graph = self.model.graph.checked_add(usize::from(graph));
let (Some(records), Some(graph)) = (records, graph) else {
return Err(Error {
code: "model_limit",
offset: at.start,
message: "Declaration count overflow.".into(),
});
};
self.model
.limits
.model
.check_counts(graph, records, 0)
.map_err(|message| Error {
code: "model_limit",
offset: at.start,
message,
})?;
self.model.records = records;
self.model.graph = graph;
Ok(())
}
pub(in crate::language) fn include(&mut self, at: Span) -> Result<(), Error> {
if self.model.includes >= self.model.limits.max_include_edges {
return Err(Error {
code: "project_limit",
offset: at.start,
message: format!(
"Include graph exceeds {} edges.",
self.model.limits.max_include_edges
),
});
}
self.model.includes += 1;
Ok(())
}
pub(in crate::language) fn ordinary_number(
&self,
at: &mut usize,
) -> Result<serde_json::Value, Error> {
let first = self.tokens[*at];
let mut end = first.span.end;
let mut after = *at + 1;
// Match the old greedy numeric alphabet only across byte-adjacent raw
// tokens. This is consumption of the shared stream, not another scan.
while let Some(next) = self.tokens.get(after) {
if next.span.start != end
|| !self.source[next.span.start..next.span.end]
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || b"_.+-".contains(&byte))
{
break;
}
end = next.span.end;
after += 1;
}
let value: serde_json::Number = serde_json::from_str(&self.source[first.span.start..end])
.map_err(|_| Error {
code: "invalid_source",
offset: first.span.start,
message: "Expected a byte-adjacent finite JSON decimal number.".into(),
})?;
*at = after;
Ok(serde_json::Value::Number(value))
}
pub(in crate::language) fn typed_header(&self, at: usize, root: bool) -> bool {
let Some(token) = self.tokens.get(at) else {
return false;
};
if declarations::declaration_kind(&self.source[token.span.start..token.span.end]).is_none()
{
return false;
}
// A generic field named `unit` remains an ordinary field. A declaration
// header has an alias and display-name token; only that shape is nested.
root || (self
.tokens
.get(at + 1)
.is_some_and(|t| t.class == lex::Class::Word)
&& self
.tokens
.get(at + 2)
.is_some_and(|t| t.class == lex::Class::String))
}
pub(in crate::language) fn parse_typed(
&mut self,
at: &mut usize,
owner: Option<Id>,
) -> Result<(), Error> {
// Transfer the one active position into the typed consumer and back;
// neither consumer scans, clones a stream, nor runs concurrently.
self.record(true, self.tokens[*at].span)?;
let mut cursor = Cursor::new(self.source, self.tokens, self.budget);
cursor.cursor = *at;
let result = declarations::parse(&mut cursor, 8, owner);
*at = cursor.cursor;
let declaration = result.map_err(ordinary_error)?;
let missing = declaration.header.identity.value.is_none()
|| matches!(&declaration.body, declarations::Body::Calculation { parameters, .. }
if parameters.iter().any(|p| p.identity.value.is_none()));
if missing {
return Err(Error {
code: "invalid_source",
offset: declaration.span.start,
message: "Private mixed syntax requires explicit IDs.".into(),
});
}
self.typed.push(declaration);
Ok(())
}
}
pub(in crate::language) fn long_analysis_word(word: &str) -> bool {
matches!(
word,
"oa_engineering_property"
| "sa_engineering_property"
| "oa_analysis_check"
| "sa_analysis_check"
| "oa_analysis_case"
| "sa_analysis_case"
)
}
pub(super) fn parse(
bundle: &SourceBundle,
limits: ProjectLimits,
) -> Result<ParsedMixedSyntax<'_>, Vec<SourceDiagnostic>> {
let profile = Limits::source8(ParseLimits {
// Storage is not the caller's final model record/ordered-step accounting.
max_source_bytes: limits.model.max_source_bytes,
max_step_references: limits.model.max_step_references,
..ParseLimits::UNLIMITED
});
parse_with_profile(bundle, limits, profile)
}
fn located(path: &str, source: &str, error: Error) -> Vec<SourceDiagnostic> {
vec![SourceDiagnostic {
path: path.into(),
diagnostic: language::diagnostic(source, error),
related: Vec::new(),
}]
}
fn parse_with_profile(
bundle: &SourceBundle,
limits: ProjectLimits,
profile: Limits,
) -> Result<ParsedMixedSyntax<'_>, Vec<SourceDiagnostic>> {
let entry_source = bundle.files.get(&bundle.entry).map_or("", String::as_str);
let at_entry = |error| located(&bundle.entry, entry_source, ordinary_error(error));
let maximum = Limits::source8(ParseLimits {
max_source_bytes: limits.model.max_source_bytes,
max_step_references: limits.model.max_step_references,
..ParseLimits::UNLIMITED
});
if profile.bytes > maximum.bytes
|| profile.tokens > maximum.tokens
|| profile.entries > maximum.entries
|| profile.references > maximum.references
|| profile.frames > maximum.frames
|| profile.numeric_units > maximum.numeric_units
{
return Err(at_entry(super::error(
Span::default(),
"Mixed test profile may only tighten inherited limits.",
)));
}
let mut budget = Budget::new(profile).map_err(at_entry)?;
let bytes = bundle
.files
.values()
.try_fold(0usize, |n, s| n.checked_add(s.len()))
.ok_or_else(|| at_entry(super::error(Span::default(), "Source size overflow.")))?;
budget
.index_bytes(bytes, Span::default())
.map_err(at_entry)?;
let mut metadata = bundle.files.len();
budget
.lowering_entries(metadata, Span::default())
.map_err(at_entry)?;
// Tighter private byte/file-slot limits precede folded-path table and
// traversal allocations. Public Traversal's own validation order is intact.
let mut traversal = Traversal::new(bundle, limits)?;
let mut parsed = BTreeMap::new();
let mut model = ModelCounts {
limits,
graph: 0,
records: 0,
includes: 0,
};
while let Some(path) = traversal.next()? {
let (path, source) = bundle
.files
.get_key_value(&path)
.expect("traversal checked source");
let failure = |error| located(path, source, ordinary_error(error));
let tokens = lex::tokens(source, &mut budget).map_err(failure)?;
let mut parser = Parser {
source,
tokens: Vec::new(),
cursor: 0,
value_depth: 0,
limits: limits.model,
mixed: Some(State {
source,
tokens: &tokens,
budget: &mut budget,
model: &mut model,
typed: Vec::new(),
value_ends: BTreeMap::new(),
lexical_owners: Vec::new(),
}),
};
let ordinary = parser
.unit(path == &bundle.entry, true)
.map_err(|e| located(path, source, e))?;
let state = parser.mixed.take().expect("private parser state");
drop(parser);
// Projected ordinary AST nodes/fields are bounded before constructing a
// second position representation. Count references as slots, not syntax.
let projection_slots = ordinary_position_slots(&ordinary);
metadata = metadata
.checked_add(tokens.len())
.and_then(|n| n.checked_add(state.value_ends.len()))
.and_then(|n| n.checked_add(projection_slots))
.and_then(|n| n.checked_add(state.lexical_owners.len()))
.ok_or_else(|| failure(super::error(Span::default(), "Metadata size overflow.")))?;
state
.budget
.lowering_entries(metadata, Span::default())
.map_err(failure)?;
let positions = language::edit_positions_from_tokens(
source,
&ordinary,
|at| lexeme(tokens[at]),
tokens.len(),
Some(&state.value_ends),
)
.map_err(|e| {
vec![SourceDiagnostic {
path: path.clone(),
diagnostic: *e,
related: Vec::new(),
}]
})?;
for (raw, span) in &ordinary.includes {
traversal.include(path, raw, *span)?;
}
traversal.finish_file(path.clone());
let State {
typed,
lexical_owners,
..
} = state;
parsed.insert(
path.as_str(),
FileSyntax {
path,
source,
tokens,
ordinary,
typed,
positions,
lexical_owners,
},
);
}
traversal.finish()?;
budget.active().map_err(at_entry)?;
Ok(ParsedMixedSyntax {
bundle,
limits,
files: parsed.into_values().collect(),
budget,
})
}
fn ordinary_position_slots(unit: &SourceUnit) -> usize {
// Iterative accounting avoids adding a second recursive walk before the
// already depth-bounded position projection. Each reference is one value.
let mut slots = 0usize;
let mut stack = Vec::new();
for declaration in &unit.declarations {
slots = slots.saturating_add(7); // declaration/body + at most five header roles
slots = slots.saturating_add(declaration.fields.len());
stack.extend(declaration.fields.values());
}
while let Some(value) = stack.pop() {
slots = slots.saturating_add(1);
match value {
Ast::Array(items, _) => stack.extend(items),
Ast::Object(fields, _) => {
slots = slots.saturating_add(fields.len());
stack.extend(fields.values());
}
_ => {}
}
}
slots
}
#[cfg(test)]
mod tests;
File diff suppressed because it is too large Load Diff
@@ -25,17 +25,6 @@ impl<'i, 's> Plan<'i, 's> {
renames: &BTreeMap<Target, String>,
budget: &Budget,
) -> Result<Self, Error> {
let plan = Self::prepare(index, renames, budget)?;
plan.validate(index.sources(), budget)?;
plan.output_lengths(index.sources(), budget)?;
Ok(plan)
}
fn validate_names(
index: &Index<'_>,
renames: &BTreeMap<Target, String>,
budget: &Budget,
) -> Result<(), Error> {
budget.active()?;
budget.lowering_entries(renames.len(), Span::default())?;
let mut bytes = 0usize;
@@ -72,16 +61,6 @@ impl<'i, 's> Plan<'i, 's> {
));
}
}
Ok(())
}
fn prepare(
index: &'i Index<'s>,
renames: &BTreeMap<Target, String>,
budget: &Budget,
) -> Result<Self, Error> {
// Validate record/name storage before copying any owned final names.
Self::validate_names(index, renames, budget)?;
let mut edits = Vec::new();
for (key, location) in index.occurrences() {
if renames
@@ -102,20 +81,26 @@ impl<'i, 's> Plan<'i, 's> {
edit.location.span.end,
)
});
Ok(Self {
let plan = Self {
index,
names: renames.clone(),
edits,
})
};
plan.validate(index.sources(), budget)?;
Ok(plan)
}
pub fn edit_count(&self) -> usize {
self.edits.len()
}
/// Recheck the complete original witness and complete rename intent.
/// A changed unrelated file is stale too. This does not construct output.
fn validate(&self, current: &BTreeMap<usize, String>, budget: &Budget) -> Result<(), Error> {
/// Recheck all files, all edit ownership and all final sizes before any
/// output String is allocated. A changed unrelated file is stale too.
fn validate(
&self,
current: &BTreeMap<usize, String>,
budget: &Budget,
) -> Result<BTreeMap<usize, usize>, Error> {
budget.active()?;
if current != self.index.sources() {
return Err(error(
@@ -125,6 +110,10 @@ impl<'i, 's> Plan<'i, 's> {
}
budget.lowering_entries(current.len(), Span::default())?;
budget.lowering_occurrences(self.edits.len(), Span::default())?;
let mut lengths: BTreeMap<_, _> = current
.iter()
.map(|(&file, source)| (file, source.len()))
.collect();
let mut previous: Option<Location> = None;
let mut seen = BTreeSet::new();
for edit in &self.edits {
@@ -141,7 +130,8 @@ impl<'i, 's> Plan<'i, 's> {
{
return Err(error(at.span, "Stale original token spelling."));
}
self.names
let replacement = self
.names
.get(&edit.key.target)
.ok_or_else(|| error(at.span, "Edit has no final target name."))?;
if previous.is_some_and(|prior| {
@@ -150,6 +140,13 @@ impl<'i, 's> Plan<'i, 's> {
return Err(error(at.span, "Overlapping or unordered source edits."));
}
previous = Some(at);
let length = lengths
.get_mut(&at.file)
.ok_or_else(|| error(at.span, "Unknown edit file."))?;
*length = length
.checked_sub(old.len())
.and_then(|size| size.checked_add(replacement.len()))
.ok_or_else(|| error(at.span, "Updated source size overflow."))?;
}
for (key, at) in self.index.occurrences() {
let required = self
@@ -163,28 +160,6 @@ impl<'i, 's> Plan<'i, 's> {
));
}
}
Ok(())
}
/// Standalone output preflight, after ownership validation. Composition
/// defers this hypothetical typed-only size to its future combined emitter.
fn output_lengths(
&self,
current: &BTreeMap<usize, String>,
budget: &Budget,
) -> Result<BTreeMap<usize, usize>, Error> {
let mut lengths: BTreeMap<_, _> = current
.iter()
.map(|(&file, source)| (file, source.len()))
.collect();
for edit in &self.edits {
let at = edit.location;
let length = lengths.get_mut(&at.file).expect("validated edit file");
*length = length
.checked_sub(at.span.end - at.span.start)
.and_then(|size| size.checked_add(self.names[&edit.key.target].len()))
.ok_or_else(|| error(at.span, "Updated source size overflow."))?;
}
let mut total = 0usize;
for length in lengths.values() {
total = total
@@ -200,8 +175,7 @@ impl<'i, 's> Plan<'i, 's> {
current: &BTreeMap<usize, String>,
budget: &Budget,
) -> Result<BTreeMap<usize, String>, Error> {
self.validate(current, budget)?;
let lengths = self.output_lengths(current, budget)?;
let lengths = self.validate(current, budget)?;
let mut result = BTreeMap::new();
let mut edits = self.edits.iter().peekable();
for (&file, source) in current {
@@ -220,97 +194,5 @@ impl<'i, 's> Plan<'i, 's> {
}
}
/// Private typed-only composition planning. The caller must supply its inherited
/// parse Budget; this type keeps that same immutable budget through export.
/// It does not establish the future complete bundle/file-table/parse-session
/// witness. In particular, an Index currently does not store its parse profile.
pub(super) struct Composition<'b, 'i, 's> {
plan: Plan<'i, 's>,
budget: &'b Budget,
}
impl std::fmt::Debug for Composition<'_, '_, '_> {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.debug_struct("Composition")
.field("plan", &self.plan)
.finish_non_exhaustive()
}
}
impl<'b, 'i, 's> Composition<'b, 'i, 's> {
pub fn renames(
index: &'i Index<'s>,
renames: &BTreeMap<Target, String>,
inherited: &'b Budget,
) -> Result<Self, Error> {
let plan = Plan::prepare(index, renames, inherited)?;
plan.validate(index.sources(), inherited)?;
Ok(Self {
plan,
budget: inherited,
})
}
/// Export the entire validated plan against exact original numeric file keys
/// and bytes. There is no output allocation, application or subset-accept API.
/// A future collector must check this actual index witness and the complete
/// combined output bounds before applying either producer's edits once.
pub fn original_edits(
&self,
current: &BTreeMap<usize, String>,
) -> Result<OriginalEdits<'_, 'i, 's>, Error> {
Plan::validate_names(self.plan.index, &self.plan.names, self.budget)?;
self.plan.validate(current, self.budget)?;
Ok(OriginalEdits { plan: &self.plan })
}
}
#[derive(Debug)]
pub(super) struct OriginalEdits<'p, 'i, 's> {
plan: &'p Plan<'i, 's>,
}
impl<'s> OriginalEdits<'_, '_, 's> {
pub fn index(&self) -> &Index<'s> {
self.plan.index
}
pub fn len(&self) -> usize {
self.plan.edits.len()
}
pub fn is_empty(&self) -> bool {
self.plan.edits.is_empty()
}
pub fn iter(&self) -> impl ExactSizeIterator<Item = OriginalEdit<'_>> {
self.plan.edits.iter().map(|edit| OriginalEdit {
key: &edit.key,
location: edit.location,
original: &self.plan.index.sources()[&edit.location.file]
[edit.location.span.start..edit.location.span.end],
replacement: &self.plan.names[&edit.key.target],
})
}
}
/// A borrowed inspection record, never independent edit authority. Only an
/// intact OriginalEdits batch can carry this producer's complete rename intent.
#[derive(Debug)]
pub(super) struct OriginalEdit<'p> {
key: &'p OccurrenceKey,
location: Location,
original: &'p str,
replacement: &'p str,
}
impl OriginalEdit<'_> {
pub fn key(&self) -> &OccurrenceKey {
self.key
}
pub fn location(&self) -> Location {
self.location
}
pub fn original(&self) -> &str {
self.original
}
pub fn replacement(&self) -> &str {
self.replacement
}
}
#[cfg(test)]
mod tests;
@@ -322,423 +322,3 @@ fn output_and_plan_limits_are_aggregate_and_do_not_reset_parse_work() {
limits.entries = 0;
assert!(Plan::renames(&index, &rename_map(), &Budget::new(limits).unwrap()).is_err());
}
#[test]
fn composition_exports_complete_borrowed_swap_witnesses_and_empty_noops() {
use super::super::index::TokenRole;
use super::super::lower::{Child, Field, Role};
type Key = (Id, Option<Field>, Option<Vec<Child>>, TokenRole, Target);
type Row = (Location, String, String, Key);
let sources = sources();
let mut work = budget();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
let usage = work.usage;
let numeric = work.numeric_units();
let composition = Composition::renames(&index, &rename_map(), &work).unwrap();
assert!(std::ptr::eq(composition.budget, &work));
// A byte-identical snapshot is valid; records still borrow the actual index.
let batch = composition.original_edits(&sources.clone()).unwrap();
assert!(std::ptr::eq(batch.index(), &index));
assert_eq!(batch.len(), 13);
assert!(!batch.is_empty());
assert_eq!(batch.iter().len(), 13);
// Independently authored occurrence identities and full source contexts.
// No index offset/path or generated output defines this oracle.
let mut expected: Vec<Row> = Vec::new();
let mut add = |file, context: &str, prefix: &str, old: &str, new: &str, key: Key| {
let start = sources[&file].find(context).unwrap() + prefix.len();
assert_eq!(&sources[&file][start..start + old.len()], old);
expected.push((
Location {
file,
span: Span {
start,
end: start + old.len(),
},
},
old.into(),
new.into(),
key,
));
};
for (id, old, new, context) in [
(0x100, "f", "g", "calculation f "),
(0x200, "g", "f", "calculation g "),
] {
add(
11,
context,
"calculation ",
old,
new,
(
Id::from_u128(id),
None,
None,
TokenRole::Declaration,
global(id),
),
);
}
for (id, old, new, context, edge, prefix) in [
(0x200, "a", "b", "input a id", Child::Left, "body $"),
(0x201, "b", "a", "input b id", Child::Right, "body $a + $"),
] {
let target = parameter(0x100, id);
add(
11,
context,
"input ",
old,
new,
(
Id::from_u128(0x100),
Some(Field::Input(a::ParameterId(Id::from_u128(id)))),
None,
TokenRole::Use(Role::ParameterDeclaration),
target,
),
);
add(
11,
"body $a + $b;",
prefix,
old,
new,
(
Id::from_u128(0x100),
Some(Field::Body),
Some(vec![edge]),
TokenRole::Use(Role::ParameterUse),
target,
),
);
}
for (context, path) in [
("@f(b: 2, a: 1)", vec![Child::Left, Child::Left]),
("@f(a: 3, b: 4)", vec![Child::Left, Child::Right]),
] {
add(
23,
context,
"@",
"f",
"g",
(
Id::from_u128(0x300),
Some(Field::Body),
Some(path.clone()),
TokenRole::Use(Role::CallTarget),
global(0x100),
),
);
for (id, old, new) in [(0x200, "a", "b"), (0x201, "b", "a")] {
let offset = context.find(&format!("{old}:")).unwrap();
add(
23,
context,
&context[..offset],
old,
new,
(
Id::from_u128(0x300),
Some(Field::Body),
Some(path.clone()),
TokenRole::Use(Role::ArgumentLabel),
parameter(0x100, id),
),
);
}
}
add(
23,
"@g(a: 5, b: 6)",
"@",
"g",
"f",
(
Id::from_u128(0x300),
Some(Field::Body),
Some(vec![Child::Right]),
TokenRole::Use(Role::CallTarget),
global(0x200),
),
);
expected.sort_by_key(|row| (row.0.file, row.0.span.start, row.0.span.end));
let actual: Vec<Row> = batch
.iter()
.map(|edit| {
let key = edit.key();
(
edit.location(),
edit.original().into(),
edit.replacement().into(),
(
key.owner,
key.field,
key.path.as_ref().map(|path| path.edges()),
key.role,
key.target,
),
)
})
.collect();
assert_eq!(actual, expected);
let noop = Composition::renames(
&index,
&BTreeMap::from([
(global(0x100), "f".into()),
(parameter(0x200, 0x200), "a".into()),
]),
&work,
)
.unwrap();
let empty = noop.original_edits(&sources).unwrap();
assert!(empty.is_empty());
assert_eq!(empty.len(), 0);
assert_eq!(empty.iter().count(), 0);
assert_eq!(work.usage, usage);
assert_eq!(work.numeric_units(), numeric);
}
#[test]
fn composition_rejects_stale_or_damaged_complete_batches_before_export() {
let mut sources = sources();
sources.insert(99, "// unrelated 雪\r\n".into());
let original = sources.clone();
let mut work = budget();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
for damage in 0..10 {
let mut current = sources.clone();
let mut composition = Composition::renames(&index, &rename_map(), &work).unwrap();
match damage {
0 => current.get_mut(&99).unwrap().push_str("changed"),
1 => {
current.remove(&23);
}
2 => {
current.insert(101, String::new());
}
3 => composition.plan.edits[0].location.span.start += 1,
4 => composition
.plan
.edits
.insert(1, composition.plan.edits[0].clone()),
5 => {
composition.plan.edits.pop();
}
6 => composition.plan.edits.swap(0, 1),
7 => {
composition
.plan
.names
.insert(global(0x100), "bad name".into());
}
8 => {
composition.plan.names.insert(global(0x100), "f".into());
}
_ => {
composition.plan.names.insert(global(999), "missing".into());
}
}
let before = current.clone();
assert!(
composition.original_edits(&current).is_err(),
"damage {damage}"
);
assert_eq!(current, before);
}
assert_eq!(sources, original);
let failure = work.frame(257, Span::default()).unwrap_err();
assert_eq!(
Composition::renames(&index, &rename_map(), &work).unwrap_err(),
failure
);
}
#[test]
fn composition_defers_only_typed_output_size_with_independent_shrink_arithmetic() {
let sources = sources();
let cap = sources.values().map(String::len).sum();
let mut limits = Limits::source8(ParseLimits::UNLIMITED);
limits.bytes = cap;
let mut work = Budget::new(limits).unwrap();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
assert_eq!(work.usage.bytes, cap);
let usage = work.usage;
let numeric = work.numeric_units();
let names = BTreeMap::from([(global(0x100), "ff".into())]);
assert_eq!(
Plan::renames(&index, &names, &work).unwrap_err().message,
"Source8 index/output byte limit exceeded."
);
let composition = Composition::renames(&index, &names, &work).unwrap();
let batch = composition.original_edits(&sources).unwrap();
assert_eq!(batch.len(), 3); // declaration and two distinct call targets
let growth: usize = batch
.iter()
.map(|edit| edit.replacement().len() - edit.original().len())
.sum();
assert_eq!(growth, 3);
assert_eq!(
composition.plan.apply(&sources, &work).unwrap_err().message,
"Source8 index/output byte limit exceeded."
);
// Independent future ordinary/scalar producer witness on untouched display
// name bytes. This is size/offset arithmetic only: this typed-only harness
// has no ordinary collector, mixed ownership proof or accepted output.
let start = DEFINITIONS.find("\"f a b\"").unwrap() + 1;
let shrink = Location {
file: 11,
span: Span {
start,
end: start + "f a b".len(),
},
};
assert_eq!(
&sources[&shrink.file][shrink.span.start..shrink.span.end],
"f a b"
);
for edit in batch.iter() {
let at = edit.location();
assert!(
at.file != shrink.file
|| at.span.end <= shrink.span.start
|| shrink.span.end <= at.span.start
);
}
let original_len = shrink.span.end - shrink.span.start;
assert_eq!(cap + growth - original_len + "fa".len(), cap);
assert_eq!(cap + growth - original_len + "fab".len(), cap + 1);
assert_eq!(work.usage, usage);
assert_eq!(work.numeric_units(), numeric);
}
#[test]
fn composition_keeps_name_storage_and_rename_record_limits_before_allocation() {
let sources = sources();
let cap = sources.values().map(String::len).sum();
let mut limits = Limits::source8(ParseLimits::UNLIMITED);
limits.bytes = cap;
let mut work = Budget::new(limits).unwrap();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
let exact = BTreeMap::from([(global(0x100), "x".repeat(cap))]);
assert_eq!(
Composition::renames(&index, &exact, &work)
.unwrap()
.original_edits(&sources)
.unwrap()
.len(),
3
);
let over = BTreeMap::from([(global(0x100), "x".repeat(cap + 1))]);
assert_eq!(
Composition::renames(&index, &over, &work)
.unwrap_err()
.message,
"Source8 index/output byte limit exceeded."
);
for invalid in [
BTreeMap::from([(global(999), "unknown".into())]),
BTreeMap::from([(global(0x100), "bad name".into())]),
BTreeMap::from([(global(0x100), "g".into())]),
BTreeMap::from([(parameter(0x100, 0x200), "b".into())]),
] {
assert!(Composition::renames(&index, &invalid, &work).is_err());
}
// A typed-empty fixture isolates the rename-map record bound without
// syntax entry usage masking it. No ordinary edits are authorized here.
let ordinary = skeleton();
let count = ordinary.identities().len();
let files = BTreeMap::from([(1, "// empty typed unit\n".into())]);
let mut limits = Limits::source8(ParseLimits::UNLIMITED);
limits.entries = count;
limits.references = 0;
let mut inherited = Budget::new(limits).unwrap();
let (index, _) = Index::parse(&ordinary, &files, &mut inherited).unwrap();
let mut names: BTreeMap<_, _> = index
.names()
.keys()
.enumerate()
.map(|(i, target)| (*target, format!("renamed_{i}")))
.collect();
assert_eq!(names.len(), count);
assert!(
Composition::renames(&index, &names, &inherited)
.unwrap()
.original_edits(&files)
.unwrap()
.is_empty()
);
names.insert(global(999), "additional".into());
assert_eq!(
Composition::renames(&index, &names, &inherited)
.unwrap_err()
.message,
"Source8 lowering entry limit exceeded."
);
}
#[test]
fn composition_checks_edit_record_ceiling_before_duplicate_membership() {
let sources = sources();
let mut limits = Limits::source8(ParseLimits::UNLIMITED);
limits.entries = 128;
limits.references = 32;
let ceiling = limits.entries + limits.references;
let mut work = Budget::new(limits).unwrap();
let (index, _) = Index::parse(&skeleton(), &sources, &mut work).unwrap();
let mut composition = Composition::renames(&index, &rename_map(), &work).unwrap();
assert_eq!(composition.original_edits(&sources).unwrap().len(), 13);
let duplicate = composition.plan.edits[0].clone();
composition.plan.edits = vec![duplicate.clone(); ceiling];
// At the exact count bound validation proceeds to the independent ownership
// error. The damaged plan never becomes a successful/authoritative batch.
assert_eq!(
composition.original_edits(&sources).unwrap_err().message,
"Forged or duplicate edit occurrence."
);
composition.plan.edits.push(duplicate);
assert_eq!(
composition.original_edits(&sources).unwrap_err().message,
"Source8 lowering occurrence limit exceeded."
);
}
#[test]
fn standalone_noop_apply_keeps_tighter_file_only_budget_and_output_boundary() {
let sources = sources();
let mut inherited = budget();
let (index, _) = Index::parse(&skeleton(), &sources, &mut inherited).unwrap();
let plan = Plan::renames(
&index,
&BTreeMap::from([(global(0x100), "f".into())]),
&inherited,
)
.unwrap();
assert_eq!(plan.edit_count(), 0);
assert!(index.names().len() > sources.len());
let mut limits = Limits::source8(ParseLimits::UNLIMITED);
limits.entries = sources.len();
limits.references = 0;
limits.bytes = sources.values().map(String::len).sum();
// Standalone apply checks file/edit records, not the already validated
// catalog/name-storage allocation from construction. Rechecking the catalog
// under this intentionally tighter probe budget would reject this no-op.
assert_eq!(
plan.apply(&sources, &Budget::new(limits).unwrap()).unwrap(),
sources
);
limits.bytes -= 1;
assert_eq!(
plan.apply(&sources, &Budget::new(limits).unwrap())
.unwrap_err()
.message,
"Source8 index/output byte limit exceeded."
);
}
@@ -194,8 +194,6 @@ pub(super) fn row_key(source: &str, origins: &BTreeMap<Id, Id>) -> Option<String
cursor: 0,
value_depth: 0,
limits: ParseLimits::UNLIMITED,
#[cfg(test)]
mixed: None,
};
let ast = parser.value().ok()?;
let facet = serde_json::from_value(raw(&ast).ok()?).ok()?;
+147 -7
View File
@@ -8,7 +8,6 @@ use std::collections::{BTreeMap, BTreeSet};
mod source_migration;
#[path = "source_project/source_structure.rs"]
mod source_structure;
pub(crate) mod traversal;
/// A portable slash-separated path, checked at every bundle boundary.
pub type LogicalPath = String;
@@ -305,7 +304,71 @@ fn compile_once(
bundle: &SourceBundle,
limits: ProjectLimits,
) -> Result<(CompiledSourceProject, Edits), Vec<SourceDiagnostic>> {
let mut traversal = traversal::Traversal::new(bundle, limits)?;
let fail = |code, message| {
vec![problem(
bundle,
&bundle.entry,
Span::default(),
code,
message,
)]
};
if bundle.entry.contains('/') {
return Err(fail(
"invalid_path",
"The entry file must be at the source project root.".into(),
));
}
if bundle.files.len() > limits.max_files {
return Err(fail(
"project_limit",
format!("Source project exceeds {} files.", limits.max_files),
));
}
let bytes = bundle
.files
.values()
.fold(0usize, |sum, text| sum.saturating_add(text.len()));
if bytes > limits.model.max_source_bytes {
return Err(fail(
"source_limit",
format!(
"Combined source exceeds {} bytes.",
limits.model.max_source_bytes
),
));
}
let mut folded = BTreeMap::new();
for path in std::iter::once(&bundle.entry).chain(bundle.files.keys()) {
if resolve_include("", path).as_ref() != Ok(path) {
return Err(fail(
"invalid_path",
format!("Non-canonical source path '{path}'."),
));
}
if let Some(previous) = folded.insert(path.to_lowercase(), path)
&& previous != path
{
return Err(fail(
"duplicate_path",
format!("Case-colliding source paths '{previous}' and '{path}'."),
));
}
}
if !bundle.files.contains_key(&bundle.entry) {
return Err(fail(
"missing_include",
"The entry file is missing from the source bundle.".into(),
));
}
enum Visit {
Enter(LogicalPath, usize, Option<SourceLocation>),
Leave,
}
let mut stack = vec![Visit::Enter(bundle.entry.clone(), 0, None)];
let mut active: Vec<LogicalPath> = Vec::new();
let mut visited = BTreeSet::new();
let mut declarations = Vec::new();
let mut declaration_paths = Vec::new();
let mut aliases = BTreeMap::new();
@@ -318,8 +381,51 @@ fn compile_once(
let mut includes = Vec::new();
let mut insertions: Edits = BTreeMap::new();
let mut graph_count = 0usize;
while let Some(path) = traversal.next()? {
let source = &bundle.files[&path];
while let Some(visit) = stack.pop() {
let Visit::Enter(path, depth, origin) = visit else {
active.pop();
continue;
};
let at = origin.unwrap_or(SourceLocation {
path: path.clone(),
span: Span::default(),
});
if let Some(start) = active.iter().position(|file| file == &path) {
let chain = active[start..]
.iter()
.chain(std::iter::once(&path))
.cloned()
.collect::<Vec<_>>()
.join(" -> ");
return Err(vec![problem(
bundle,
&at.path,
at.span,
"include_cycle",
format!("Include cycle: {chain}."),
)]);
}
if depth > limits.max_include_depth {
return Err(vec![problem(
bundle,
&at.path,
at.span,
"project_limit",
format!("Include depth exceeds {}.", limits.max_include_depth),
)]);
}
if !visited.insert(path.clone()) {
continue;
}
let Some(source) = bundle.files.get(&path) else {
return Err(vec![problem(
bundle,
&at.path,
at.span,
"missing_include",
format!("Included file '{path}' is missing."),
)]);
};
let mut unit = language::parse_unit(source, path == bundle.entry, limits.model).map_err(
|diagnostic| {
vec![SourceDiagnostic {
@@ -407,12 +513,46 @@ fn compile_once(
declarations.push(declaration);
declaration_paths.push(path.clone());
}
let mut children = Vec::new();
for (raw, span) in unit.includes {
includes.push(traversal.include(&path, &raw, span)?);
let target = resolve_include(&path, &raw)
.map_err(|message| vec![problem(bundle, &path, span, "invalid_path", message)])?;
let location = SourceLocation {
path: path.clone(),
span,
};
includes.push(IncludeLocation {
location: location.clone(),
target: target.clone(),
});
if includes.len() > limits.max_include_edges {
return Err(vec![problem(
bundle,
&path,
span,
"project_limit",
format!("Include graph exceeds {} edges.", limits.max_include_edges),
)]);
}
children.push(Visit::Enter(
target,
depth.saturating_add(1),
Some(location),
));
}
traversal.finish_file(path);
active.push(path);
stack.push(Visit::Leave);
stack.extend(children.into_iter().rev());
}
if let Some(path) = bundle.files.keys().find(|path| !visited.contains(*path)) {
return Err(vec![problem(
bundle,
path,
Span::default(),
"unreachable_file",
"Supplied source file is not reachable from the entry file.",
)]);
}
traversal.finish()?;
let header = project_header.expect("validated entry header");
let index = SourceIndex {
declarations: declaration_locations,
@@ -1,258 +0,0 @@
//! Shared include traversal policy for ordinary compilation and private syntax.
use super::*;
enum Visit {
Enter(LogicalPath, usize, Option<SourceLocation>),
Leave,
}
pub(crate) struct Traversal<'a> {
bundle: &'a SourceBundle,
limits: ProjectLimits,
stack: Vec<Visit>,
active: Vec<LogicalPath>,
visited: BTreeSet<LogicalPath>,
children: Vec<Visit>,
depth: usize,
edges: usize,
}
impl<'a> Traversal<'a> {
pub(crate) fn new(
bundle: &'a SourceBundle,
limits: ProjectLimits,
) -> Result<Self, Vec<SourceDiagnostic>> {
let fail = |code, message| {
vec![problem(
bundle,
&bundle.entry,
Span::default(),
code,
message,
)]
};
if bundle.entry.contains('/') {
return Err(fail(
"invalid_path",
"The entry file must be at the source project root.".into(),
));
}
if bundle.files.len() > limits.max_files {
return Err(fail(
"project_limit",
format!("Source project exceeds {} files.", limits.max_files),
));
}
let bytes = bundle
.files
.values()
.fold(0usize, |sum, text| sum.saturating_add(text.len()));
if bytes > limits.model.max_source_bytes {
return Err(fail(
"source_limit",
format!(
"Combined source exceeds {} bytes.",
limits.model.max_source_bytes
),
));
}
let mut folded = BTreeMap::new();
for path in std::iter::once(&bundle.entry).chain(bundle.files.keys()) {
if resolve_include("", path).as_ref() != Ok(path) {
return Err(fail(
"invalid_path",
format!("Non-canonical source path '{path}'."),
));
}
if let Some(previous) = folded.insert(path.to_lowercase(), path)
&& previous != path
{
return Err(fail(
"duplicate_path",
format!("Case-colliding source paths '{previous}' and '{path}'."),
));
}
}
if !bundle.files.contains_key(&bundle.entry) {
return Err(fail(
"missing_include",
"The entry file is missing from the source bundle.".into(),
));
}
Ok(Self {
bundle,
limits,
stack: vec![Visit::Enter(bundle.entry.clone(), 0, None)],
active: Vec::new(),
visited: BTreeSet::new(),
children: Vec::new(),
depth: 0,
edges: 0,
})
}
pub(crate) fn next(&mut self) -> Result<Option<LogicalPath>, Vec<SourceDiagnostic>> {
while let Some(visit) = self.stack.pop() {
let Visit::Enter(path, depth, origin) = visit else {
self.active.pop();
continue;
};
let at = origin.unwrap_or(SourceLocation {
path: path.clone(),
span: Span::default(),
});
if let Some(start) = self.active.iter().position(|file| file == &path) {
let chain = self.active[start..]
.iter()
.chain(std::iter::once(&path))
.cloned()
.collect::<Vec<_>>()
.join(" -> ");
return Err(vec![problem(
self.bundle,
&at.path,
at.span,
"include_cycle",
format!("Include cycle: {chain}."),
)]);
}
if depth > self.limits.max_include_depth {
return Err(vec![problem(
self.bundle,
&at.path,
at.span,
"project_limit",
format!("Include depth exceeds {}.", self.limits.max_include_depth),
)]);
}
if !self.visited.insert(path.clone()) {
continue;
}
if !self.bundle.files.contains_key(&path) {
return Err(vec![problem(
self.bundle,
&at.path,
at.span,
"missing_include",
format!("Included file '{path}' is missing."),
)]);
}
self.depth = depth;
return Ok(Some(path));
}
Ok(None)
}
pub(crate) fn include(
&mut self,
path: &str,
raw: &str,
span: Span,
) -> Result<IncludeLocation, Vec<SourceDiagnostic>> {
let target = resolve_include(path, raw)
.map_err(|message| vec![problem(self.bundle, path, span, "invalid_path", message)])?;
let location = SourceLocation {
path: path.into(),
span,
};
if self.edges >= self.limits.max_include_edges {
return Err(vec![problem(
self.bundle,
path,
span,
"project_limit",
format!(
"Include graph exceeds {} edges.",
self.limits.max_include_edges
),
)]);
}
self.edges += 1;
self.children.push(Visit::Enter(
target.clone(),
self.depth.saturating_add(1),
Some(location.clone()),
));
Ok(IncludeLocation { location, target })
}
pub(crate) fn finish_file(&mut self, path: LogicalPath) {
self.active.push(path);
self.stack.push(Visit::Leave);
self.stack.extend(self.children.drain(..).rev());
}
pub(crate) fn finish(self) -> Result<(), Vec<SourceDiagnostic>> {
if let Some(path) = self
.bundle
.files
.keys()
.find(|path| !self.visited.contains(*path))
{
return Err(vec![problem(
self.bundle,
path,
Span::default(),
"unreachable_file",
"Supplied source file is not reachable from the entry file.",
)]);
}
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn ordinary_compile_once_keeps_include_order_and_cycle_source_location() {
let mut bundle = SourceBundle {
entry: "main.arcadia".into(),
files: BTreeMap::from([
("main.arcadia".into(), "arcadia 7; project p \"P\" id \"00000000-0000-0000-0000-000000000001\" { include \"dir/left.arcadia\"; include \"dir/right.arcadia\"; }".into()),
("dir/left.arcadia".into(), "include \"../shared.arcadia\";".into()),
("dir/right.arcadia".into(), "include \"../shared.arcadia\";".into()),
("shared.arcadia".into(), "// one shared ordinary fragment\n".into()),
]),
};
let original = bundle.clone();
// This is the existing initial ordinary pass, before schema migration's
// separate output/reparse. It makes no public source8 acceptance claim.
let (compiled, insertions) =
super::super::compile_once(&bundle, ProjectLimits::default()).unwrap();
assert!(insertions.is_empty());
assert_eq!(compiled.bundle, original);
assert_eq!(compiled.index.declarations.len(), 1);
assert_eq!(
compiled
.index
.includes
.iter()
.map(|i| (i.location.path.as_str(), i.target.as_str()))
.collect::<Vec<_>>(),
[
("main.arcadia", "dir/left.arcadia"),
("main.arcadia", "dir/right.arcadia"),
("dir/left.arcadia", "shared.arcadia"),
("dir/right.arcadia", "shared.arcadia"),
]
);
let cycle = "// 雪\r\ninclude \"main.arcadia\";";
bundle.files.insert("shared.arcadia".into(), cycle.into());
let before = bundle.clone();
let errors = super::super::compile_once(&bundle, ProjectLimits::default()).unwrap_err();
assert_eq!(errors[0].diagnostic.code, "include_cycle");
assert_eq!(errors[0].path, "shared.arcadia");
assert_eq!(errors[0].diagnostic.line, 2);
assert_eq!(
&cycle[errors[0].diagnostic.span.start..errors[0].diagnostic.span.end],
"include \"main.arcadia\";"
);
assert_eq!(
errors[0].diagnostic.message,
"Include cycle: main.arcadia -> dir/left.arcadia -> shared.arcadia -> main.arcadia."
);
assert_eq!(bundle, before);
}
}
@@ -0,0 +1 @@
* -text
@@ -0,0 +1,46 @@
# Private S3 Windows evidence
Source `1b1c3e3abb6fc3d0934c9dbfbd21fddc8ae4f594`, following `90fae14818f684bf778640be2e613f34e3c7e999`. This checkpoint verifies the private typed index/rename slice and the rejected-depth syntax cleanup. **It does not establish current full-core runtime, public source8, integration, native UI or release acceptance.** Parent owns review/publication and integration.
| Recorded scope | Result |
| --- | --- |
| Full library listing | 167 test names; original 166 retained plus one cleanup test |
| Typed source runtime | 59 passed, including all 9 index and 5 update tests |
| Existing raw-token control | 1 passed |
| Three focused 128 KiB controls | Original whole-flow, eight-parent cleanup and synthetic 256-frame Path tests passed |
| Strict core, all targets | Clippy with `-D warnings`, exit0 |
| Original90fae typed gate | Stack overflow retained; aborted run is not a passing full gate |
| Temporary diagnostics/probes | Two dirty overflow reproductions; four dirty direct-parser malformed-input probes passed |
Actual platform was Windows x86_64-pc-windows-gnullvm, Rust/Cargo 1.98.1, one compiler job. Every clean gate binds 538 committed inputs before/after, exact commands and source. Raw source manifests, logs, receipts, helper scripts and diagnostic diffs remain byte-for-byte unchanged in [evidence.zip](evidence.zip). Its SHA256 is `4486f4c42b140ae4b9b3caf0aacdf151a2acb1fd85eda09a57764fd69e980574`; 658251 bytes, 61 members. No executables, caches or profiles are included.
The failure occurred when `Cursor::node` recursively dropped a rejected depth129 syntax tree below parser frames. Complete depth128 parsing/indexing/renaming/output/model equality/admission and earlier cleanup had finished. The source fix uses iterative rejected-kind disposal and preserves the existing stack request, depth limit and assertions. The 14 original S3 tests and the added mixed-shape cleanup test now have actual runtime evidence. Ordinary AST destruction and all possible parser-error paths are not claimed iterative or universally stack-safe.
## Reproduce evidence verification
With Python 3 standard library and Git, from any working directory:
```text
python -B PATH/verify.py --repo PATH/TO/GIT/REPOSITORY
python -B PATH/negative_controls.py --repo PATH/TO/GIT/REPOSITORY
```
The repository needs immutable commits `1b1c3e3`, `90fae148` and `1f828323` (the publication commit contains that ancestry). Verification reads commit blobs; it neither depends on nor changes the current checkout. `--evidence-dir PATH` optionally selects another complete extracted payload directory. No archived helper, compiler or native artifact is executed. Optimized Python is refused.
The verifier checks both manifests/ZIP byte integrity, actual listing and exact selected runtime membership, 538 committed inputs, recorded commands/times/environment, strict completion, historical source-artifact manifests and the original aborted gate. It reconstructs the dirty source patches in memory and verifies the corresponding 538 test-input hashes. Broader tracked-file hashes also remain in the original captures; the verifier does not claim to audit unrelated tracked files outside that test-input scope. Binary identities are checked against raw executable-path receipts, not rehashed executable bytes, because binaries are deliberately absent. Initial focused tests and final restored typed/token/listing tests were separate builds of identical source and have their own binary receipts.
[checks.json](checks.json) records the positive working evidence audit and 13 discriminating negative controls. Those controls corrupt evidence only in memory; raw archives stay intact. Semantic cases reach their specific checks instead of failing only at the outer ZIP checksum. Package verification also runs against the saved commit and a fresh extraction; exact publication-commit readback is reported in the handoff to the parent, since embedding a commit's own hash in its contents is not possible.
## Review and historical evidence navigation
Extract the ZIP only if desired; the verifier itself inspects it in memory. Current authoritative raw closure is `closure.json` plus `raw/`. Relevant retained notes inside the ZIP:
- `notes/analysis-source8-s3-index-independent-review-20260913.md`: original S3 source review and required corrections.
- `notes/analysis-source8-s3-correction-independent-review-20260922.md`: corrected Path/index source review before runtime.
- `notes/analysis-source3-stack-cleanup-review-20260922.md`: independent review of iterative rejected-kind cleanup.
- `notes/analysis-source3-stack-cleanup-handoff-20260922.md`: exact failure isolation, fix and completed Windows gates.
- Original index/correction source manifests bind their respective immutable Git versions; their historical `unrun` labels are preserved and superseded only by the later actual receipts.
The ZIP's initial `README.md` contains the earlier portable-verifier proposal. It is retained as historical package text; this outer README and actual `verify.py` describe the completed verifier. No peer should execute the archived local orchestration helpers to verify evidence.
Source-bound original-span export, mixed ordinary/typed composition, complete output recompile/admission/intended-Project equality and public dispatch/migration/save integration remain separate work. This evidence does not close those requirements.
@@ -0,0 +1,107 @@
{
"source": "1b1c3e3abb6fc3d0934c9dbfbd21fddc8ae4f594",
"positive_audit": {
"source": "1b1c3e3abb6fc3d0934c9dbfbd21fddc8ae4f594",
"verified_git_inputs": 538,
"listing": 167,
"typed_pass": 59,
"index_pass": 9,
"update_pass": 5,
"token_pass": 1,
"strict_exit": 0,
"historical_overflow_retained": true,
"dirty_diagnostics": 2,
"dirty_direct_parser_probes_pass": 4,
"recorded_test_artifacts": {
"stack-fixed-original-1b1c3e3": {
"path": "C:\\Users\\User\\Documents\\Git\\ARCADIA-Studio\\.local\\analysis-checked-save-target-20260913\\debug\\deps\\oa_core-73987e0d7089bf1e.exe",
"bytes": 39014400,
"sha256": "56c0e5cd44ac5cb96b814d25ea403356195271a1aa9b6c6d1165fed4ffd09ddf"
},
"stack-fixed-shapes-1b1c3e3": {
"path": "C:\\Users\\User\\Documents\\Git\\ARCADIA-Studio\\.local\\analysis-checked-save-target-20260913\\debug\\deps\\oa_core-73987e0d7089bf1e.exe",
"bytes": 39014400,
"sha256": "56c0e5cd44ac5cb96b814d25ea403356195271a1aa9b6c6d1165fed4ffd09ddf"
},
"stack-fixed-paths-1b1c3e3": {
"path": "C:\\Users\\User\\Documents\\Git\\ARCADIA-Studio\\.local\\analysis-checked-save-target-20260913\\debug\\deps\\oa_core-73987e0d7089bf1e.exe",
"bytes": 39014400,
"sha256": "56c0e5cd44ac5cb96b814d25ea403356195271a1aa9b6c6d1165fed4ffd09ddf"
},
"typed-fixed-1b1c3e3": {
"path": "C:\\Users\\User\\Documents\\Git\\ARCADIA-Studio\\.local\\analysis-checked-save-target-20260913\\debug\\deps\\oa_core-73987e0d7089bf1e.exe",
"bytes": 39014400,
"sha256": "7b38b54ad94678171bd8b5964b1e00dd7da8346d96276c1ad790d1a6eb335208"
},
"token-fixed-1b1c3e3": {
"path": "C:\\Users\\User\\Documents\\Git\\ARCADIA-Studio\\.local\\analysis-checked-save-target-20260913\\debug\\deps\\oa_core-73987e0d7089bf1e.exe",
"bytes": 39014400,
"sha256": "7b38b54ad94678171bd8b5964b1e00dd7da8346d96276c1ad790d1a6eb335208"
},
"list-fixed-1b1c3e3": {
"path": "C:\\Users\\User\\Documents\\Git\\ARCADIA-Studio\\.local\\analysis-checked-save-target-20260913\\debug\\deps\\oa_core-73987e0d7089bf1e.exe",
"bytes": 39014400,
"sha256": "7b38b54ad94678171bd8b5964b1e00dd7da8346d96276c1ad790d1a6eb335208"
}
},
"scope": "Private S3 only; no full-core runtime/public8/integration/native/release acceptance. Recorded binary identities verified against receipts, not rehashed executable bytes; binaries intentionally absent."
},
"negative_control_count": 13,
"negative_controls": [
{
"name": "changed_zip_bytes",
"refusal": "Evidence ZIP SHA256 differs"
},
{
"name": "duplicate_json_key",
"refusal": "Duplicate JSON key: source"
},
{
"name": "duplicate_zip_member_with_test_only_pin",
"refusal": "Exact unique ZIP membership"
},
{
"name": "unsafe_zip_member_with_test_only_pin",
"refusal": "Unsafe ZIP path"
},
{
"name": "truncated_strict_completion",
"refusal": "Closed clean strict log"
},
{
"name": "missing_typed_runtime_member",
"refusal": "Exact ordered runtime results: typed-fixed-1b1c3e3"
},
{
"name": "altered_gate_command",
"refusal": "Exact command: token-fixed-1b1c3e3"
},
{
"name": "altered_committed_input_hash",
"refusal": "Exact538 source bindings: typed-fixed-1b1c3e3"
},
{
"name": "dirty_run_mislabeled_as_fixed_source",
"refusal": "Dirty capture identity/stability"
},
{
"name": "historical_abort_mislabeled_success",
"refusal": "Historical aborted gate"
},
{
"name": "diagnostic_patch_disagrees_with_captured_hash",
"refusal": "Reconstructed diagnostic input: crates/oa-core/src/language/analysis_source/index/tests.rs"
},
{
"name": "probe_case_mislabeled",
"refusal": "Dirty probe result/case"
},
{
"name": "optimized_python_refused",
"refusal": "Optimized Python refused"
}
],
"verifier_sha256": "4da79ba0c6fe9dc0ae9b81da6589756d3a215620f2e2fd6cbefaa621f646ebc9",
"controls_sha256": "a969264c98ab2fb95a0585b81c9d27a277ebe629488f20f45216d4483c26b396",
"method": "Raw archive untouched. Semantic corruptions pass directly to audit() after authentic archive validation; ZIP structural negatives temporarily substitute only the test module pin. No compiler, artifact execution or worktree write."
}
@@ -0,0 +1,36 @@
{
"source": "1b1c3e3abb6fc3d0934c9dbfbd21fddc8ae4f594",
"archive_sha256": "4486f4c42b140ae4b9b3caf0aacdf151a2acb1fd85eda09a57764fd69e980574",
"files": [
{
"path": ".gitattributes",
"bytes": 8,
"sha256": "705fd4d6451a31d36b3df7de96f83f30ac976c9b4a6d1e51671d8e2f33e2d0da"
},
{
"path": "README.md",
"bytes": 5456,
"sha256": "8059aa1794544272e3cdaa622ff48a9d0fdf4c257a4a1ca0daf9cc1adcc8b061"
},
{
"path": "verify.py",
"bytes": 17589,
"sha256": "4da79ba0c6fe9dc0ae9b81da6589756d3a215620f2e2fd6cbefaa621f646ebc9"
},
{
"path": "negative_controls.py",
"bytes": 5767,
"sha256": "a969264c98ab2fb95a0585b81c9d27a277ebe629488f20f45216d4483c26b396"
},
{
"path": "checks.json",
"bytes": 4506,
"sha256": "295acaadd2b1fe0f55c08c71c66c193635f70c2f7c63311ea42b039035065139"
},
{
"path": "evidence.zip",
"bytes": 658251,
"sha256": "4486f4c42b140ae4b9b3caf0aacdf151a2acb1fd85eda09a57764fd69e980574"
}
]
}
@@ -0,0 +1,98 @@
"""Read-only in-memory falsifiers for verify.py; does not modify archived evidence."""
import argparse
import importlib.util
import io
import json
from pathlib import Path
import subprocess
import sys
import warnings
import zipfile
def run(repo, directory):
spec = importlib.util.spec_from_file_location('s3_evidence_verifier', directory / 'verify.py')
verifier = importlib.util.module_from_spec(spec)
spec.loader.exec_module(verifier)
verifier.require(__debug__, 'Optimized Python refused')
raw = (directory / 'evidence.zip').read_bytes()
payload = verifier.archive_payload(raw)
positive = verifier.audit(repo, payload)
cases = []
def refused(name, expected, action):
try:
action()
except RuntimeError as error:
verifier.require(expected in str(error), f'{name}: wrong refusal: {error}')
cases.append({'name': name, 'refusal': str(error)})
else:
raise RuntimeError(name + ': invalid evidence was accepted')
def changed_json(name, change):
candidate = dict(payload)
data = verifier.document(candidate[name])
change(data)
candidate[name] = json.dumps(data).encode()
return candidate
refused('changed_zip_bytes', 'Evidence ZIP SHA256 differs', lambda: verifier.archive_payload(raw + b'tampered'))
refused('duplicate_json_key', 'Duplicate JSON key', lambda: verifier.document(b'{"source":1,"source":2}'))
def changed_zip(mode):
buffer = io.BytesIO()
with zipfile.ZipFile(buffer, 'w') as archive:
for name, content in payload.items():
archive.writestr('../README.md' if mode == 'unsafe' and name == 'README.md' else name, content)
if mode == 'duplicate':
with warnings.catch_warnings():
warnings.simplefilter('ignore', UserWarning)
archive.writestr('README.md', payload['README.md'])
malformed = buffer.getvalue()
# Test-only pin substitution reaches ZIP structure checks. Production
# archive_payload always requires its checked-in immutable ZIP digest.
original_pin = verifier.ZIP_SHA
try:
verifier.ZIP_SHA = verifier.sha(malformed)
verifier.archive_payload(malformed)
finally:
verifier.ZIP_SHA = original_pin
refused('duplicate_zip_member_with_test_only_pin', 'Exact unique ZIP membership', lambda: changed_zip('duplicate'))
refused('unsafe_zip_member_with_test_only_pin', 'Unsafe ZIP path', lambda: changed_zip('unsafe'))
changed = dict(payload)
changed['raw/strict-fixed-1b1c3e3.log'] = b' Compiling oa-core\n'
refused('truncated_strict_completion', 'Closed clean strict log', lambda: verifier.audit(repo, changed))
changed = dict(payload)
name = 'raw/typed-fixed-1b1c3e3.log'
changed[name] = b'\n'.join(line for line in changed[name].split(b'\n') if verifier.NEW.encode() not in line)
refused('missing_typed_runtime_member', 'Exact ordered runtime results', lambda: verifier.audit(repo, changed))
changed = changed_json('raw/token-fixed-1b1c3e3.result.json', lambda data: data['command'].remove('--offline'))
refused('altered_gate_command', 'Exact command', lambda: verifier.audit(repo, changed))
changed = changed_json('raw/typed-fixed-1b1c3e3.inputs.json', lambda data: data['inputs'][0].update(sha256='0' * 64))
refused('altered_committed_input_hash', 'Exact538 source bindings', lambda: verifier.audit(repo, changed))
changed = changed_json('raw/stack-diag-02-dirty.inputs.json', lambda data: data.update(head=verifier.SOURCE))
refused('dirty_run_mislabeled_as_fixed_source', 'Dirty capture identity/stability', lambda: verifier.audit(repo, changed))
changed = changed_json('raw/typed-90fae148.result.json', lambda data: data.update(exit=0))
refused('historical_abort_mislabeled_success', 'Historical aborted gate', lambda: verifier.audit(repo, changed))
changed = changed_json('raw/stack-diag-01-dirty.inputs.json', lambda data: data.update(dirty=data['dirty'].replace('STAGE entered', 'STAGE changed')))
refused('diagnostic_patch_disagrees_with_captured_hash', 'Reconstructed diagnostic input', lambda: verifier.audit(repo, changed))
changed = dict(payload)
changed['raw/stack-probe-if-01-dirty.log'] = changed['raw/stack-probe-if-01-dirty.log'].replace(b'PROBE after malformed case 1:', b'PROBE after malformed case 0:')
refused('probe_case_mislabeled', 'Dirty probe result/case', lambda: verifier.audit(repo, changed))
result = subprocess.run([sys.executable, '-B', '-O', str(directory / 'verify.py'), '--repo', str(repo)], capture_output=True, text=True)
verifier.require(result.returncode != 0 and 'Optimized Python refused' in result.stderr, 'Optimized verifier must refuse before audit')
cases.append({'name': 'optimized_python_refused', 'refusal': 'Optimized Python refused'})
return {'source': verifier.SOURCE, 'positive_audit': positive, 'negative_control_count': len(cases), 'negative_controls': cases, 'verifier_sha256': verifier.sha((directory / 'verify.py').read_bytes()), 'controls_sha256': verifier.sha(Path(__file__).read_bytes()), 'method': 'Raw archive untouched. Semantic corruptions pass directly to audit() after authentic archive validation; ZIP structural negatives temporarily substitute only the test module pin. No compiler, artifact execution or worktree write.'}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--repo', type=Path, required=True)
args = parser.parse_args()
print(json.dumps(run(args.repo.resolve(), Path(__file__).resolve().parent), indent=2))
if __name__ == '__main__':
main()
@@ -0,0 +1,257 @@
"""Read-only portable S3 evidence verification; never compiles or runs artifacts."""
import argparse
from datetime import datetime
import hashlib
import io
import json
from pathlib import Path, PurePosixPath, PureWindowsPath
import re
import subprocess
import zipfile
SOURCE = '1b1c3e3abb6fc3d0934c9dbfbd21fddc8ae4f594'
PRIOR = '90fae14818f684bf778640be2e613f34e3c7e999'
ZIP_SHA = '4486f4c42b140ae4b9b3caf0aacdf151a2acb1fd85eda09a57764fd69e980574'
RECORDER = '328970b226d9f4c3b8d3feb7d64478fb9756310ae5dfd3caa43b23c8b9d6f245'
PREFIX = ['cargo', 'test', '--locked', '--offline', '-p', 'oa-core', '--lib']
SCOPES = ['Cargo.toml', 'Cargo.lock', 'rust-toolchain.toml', '.cargo', 'build', 'crates', 'assets', 'docs', 'examples', 'harness/fixtures', 'vendor']
STACK = 'deep_parse_index_plan_and_failed_build_cleanup_on_128k_stack'
NEW = 'every_parent_shape_rejects_depth129_and_cleans_children_on_128k_stack'
PATHS = 'maximum_index_paths_compare_lookup_debug_and_drop_on_128k_stack'
OVERFLOW = 3221225725
def require(ok, message):
if not ok:
raise RuntimeError(message)
def sha(raw):
return hashlib.sha256(raw).hexdigest()
def unique(pairs):
value = {}
for key, item in pairs:
require(key not in value, 'Duplicate JSON key: ' + key)
value[key] = item
return value
def document(raw):
return json.loads(raw, object_pairs_hook=unique)
def git(repo, *args, data=None):
return subprocess.check_output(['git', *args], cwd=repo, input=data)
def source_inputs(repo, commit):
paths = git(repo, 'ls-tree', '-r', '--name-only', commit, '--', *SCOPES).decode().splitlines()
require(len(paths) == len(set(paths)) == 538, 'Exactly538 committed test inputs')
batch = git(repo, 'cat-file', '--batch', data=''.join(f'{commit}:{p}\n' for p in paths).encode())
offset, inputs, blobs = 0, [], {}
for path in paths:
end = batch.index(b'\n', offset)
_, kind, size = batch[offset:end].split()
size = int(size)
raw = batch[end + 1:end + 1 + size]
require(kind == b'blob' and len(raw) == size and batch[end + 1 + size:end + 2 + size] == b'\n', 'Git blob framing: ' + path)
inputs.append({'path': path, 'bytes': size, 'sha256': sha(raw)})
blobs[path] = raw
offset = end + size + 2
require(offset == len(batch), 'Complete Git blob stream')
return inputs, blobs
def archive_payload(raw):
require(sha(raw) == ZIP_SHA, 'Evidence ZIP SHA256 differs')
with zipfile.ZipFile(io.BytesIO(raw)) as archive:
names = archive.namelist()
require(len(names) == len(set(names)) == 61, 'Exact unique ZIP membership')
for name in names:
path = PurePosixPath(name)
require(not path.is_absolute() and '..' not in path.parts and str(path) == name and '\\' not in name and ':' not in name, 'Unsafe ZIP path')
payload = {name: archive.read(name) for name in names}
manifest = document(payload['manifest.json'])
require(manifest['source'] == SOURCE, 'Archive manifest source')
listed = [item['path'] for item in manifest['files']]
require(len(listed) == len(set(listed)) and set(listed) | {'manifest.json'} == set(payload), 'Archive manifest membership')
for item in manifest['files']:
require(len(payload[item['path']]) == item['bytes'] and sha(payload[item['path']]) == item['sha256'], 'Archive member binding: ' + item['path'])
return payload
def apply_recorded_diff(blobs, diff):
"""Reconstruct diagnostic source without touching an index or worktree."""
changed = {}
sections = re.split(r'(?m)^diff --git ', diff)[1:]
require(sections, 'Missing dirty diff')
for section in sections:
lines = section.splitlines(keepends=True)
names = re.fullmatch(r'a/(\S+) b/(\S+)\n', lines[0])
require(names is not None and names[1] == names[2] and names[1] in blobs, 'Unexpected diagnostic diff path')
path = names[1]
old = blobs[path].decode().splitlines(keepends=True)
output, cursor, index = [], 0, 1
while index < len(lines) and not lines[index].startswith('@@ '):
index += 1
require(index < len(lines), 'Missing diagnostic hunk')
while index < len(lines):
match = re.match(r'@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@', lines[index])
require(match is not None, 'Invalid diagnostic hunk header')
old_start, old_count = int(match[1]), int(match[2] or 1)
new_start, new_count = int(match[3]), int(match[4] or 1)
start = old_start - 1 if old_count else old_start
require(cursor <= start <= len(old), 'Diagnostic hunk position')
output.extend(old[cursor:start])
cursor = start
require(len(output) == (new_start - 1 if new_count else new_start), 'Diagnostic new hunk position')
consumed = emitted = 0
index += 1
while index < len(lines) and not lines[index].startswith('@@ '):
line = lines[index]
require(line[:1] in [' ', '+', '-'], 'Unsupported diagnostic patch line')
if line[0] in ' -':
require(cursor < len(old) and old[cursor] == line[1:], 'Diagnostic old source mismatch')
cursor += 1
consumed += 1
if line[0] in ' +':
output.append(line[1:])
emitted += 1
index += 1
require((consumed, emitted) == (old_count, new_count), 'Diagnostic hunk counts')
output.extend(old[cursor:])
require(path not in changed, 'Repeated diagnostic path')
changed[path] = ''.join(output).encode()
return changed
def audit(repo, payload):
read = lambda name: payload['raw/' + name]
doc = lambda name: document(read(name))
closure = document(payload['closure.json'])
expected, blobs = source_inputs(repo, SOURCE)
prior_inputs, prior_blobs = source_inputs(repo, PRIOR)
require(git(repo, 'rev-parse', SOURCE + '^').decode().strip() == PRIOR, 'Correction parent')
require(sha(payload['helpers/run-analysis-source3-gate-20260922.py']) == RECORDER, 'Recorder bytes')
for manifest_name, commit, count in [
('analysis-source8-s3-index-source-manifest-20260913.json', '1f8283236f8ccdefca13814003e7f046860b4f22', 7),
('analysis-source8-s3-index-correction-source-manifest-20260913.json', PRIOR, 2),
]:
manifest = document(payload['notes/' + manifest_name])
require(manifest['source'] == commit and len(manifest['files']) == count, 'Historical source artifact manifest')
for item in manifest['files']:
raw = git(repo, 'show', commit + ':' + item['path'])
require(len(raw) == item['bytes'] and sha(raw) == item['sha256'] and git(repo, 'rev-parse', commit + ':' + item['path']).decode().strip() == item['git_blob'], 'Historical source artifact: ' + item['path'])
names = re.findall(r'^([^\r\n]+): test\r?$', read('list-fixed-1b1c3e3.log').decode(), re.M)
old_names = re.findall(r'^([^\r\n]+): test\r?$', read('list-90fae148.log').decode(), re.M)
require(len(names) == len(set(names)) == 167 and len(old_names) == len(set(old_names)) == 166, 'Actual listing sizes')
qualified_new = 'language::analysis_source::tests::' + NEW
require(set(names) == set(old_names) | {qualified_new} and qualified_new not in old_names, 'Retained166 plus one new case')
require(len([n for n in names if 'language::analysis_source' in n]) == 59 and len([n for n in names if 'language::raw_tokens' in n]) == 1, 'Typed/token listing counts')
for module, count in [('index', 9), ('update', 5)]:
path = f'crates/oa-core/src/language/analysis_source/{module}/tests.rs'
authored = [f'language::analysis_source::{module}::tests::{name}' for name in re.findall(r'#\[test\]\s*fn (\w+)\(', blobs[path].decode())]
require(len(authored) == count and set(authored) == {n for n in names if n.startswith(f'language::analysis_source::{module}::tests::')}, 'Source-derived membership: ' + module)
selectors = {'stack-fixed-original-1b1c3e3': STACK, 'stack-fixed-shapes-1b1c3e3': NEW, 'stack-fixed-paths-1b1c3e3': PATHS, 'typed-fixed-1b1c3e3': 'language::analysis_source', 'token-fixed-1b1c3e3': 'language::raw_tokens', 'list-fixed-1b1c3e3': None, 'strict-fixed-1b1c3e3': 'clippy'}
environment, artifacts, last_end = None, {}, None
for stem, selector in selectors.items():
record, log = doc(stem + '.result.json'), read(stem + '.log').decode()
require(record['source'] == record['head_after'] == SOURCE and type(record['exit']) is int and record['exit'] == 0 and record['changed_inputs'] == [], 'Clean source/result: ' + stem)
require(record['inputs'] == stem + '.inputs.json' and doc(record['inputs']) == {'source': SOURCE, 'inputs': expected}, 'Exact538 source bindings: ' + stem)
require(record['launcher_sha256'] == RECORDER, 'Recorded launcher')
start, end = (datetime.fromisoformat(record[k]) for k in ['start', 'end'])
require(start.tzinfo is not None and end.tzinfo is not None and start <= end and (last_end is None or last_end <= start), 'Closed ordered gate times')
last_end = end
if environment is None:
environment = record['environment']
require(record['environment'] == environment and environment['CARGO_BUILD_JOBS'] == '1' and PureWindowsPath(environment['CARGO_TARGET_DIR']).name == 'analysis-checked-save-target-20260913', 'Recorded environment')
if selector == 'clippy':
command = ['cargo', 'clippy', '--locked', '--offline', '-p', 'oa-core', '--all-targets', '--', '-D', 'warnings']
require(record['artifacts'] == [] and 'Finished `dev` profile [unoptimized]' in log and not re.search(r'^\s*(warning|error)(\[|:)', log, re.M), 'Closed clean strict log')
selected = []
else:
require(len(record['artifacts']) == 1, 'One recorded test artifact')
artifact = record['artifacts'][0]
emitted = re.findall(r'Running unittests [^\r\n]+ \(([^\r\n]+\.exe)\)', log)
require(len(emitted) == 1 and PureWindowsPath(emitted[0]) == PureWindowsPath(artifact['path']) and artifact['bytes'] > 0 and re.fullmatch('[0-9a-f]{64}', artifact['sha256']), 'Recorded executable/log binding')
artifacts[stem] = artifact
if selector is None:
command = PREFIX + ['--', '--list']
selected = names
require(record['listing_members'] == names and record['listing_count'] == 167 and re.findall(r'^(\d+) tests, (\d+) benchmarks\s*$', log, re.M) == [('167', '0')], 'Listing receipt and summary')
else:
command = PREFIX + [selector, '--', '--test-threads=1']
selected = [name for name in names if selector in name]
rows = re.findall(r'^test ([^\r\n]+) \.\.\. (ok|FAILED|ignored)\r?$', log, re.M)
require(rows == [(name, 'ok') for name in selected], 'Exact ordered runtime results: ' + stem)
require(re.findall(r'^test result: ok\. (\d+) passed; (\d+) failed; (\d+) ignored; (\d+) measured; (\d+) filtered out;', log, re.M) == [(str(len(selected)), '0', '0', '0', str(167-len(selected)))], 'Runtime summary')
require(record['command'] == command, 'Exact command: ' + stem)
if selector is not None:
require(record['listing_count'] == 0 and record['listing_members'] == [], 'Non-list metadata')
saved = closure['gates'][stem]
require([saved[k] for k in ['command', 'exit', 'start', 'end', 'names']] == [command, 0, record['start'], record['end'], selected], 'Closure gate facts')
require(saved['pins'] == {stem+s: sha(read(stem+s)) for s in ['.inputs.json', '.log', '.result.json']}, 'Closure raw pins')
require(artifacts['stack-fixed-original-1b1c3e3'] == artifacts['stack-fixed-shapes-1b1c3e3'] == artifacts['stack-fixed-paths-1b1c3e3'], 'First focused build receipt')
require(artifacts['typed-fixed-1b1c3e3'] == artifacts['token-fixed-1b1c3e3'] == artifacts['list-fixed-1b1c3e3'], 'Final restored clean build receipt')
require([closure[k] for k in ['source', 'parent', 'verified_inputs', 'listing', 'typed_pass', 'index_pass', 'update_pass', 'token_pass', 'strict_exit']] == [SOURCE, PRIOR, 538, 167, 59, 9, 5, 1, 0], 'Closure counts')
for stem, mode in [('list-90fae148', 'list'), ('typed-90fae148', 'failed')]:
record, log = doc(stem + '.result.json'), read(stem + '.log').decode()
require(record['source'] == record['head_after'] == PRIOR and record['changed_inputs'] == [] and record['launcher_sha256'] == RECORDER, 'Historical frozen source')
require(doc(stem + '.inputs.json') == {'source': PRIOR, 'inputs': prior_inputs}, 'Historical538 inputs')
if mode == 'list':
require(record['exit'] == 0 and record['listing_members'] == old_names and record['listing_count'] == 166 and record['command'] == PREFIX + ['--', '--list'], 'Historical listing')
else:
require(record['exit'] == OVERFLOW and record['command'] == PREFIX + ['language::analysis_source', '--', '--test-threads=1'] and 'STATUS_STACK_OVERFLOW' in log and STACK + ' ...' in log and 'test result:' not in log, 'Historical aborted gate')
selected = [n for n in old_names if 'language::analysis_source' in n]
failing = 'language::analysis_source::index::tests::' + STACK
rows = re.findall(r'^test ([^\r\n]+) \.\.\. (ok|FAILED|ignored)\r?$', log, re.M)
require(rows == [(n, 'ok') for n in selected[:selected.index(failing)]] and 'running 58 tests' in log, 'Historical completed prefix before abort')
dirty = ['stack-diag-01-dirty', 'stack-diag-02-dirty', 'stack-probe-sum-01-dirty', 'stack-probe-if-01-dirty', 'stack-probe-power-01-dirty', 'stack-probe-trailing-01-dirty']
for index, stem in enumerate(dirty):
capture, result, log = doc(stem + '.inputs.json'), doc(stem + '.result.json'), read(stem + '.log').decode()
diagnostic = index < 2
commit, original = (PRIOR, prior_blobs) if diagnostic else (SOURCE, blobs)
require(capture['head'] == commit and result['changed'] == [], 'Dirty capture identity/stability')
patched = apply_recorded_diff(original, capture['dirty'])
expected_paths = {'crates/oa-core/src/language/analysis_source/index/tests.rs'} if diagnostic else {'crates/oa-core/src/language/analysis_source/tests.rs'}
if index == 1:
expected_paths.add('crates/oa-core/src/language/analysis_source/syntax.rs')
require(set(patched) == expected_paths, 'Bounded diagnostic paths')
for path, raw in original.items():
require(capture['inputs'][path] == sha(patched.get(path, raw)), 'Reconstructed diagnostic input: ' + path)
expected_filter = STACK if diagnostic else 'diagnostic_malformed_cleanup_on_128k_stack'
require(capture['command'] == PREFIX + [expected_filter, '--', '--test-threads=1', '--nocapture'], 'Diagnostic command')
if diagnostic:
require(result['exit'] == OVERFLOW and 'STATUS_STACK_OVERFLOW' in log and 'STAGE failed builds dropped' in log and 'STAGE finished' not in log, 'Diagnostic abort')
if index == 1:
require('STAGE node rejecting depth 129 before kind drop' in log and 'STAGE node rejected kind dropped' not in log, 'Isolated rejected-kind cleanup')
else:
case = index - 2
require(result['exit'] == 0 and f'PROBE before malformed case {case}' in log and f'PROBE after malformed case {case}:' in log and '1 passed; 0 failed; 0 ignored' in log, 'Dirty probe result/case')
return {'source': SOURCE, 'verified_git_inputs': 538, 'listing': 167, 'typed_pass': 59, 'index_pass': 9, 'update_pass': 5, 'token_pass': 1, 'strict_exit': 0, 'historical_overflow_retained': True, 'dirty_diagnostics': 2, 'dirty_direct_parser_probes_pass': 4, 'recorded_test_artifacts': artifacts, 'scope': 'Private S3 only; no full-core runtime/public8/integration/native/release acceptance. Recorded binary identities verified against receipts, not rehashed executable bytes; binaries intentionally absent.'}
def verify_package(directory):
manifest = document((directory / 'manifest.json').read_bytes())
require(manifest['source'] == SOURCE and manifest['archive_sha256'] == ZIP_SHA, 'Outer manifest identity')
names = [item['path'] for item in manifest['files']]
require(set(names) == {'.gitattributes', 'README.md', 'verify.py', 'negative_controls.py', 'checks.json', 'evidence.zip'} and len(names) == len(set(names)), 'Outer manifest membership')
for item in manifest['files']:
raw = (directory / item['path']).read_bytes()
require(len(raw) == item['bytes'] and sha(raw) == item['sha256'], 'Outer manifest bytes: ' + item['path'])
return archive_payload((directory / 'evidence.zip').read_bytes())
def main():
require(__debug__, 'Optimized Python refused')
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--repo', type=Path, required=True, help='Git repo containing immutable source commits; current checkout is not used')
parser.add_argument('--evidence-dir', type=Path, default=Path(__file__).resolve().parent)
args = parser.parse_args()
print(json.dumps(audit(args.repo, verify_package(args.evidence_dir)), indent=2))
if __name__ == '__main__':
main()