delete the mappings ledger — per-job in-memory resolution + notes fingerprint guard
The persisted (chatId, sourceId)→node table was a shared namespace across independent chunk/batch jobs: each chunk's extractor mints local ids fresh (p1..pN), so later chunks collided with earlier ones — propositions silently skipped as 'already resolved' (coverage checkpointed anyway: permanent silent loss) and edges resolved onto the wrong nodes (the 'everything links to everything' mush). Owner ruling 2026-09-29: mergeFindings resolves THIS job's ids in memory, nothing persists between jobs; cross-job identity is the auditor's linkTo verdicts alone. - contracts: NodeMapping dies; NotesSource.contentFp — sha256 of the text at the last run, the re-import guard (unchanged re-import = TRUE no-op, no model calls, no usage; changed text re-runs; legacy unstamped sources re-run once) - merger: job-local resolution map, in-submit duplicate ids skip once, edges resolve extractor-local ids only - ports/store: appendMapping/readMappings, LEDGER_FILES.mappings, rewriteLedgers.mappings gone (stale data/graph/mappings.jsonl just stops being read) - session-clusters: birthSessionOf is provenance-only — the legacy mapping-line fallback for provenance-empty notes nodes dies - deletion: no mapping rows to strip; stats lose mappingsRemoved - replay-layout batches nodes by first provenance point - graph.test pins the new independence semantics (re-merge re-creates; duplicate ids within one submit resolve once); extraction tests pin per-chunk merges + the fingerprint guard
This commit is contained in:
@@ -1221,11 +1221,19 @@ export function createExtractionPipeline(deps: ExtractionPipelineDeps): Extracti
|
||||
|
||||
/** Import ONE raw notes file: register the source (deterministic id per
|
||||
* file path — re-import keeps the same identity and its usage history)
|
||||
* and run the pipeline over it. */
|
||||
* and run the pipeline over it. Re-import guard (owner, 2026-09-29):
|
||||
* the file's content fingerprint decides — unchanged text is a TRUE
|
||||
* no-op (the mappings ledger used to dedupe this at merge time by
|
||||
* accident); changed text re-runs the whole file. */
|
||||
async function importNotes(input: NotesImportInput): Promise<EmissionStats> {
|
||||
const id = `src_${crypto.createHash("sha256").update(input.filePath).digest("hex").slice(0, 12)}`;
|
||||
const contentFp = crypto.createHash("sha256").update(input.text).digest("hex").slice(0, 16);
|
||||
if (graphStore.readSources().find((s) => s.id === id)?.contentFp === contentFp) {
|
||||
console.info(`[extraction ${id}] re-import with unchanged content (fp match) — skipping`);
|
||||
return { nodes: 0, edges: 0 };
|
||||
}
|
||||
const importedAt = new Date().toISOString();
|
||||
graphStore.upsertSource({ id, filePath: input.filePath, importedAt });
|
||||
graphStore.upsertSource({ id, filePath: input.filePath, importedAt, contentFp });
|
||||
return runNotesExtraction({ id, ...input, importedAt });
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user