learnharness/docs/research/graph-rendering.md

Graph rendering & cluster-first layout — research findings

Compiled 2026-09-13 from four research passes (library fact-check ×2, prior-art ×2), all claims source-verified by subagents against shipped package code (npm pack tarballs of sigma@3.0.3, @sigma/layer-webgl@3.0.0), the sigma.js GitHub storybook source, and primary docs. Speculation is marked where it appears. This document is the decision basis for the rendering rebuild; nothing has been implemented yet.

---

1. Requirements (as of this research)

Current symptoms to eliminate:

• Hull layers lag behind nodes during pan (desync).
• Settings changes shift hulls up-left until the next interaction (canvas-sizing race).
• GPU ~50–100% during pan/zoom (redundant full-canvas 2D repaints per camera event + WebGL re-render per frame; the earlier ctx.filter: blur per frame caused coil whine).
• Stale/garbage boundary shapes at extreme zoom-out.

Feature set:

• Circles + directed arrows; precomputed layout (no runtime physics).
• Cluster regions: hull boundary around each community, dashed stroke, soft translucent fill, stable color per region, region name label.
• Inverse zoom gates: node labels when zoomed IN, region labels when zoomed OUT, cross-fading over a shared span.
• Pan, zoom-to-cursor, hover glow, click → detail card, edge-type filtering, camera survives UI changes. 60fps, minimal GPU, ordinary laptop.

New direction (user, 2026-09-13):

• Cluster-first placement: layout depends on clustering — clusters placed as units, nodes within a cluster close together (compact, natural hull shapes as a consequence).
• Freeform experimentation: a dev-only playground to try layouts/shapes/styles against real and synthetic data; renderer must be dumb, data-driven, swappable.

Scale: 338 nodes / 211 edges today; low thousands expected (notes imports).

2. Verified facts about the current stack (sigma.js v3)

• v3.0.3 ships a PUBLIC custom-layer API — createLayer / createCanvas / createCanvasContext / killLayer (typed in shipped .d.ts). Our current firstChild DOM injection was improvising a worse version of a supported API.
• Sigma sizes and DPR-scales ALL registered canvases in its own resize() — using the public API eliminates the canvas-sizing race by construction.
• afterRender fires once per rendered frame (rAF-debounced scheduleRender()). Camera motion → full re-render each rAF. Hover is cheap: it only redraws the hoverNodes WebGL layer + edgeLabels canvas via a separate rAF; no afterRender, no full re-render. Sigma is strictly event-driven: no continuous loop, no rendering when idle (verified: zero setInterval, all rAF sites are schedulers).
• renderer.graphToViewport() is public and typed — the sanctioned graph→pixel path. renderer.matrix is private: building a ctx.setTransform() from it would be guessing at internals; per-vertex graphToViewport() is the supported route.
• The official cluster-label storybook example is exactly our pattern: custom DOM layer + afterRender + per-cluster graphToViewport() re-projection, maintained in the sigma repo. Our minimal-change fix path is a demonstrated usage.
• Real v3 layer order (verified in source, NOT in docs): edges (WebGL) → edgeLabels → nodes (WebGL) → labels → hovers → hoverNodes (WebGL) → mouse.

@sigma/layer-webgl@3.0.0 (official background-layer package)

• bindWebGLLayer inserts a background WebGL layer (beforeLayer: "edges"), redrawn on afterRender with the same camera matrix — fully in-loop, no desync.
• createContoursProgram renders METABALL-style implicit fields, not hull polygons (verified from the fragment shader: fullscreen quad, summed radial falloffs, analytic contour lines via dFdx/dFdy). Translucent fill: yes (arbitrary CSS color with alpha). Dashed stroke: NO — solid anti-aliased line only; no dash parameter exists. Per-community color: yes, but one program + one WebGL context per community, and each runs a fullscreen fragment pass per frame (browsers cap ~16 WebGL contexts — a problem for dozens of regions).
• Decisive fact: our exact visual (hull polygon + translucent fill + dashed stroke) is NOT achievable with shipped layer-webgl programs. It would require a custom WebGLLayerProgram (supported extension point, moderate effort, dashes DIY in GLSL) — whereas Canvas 2D gives it for free with ctx.setLineDash().
• v4-era programs (heatmap, density-splat, dashed/fill layer factories) are NOT in the v3 package.

sigma v4

• Real but beta (4.0.0-beta.5; docs site self-describes as alpha). Single stage canvas + public createLayer, GPU picking with downsizing, SDF labels, depth layers. Breaking migration. Not a production bet today.

Pan GPU cost at our scale

• Structural facts verified (event-driven idle; full re-render per camera frame; synchronous gl.readPixels picking per mousemove pre-#1546, since fixed).
• Absolute GPU cost of panning 338–3k nodes in v3: UNVERIFIED — no public benchmark exists; nobody has measured it. If it matters, we measure it ourselves.

3. Verified facts about the candidates

Candidate	Renderer	True hull fill+dash	Picking	Camera	Maintenance	License	Fit
sigma v3 + public 2D canvas layer	WebGL (+our 2D)	yes (our canvas, free)	GPU color-pick	built-in	active (3.0.3, 2026-04)	MIT	7/10
sigma v3 + layer-webgl contours	WebGL	no (metaballs, solid border)	—	built-in	active	MIT	6/10
sigma v4 + layers	WebGL, SDF	via custom program (beta risk)	GPU picking	built-in	beta	MIT	6 (risky)
d3-zoom + Canvas 2D (hand-rolled loop)	Canvas 2D	yes, trivially	distance check	d3-zoom (verified math)	d3 very active	ISC	8/10
PixiJS v8 + pixi-viewport	WebGL/WebGPU/Canvas	yes (Graphics; dashes via shim pixi-dashed-line)	built-in events	pixi-viewport 6.0.3	very active	MIT	7/10
cytoscape.js 3.34	Canvas 2D	via bubblesets/layers extensions	built-in	built-in	active	MIT	5/10
vis-network 10.1	Canvas 2D	no — clustering = supernodes (collapse, opposite of our regions)	built-in	built-in	sporadic	MIT	2/10
@cosmos.gl/graph (Cosmograph engine)	WebGL (luma.gl)	no (cluster = layout force only)	GPU	built-in	active	MIT	3/10

Fact nuances we must not over-trust:

• "Canvas 2D is GPU-rasterized": half-true. Skia may raster on CPU or GPU depending on browser/OS config. Either way the cost driver is repaint count × pixel area (DPR²), not "Skia = GPU".
• Canvas 2D at 60fps for hundreds–low-thousands of elements: supported by practitioner evidence (cytoscape is fully canvas-based into the thousands; 10k animated canvas elements ≈ 55fps in one 2025/26 benchmark). Pixel area matters more than node count at our scale.
• Obsidian's graph view renderer: UNVERIFIED — no primary source. Do not cite.
• PixiJS exact bundle size: unmeasured. yFiles: commercial, unverified details.

4. Prior art for our specific features

Region/hull visuals

• Nobody in the graph-tool world ships our region aesthetic: Gephi desktop, Gephi Lite (the flagship sigma app — zero hull code in its tree), Obsidian — all partition by node coloring only. The region feature is a differentiator, not a copied pattern.
• The aesthetic reference: ggforce geom_mark_hull (expand + round the hull, low-alpha fill, integrated label placement).
• Bostock's thick-stroke hull trick: one path, wide stroke (e.g. 32px) + linejoin: round = padded, rounded hull with zero polygon-offset math. Replaces our offset-normal code outright.
• Named-region UX validation: Excalidraw frames / tldraw regions (dashed titled container around content).
• Closest feature twin: Cosmograph — documents "cluster labels when zoomed out, point labels when zoomed in" as a first-class use case (our exact inverse gate).

Zoom-gated labels

• Consensus pattern = map-style per-layer zoom ranges (Mapbox/MapLibre minzoom/maxzoom on symbol layers). State labels at far zoom, city labels at near zoom is the standard cartographic solution; our inverse gates are the same mechanism with opposite ranges. Sigma's own labelRenderedSizeThreshold is the zoom-in gate.
• Cross-fade + hysteresis over a shared span: no single canonical writeup (medium confidence); nearest established practice is map label collision systems.

The sync pattern (fixes our desync class structurally)

• Canonical and repeatedly documented: one rAF loop with a dirty flag; input/DOM events only mutate state. All layers drawn in the same frame, under the same camera transform — either same canvas (setTransform once, draw back-to-front) or stacked canvases all redrawn by the same loop. Sources: browser.engineering, MDN "Optimizing canvas", web.dev canvas performance.

Picking

• Canvas 2D at our scale: nearest-node distance check per pointer event is the norm, trivially fast (µs for 3k nodes; spatial grid only past ~10k).

5. Cluster-first layout (the new direction)

• Two-level pattern is well-trodden: place cluster anchors first (quotient graph — one super-node per community, Gephi's "Groups by Partition" plugin is the desktop prior art), layout nodes within each cluster, compose by translating local layouts to cluster anchors. Bostock's Clustered Force Layout lineage; cosmos.gl's cluster force verified from source (per-frame center-of-mass pull, or user-pinned positions — cluster anchors as data independent of member layout).
• graphology FA2 has NO cluster-aware settings (full settings list read; gravity is global). LinLog mode (linLogMode: true) is the documented "communities separate tightly" switch — a one-afternoon A/B candidate before any custom pipeline.
• No turnkey JS package does the composed two-level pipeline — assemble from graphology primitives (~150 lines of glue: quotient build, per-cluster FA2 with circular init, compose transform, noverlap cleanup). The combination is our synthesis from verified primitives.
• Deterministic given fixed init positions (no RNG dependency in FA2).
• Server-side at extraction time is the better fit for our architecture: positions become data, rendering stays dumb. Browser worker fallback for live re-layout.
• Harness precedents: yFiles layout playground (polished), Observable/bl.ocks lineage (notebook = graph + sliders + canvas), react-sigma layouts example. No canonical OSS playground for our stack — copy the config-driven pattern (algorithm dropdown + parameter sliders + seed input + live canvas).

6. Options (the decision)

Option A — Stay on sigma v3; mount the hull canvas via the public createCanvas API; redraw on afterRender via graphToViewport(). Supported, minimal-change, demonstrated by sigma's own cluster-label storybook. Kills the desync, the sizing race, and the multi-event redraws (once per frame, sigma-sized). Our exact visuals stay (dashes free in 2D). Keeps sigma's camera/labels/picking machinery. Residual: full-canvas 2D repaint per interaction frame remains (raster cost, mitigable with move-gating like sigma's own hideLabelsOnMove); WebGL + 2D raster still both run per frame. Est: ~1 day.

Option A′ — sigma + custom WebGLLayerProgram for hulls. Supported extension point, but dashes are DIY GLSL and we'd rebuild in shaders what Canvas 2D gives free. No reason given A exists. Est: several days, for a worse iteration loop.

Option B — d3-zoom + hand-rolled single Canvas 2D renderer. One canvas, one rAF + dirty flag, world-space geometry + one transform, all layers back-to-front in the same frame. Lowest GPU/CPU, total control, perfectly matches the new requirements: renderer is a dumb draw function over data (layout experiments can't fight it), playground is trivial, cluster-first layout slots in as pure data. Costs ~600 lines + camera feel + picking + label gating owned by us. Est: ~1 focused day.

Option C — PixiJS v8 + pixi-viewport. Cleanest scene-graph answer (one ticker, sync by construction), WebGPU headroom. But: dashes need a third-party shim, no graph semantics, bundle, more machinery than Canvas 2D needs at our scale. Est: 1–2 days.

Recommendation: Option B as the target — it is the only option that structurally matches all three requirement sets (bugs, cluster-first placement, freeform experimentation), scored highest (8/10), and removes the vendor-internals class of problems permanently. Option A is a legitimate ~1-day stabilization if we want the current bugs dead immediately while deferring the rebuild; it reuses all our existing visual code and is the officially sanctioned pattern, but it keeps us on sigma's opinionated machinery and the experimentation direction will keep pushing against it.

Layout work rides along independent of renderer choice: first A/B LinLog-only FA2 (one afternoon), then the two-level pipeline (quotient → per-cluster FA2 → compose → noverlap) if needed, developed in the playground.

7. Facts we should NOT rely on

• "Obsidian's graph view uses renderer X" — unverified.
• "Sigma's documented canvas DOM order" — source-only, implementation detail, changed in v4.
• "Canvas 2D is always GPU-rasterized" — false as stated.
• "Sigma v4 is ready" — beta.
• "Sigma pan cost at our scale is X%" — nobody has measured it publicly.
• "@cosmograph/cosmos is the maintained Cosmograph engine" — stale npm channel (Dec 2024); current engine is @cosmos.gl/graph v3.
• Any PixiJS bundle-size number — unmeasured.

8. Open items for the implementation phase

• Measure pan GPU/CPU ourselves (Chrome profiler) before/after whichever path is chosen.
• Playground design: /playground dev-only route, config-driven (layout algorithm dropdown, parameter sliders, seed input, synthetic + real datasets), same renderer.
• Hull styling decision inside Option B: Bostock thick-stroke trick vs offset polygons (geometry already unit-tested in components/cluster-geometry.ts).
• Label-gate hysteresis: small dead-band in log-zoom to avoid threshold flicker.
• Whether layout runs server-side at extraction (persisted as data) or in-browser.