Sources, pushdown and derived grids

The request also carries groupBy, groupPath, pivotBy, totals and your own context, so a server that can group and aggregate does that work instead of the browser. Blocks are fetched as the viewport reaches them and cached; changing the sort or the filter invalidates the cache and re-queries.

The filters your callback receives is the same condition tree documented above. You are not handed an opaque object to reverse-engineer, it is the published format, and the same shape you would have written by hand.

An update is a patch. Fields absent from it are untouched, so a delta arriving from a websocket or coming back from a save can be applied as-is without reading the row first. This has to be said explicitly because the opposite: assigning the patch over the row: looks identical for a caller who happens to send whole rows and silently destroys data for one who does not.

Coalescing merges fields rather than keeping the last message. A feed sending {price} and {volume} as separate messages inside one window keeps both. Coalescing may reorder work; it may not lose it.

Flushing happens on a frame, with a timer behind it. A queued batch lands on a paint boundary, which is what makes "ten thousand updates, one repaint" true rather than usually true, a timer can fire twice between two paints. But requestAnimationFrame is not guaranteed to fire at all: a backgrounded tab stops firing it entirely. So a frame and a timer are armed together and the first to arrive wins. In a foreground tab the frame always wins, at ~16ms against a 50ms fallback; in a hidden tab the timer keeps the feed applying instead of the grid silently stalling with every caller's promise unresolved.

A long flush defers rather than blocks. updates.budgetMs caps how long one flush spends applying; over budget, the remainder returns to the queue and lands next frame, and the promise a caller is holding resolves when their rows actually land rather than when the first slice does. Slicing is by row and only for updates, a partially applied row is not a state the store should be in, and splitting a structural change would re-run the pipeline twice for one batch. stats().deferrals rising steadily means the feed is arriving faster than the grid can apply it.

Rejections are reported, never thrown. Throwing would abandon the rows that were fine. An update or remove naming a row that is not here is unknown-id; an add whose key already exists is duplicate-id and is refused, because selection, expansion, comments and the key index all resolve one key to one row and admitting a second corrupts every one of them at once.

This runs the minimum pipeline. An update touching no sorted, filtered or grouped column skips those stages entirely and only the totals and the affected cells refresh. Adds and removals are structural and re-run everything.

Removals tombstone in place rather than compacting, so every existing index stays valid, which is what lets selection, expansion state and cached permutations survive a delete.

A remote or paged source holds the page it has fetched, not the whole set, so there is nothing there to walk past the filters. It warns and walks what it has rather than quietly returning the filtered rows, a caller who asked for everything and silently received a subset gets a number that looks entirely plausible and is wrong.

Pausing is a button, not a guess. Inferring it from whether the user looks busy sounds friendlier and is wrong in both directions: too broad and a mouse resting on the grid freezes the feed until someone reloads, too narrow and rows jump the instant somebody stops moving in order to read. An explicit control has no heuristic to get wrong and no invisible state to explain.

Merging continues while paused, so a long pause costs one entry per changed row rather than one per update. Forty updates to one row are one row of work when play is pressed, and coalesced is the thirty-nine, which is the number nobody could see before.

Bounding the rows themselves is separate. The log bounds change history; a streaming source also needs to bound row retention, or a grid left up overnight holds every row it was ever sent. Set source.maxRows and the stream becomes a sliding window, dropping the oldest as new ones arrive and reporting how many it let go through evicted on the progress report.

The log keeps the raw sequence, not the merged one. Merging is right for applying a backlog quickly and wrong for looking at what happened, because the intermediate states are exactly what a time scrubber would move between. It survives the flush, pending is what is waiting, the log is what happened, and it is capped, so a grid paused over lunch holds the recent past and reports how much it dropped rather than taking the tab with it.

Everything is off unless declared. An adapter that declares nothing still works: the grid fetches and does all the work itself. That is the safe direction to be wrong in. Declaring an operator the engine does not really apply is the unsafe one, because the grid will trust it and stop checking.

Splitting a filter is not symmetric. An and group narrows with each condition, so the supported conjuncts go to the engine and the rest stay behind: the engine returns a superset and the grid narrows it. An or group widens with each branch, so pushing only the supported branches would return fewer rows than the filter allows, and the grid cannot recover rows that were never fetched. A disjunction that is not fully supported therefore stays whole on the client. The same asymmetry governs column pruning in the facet path.

Residual work needs the whole result. When anything is left over, the source stops asking for windows and asks for everything, applies the remainder, and pages from what it holds. Filtering a window on the client is not a slower route to the right answer, it is a fast route to a wrong one: the rows that belong on page one may sit on page nine, and the total is whatever the engine happened to count.

from is any FROM expression, so read_parquet('s3://bucket/*.parquet') is as valid as a table name. Values are bound through prepared statements; a connection without prepare is used only for unfiltered queries, because interpolating a user's filter into SQL is the one thing worse than not filtering at all.

Column and table names are checked against an identifier pattern rather than escaped, and a name that fails is refused. Integers past the safe range are kept as strings instead of being rounded into a plausible lie.

demo/duckdb.html runs this against a Parquet file of several million readings with no server involved.

A change is patched, not re-derived. When a row changes in the source, the derived grid updates the groups that row belongs to rather than rebuilding the lot. Five hundred updates against a two hundred thousand row source cost under 300 ms in total. Set refresh to live, manual or a number of milliseconds to change the coalescing; idle is the default and settles to a frame.

Derived grids are read-only. There is one copy of the data and it lives in the source. Write there and the derived grid follows.

The key comes for free. A derived grid keys on __key, which the source writes onto every row it produces: the group value, the profiled column, or the source row's own key when nothing is grouped. Set rowKey only to override it.

A left join is usually the one you want. An inner join quietly drops the rows that did not match, and those are often the finding: the site with no owner, the payment with no invoice. left keeps them visible with the partner's fields empty, so the gap is something you can see and sort by rather than something you have to notice is missing.

First match wins. The join is a lookup, not a cross product: a row on the left produces exactly one row out, so a grid of ten thousand rows stays a grid of ten thousand rows and cannot silently multiply.

Both sides are live. A change on either grid updates the join, and it is patched from whichever side changed rather than rebuilt.

A panel does not filter itself. The filter a summary pushes onto its source is excluded when that same summary re-derives. Without that, clicking one region collapses the panel to the single row you just clicked, and there is nothing left to click next. With it, the panel keeps its full set of regions with the chosen one marked, which is what makes a second click possible at all.

Several panels compose. Each pushes its own filter onto the shared source and each excludes only its own, so region and status narrow the detail together while both panels stay navigable.

The API came first and the panel second, on purpose. Instrumentation written behind a UI gets shaped by the layout: it reports what is convenient to display rather than what is true, and it cannot be tested. An API that stands on its own can be asserted against, and the assertion above is not one most grids can support.

Most of it already existed inside the grid. The renderer had counted cell writes, row updates and paints all along; the column store could already report a real byte footprint per column, summing backing array, presence bitset and dictionary. Neither was reachable from the public API. Exposing what a system already knows is usually a better first move than measuring something new.

Warnings are mostly collection, not detection. The grid has 160 places that warn once per cause, each already carrying a stable de-duplication key, which is exactly the stable identifier a support conversation needs. They went to the console and nowhere else. The console interleaves with your own logging, does not survive a reload, and cannot be asked what it has already complained about. They are now kept as records too.

Every warning names values, not just a condition. "Something is slow" is a warning nobody can act on. Each carries the specific numbers, a stable id, and is dismissible for the session but not permanently, a permanently dismissible warning is one nobody sees again after the person who dismissed it leaves.

The checks are tested for silence as much as for detection. A clean grid must raise nothing. A checker that cries wolf is one developers learn to ignore, and then it is worth less than no checker at all. The accessibility checks found two false positives in themselves during development: first comparing aria-rowcount against the row count when ARIA counts header rows too, then counting header row *elements*, of which a single-level header has three because the header is built once per pinned region.

Instrumentation must not change what it measures. Counters are integers incremented where the work already happened. Render phases are four performance.now() marks around existing sections. Timings are sampled, a bounded window of recent operations, and every report names which of its figures are sampled, because a number whose provenance is unclear is worse than no number. Paint wait is the browser's and is deliberately not claimed.

Render causes are captured, not inferred. By the time a paint runs, several distinct causes have collapsed into the same dirty flags, so working backwards gives a plausible answer rather than a true one. The renderer records the structural reason when the invalidation arrives; the semantic one (filter, sort, data) is only knowable from the event that preceded it, so the DOM layer supplies that and the structural reason is the fallback.

Providers are wrapped where they are installed, not at each call site, so one added later is instrumented by construction. The wrapper returns exactly what the original returned and re-throws exactly what it threw. Failures are kept, because a provider failure is usually swallowed by the host application's own error handling before a developer sees it, and "the grid is behaving strangely" is where that conversation otherwise starts.

The heat overlay uses two colours because there are two findings. A cell written with a value it did not hold is work the grid had to do. A cell rewritten with the value already in it is work it did not. Only the second is waste, and a counter alone will never tell you where it is. The overlay records a baseline when switched on, because otherwise the first paint has nothing to compare against and tints nothing, which reads as broken rather than as empty.

The module imports nothing. Not the grid, not a shared helper. The bundler inlines whatever a module imports: the framework adapters stay small because createGrid is handed to them, while the web component carries the grid with it because it imports it. A devtools module that imported anything from core would put the whole grid inside a bundle whose entire promise is that deployments not using it pay nothing.

The support bundle carries no row data. Configuration, query state, timing, warnings, provider statistics, version and environment, and nothing from your data. Stated as a guarantee because a bundle that had to be inspected for confidential values before sending is a bundle that never gets attached to the ticket.

It observes and never mutates. A configuration editor in a debug panel is tempting and would create a second path into grid state that has to be kept correct forever. No telemetry: nothing leaves the browser unless you export a bundle yourself.