Rows and data sources: JavaScript Data Grid API

API reference › Rows and data sources

grid.rows

Data usually arrives after the grid does. Build it with rows: [], then load when your fetch resolves, the sort, filters, grouping and column layout you set up in the meantime all survive, and apply to the new data.

const grid = createGrid(el, { columns, rowKey: 'id', rows: [] });
grid.overlay.show('loading');

const data = await fetch('/api/circuits').then(r => r.json());
grid.rows.load(data);          // replaces whatever was there
grid.overlay.hide();

grid.columns

const { createHeadlessGrid } = await import('../packages/core/src/index.js');

const grid = createHeadlessGrid({
  rowKey: 'id',
  rows: [{ id: 1 }],
  columns: ['a', 'b', 'c', 'd', 'e'].map((id) => ({ id, field: id, layout: { width: 100 } })),
});

const resized = [];
grid.on('column:resized', (e) => resized.push(e.ids ? e.ids.join(',') : e.id));

// A map of id to width is one call, one history entry, one event naming
// every id. Two more single calls land in the same tick - on a real
// renderer the three would coalesce into one relayout on the next frame
// instead of three, but `column:resized` still fires once per call, here
// and for `show`/`hide`/`pin` below, and every width read back is already
// the new one.
grid.columns.resize({ a: 120, b: 80 });
grid.columns.resize('c', 90);
grid.columns.resize('d', 70);

grid.columns.hide(['a', 'b']);
grid.columns.show('a');
grid.columns.pin(['c', 'd'], 'start');

const widths = ['a', 'b', 'c', 'd'].map((id) => grid.columns.get(id).layout.width).join('/');
const visible = grid.columns.visible().map((c) => c.id).join(',');
const pinned = grid.columns.all().filter((c) => c.layout.pin).map((c) => c.id).join(',');
grid.destroy();

return `resized ${resized.join('|')}; widths ${widths}; visible ${visible}; pinned ${pinned}`;

Header bands in the Columns tool panel

The columns tool panel lists every header band in header order, with its columns nested under it and nested bands nested again, so a band made in the header, by dragging onto a heading's centre third or from the column menu's Group with adjacent column, can be seen and undone from the panel too. Each band row has a collapse toggle, a tri-state checkbox that shows or hides every column in the band (mixed while only some are showing; columns with lockVisible are left alone), and Rename and Dissolve band, which call grid.columns.renameGroup and grid.columns.dissolveGroup exactly as the column menu does. Dissolving leaves the columns where they are and asks for no confirmation, as the menu does not; Undo restores the band.

Drag a column onto a band row, or onto a column of another band, to join it (groupColumns with groupId); drag a banded column onto a column outside the band to take it out (ungroupColumn, one level up). The panel does not form a new band from a drop on a loose column: that gesture belongs to the header. Keyboard: the band row is focusable, Enter collapses or expands it, Delete dissolves it, ↑/↓ walk the list. The panel repaints on the same events as the rest of the panel, so a band edit from either side shows on the other (a panel the keyboard is inside is left alone until focus leaves, as every panel is); collapsed bands stay collapsed across a repaint. The search box filters inside bands and keeps a band while any of its columns, or its own title, matches. Each band is a role="group" named by its title.

Sources

Where rows come from. memory is the default and needs no configuration.

A memory grid's rows may be declared either way. The top-level rows option is the usual one and what every example here uses; the same array inside the block works identically - createGrid(el, { columns, source: { mode: 'memory', rows } }) opens with exactly those rows, with the same count(), value(), text(), type inference and bound KPI and chart readings as createGrid(el, { columns, rows }), and rows.load() and rows.apply() behave the same afterwards. Either array is copied on ingest, so the one you passed is never written to (see rows under Configuration). Declare them once: a grid given both uses the top-level rows, ignores the block's, and warns once naming both places. Only memory reads a rows key from the block - the fetching modes take theirs from the server.

Load from a URL: a JSON or NDJSON file

createUrlSource(url, opts) points the grid straight at a file. A JSON file (a top-level array, or a nested array picked out with rowsPath or map) is read whole and handed over as rows. An NDJSON / JSONL file - one JSON value per line - is streamed in incrementally: the first rows render while the rest is still arriving, and a large file never sits in memory as one string. It is built on the stream source, so it inherits the frame-coalesced render, the stream:chunk / stream:end progress events and promotion to memory once a small file has fully landed.

Format is resolved in order: an explicit format: 'json' | 'ndjson' wins; else the URL extension (.ndjson/.jsonl vs .json); else the Content-Type (application/x-ndjson vs application/json); else a sniff of the first bytes, or a clear error asking for an explicit format. Pass fetch to supply auth or a proxy (default is the global fetch), headers to merge request headers, batchSize to tune NDJSON chunking (default 500), lenient: true to skip a malformed NDJSON line with a warning rather than failing, and poll (ms) to re-fetch on an interval, replacing the rows each pass. Every failure - a non-2xx status, a network error, a bad body, a malformed line - surfaces as source:error and leaves the grid usable, never an uncaught throw. The fetch is aborted when the grid is destroyed. Zero new dependencies: fetch, response.body.getReader() and TextDecoder.

A JSON file loaded through a mock transport so the example runs headless with no network; in an app, omit fetch and the global one is used. Run on every build.

const { createHeadlessGrid, createUrlSource } = await import('../packages/core/src/index.js');

// A mock transport, so this runs with no network. In your app, drop `fetch`.
const file = JSON.stringify([{ id: 1, city: 'Oslo' }, { id: 2, city: 'Lima' }, { id: 3, city: 'Cairo' }]);
const fetchImpl = async () => new Response(file, { headers: { 'content-type': 'application/json' } });

const grid = createHeadlessGrid({
  rowKey: 'id',
  columns: [{ field: 'id' }, { field: 'city' }],
  source: createUrlSource('https://example.test/cities.json', { fetch: fetchImpl }),
});

await new Promise((r) => setTimeout(r, 50)); // let the file load
const count = grid.rows.count();
grid.destroy();
return count; // 3

For NDJSON, point it at a .ndjson or .jsonl URL - createUrlSource('/events.ndjson') - and the rows stream in as they parse.

Derived sources: a grid built from another grid

Most dashboards put a summary panel beside a table, the top five sales people, the breakdown by region, the exceptions list. Built by hand, that panel runs its own query, and sooner or later somebody filters the table and the panel does not follow. Everyone who has shipped a dashboard has been in the meeting where two numbers on one screen disagree.

A derived grid removes the possibility. It is a second grid whose rows are built from the first (grouped, unnested, filtered, ranked or profiled) so the panel is the table, one derivation later, and one filter moves both. It answers the questions a summary panel exists for: the top five sales people, the most-sold SKUs, a statistical profile of whatever the user has filtered to.

It is a source rather than a new kind of grid, so everything downstream: its own sorting and filters, totals, shadow columns, formatting, export, themes: works on the result and knows nothing about where the rows came from. Charts bind to one as readily as to any grid.

source: {
  mode: 'derived',
  from: salesGrid,
  follow: 'filtered',

  groupBy: 'rep',
  select: { revenue: { of: 'amount', fn: 'sum' }, deals: { fn: 'count' } },
  sort: [{ col: 'revenue', dir: 'desc' }],
  limit: 5,
}

The pipeline runs in one order, and the order is the contract: unnest → where → bucket → group → reduce → sort → limit. where sits before grouping deliberately: filtering afterwards is a different question, which groups, not which rows, and one key cannot mean both.

// refresh: 'manual' - the summary re-derives only when the host asks.
refreshButton.addEventListener('click', () => summary.rows.load());

The relational statistics, as rows

A single-column statistic already has a route: select reduces a group with any kernel the totals row uses, and that table is a superset of the statistics one, so select: { p95: { of: 'amount', fn: 'p95' } } works, along with median, stddev, gini and the rest. statistics is for what select structurally cannot reach: the figures needing two or more columns, or a second grid. Like profile it is a terminal producer - it replaces the pipeline rather than joining it, and the two cannot be used together. Every row carries n, the rows the figure covered. Full detail, including the measured re-derive cost of each producer, is in the detail reference.

const { createHeadlessGrid } = await import('../packages/core/src/index.js');

const readings = Array.from({ length: 20 }, (_, i) => ({ id: i, t: i, v: 100 + i * 3, w: 50 - i }));
const plant = createHeadlessGrid({ rowKey: 'id',
  columns: [{ field: 't', type: 'number' }, { field: 'v', type: 'number' }, { field: 'w', type: 'number' }],
  rows: readings });
plant.rows.count();

// One row per column PAIR: { a, b, coefficient, n }. `w` runs exactly against `t`.
const pairs = createHeadlessGrid({ rowKey: '__key',
  columns: [{ field: 'a' }, { field: 'b' }, { field: 'coefficient', type: 'number' }, { field: 'n', type: 'number' }],
  source: { mode: 'derived', from: plant, refresh: 'live',
    statistics: { fn: 'correlation', columns: ['t', 'v', 'w'] } } });
let tw = null;
pairs.rows.forEach((row) => {
  if (pairs.rows.value(row.key, 'a') === 't' && pairs.rows.value(row.key, 'b') === 'w') {
    tw = pairs.rows.value(row.key, 'coefficient');
  }
});

// One row per METRIC of the series summary, not one per point.
const series = createHeadlessGrid({ rowKey: '__key',
  columns: [{ field: 'metric' }, { field: 'value', type: 'number' }],
  source: { mode: 'derived', from: plant, refresh: 'live',
    statistics: { fn: 'series', of: 'v', by: 't' } } });

// One row per compared column, against a second grid. The peer is watched.
// `t` is on this grid only, so it is reported as unmatched rather than dropped.
const peer = createHeadlessGrid({ rowKey: 'id',
  columns: [{ field: 'v', type: 'number' }, { field: 'w', type: 'number' }],
  rows: readings.map((r) => ({ id: r.id, v: r.v * 2, w: r.w })) });
peer.rows.count();
const compared = createHeadlessGrid({ rowKey: '__key',
  columns: [{ field: 'column' }, { field: 'magnitude', type: 'number' }],
  source: { mode: 'derived', from: plant, refresh: 'live',
    statistics: { fn: 'datasetVsDataset', with: peer, columns: ['v', 'w'] } } });

let shared = 0, unmatched = 0;
compared.rows.forEach((row) => {
  if (compared.rows.value(row.key, 'magnitude') === null) unmatched += 1; else shared += 1;
});

return `t/w ${tw}; pairs ${pairs.rows.count()}; metrics ${series.rows.count()}; compared ${shared} + ${unmatched} unmatched`;

Union sources: combining several grids into one

"Worst performers across two datasets" is easy when the two datasets share a key: a join brings the second grid's fields onto the first. It is not expressible at all when they do not: incidents from two regions with no shared identifier, orders from two systems, this quarter and last as one ranked list. from takes an array of sources for exactly this: stack several row sets into one, then rank, group or filter the combined set with the same pipeline a single from already runs.

source: {
  mode: 'derived',
  from: [
    { grid: eastIncidents, label: 'east' },
    { grid: westIncidents, label: 'west' },
  ],
  sort: [{ col: 'severity', dir: 'desc' }],
  limit: 10,
}

Every source is read (each narrowed by its own follow, filtered by default) and concatenated in declaration order, deterministic rather than interleaved, before unnest/join/where/bucket/ groupBy/select/sort/limit/limitPer/ cumulative run once over the result - so "the worst across both" is one derivation, not a hand-merged array.

__source is required, not optional. Every row carries it - the entry's label, or its declaration index when unlabelled - because without it a combined list cannot be read, filtered or grouped by where it came from, which is most of the point of stacking several sources. It is an ordinary field to where, groupBy and select, exactly like a column the data itself carries.

The union of fields, not the intersection. A field present on only one source is undefined on rows from the others - not fabricated, not coerced. Sources are not type-reconciled: if two disagree on what a field means, map is where you make them agree, before they combine, not something the union guesses at for you.

The key is namespaced. A derived grid's __key is the source row's own key when nothing is grouped, and two sources sharing the same identifiers would otherwise collide. So it is qualified by the source tag when there is no groupBy. Grouped, the key is the group value exactly as it always has been - rows from different sources landing in the same group when their group values agree is the point of grouping a union, not a collision to guard against.

Not a join. There is no dedup and no merge-on-key: two sources reporting the same fact both appear as separate rows, and there is no UNION-vs-UNION-ALL distinction to draw. Reach for join when two sides share a key and you want them matched rather than stacked; use groupBy on the combined set when you want them summed together.

Empty and failing sources. A source with no matching rows contributes nothing; the rest of the union still derives. A source that throws while being read (or mapped) is named in a warnOnce and skipped for that pass - reported, never silently dropped, because a silently missing source would make "worst across both" quietly wrong.

A cycle is refused, not recursed. A source list that includes the grid being derived, directly or through a chain of other derived grids, is refused when the source is built, naming the offending source.

Not supported alongside a union. crossFilter has no single target once there is more than one parent, and profile reduces one grid's own columns, so both are refused with a warning rather than guessed at.

const { createHeadlessGrid } = await import('../packages/core/src/index.js');

const east = createHeadlessGrid({
  rowKey: 'id',
  columns: [{ field: 'title' }, { field: 'severity', type: 'number' }],
  rows: [{ id: 'e1', title: 'disk-full', severity: 9 }, { id: 'e2', title: 'slow-query', severity: 3 }],
});
// A second, unrelated incident log - its own "rating" field, no shared id with `east` at all.
const west = createHeadlessGrid({
  rowKey: 'id',
  columns: [{ field: 'name' }, { field: 'rating', type: 'number' }],
  rows: [{ id: 'w1', name: 'oom-kill', rating: 10 }, { id: 'w2', name: 'stale-cache', rating: 2 }],
});

const worst = createHeadlessGrid({
  columns: [{ field: 'title' }, { field: 'severity', type: 'number' }, { field: '__source' }],
  source: {
    mode: 'derived',
    from: [
      { grid: east, label: 'east' },
      // `map` brings west's differently-named fields into the common shape.
      { grid: west, label: 'west', map: (row) => ({ title: row.name, severity: row.rating }) },
    ],
    sort: [{ col: 'severity', dir: 'desc' }],
    limit: 2,
  },
});

const titles = [];
worst.rows.forEach((r) => titles.push(worst.rows.value(r.key, 'title')));

// A per-source breakdown reads `__source` like any other field.
const bySource = createHeadlessGrid({
  columns: [{ field: '__source' }],
  source: { mode: 'derived', from: [{ grid: east, label: 'east' }, { grid: west, label: 'west' }] },
});
const sources = [];
bySource.rows.forEach((r) => sources.push(bySource.rows.value(r.key, '__source')));

worst.destroy(); bySource.destroy(); east.destroy(); west.destroy();
return `worst=${titles.join(',')}; sources=${sources.join(',')}`;

Read-only. A derived row is an answer, not a record: there is no write-back for the sum of four hundred rows, so writes are refused with a reason rather than accepted and discarded on the next refresh.

They chain. A derived grid can be the source of another to any depth: a profile of the top five, and a change at the root travels the whole chain. A cycle is refused rather than recursed.

The key. rowKey defaults to the derived key and need not be set. It is the group value, which is what makes a live ranking readable: the row moves rather than the values under it changing.

Cost. The first derivation is linear in the rows read and largely independent of what is reduced: roughly 900 ms per 200,000 rows grouped into forty, whether the selection is one sum or four statistics. After that, a change that names the rows it touched is patched rather than re-derived: only the groups those rows entered or left are reduced again, so a live feed costs time proportional to what changed rather than to the table. Five hundred updates against that same source take under 300 ms in total, not 300 ms each. A joined derivation is maintained the same way from both sides: the lookup is held between derivations rather than rebuilt, a change to the fact table rejoins only the rows that moved, and a change to the lookup rejoins only the rows behind the keys whose match actually changed, about 2 ms per fact update and 1.5 ms per lookup edit against a 200,000-row source joined to 2,000 customers. A change that cannot be reasoned about that way, a new filter, a regrouping, a derivation using unnest or where, or a lookup row arriving for rows an inner join had dropped, falls back to a full derivation, which is correct but costs the full linear pass. Narrow with follow: 'filtered' so the derivation reads what the user is looking at rather than the whole table.

What a change firing promises

Anything maintaining state from rows:changed (a derived grid, a chart, your own cache) needs to know whether a firing names the rows that moved or merely says that something did. One rows.apply announces itself more than once: the source reports how many rows moved, the row model reports which ones, and the grid reports that a change happened. Acting on all three does the work three times over.

The default is the safe one. A firing that carries neither flag is treated as a change of unknown extent, so a listener re-reads rather than assuming nothing moved. Read the flags rather than the shape of the payload: a firing that reports a row move carries counts, because no row changed value, and only the flags distinguish that case from a duplicate announcement.

Confidence intervals: how much to trust the figure

Every other statistic here describes the data you have. An interval describes how well that data pins down the figure you actually care about, and it is the one thing a descriptive tool can honestly say about the world beyond its rows.

The line this draws. Lattice quantifies uncertainty; it does not adjudicate hypotheses. There are no p-values, no significance tests and no verdicts, and reading two non-overlapping intervals as a significance test is a mistake often enough to be worth not encouraging. An interval says "the mean is 42, and the sample pins that down to between 39 and 45". It does not say 42 differs from 40.

Every interval carries its level. The result's confidence field says what it was computed at, so a figure copied out of one cannot lose the thing that makes it readable. An interval without its level is not a smaller claim, it is an unreadable one.

A proportion is Wilson, not Wald. The textbook p ± z√(p(1−p)/n) fails exactly where a rate is most interesting: near zero it reaches below zero, and at no observed successes it collapses to the single point zero: claiming perfect certainty from the least informative sample there is. The Wilson interval stays inside 0 to 1 and stays sensible at the extremes, so "none of forty failed" correctly reads as "the failure rate is under 9%" rather than "the failure rate is zero".

Report the capability interval. Its absence is the commonest way a capability study overstates itself. A Cpk of 1.35 measured on thirty parts has a lower bound below 1.0, so a process that has "passed" a 1.33 requirement on thirty parts has demonstrated very little. The point estimate alone does not say that; the interval does.

It follows the filters. Like every statistic here, an interval reads the rows the filters left, so it narrows as the user narrows the grid. That is the correct behaviour and worth knowing: it describes the filtered population, not the whole table.

Reading an SPC chart

The figures are only half of it. A capability claim is made in a picture, and these are the two the discipline expects.

The lines are named, on opposite edges. CL, UCL and LCL at the right; LSL, USL and Target at the left. Control limits are what the process does; specification limits are what the customer asked for, and reading one as the other is the classic misreading of a control chart. A capable process puts its control limits just inside its tolerance, so the two families sit close together, which is exactly when they need telling apart, and why they are named at opposite ends rather than left to collide.

Rule breaks carry their number. With Western Electric's four rules a marked point was readable on its own; with Nelson's eight it is not. A spike (rule 1) is a bad part; a six-point trend (rule 3) is tool wear or a drifting sensor. They call for different responses, and a chart that marks both the same way has told you the less useful half of what it knows. Set the rule set with rules: 'nelson' on the chart, as on statistics.capability.

The capability report draws two curves, not one. Cp and Cpk are computed from short-term variation, Pp and Ppk from overall. When a process has drifted the two differ, and the four indices say so only as numbers a reader has to know how to compare. Drawn, the gap is the finding: a narrow solid curve inside a wide dashed one is a capable process that has been allowed to wander, a scheduling problem, not a machine problem.

What this covers, and what it does not. These are individuals charts: one reading per point, with short-term variation estimated from the moving range. That is the right instrument when readings arrive one at a time, a sensor, a test rig, a single-piece flow.

It is not the right instrument for subgrouped data, and the difference is not cosmetic. If you measure five parts an hour, the correct chart is X̄-R: its limits come from within-subgroup variation and are roughly √n tighter, which is what makes a shift in the process centre visible. Run an individuals chart over the same readings and the limits are computed from differences that mix within- and between-subgroup variation; they come out around twice as wide, and a two-sigma shift that X̄-R flags a dozen times over reads as scattered noise. Lattice does not ship X̄-R, X̄-S, or the attribute charts (p, np, c, u), and an individuals chart should not be substituted for them.

Pair the two control charts. An individuals chart asks whether the process has moved; a movingRange chart asks whether it has become less repeatable. A process can fail either without failing the other: it can drift while its point-to-point variation holds steady, and it can hold its average while shaking itself apart. The second is close to invisible on the individuals chart alone, because a wider spread pulls that chart's own limits wider with it: it rescales to accommodate the very thing that has gone wrong. Drawn one above the other, they are the standard I-MR pair.

Bind it to the readings, not to a summary of them. A whisker is computed from the values the chart can see behind each mark. Bound to a grid whose rows are already one per mark, a summary or a derived panel, the chart sees a single value per category and there is no spread to draw: the readings that produced each average are upstream and no longer reachable. Bind the chart to the rows the summary was computed from, or carry a margin yourself and use error: { of: 'margin' }. A chart asked for whiskers it cannot compute says so once rather than drawing nothing in silence.

Uncertainty on a chart. error: true draws a whisker on each mark, computed from the readings behind it. Four bars side by side invite a comparison the numbers alone cannot support, a five per cent gap between two categories of eight readings is noise, and between two of eight hundred it is the finding. The whisker is what tells them apart, and its absence is why bar charts are so often over-read. A mark with a single reading gets none, because one reading has no spread and a zero-height whisker would claim certainty rather than admit ignorance.

Fitted lines. fit: true draws a least-squares line through a scatter with its R² beside it; fit: 'line' draws the line alone. A cloud of points invites a reader to draw the line themselves, and people are consistently poor at it, the eye is pulled by the extremes, which is exactly what least squares is not.

The interval travels with the index. The statistics panel shows the bounds under Cpk and Ppk, and createStat takes an interval function that puts them under the value. A tile is where a figure is read fastest and questioned least, which makes it the place an interval earns its keep rather than the place it is least needed.

Pushdown adapters: one query, many engines

A remote source already receives a structured request: range, sort, filters, quick text, grouping, pivoting and totals. A pushdown adapter turns that request into whatever an engine speaks, so connecting a new back end is a translation layer rather than a new source.

import { createPushdownSource, odataAdapter } from '@toclocoinc/lattice-grid';

const source = createPushdownSource({
  adapter: odataAdapter({ url: 'https://api.example.com/Orders' }),
  compute,
});

createGrid(host, { source, columns: [...] });

An adapter never carries an engine. Each one takes what it needs as a parameter: restAdapter takes a fetch and bundles no HTTP library, dfqlAdapter takes a token, and duckdbAdapter takes a connection you have already made. So a grid can drive a full analytical engine without this package carrying one, and installing Lattice never installs anything else.

An adapter declares what it can answer. No engine speaks the whole query. OData takes a condition tree but only some operators; a single-term API takes one field and one value; a hand-written endpoint may take nothing but a page number. The adapter states its capabilities, the SDK divides the request accordingly, and the grid finishes whatever is left.

Anything left over means the whole result is fetched. Filtering a window of rows in the browser is not a slower way to get the right answer, it is a fast way to get a wrong one: the rows that belong on the first page may be on the ninth, and the count is whatever the engine happened to return. So when the grid has work left to do it asks the engine for the complete result, applies the remainder, and pages from what it holds. It says so once, naming the part that could not be pushed, because the fix is usually a wider adapter rather than a bigger machine. source.lastPlan() reports the division for any request.

A conjunction splits; a disjunction does not. An and group narrows with each condition, so the engine can apply the conditions it understands and the grid narrows what comes back. An or group widens with each branch, so pushing only the supported branches returns fewer rows than the filter allows, and the grid cannot recover rows that were never fetched. A disjunction the engine cannot fully answer therefore stays with the grid whole. The same asymmetry governs faceting.

A sort is pushed whole or not at all. Ordering by the first column and fixing the rest in the browser needs every row anyway, so a partial sort buys nothing and returns rows in an order that is wrong until the grid corrects it.

What each adapter takes

Every adapter is a function of one options object. The tables below list what each accepts, the type, the default where it is not obvious, and what it means. The defaults are the load-bearing part: an adapter is designed to work when handed almost nothing, so most of what you can set is about telling it what your endpoint cannot do rather than switching features on.

odataAdapter

restAdapter

The parameter names are yours, and the defaults are not zero. Paging and sorting are assumed present; filtering is assumed absent until you declare operators, because an adapter that claims to filter when the endpoint ignores it returns the wrong rows silently. The query-string names default to offset, limit, sort, order, filter and q (search); params overrides any of them.

Persisting a cell edit. With edit: true the adapter advertises mutate, so a committed cell edit is sent as an HTTP request. REST has no universal write convention, so the request is yours to shape: writeUrlFor names the per-row endpoint, encodeMutation takes full control of method and body, and writeRow reads the authoritative row back when returning: 'row'.

const { restAdapter } = await import('../packages/core/src/index.js');

// The per-row endpoint a mutation targets.
const writeUrlFor = (op) => `/api/orders/${op.key}`;

let sentMethod;
const adapter = restAdapter({
  url: '/api/orders',
  edit: true,          // opt into write-back (advertises mutate.update / delete)
  returning: 'row',    // reconcile to the server's authoritative row
  writeUrlFor,
  // Full control of the request; supersedes the default envelope.
  encodeMutation: (op) => ({
    method: 'PATCH',
    url: writeUrlFor(op),
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(op.patch),
  }),
  // Pull the authoritative row out of this service's envelope.
  writeRow: (body) => body.record,
  fetch: async (url, init) => {
    sentMethod = init.method;
    return { ok: true, status: 200, json: async () => ({ record: { id: '42', status: 'shipped' } }) };
  },
});

const result = await adapter.mutate({ kind: 'update', key: '42', patch: { status: 'shipped' } });
return result.rows[0].status;   // 'shipped', read back from the server

duckdbAdapter

Spatial pushdown (spatial). On first use the adapter runs INSTALL spatial; LOAD spatial; once per connection, inside the connection's statement queue, then reads the schema and, for a read_parquet('…') FROM, the GeoParquet geo metadata. A column is geometry when DuckDB types it GEOMETRY (its Parquet reader converts a GeoParquet column itself once the extension is loaded), when the metadata names it, or when column does; a BLOB is read through ST_GeomFromWKB, and a column in another CRS (the file's declaration, or crs, which wins) through ST_Transform(geom, crs, 'EPSG:4326', always_xy := true). Geometry is selected as ST_AsGeoJSON(geom). The operators keep their in-memory meaning: withinBbox → ST_CoveredBy(geom, ST_MakeEnvelope(?, ?, ?, ?)); withinPolygon → ST_Within of the centroid (of the geometry itself for a MultiPoint); intersects → ST_Intersects (a four-number operand is an envelope); withinDistance → the haversine distance from the centroid on the 6,371,008.8 m sphere, in SQL. Operands are bound, never interpolated: GeoJSON to ST_GeomFromGeoJSON(?), WKT to ST_GeomFromText(?), WKB bytes or hex to ST_GeomFromHEXWKB(?); one that cannot be read compiles to FALSE and source.duckdb.spatial.operand.<op> says why. A geometry column sorts by ST_Y(ST_Centroid(geom)), then ST_X; grouping by one is refused by name; a SpatialBucket { col, bucket: 'grid', size } in source.aggregate()'s groupBy pushes as floor(ST_X(ST_Centroid(geom)) / size) and the same for ST_Y, one ROLLUP unit, the key back as [x, y]. The adapter declares the operators in capabilities.operators and the bucket kind in capabilities.buckets, and exposes ready(), which createPushdownSource awaits before its first plan. The operators are declared only while the type layer has them - the modules/geometry module is loaded and a grid has resolved the type; without it a condition handed to the adapter compiles to FALSE and source.duckdb.spatial.module.<op> names the missing import. If the extension does not load, source.duckdb.spatial.load warns once, both are withdrawn, and the operators are left to the grid (pass compute to createPushdownSource to apply them), with geometry arriving as its stored WKB.

List pushdown. Over a column DuckDB types as a LIST (VARCHAR[], INTEGER[], DATE[]…), a condition keeps its in-memory meaning: hasAny → list_has_any(col, [?, …]), hasAll → list_has_all, hasNone → NOT list_has_any, listCount → len(col) <op> ? (an absent list counting as none), blank → len(col) = 0 OR col IS NULL; eq → list_contains(col, ?) (over list_transform(col, x -> lower(x)) for case-insensitive text), and any other item operator → len(list_filter(col, x -> x <op> ?)) > 0, text matching through ILIKE. A negation is written NOT COALESCE(…, false) so an absent list is kept, as it is in memory. Every item is bound, typed by the list's item type (CAST(? AS DATE) for a DATE[]), never interpolated. The list operators are declared in capabilities.operators only while the modules/list module is loaded; one reaching a column the engine does not describe as a LIST compiles to FALSE and source.duckdb.list.column.<col> says so. notContains, notIn and matches are left to the grid, which asks them item by item over the returned rows; lastPlan() shows which conditions went to the engine. A LIST value arrives in the grid as an array.

How the total is counted, and what it costs. The grid needs the size of the matching set to size its scrollbar. Until 1.51 every page query carried count(*) OVER () AS "__lattice_total", which looks free: the window is evaluated before LIMIT, so one round trip returns both the window and the size of the set it was cut from. Against a local table it is free. Against a remote Parquet it is the most expensive thing the adapter does - a window function has to see every matching row of the projected columns, so whatever row-group pruning or range reads DuckDB and the file host might otherwise manage between them (see whether a Parquet file streams or downloads whole) cannot help, and the whole file crosses the wire to produce one page. Measured in Chrome on duckdb-eh.wasm against a 10,000,000-row Parquet of 162,386,227 bytes - 162.4 MB decimal, 154.9 MiB binary, and every transfer figure on this page is decimal MB so that it can be compared with it directly - on an origin counting bytes actually served, first paint pulled 162.5 MB. Slightly more than the file, because the reads overlap.

The total is now its own statement - SELECT count(*) FROM <from> WHERE …, the same predicate through the same builder with the same typed casts and the same bound values, and no ORDER BY or LIMIT. Read it with adapter.countSqlFor(query), the counting counterpart of adapter.sqlFor(query); it returns null when count is false. The two statements are dispatched in the same tick - both started before either is awaited - so there is no browser round trip between them.

That does not make the count free on the clock, and on DuckDB-Wasm it often is not. One DuckDB-Wasm connection funnels its statements through a single worker, so the engine still runs the two in sequence; what dispatching together buys there is that the second is already queued the instant the first finishes. Measured on the file below: an unfiltered first paint takes 566 ms with the count and 558 ms without, so the count costs about 8 ms - genuinely hidden. A filtered query takes 2122 ms with the count and 573 ms without, so there the count costs about 1550 ms and is not hidden at all. (It is still faster than the 3064 ms the old window function took for the same query.) A server-side DuckDB with a thread pool runs the two at once and the distinction goes away.

Cheap, not free - and on a filtered query, not even cheap. An unfiltered count(*) over Parquet is answered from the file's footer metadata and reads no data at all: measured against the 162.4 MB file above, first paint costs the same 5.71 MB with the count on as with count: false - the count's own share is 0.00 MB. A filtered count still has to evaluate the predicate. It reads the predicate columns rather than the whole projection, and row-group statistics can prune entire groups (a predicate no row group can satisfy is answered from metadata alone - country = 'ZZ' against this file costs 0.12 MB, count and all). But a predicate whose columns are spread across every row group has to read them all: country = 'GB' AND risk_score > 70 costs 45.2 MB with the count and 3.2 MB without, so 41.9 MB of it is the count. Against the 191.2 MB the old window function cost, that is still a 4× saving - and if your grid never shows a count, count: false makes the same query a 59× one.

One case where 1.51 transfers more, not less: a session that eventually reads the whole table anyway. Every individual query above is cheaper than or equal to its 1.50.0 counterpart, but a whole session need not be. 1.50.0 dragged the entire file down on first paint in a handful of large sequential reads, after which everything was cache-warm and every later query cost nothing. 1.51 reads lazily, and lazy range reads over a big Parquet overlap where one eager read did not - so bytes already paid for can be paid for again. Measured over one browser session doing first paint, a selective filter, a full-table ORDER BY, a deep page and two more filters, with the counter reset between each: 1.50.0 transferred 162.5 MB in total and 1.51 transferred 209.8 MB. The user who never sorts the whole table pays 46.7 MB instead of 162.5 MB and sees a first paint in 566 ms instead of 5808 ms; the user who does sort the whole table has to read the whole file either way, and now pays some of it twice. A full ORDER BY over an unindexed column with SELECT * is unchanged at 162.5 MB before and after - this card neither helps nor hurts it.

Turning the count off changes what the grid knows, on purpose. With count: false the adapter reports no total, and the grid does what it already does for any source of unknown length: it scrolls open-ended and discovers the end when a short page arrives. It does not substitute the page length for the total - a page presented as the whole is a wrong number where a right one goes, and every “showing X of Y”, scrollbar and row count would be wrong with nothing said.

So count: false is a trade, not a free win, and here is the part you will notice first: the grid can only scroll as far as it has discovered. The scrollbar is open-ended rather than proportional, a “showing X of Y” readout has no Y, and grid.scroll.toRow(n) cannot jump to a row beyond the discovered end - asking for row 900 of a not-yet-discovered million lands at the furthest row known so far, and reaching the real row 900 means paging to it. Turn the count off for a grid whose users scroll; leave it on for one whose users jump.

Whether a Parquet file streams or downloads whole is DuckDB's and the file host's doing, not the grid's. duckdbAdapter only writes SQL; it never opens a file, so it has no say in whether read_parquet(...) reads the whole thing or only the row groups a query needs. Measured against a 1.5 MB Parquet on GitHub Pages: DuckDB-Wasm 1.32.0's default HTTP path issued 0 Range requests and read 100% of the file, for every query including a single-value chip filter. Running LOAD httpfs; on the connection before the first read_parquet(...) call changed that to 25 Range requests and 30% of the file for the same query. (DuckDB-Wasm 1.29.0 read 119% of the file on the same test - its ranges overlapped - so the exact figures are a version's, not a promise.) A file registered with db.registerFileBuffer(...) is always read whole, whatever version is loaded: a buffer has already been downloaded in full before DuckDB ever sees it. And the file host has to cooperate: it must answer HEAD, advertise Accept-Ranges: bytes and answer a range request with 206 Partial Content and a Content-Range header - GitHub Pages does all three - and, cross-origin, its CORS policy must expose Content-Range and Content-Length or the browser cannot read them back. Any of that missing and DuckDB falls back to reading the file whole, silently.

Typed binding for timestamp and date columns. A prepared statement binds a filter value with the value's own type, not the column's: the grid sends an instant as an ISO-8601 string, the client binds it as VARCHAR, and DuckDB refuses "ts" >= ? against a TIMESTAMP column (Binder Error: Cannot compare values of type TIMESTAMP and type VARCHAR). The adapter therefore types the placeholder: a comparison or IN member against a TIMESTAMP, TIMESTAMP WITH TIME ZONE, DATE, TIME or TIMESTAMP_S/_MS/_NS column is written CAST(? AS <that type>), and the value is still bound, never interpolated. The column's type comes from the engine - one DESCRIBE SELECT * FROM <from> on the first query, cached for the adapter's life and exposed as adapter.describe() - so an untyped grid column over a timestamp is covered. When the schema does not name the column (a DESCRIBE that failed, said once), the grid column's declared type on the condition is the fallback: timestamp/datetime cast to TIMESTAMP, date/dateString to DATE, time to TIME. The engine's type wins when both are known. A Date or an epoch-milliseconds number is bound as its ISO instant, because DuckDB has no cast from a number to a timestamp. The same schema fixes blank: = '' is a conversion error on any non-text column, so a typed column's blank test is IS NULL alone. Text and numeric comparisons (VARCHAR, BIGINT, DOUBLE, DECIMAL, HUGEINT) are written exactly as before, with no cast.

Time zones, honestly. The cast is the engine's, so its zone rules apply. Against a naive TIMESTAMP column the wall-clock digits of the bound string are compared; an instant ending in Z - which is what the grid's own date filter sends - therefore matches a column that stores UTC wall time, the usual convention for log and event data. A non-zero offset in the string is engine-version dependent (DuckDB 1.1 converts it to UTC, 1.5 keeps the digits as written), so send Z instants, not local offsets. Against a TIMESTAMP WITH TIME ZONE column an offset or Z is honoured exactly, and a string with no zone is interpreted in the engine's session TimeZone (UTC in DuckDB-Wasm unless the ICU extension is loaded and the setting changed). Against a DATE column an instant is truncated to its UTC day.

duckdbAdapter: grouping runs in the engine, one level at a time

Grouping a hundred million Parquet rows used to mean fetching a hundred million Parquet rows: the adapter declared no group capability, so the push router never sent it a grouped request and the grid grouped whatever it held. duckdbAdapter now declares group: true and answers the grouped view with GROUP BY - one statement per grid level, paged like any other window.

Nothing is configured. Group a grid over a DuckDB source and the grouping is pushed:

const grid = createGrid(el, {
  columns: [
    { id: 'region' },
    { id: 'tier' },
    { id: 'amount', type: 'number', total: 'sum' },
  ],
  groupBy: ['region', 'tier'],
  source: createPushdownSource({
    adapter: duckdbAdapter({ connection, from: "read_parquet('s3://bucket/sales/*.parquet')" }),
  }),
});

The root level is one statement - SELECT "region", count(*), sum("amount") … GROUP BY "region" ORDER BY "region" ASC NULLS LAST LIMIT ? OFFSET ? - so the group rows on screen cost a grouped scan and no leaf crosses the wire. Expanding a group narrows the next level by its parent's key; expanding the deepest one runs the ordinary row query with the same predicate ANDed on, so the leaves arrive paged and sorted exactly as they would without grouping. Every identifier goes through the same validation the row query uses and every value is bound, so the grouped path is no more exposed than the read path.

Subtotals come from the engine, and a statistic it cannot express shows nothing rather than something. Each totalled column contributes one aggregate expression, taken from the same verified pushdown map the statistics panel uses (see pushing statistics down) - sum, avg, min, max, count, the quantiles, and the rest. A column whose total is a host function, a two-column statistic such as weightedAvg (a grouped request carries no weight column), and the one genuine fallback weightedQuantile are not sent, are named in source.lastPlan().aggregates.client with the reason, and warn once. The group row then carries no value for that column. That is deliberate: the leaves of an unexpanded group are not in the browser, so the only alternative to the engine's figure is a figure computed over something that is not the group.

The counts a grouped grid shows are the engine's. Under grouping the display count is group headers plus whatever is expanded, which is how a grid showing three rows under one group once reported “4 of 3”. The root level's fetch therefore also asks for count(*) over the matching set and the grand total over it, in one extra statement, and an unfiltered count(*) once per adapter (on a Parquet file that is a footer read, not a scan). rows.matchCount() and rows.totalCount() read them, and grandTotalRow: 'bottom' draws its row from them.

Group order, and the collation decision. Group rows are ordered by their own key, ascending unless the sort names that column, which is exactly what the grid does to sibling group rows in memory - a sort naming some other column does not reorder groups in either. Absent keys sort last ascending and first descending, written as explicit NULLS LAST/NULLS FIRST rather than left to the connection's default_null_order. No COLLATE is written and the ICU extension is not loaded: the grid compares group keys with < on the JavaScript string (UTF-16 code-unit order) and DuckDB's default VARCHAR ordering is UTF-8 byte order, and the two agree for every character in the Basic Multilingual Plane. They part only for supplementary-plane characters (emoji, CJK extension B and above) compared against U+E000-U+FFFF. An ICU collation would disagree with the grid everywhere instead, so binary ordering is the pin. Practically: 'North' sorts before 'north' in both.

When grouping is not pushed, it says so. Grouping is all or nothing - group rows counted over the wrong set are wrong rows, not slow ones - so the whole level is refused if anything else in the query stayed behind: a filter that did not fully push, a sort the engine could not take, a quick search, a host where predicate, a grouping key that is not a plain column, or fullDataset (which holds the whole set and groups it client-side on purpose). source.lastPlan() then reports grouped: false, 'group' in unpushed and a groupReason sentence, and a one-time warning names it. Outside fullDataset the grid then stays ungrouped rather than grouping a window - see grouping the engine was not asked for.

One known difference from a memory grid. A column holding both NULL and the empty string produces two groups in the engine and one in a memory grid, because the grid keys its group nodes on a display path where an absent value and an empty string are both ''. The engine's answer is the right one; the two are otherwise identical group for group, count for count and subtotal for subtotal, which the parity suite asserts at every level.

A chart over a paged relation reads through source.aggregate(), not the loaded window (GEO-5). A bar chart drawn from a grid holding the first hundred rows of a hundred thousand cannot sum those hundred and call it the total - that is a confident, wrong picture. The charts module's grouped types (a bar, a line, a heatmap and the rest of the aggregate types) ask the same group capability this section describes: x (and series) become the groupBy, each measure an aggregate in the request, and the grid's filter and quick search are pushed exactly as they are for a grouped row above - refused wholesale, never partially, on the same conditions: an unpushed filter or quick search, a grouping key that is not a plain column, or a statistic the source's own aggregates config keeps in the browser. The one difference from a group row is the trigger: a chart asks only when the grid holds fewer rows than match, so a fully-loaded memory grid never pays for a request whose answer it already has.

Maps and KPI tiles ask the same way (GEO-6). A map over a paged pushdown source counts the rows inside its view with an ungrouped source.aggregate(), fetches them through the adapter's execute capped at its viewportCap, and past the cap groups by a spatial buckets key for density cells; a KPI tile's sum, avg, min, max or countDistinct is one ungrouped aggregate. The refusals are the chart's: an unpushed filter or quick search, a where predicate, or a statistic the aggregates config keeps in the browser sends the viewer back to the loaded rows, said by name - except a countDistinct KPI tile, which shows no number at all rather than the loaded rows': unlike a sum, a page's distinct count is not a partial reading of the relation's.

dfqlAdapter

graphqlAdapter

GraphQL has no fixed query semantics, so this adapter is configured. A filter, a sort and pagination are whatever the schema defines, so the two hooks are yours to write: buildQuery turns the pushed plan into the { query, variables } body the endpoint is POSTed, and parseResponse reads its data back into { rows, total }. The defaults cover an offset/limit list with a totalCount and a Relay cursor connection; either is replaced whole by passing the hook. The default query pushes only the window and asks for the total, which is why the default capabilities are range and total and nothing more - declare operators or capabilities for filter or sort only alongside a buildQuery that genuinely emits them, or the grid returns the wrong rows silently.

If your schema does not expose totalCount. The adapter reports no total, rather than the number of rows in the page, and the grid scrolls open-ended; the endpoint is named once in a warning so the silence is not mistaken for a working count. Before 1.51 the page length was reported as the total, and that was not only a wrong number on screen - it truncated results. When the grid has residual work to finish it asks for the whole result and the adapter walks offset/limit to get it, stopping when it has as many rows as the total says exist. With the total invented from page one, the walk stopped at page one: a 337-row connection came back as 100 rows, reported as complete, and any client-side filter or sort then ran over that fraction. The walk now stops on a short page or an exhausted cursor, so it returns everything and its count is exact. A schema that does report totalCount was never affected.

splunkAdapter

Splunk’s index format is closed, so the adapter speaks the search API. There is no file to read and no bucket to open: splunkAdapter dispatches a search job with POST /services/search/v2/jobs, watches it, reads a window of its results with offset and count, and takes the match count from the job’s own resultCount. Paging reuses the job rather than dispatching a second search, and the moment the grid asks a different question - a new filter, a new sort - the job it walked away from is DELETEd rather than left to expire. An aborted request cancels its job too, and so does a finished one: the moment the last page of a DONE job is read, it is DELETEd too, by default, rather than left for its 600-second TTL to reclaim - a read-only role’s search-artifact quota is usually far smaller than an interactive user’s, and a grid that only ever asks one question should not spend a slot on it for ten minutes nobody reads. Set releaseJobs: false to keep the job open instead.

What pushes down. The filter tree becomes one SPL search expression: = for eq, NOT field=x for ne (Splunk’s != drops events that do not carry the field at all, and the grid’s ne keeps them), </<=/>/>= for the comparisons, wildcards for contains/startsWith/endsWith, an OR of equalities for in, field=* and its negation for notBlank/blank, and AND/OR/NOT for the groups. A multi-column sort becomes | sort 0 … - the explicit 0 matters, because | sort keeps only the first 10,000 rows otherwise and a silently truncated result is the one thing a sorted grid must never be handed. Everything else stays with the grid through the ordinary residual path: | stats grouping is not pushed, and neither is the quick filter, because an SPL bare term matches whole tokens against the index and the grid’s quick filter is a substring match - a different question with a different answer.

A time filter becomes the job’s window where that is exact. earliest_time is inclusive and latest_time is exclusive, so >= and < on the timeColumn leave the expression and become the window, which is the difference between Splunk seeking to a time range and Splunk reading everything to discard most of it. > and <= do not map onto those boundaries, and a time bound inside an or would narrow the other branch too, so both stay in the expression as an exact numeric comparison on the field. Where you have set earliest/latest yourself, the window is the narrower of the two when both are absolute instants, and yours plus an expression clause when yours is a relative token only Splunk can resolve - either way the result is your window intersected with the grid’s filter, never wider than both.

No value is interpolated. Strings are double-quoted with \, " and * escaped - the asterisk because an unescaped one inside a quoted SPL value is a wildcard, so a user who typed foo*bar into a filter box would otherwise silently match far more than they asked for. The wildcards contains needs are added outside the escaped text, so ours are wildcards and yours are literal. Numbers are written only when finite, times only after they parse as a real instant, and a field name is validated against ^[A-Za-z_][A-Za-z0-9_.:-]*$ and refused by name otherwise - an identifier is the one part of an expression that cannot be quoted away.

The grid never holds your credential. A fixed token or session key goes in headers (Authorization: Bearer …, or Authorization: Splunk <sessionKey>); a token that expires goes in a fetch wrapper, which is the only thing called afresh per request. The adapter stores nothing, prompts for nothing and refreshes nothing, so there is nothing in it for a credential to leak from - and a 401 or 403 is reported as a refused token naming that fix, rather than as an empty result. Any other refusal - a 503 on an exhausted artifact quota, a 400 on a malformed search - reads Splunk’s own messages[].text out of the response body and carries it in both the thrown error and a named warning, rather than leaving only the HTTP status to go on. See authenticating.

The limits worth knowing. A search job is capped by the instance’s limits.conf maxresultrows (500,000 by default): past it Splunk truncates, and a truncated count presented as a total would make every “showing X of Y” wrong, so tell the adapter your ceiling with maxResultRows and it says so once instead. In mode: 'export' the same option stops the stream once that many rows have arrived - without it, an export is unbounded, however many rows the search matches, and lands in one in-memory array. A job also has a TTL (600 seconds by default) after which its results are gone; the adapter re-dispatches rather than failing when a held job has expired, and releases a finished one itself once its last page is read (releaseJobs, default true) so a read-only role’s artifact quota is not spent by jobs nobody will read again. A case-sensitive condition is pushed without its case constraint, because Splunk’s search matching is case-insensitive, and that too is said once rather than left to be discovered.

An executed example. The block below runs against test/fixtures/splunk-mock.js, the repository’s own test double for the search REST API: it speaks the four endpoints and records what was sent, so the example proves the SPL and the paging without a live instance and without a credential anywhere. Swap createSplunkMock(…).fetch for your own fetch (or leave it out) and the same call runs against Splunk.

const { splunkAdapter } = await import('../packages/core/src/index.js');
const { createSplunkMock } = await import('../test/fixtures/splunk-mock.js');

// A mock Splunk: four firewall events, no network, no credential.
const splunk = createSplunkMock({ events: [
  { _time: 1758844800, host: 'fw1', status: 403, uri: '/admin/login' },
  { _time: 1758844860, host: 'fw2', status: 404, uri: '/admin.php' },
  { _time: 1758844920, host: 'fw1', status: 500, uri: '/admin/api' },
  { _time: 1758844980, host: 'fw3', status: 403, uri: '/wp-admin' },
] });

const adapter = splunkAdapter({
  url: 'https://splunk.example.com:8089',
  search: 'index=security sourcetype=fw',
  earliest: '-24h',
  // Your token, your header. The adapter stores nothing.
  headers: { Authorization: 'Bearer your-token-here' },
  fetch: splunk.fetch,
});

// What the grid's filter and sort become, before anything is dispatched.
const query = {
  filters: { op: 'and', conditions: [
    { col: 'status', op: 'gte', value: 400 },
    { col: 'uri', op: 'contains', value: 'admin' },
  ] },
  sort: [{ col: '_time', dir: 'desc' }],
  range: { start: 0, end: 2 },
};
const { search } = adapter.searchFor(query);
const expected = 'search index=security sourcetype=fw (status>=400 AND uri="*admin*") | sort 0 -_time';

// One window of the result, and the job's own match count.
const page = await adapter.execute(query, {});
return `${search === expected ? 'SPL as documented' : search}, ${page.rows.length} of ${page.total} events`;

Two ways the adapter cleans up after itself. mode: 'export' stops reading the stream once maxResultRows is reached, rather than holding every row an unbounded search would produce; and a finished blocking-mode job is DELETEd the moment its last page is read, by default, rather than left for its TTL - the failure mode that exhausted a read-only role’s search-artifact quota before this existed. Set releaseJobs: false to keep a job around instead.

const { splunkAdapter } = await import('../packages/core/src/index.js');
const { createSplunkMock } = await import('../test/fixtures/splunk-mock.js');

// Five events shared by both halves of this example.
const mock = createSplunkMock({ events: Array.from({ length: 5 }, (_v, i) => ({ seq: i })) });

// mode: 'export' with a cap stops the stream rather than reading it unbounded.
const capped = splunkAdapter({
  url: 'https://splunk.example.com:8089', search: 'index=security',
  mode: 'export', maxResultRows: 2, fetch: mock.fetch,
});
const exported = await capped.execute({}, {});

// releaseJobs defaults to true: the job is DELETEd once its last page is read.
const blocking = splunkAdapter({
  url: 'https://splunk.example.com:8089', search: 'index=security', fetch: mock.fetch,
});
const read = await blocking.execute({ range: { start: 0, end: 5 } }, {});

// The release DELETE is fire-and-forget; give it a tick to land.
await new Promise((resolve) => setTimeout(resolve, 0));

return `capped at ${exported.rows.length} of ${read.total}, job released: ${mock.cancelled.length === 1}`;

clickhouseAdapter

Transport. ClickHouse answers SQL over plain HTTP, so the adapter needs no driver: each statement is POSTed to {url}/?default_format=JSONEachRow with the SQL as the body and read back one JSON row per line. The page and its count are two statements sent together - SELECT * … ORDER BY … LIMIT n OFFSET m and SELECT count() … over the same WHERE. Give table (events or analytics.events, each part validated as an identifier and backtick-quoted) or query, any SELECT, which is read as a subquery. The adapter sets no Content-Type of its own, so a browser request without an Authorization header is a simple request and needs no CORS preflight.

No value is written into the SQL. Every filter value is a ClickHouse query parameter: the statement says {p0:String}, {p1:Float64}, {p2:DateTime64(3, 'UTC')} or {p3:Array(String)}, and the value travels separately as param_p0=…, parsed by the server as a value of that type. A backslash in a value is doubled on the way out, because ClickHouse reads a parameter in its escaped format and would otherwise turn a\b into a backspace. Column names are validated against ^[A-Za-z_][A-Za-z0-9_$]*$ and refused by name otherwise.

What pushes down, and how it matches the grid. A text comparison folds case the way the grid's own filter does, through lowerUTF8() on both sides (lower() folds ASCII only), unless the condition sets caseSensitive, when it is exact. contains/startsWith/endsWith use position/startsWith/endsWith rather than LIKE, so a % or _ someone typed is a character, not a wildcard. ne, notIn and notBetween add an explicit IS NULL branch, because the grid keeps a row with no value there and SQL's NULL <> x would drop it. A sort writes its NULL placement out - last ascending, first descending - which is the grid's rule. Anything not in the table below - matches, notContains, the quick filter - stays with the grid through the ordinary residual path and is named in lastPlan(). The grid's grouped view is built from grouped aggregates (see grouping from grouped aggregates), and source.aggregate() is pushed.

The grid never holds your credential. The rule splunkAdapter follows. A fixed user and password go in headers (X-ClickHouse-User/X-ClickHouse-Key, or Authorization: Basic …); a credential that expires goes in a fetch wrapper, called afresh per request; and a page that must not see a credential at all points url at a proxy of your own that adds one (a relative url such as /api/clickhouse is kept relative). A public read-only user can ride in the URL's own query string - url: 'https://play.clickhouse.com/?user=play' - which the adapter keeps. See authenticating.

Settings. readonly=1 is sent with every statement, because the grid only reads - and nothing else is sent by default. A user whose profile is already read-only (the public playground's play user, most analyst accounts) may change no setting at all: ClickHouse refuses such a request with READONLY if it carries, say, max_execution_time, while re-sending readonly=1 is accepted. So the default works for that common case with nothing to configure. max_execution_time is optional: on a writable or service account, set it - settings: { max_execution_time: 30 } - so a filter a stranger can steer cannot hold the server longer than a page is worth waiting for. settings adds any other ClickHouse setting; null drops any, readonly included. 64-bit integers past JavaScript's safe range arrive as their exact text rather than a rounded number, where the runtime gives JSON.parse the source text (current Chrome, Node 22).

Errors carry ClickHouse's message. A non-2xx answer, and an exception ClickHouse writes into a 200 after it has started streaming, reject with an Error named ClickHouseError carrying status, code (60), type (UNKNOWN_TABLE) and clickhouseMessage, the server's own text - surfaced by the grid as source:error, never as an empty page.

An executed example. The fetch below stands in for the server so the example can show exactly what is sent - the SQL, the parameter beside it and the settings - without a network or a credential. Leave it out, and the same adapter talks to ClickHouse.

const { clickhouseAdapter } = await import('../packages/core/src/index.js');

const sent = [];
const adapter = clickhouseAdapter({
  url: 'https://clickhouse.example.com:8443',
  table: 'logs.requests',
  headers: { 'X-ClickHouse-User': 'grid_reader', 'X-ClickHouse-Key': 'your-password' },
  // A stand-in server: records the request and answers in JSONEachRow.
  fetch: async (url, init) => {
    sent.push({ url: new URL(url), sql: init.body });
    const body = init.body.startsWith('SELECT count()')
      ? '{"__lattice_total":"3"}'
      : '{"id":1,"path":"/admin"}\n{"id":2,"path":"/Admin/login"}';
    return { ok: true, status: 200, text: async () => body };
  },
});

const page = await adapter.execute({
  filters: { col: 'path', op: 'contains', value: 'admin' },
  sort: [{ col: 'id', dir: 'asc' }],
  range: { start: 0, end: 2 },
}, {});

// The value is a parameter; the SQL only names it.
const [first] = sent;
const sql = 'SELECT * FROM `logs`.`requests` WHERE position(lowerUTF8(toString(`path`)), lowerUTF8({p0:String})) > 0 ORDER BY `id` ASC NULLS LAST LIMIT 2 OFFSET 0';
if (first.sql !== sql) return first.sql;
return `parameter p0=${first.url.searchParams.get('param_p0')}, readonly=${first.url.searchParams.get('readonly')}, ${page.rows.length} of ${page.total} rows`;

elasticsearchAdapter

Transport. Elasticsearch and OpenSearch answer the same Query DSL over HTTP, so the adapter needs no client library. A page is one POST {url}/{index}/_search with from, size and track_total_hits: true, so the rows and their exact count come back together. GET {index}/_mapping is read once, before the first request, so the adapter knows which fields are keyword, text, numbers or dates; pass mapping to skip it. index is a name, an alias, a pattern such as logs-* or a comma-separated list, and is refused if it is anything else, because it is written into the request path. Each hit arrives as its _source with its _id beside it, so rowKey: '_id' works for any index.

No value is spliced into a string. The request body is JSON the adapter builds as an object, and every filter value is a JSON value inside it. There is no query_string: a user who types a OR b is searching for that text. The one grammar that reaches the engine is the wildcard pattern of contains/startsWith/endsWith, and a *, ? or \ the user typed is escaped, so it is a character and not a wildcard. Field names are validated (letters, digits, _, @ and - in dot-separated segments) and refused by name otherwise.

What pushes down, and how it matches the grid. A text comparison folds case the way the grid's own filter does, with case_insensitive: true, unless the condition sets caseSensitive. The engines fold A-Z only, so a case-insensitive value holding a cased letter outside ASCII (Zoë) is pushed and named once (source.elasticsearch.casefold). A text field holds analysed words rather than a whole value, so it is compared through its keyword sub-field (the default dynamic mapping always adds .keyword; a value longer than its ignore_above is not in that sub-field and cannot match). A text field with no keyword sub-field is matched word by word with match_phrase and named once per field (source.elasticsearch.analyzed.*). ne, notIn and notBetween are must_not, which keeps a document with no value, as the grid does. A substring test on a number or a date, and any filter on a nested field, is refused by name rather than sent to fail or to match nothing. matches, notContains, listCount and the quick filter stay with the grid through the ordinary residual path and are named in lastPlan().

Paging past the result window. An index answers from + size only up to its max_result_window (10,000 by default; tell the adapter yours with maxResultWindow). A page inside it is one _search. A page past it - or the whole result, which the grid asks for when part of the filter stays with it - is counted with _count and read with search_after over a point in time: the adapter opens one, hops to the page's start without fetching documents, reads the page, remembers where each hop ended so scrolling on does not walk again, and closes the point in time as soon as the grid asks a different question. The sort gains a tiebreaker so the walk never repeats or skips a row: _shard_doc on Elasticsearch; on OpenSearch set engine: 'opensearch' (its point-in-time endpoints differ) and tiebreaker to a field unique per document.

The grid never holds your credential. An API key or a user and password go in headers (Authorization: ApiKey … or Basic …); a credential that expires goes in a fetch wrapper, called afresh per request. In production put a proxy of your own in front of the cluster that authenticates your user, adds the credential and forwards only GET {index}/_mapping, POST {index}/_search, POST {index}/_count and the point-in-time calls, and point url at it. A browser that talks to the cluster directly needs http.cors.enabled: true, http.cors.allow-origin set to your page's origin and http.cors.allow-headers including Authorization and Content-Type in elasticsearch.yml (the same settings in opensearch.yml); the JSON body always makes the request preflighted.

Errors carry the engine's reason. A refusal rejects with an Error named ElasticsearchError carrying status, type (index_not_found_exception) and reason, the engine's own text - surfaced by the grid as source:error, never as an empty page. A 401 or 403 is also named once (source.elasticsearch.auth).

Export and stream. execute(query, { stream: true }) is always a point-in-time + search_after walk, page by page, whatever the window's size: a stream does not know a later page will not need one, and counting the matching set on every page asked for would rescan it once per page. No _count runs there, so the answer carries no total - exactly as the DuckDB and Splunk adapters' export mode leave it. Given, maxResultRows stops the walk once that many rows have arrived, named once (source.elasticsearch.truncated); the point in time it opened closes the moment the walk is known finished - every requested row arrived, the cap was reached, or the request's own signal aborted.

An executed example. It drives the suite's mock cluster, which evaluates the Query DSL the adapter sends, so the example shows the exact query and a real answer without a network or a credential. Leave fetch out, and the same adapter talks to your cluster.

const { elasticsearchAdapter } = await import('../packages/core/src/index.js');
const { createElasticsearchMock } = await import('../test/fixtures/elasticsearch-mock.js');

const cluster = createElasticsearchMock({
  types: { service: 'keyword', status: 'long', message: 'text', 'message.keyword': 'keyword' },
  docs: [
    { service: 'api', status: 503, message: 'Upstream timeout' },
    { service: 'API', status: 500, message: 'Disk full' },
    { service: 'web', status: 200, message: 'OK' },
  ],
});
const adapter = elasticsearchAdapter({
  url: 'https://es.example.com:9200',
  index: 'logs-*',
  headers: { Authorization: 'ApiKey your-key' },
  fetch: cluster.fetch, // leave out to talk to the cluster
});

const page = await adapter.execute({
  filters: { op: 'and', conditions: [
    { col: 'service', op: 'eq', value: 'api' },
    { col: 'status', op: 'gte', value: 500 },
  ] },
  sort: [{ col: 'status', dir: 'desc' }],
  range: { start: 0, end: 50 },
}, {});

// The query is JSON: the value is a value, and "api" matches "API" as the grid would.
const sent = cluster.calls.find((c) => c.path.endsWith('/_search')).body;
const expected = '{"bool":{"filter":[{"term":{"service":{"value":"api","case_insensitive":true}}},{"range":{"status":{"gte":500}}}]}}';
if (JSON.stringify(sent.query) !== expected) return JSON.stringify(sent.query);
return `${page.rows.length} of ${page.total} hits, ${page.rows.map((r) => r.status).join(' then ')}`;

Authenticating a remote adapter

Two shapes cover almost every endpoint. A fixed credential - an API key or a long-lived token - goes in headers, which odataAdapter and restAdapter send on every request. A credential that expires - a short-lived bearer token you refresh - goes in a custom fetch, which is the one place that can mint a fresh value per request. dfqlAdapter takes its token directly, and headers for anything a gateway adds on top.

A fixed token in headers. The map is sent on every request, so an Authorization header authenticates the whole grid. Below, a custom fetch stands in for the network only so the example can prove the header arrived:

const { odataAdapter } = await import('../packages/core/src/index.js');

let seen;
const adapter = odataAdapter({
  url: 'https://api.example.com/Orders',
  // A fixed credential authenticates every request.
  headers: { Authorization: 'Bearer static-token-123' },
  // Only here to capture what the adapter sent; in a browser, omit it.
  fetch: async (url, init) => {
    seen = init.headers.Authorization;
    return { ok: true, json: async () => ({ value: [], '@odata.count': 0 }) };
  },
});

await adapter.execute({ range: { start: 0, end: 20 } }, {});
return seen;   // the header reached the request

An expiring token in a custom fetch. A token with a lifetime cannot sit in a fixed map, because the map is read once and the token outlives no request that matters. A custom fetch is called afresh for every request, so it is where you refresh the credential and set the header on the outgoing call:

const { restAdapter } = await import('../packages/core/src/index.js');

// Stands in for a token service that hands out a new value each time.
let issued = 0;
const freshToken = async () => `token-${++issued}`;

let lastAuth;
const adapter = restAdapter({
  url: '/api/orders',
  fetch: async (url, init) => {
    // Refreshed per request, then merged over whatever headers the adapter set.
    const headers = { ...init.headers, Authorization: `Bearer ${await freshToken()}` };
    lastAuth = headers.Authorization;
    return { ok: true, json: async () => ({ rows: [], total: 0 }) };
  },
});

await adapter.execute({ range: { start: 0, end: 20 } }, {});   // token-1
await adapter.execute({ range: { start: 20, end: 40 } }, {});  // token-2
return lastAuth;   // a fresh token on the second request

Wiring it to the API you already have

Most data sits behind a service someone on your team wrote. The adapter below sends four parameters and expects { rows, total } back. Start by declaring only what the endpoint genuinely does, and widen it as you teach the endpoint more.

const source = createPushdownSource({
  compute,
  adapter: restAdapter({
    url: '/api/orders',
    // Only the comparisons the endpoint really applies. Claiming more here
    // returns the wrong rows rather than merely running slowly.
    operators: ['eq', 'gt', 'lt', 'contains'],
    params: { offset: 'skip', limit: 'take' },
  }),
});

The request that reaches your service, and the answer it owes:

// Express. FastAPI and ASP.NET differ only in how the query string is read.
app.get('/api/orders', async (req, res) => {
  const { skip = 0, take = 100, sort, order, filter } = req.query;

  let q = db('orders');
  if (filter) q = applyConditions(q, JSON.parse(filter));   // your translation
  if (sort) {
    sort.split(',').forEach((col, i) => {
      q = q.orderBy(col, (order || '').split(',')[i] === 'desc' ? 'desc' : 'asc');
    });
  }

  // The count is of everything matching, not of the page. A grid scrollbar is
  // sized from it, so a page-sized total makes the grid look empty below.
  const [{ count }] = await q.clone().clearOrder().count({ count: '*' });
  const rows = await q.offset(Number(skip)).limit(Number(take));

  res.json({ rows, total: Number(count) });
});

The total is the commonest mistake. It is the number of rows matching the filter, not the number returned in this page. The grid sizes its scrollbar from it and requests windows against it, so returning the page length makes a large result look like one page.

A worked example, executed on every build so it cannot go stale (data-run, PRD  C3):

const { capabilitiesOf, splitFilters, resolveMutate } = await import('../packages/core/src/source/pushdown.js');

// An adapter that understands three comparisons and nothing else.
const caps = capabilitiesOf({ filter: 'tree', operators: ['eq', 'gt', 'lt'] });

// Read-only by declaration: an adapter that says nothing about writing
// cannot mutate, and one that opts in resolves to a complete capability.
if (resolveMutate() !== false) throw new Error('a silent adapter must stay read-only');
if (!resolveMutate({ update: true }).update) throw new Error('an opt-in must resolve');

// A conjunction splits: what the engine knows goes to it, the rest stays here.
const { pushed } = splitFilters({
  op: 'and',
  conditions: [
    { col: 'a', op: 'eq', value: 1 },
    { col: 'b', op: 'gt', value: 2 },
    { col: 'c', op: 'lt', value: 3 },
  ],
}, caps);

return pushed.conditions.length;   // all three are supported

Building an adapter from the parts

createPushdownSource is the whole story for most callers. When an engine needs a source of its own, the four pieces it is assembled from are exported separately, so a custom source can plan and finish work the same way rather than reimplementing the split.

Residual work needs the complete result. applyResidual expects every matching row, not a window. Filtering a window is not a slower route to the right answer, it is a fast route to a wrong one: the rows that belong on page one may sit on page nine. planQuery sets needsAll whenever that applies, and createPushdownSource switches to fetching everything and paging from what it holds.

Whole-dataset statistics: fullDataset

A windowed source computes a total, statistic or group over the loaded window - the rows on screen - not the whole matching set, unless residual work already forced a whole-result fetch. “Median revenue” in the footer becomes the median of ~200 rows, wrong and looking right. fullDataset.enabled makes the whole-result fetch sticky and explicit: the entire matching set is held client-side once per query and every window, total and statistic is served from it, so the figures are computed over everything. It reuses the same whole-result path residual work already takes - needsAll - rather than a parallel mechanism. It is off by default and strictly opt-in. For a restAdapter, which cannot compute, it is the only way to get a correct whole-dataset statistic at all.

Memory-guarded, refused loudly. A matching set past maxRows or maxBytesEstimate is refused - thrown, surfaced as a source:error with no rows shown - never silently truncated. Presenting a fraction as the whole is the exact failure whole-dataset pull exists to prevent, so it is never the failure mode of the fix itself. Narrow the filter or raise the limit.

// An adapter that can page but reports the whole matching count. Without
// fullDataset a stat would see only the window; with it, the whole set.
const { createPushdownSource } = await import('../packages/core/src/source/pushdown.js');
const all = Array.from({ length: 100 }, (_, i) => ({ id: i, amount: i }));
const adapter = {
  name: 'demo',
  capabilities: { range: true, total: true },
  execute: async (query) => {
    const start = query.range ? query.range.start : 0;
    const end = query.range ? query.range.end : all.length;
    return { rows: all.slice(start, end), total: all.length };
  },
};
const source = createPushdownSource({ adapter, fullDataset: { enabled: true, maxRows: 1000, maxBytesEstimate: 5_000_000 } });
// Ask for a 10-row window; fullDataset holds all 100, so the mean is the true one.
const block = await source.fetch({ range: { start: 0, end: 10 }, filters: null, sort: [], quick: '' });
const held = block.total; // 100: the whole set is held, not the 10-row window
return all.reduce((s, r) => s + r.amount, 0) / held; // 49.5, the true whole-dataset mean

Refusing a partial result: allowPartialResults

When a query has residual work - a filter, sort or quick search the engine could not push - the source asks the adapter for the whole matching set, applies the residual here, and pages from what it holds. If the adapter instead returns a page of that result (it paged when told not to), the client-side filter or sort runs over the wrong rows: the rows that belong on page one may be in the fraction that was never fetched, so a page is presented as the full filtered set. That is a wrong answer, not a slow one.

By default the source refuses such a shortfall - it throws, and the remote source surfaces a source:error with no rows shown, rather than filter a fraction and lie. The fix is to make the adapter follow the engine's own paging before returning, or hold the data in memory. allowPartialResults: true is the knowing escape hatch: a caller who accepts the permissive behaviour - an adapter that genuinely cannot page and a result small enough not to matter, or a diagnostic run - keeps the old warn-once-and-proceed path. It is off by default, because a silent wrong answer is the one thing the design refuses. It does not affect the fullDataset memory guard, nor the no-residual short-return warning.

// An eq-only engine: the `gt` filter is residual and runs in the browser. The
// adapter reports 40 matching but returns only 1 row - a page shown as the whole.
const { createPushdownSource } = await import('../packages/core/src/source/pushdown.js');
const adapter = {
  name: 'shortfall',
  capabilities: { filter: 'tree', operators: ['eq'] },
  execute: async () => ({ rows: [{ id: 1, a: 5 }], total: 40 }),
};
const req = { range: { start: 0, end: 10 }, filters: { col: 'a', op: 'gt', value: 1 }, sort: [], quick: '' };

// Default: refused. Filtering 1 of 40 rows would return the wrong rows.
const strict = createPushdownSource({ adapter });
let refused = false;
try { await strict.fetch(req); } catch (e) { refused = /returned 1 of 40|refused/.test(e.message); }

// Knowing opt-in: warns once and proceeds with the fraction.
const lax = createPushdownSource({ adapter, allowPartialResults: true });
const block = await lax.fetch(req);
return refused ? `refused; opted in: ${block.rows.length} rows` : 'not refused';

Running a host predicate: whereRowLimit

A where predicate is a host function - whether this user may see the row, whether you hold a rate for its currency. No engine can evaluate one, so the only way a pushdown source can honour it is to fetch every matching row and filter here. That is a real answer, and it is also a windowed grid quietly turning into a whole-dataset download - the one thing a pushdown source exists to avoid.

So it is gated rather than done on your behalf. Under whereRowLimit (default 50_000, the same anchor as the grid's workerThreshold) the predicate runs and the counts are whole-dataset counts. At or past it - or when the adapter reports no row total, since the only way to learn the size from such an adapter is to fetch the set - the source refuses: the predicate is not applied, the rows it would exclude stay on screen, and one warning names the adapter, the size, the limit and the way out. Raise the limit when you want the download.

// Three rows, two of them ana's. The predicate is a host function with no twin,
// so the engine cannot narrow the fetch and the source must hold the set to run it.
const { createPushdownSource } = await import('../packages/core/src/source/pushdown.js');
const all = [{ id: 1, owner: 'ana' }, { id: 2, owner: 'bo' }, { id: 3, owner: 'ana' }];
const adapter = {
  name: 'demo',
  capabilities: { range: true, total: true },
  execute: async (q) => ({ rows: q.range ? all.slice(q.range.start, q.range.end) : all, total: all.length }),
};
const where = { active: true, names: ['mine'], version: 1, passes: (row) => row.owner === 'ana' };
const req = { range: { start: 0, end: 10 }, filters: null, sort: [], quick: '', where };

// Under the limit: the whole matching set is fetched and the predicate runs.
const under = createPushdownSource({ adapter, whereRowLimit: 1000 });
const applied = await under.fetch(req);

// At or past it: refused and warned about, and every row stays on screen.
const over = createPushdownSource({ adapter, whereRowLimit: 2 });
const refused = await over.fetch(req);

return `applied: ${applied.rows.length}, refused: ${refused.rows.length}`;

Pushing statistics down: aggregates

A DuckDB-class engine can compute a median or a standard deviation over the whole matching set far faster than pulling every row to do it here. The aggregates config decides, at grid setup, which statistics are computed by the engine and which by the grid. It is a design-time developer choice - fixed for the life of the grid, never a runtime toggle, never shown to an end user. Absent, every aggregate is computed client-side, so no existing grid changes behaviour.

Each statistic is classified IDENTICAL (the engine's result equals the grid's own kernel, verified against it) or MAY-DIFFER (the engine computes it by a method that can differ from the grid's definition). The classification drives this documentation and build-time provenance, not whether a stat is pushed - that is your choice. Only weightedQuantile is a genuine fallback: the engine cannot express the grid's midpoint convention, so it is always computed client-side.

No mixed provenance. An engine number and a client number never appear in one result set. Aggregates are pushed only when the filter and the quick search are both fully pushed; a residual filter the engine could not apply, or a quick search its capabilities.quick refuses, forces every aggregate client-side, because an engine figure computed over a superset beside a client figure over the real set would be wrong-but-plausible. lastPlan().aggregates reports, per statistic, whether the engine or the client computed it and the class it was assigned - build-time inspection, not a per-figure runtime marker.

The classification table below is generated from the single pushdown map (STAT_PUSHDOWN), so it cannot drift from what the adapter actually emits:

// Push the verified-identical stats to the engine; keep the fallback here.
// STAT_PUSHDOWN is the published map every stat's class and SQL comes from.
const { createPushdownSource, STAT_PUSHDOWN } = await import('../packages/core/src/source/index.js');
void STAT_PUSHDOWN;   // the single source of truth for the classification table above
const adapter = {
  name: 'demo',
  capabilities: { filter: 'tree', operators: ['eq'] },
  execute: async () => ({ rows: [], total: 0 }),
  // A real duckdbAdapter runs SQL; here we just echo which stats arrived.
  executeAggregates: async (query, aggs) => Object.fromEntries(aggs.map((a) => [a.id, 1])),
};
const source = createPushdownSource({ adapter, aggregates: {
  default: 'engine-if-identical',   // push only the verified-identical stats
  overrides: { mode: 'client' },     // but always keep mode's exact definition
} });
const split = await source.aggregate(
  { filters: null, sort: [], range: null },
  [{ id: 'a', col: 'revenue', fn: 'median' }, { id: 'b', col: 'size', fn: 'weightedQuantile' }],
);
// median is IDENTICAL so it pushes; weightedQuantile is a fallback so it stays here.
return `engine=${split.engine[0].class} client=${split.client[0].class}`;

Pivoting in the engine

A pivot on a pushdown source is answered by the engine, from grouped aggregates. The level's grouping column and the pivot columns become the GROUP BY keys, the engine returns one aggregated row per occupied group and pivot key, and the grid reshapes those rows into its pivot columns. No leaf row is fetched to do it: over the 10-million-row transactions sample below, a pivot of country by category with a sum, a count and an average moves 90 aggregated rows, where a memory source moves all ten million. duckdbAdapter, clickhouseAdapter, elasticsearchAdapter and splunkAdapter (blocking mode) declare capabilities.pivot.

The columns come from the answer. The root level also asks for the pivot's axis - every pivot key over the whole matching set - so the column headings are the keys the engine found, not a list you declare. Groups still open on demand. In a two-level grouping the root fetches the outer groups and their cells; opening a group fetches its children then, narrowed by the typed parent key. The innermost level is the pivot's row axis and has no leaves to open, as in a memory pivot. Everything else must push too. A filter, sort or quick search the engine cannot take, or a host where predicate, would make the engine pivot the wrong set, so the pivot is refused rather than half-pushed.

An adapter that cannot pivot refuses by name. restAdapter, odataAdapter, graphqlAdapter and dfqlAdapter have no grouped aggregate to ask for, and a remote source holds a window, not the leaves, so there is nothing to pivot in the browser either. The fetch rejects with code: 'pushdown:pivot-unsupported', the grid shows the refusal in its error overlay where the rows would have been, lastPlan().unpushed names pivot and lastPlan().pivotReason says why. No row is requested.

The statements duckdbAdapter sends for the root level of that pivot, the cells statement bound to the window's group keys:

-- The level: one row per country, with its row totals, windowed.
SELECT "country" AS "__lattice_key", count(*) AS "__lattice_leaves", sum("amount") AS "amount",
       count(*) AS "transaction_id", avg("risk_score") AS "risk_score"
FROM read_parquet('https://data.latticegrid.dev/samples/transactions-10m.parquet')
GROUP BY "country" ORDER BY "country" ASC NULLS LAST LIMIT 100 OFFSET 0

-- The axis: every category over the matching set - the column set.
SELECT "category" AS "__lattice_p0", count(*) AS "__lattice_leaves", sum("amount") AS "amount",
       count(*) AS "transaction_id", avg("risk_score") AS "risk_score"
FROM read_parquet('https://data.latticegrid.dev/samples/transactions-10m.parquet')
GROUP BY "category" ORDER BY "category" ASC NULLS LAST

-- The cells: country × category for the countries in the window, keys bound.
SELECT "country" AS "__lattice_key", "category" AS "__lattice_p0", count(*) AS "__lattice_leaves",
       sum("amount") AS "amount", count(*) AS "transaction_id", avg("risk_score") AS "risk_score"
FROM read_parquet('https://data.latticegrid.dev/samples/transactions-10m.parquet')
WHERE ("country" = ? OR "country" = ? OR …) GROUP BY "country", "category"

clickhouseAdapter writes the same statements with its statistics aliased by position (sum(`price`) AS `__lattice_a0`), because ClickHouse reads an alias back anywhere in a statement. elasticsearchAdapter sends one size: 0 search of nested terms aggregations; splunkAdapter runs | stats … by pipelines as jobs of their own and cancels each once read. adapter.sqlFor's siblings - pivotCellsSqlFor, pivotAxisSqlFor, pivotSqlFor, pivotBodyFor, pivotSearchFor - show the statement without running it. An adapter of your own declares capabilities.pivot and implements executePivotLevel(query, aggregates, request), returning the level's rows and pivot: { axis, cells }:

const { createPushdownSource, planQuery, capabilitiesOf } = await import('../packages/core/src/source/index.js');
const request = {
  range: { start: 0, end: 50 }, groupPath: [], groupValues: [], groupBy: ['region'],
  pivotBy: ['product'], pivotMode: true, totals: ['qty'], totalFns: { qty: 'sum' },
  filters: null, sort: [],
};
// An engine that pivots: aggregated rows back, one per region × product.
const engine = createPushdownSource({ adapter: {
  capabilities: { filter: 'tree', sort: 'multi', range: true, pivot: true },
  execute: async () => { throw new Error('no leaf is ever asked for'); },
  executePivotLevel: async () => ({
    rows: [{ region: 'North', leafCount: 7, totals: { qty: 9 } }], total: 1,
    pivot: { axis: [['apple'], ['pear']], cells: [
      { group: ['North'], pivot: ['apple'], values: { qty: 5 } },
      { group: ['North'], pivot: ['pear'], values: { qty: 4 } },
    ] },
  }),
} });
const level = await engine.fetch(request);
const cell = level.pivot.cells.find((c) => c.pivot[0] === 'apple');
// An adapter that cannot pivot is refused by name, and nothing is fetched.
const rest = createPushdownSource({ adapter: {
  capabilities: { filter: 'tree', range: true },
  execute: async () => { throw new Error('not reached'); },
} });
const refused = await rest.fetch(request).catch((error) => error.code);
const plan = planQuery(request, capabilitiesOf({ pivot: true, range: true }));
return `pivoted=${plan.pivoted} unpushed=${engine.lastPlan().unpushed.length} `
  + `${cell.group[0]}|${cell.pivot[0]}=${cell.values.qty} refused=${refused}`;

Grouping the engine was not asked for

A remote source groups nothing itself: a grouped level is the engine's answer, one level at a time. When the engine is not asked - the adapter declares neither capabilities.group nor the statistics to build the level from grouped aggregates (splunkAdapter, restAdapter, odataAdapter, graphqlAdapter, dfqlAdapter), or this query's filter, sort, quick search or where predicate did not push - the rows that come back are a window of leaves, and group rows built from them would show a page's figures as a group's. So the grid stays ungrouped: every display row is a leaf, grid.rows.count() is the source's count, no row carries a subtotal, and the warning pushdown:group-unsupported names the adapter and the reason, which lastPlan().groupReason repeats. The next query the engine can group is shown grouped again. Under fullDataset the grid holds every matching row and groups them itself, so nothing is refused there. An expanded group whose children are sub-groups shows one placeholder row until its level arrives.

const { createTestDom, flushFrames } = await import('../packages/dom/src/renderer/testdom.js');
const { createGrid } = await import('../packages/dom/src/index.js');
const { createPushdownSource } = await import('../packages/core/src/source/index.js');
const { reportedWarnings } = await import('../packages/core/src/internal/util.js');
const table = Array.from({ length: 1000 }, (_, id) => ({ id, county: ['KENT', 'DEVON'][id % 2], price: 1000 + id }));
// An engine that windows and counts, and cannot group.
const source = createPushdownSource({ pageSize: 100, adapter: {
  name: 'sales', capabilities: { filter: 'tree', sort: 'multi', range: true, total: true },
  execute: async (q) => ({ rows: table.slice(q.range.start, q.range.end), total: table.length }),
} });
const { root } = createTestDom({ width: 800, height: 400 });
const grid = createGrid(root, { rowKey: 'id', source, columns: [
  { field: 'id', type: 'number' }, { field: 'county', group: { enabled: true } },
  { field: 'price', type: 'number', total: 'sum' },
] });
for (let i = 0; i < 8; i++) { await new Promise((r) => setTimeout(r, 20)); flushFrames(); }
const rows = Array.from({ length: 100 }, (_, i) => grid.rows.get(i));
const out = `count=${grid.rows.count()} groups=${rows.filter((r) => r.group).length} first=${rows[0].data.id} `
  + `reason=${/does not declare/.test(source.lastPlan().groupReason)} `
  + `named=${reportedWarnings().some((w) => w.message.includes('pushdown:group-unsupported: the sales adapter'))}`;
grid.destroy();
return out;

Grouping from grouped aggregates

An adapter that cannot answer a grouped level but computes grouped aggregates - clickhouseAdapter and elasticsearchAdapter - still gives a grouped grid real groups. It qualifies when it declares capabilities.aggregates including count, a tree filter with the eq operator, and implements executeGroupedAggregates. Each group level is then one executeGroupedAggregates(query, [levelColumn], aggregates) call over the pushed filter: the group keys, the requested subtotals and a count per group come back computed by the engine and become the level's group rows (totals, leafCount). No leaf row is fetched for a level. Expanding a group fetches its rows through the ordinary execute row query, a page at a time, with each ancestor's key ANDed onto the filter as an exact, case-sensitive eq; a nested grouping asks one more grouped-aggregate query per expanded level. At the root, grid.rows.count() is the number of groups - never more than the source's rows - and the engine's grand row gives rows.matchCount() and the pinned grand total.

Order. The call answers the whole level, so the level is ordered the way the grid orders sibling groups - by the key, descending when the sort names the grouping column descending, with the grid's own comparator - and later windows of the level reuse the one answer (a window from row 0, as a reload asks, asks again). The sort travels in every pushed query, so a sort on any other column, a subtotalled one included, orders the rows inside an expanded group in the engine. As in a memory grid, it does not reorder the groups. A statistic outside the adapter's declared aggregates is absent from every group row and named in lastPlan().aggregates.client. Everything else that refuses a pushed grouping - a residual filter, sort or quick search, a where predicate, a key that is not a plain column - refuses this one too, with the pushdown:group-unsupported warning above.

const { createPushdownSource } = await import('../packages/core/src/source/index.js');
const compute = await import('../packages/core/src/compute/index.js');
const table = Array.from({ length: 1000 }, (_, id) => ({ id, county: ['KENT', 'DEVON'][id % 2], price: 2500 }));
const calls = [];
// An engine that aggregates but cannot answer a grouped level.
const adapter = {
  name: 'sales',
  capabilities: { filter: 'tree', operators: ['eq'], sort: 'multi', range: true, total: true, aggregates: ['sum', 'count'] },
  execute: async (q) => { calls.push(q); return { rows: table.slice(q.range.start, q.range.end), total: 500 }; },
  executeGroupedAggregates: async (q, [key], aggs) => [...new Set(table.map((r) => r[key]))].map((k) => {
    const rows = table.filter((r) => r[key] === k);
    return { keys: [k], level: 1, values: Object.fromEntries(aggs.map((a) =>
      [a.id, a.fn === 'count' ? rows.length : rows.reduce((t, r) => t + r[a.col], 0)])) };
  }),
};
const source = createPushdownSource({ adapter, compute, pageSize: 100 });
// The two requests a grid grouped by county sends: the root level, then KENT's rows.
const ask = { groupBy: ['county'], totals: ['price'], totalFns: { price: 'sum' }, sort: [], filters: null, range: { start: 0, end: 100 } };
const level = await source.fetch({ ...ask, groupPath: [], groupValues: [] });
const kent = level.rows.find((r) => r.county === 'KENT');
const leafCalls = calls.length;
const leaves = await source.fetch({ ...ask, groupPath: ['KENT'], groupValues: ['KENT'] });
const f = calls[0].filters;
return `groups=${level.total} KENT=${kent.totals.price} leaves=${kent.leafCount} leafCalls=${leafCalls} `
  + `filter=${f.col}=${f.value} page=${leaves.rows.length}`;

Joining two grids

Two grids each holding their own data, and a third showing where they meet. Orders against customers; shipments against carriers; enrolments against students. The third grid derives from one side and names the other as its join partner.

const joined = createGrid(host, {
  source: {
    mode: 'derived',
    from: orders,
    join: {
      with: customers,
      on: { left: 'customerId', right: 'id' },
      select: ['name', 'tier'],
    },
  },
  columns: [{ field: 'ref' }, { field: 'name' }, { field: 'tier' }, { field: 'amount' }],
});

The row count does not change. A key appearing twice on the right keeps the first match rather than emitting a row per pair. SQL would multiply them out; here that would change the row count of a grid the reader thinks of as “the orders” and quietly double every total taken from it.

Both sides are live. The partner is read at derivation time, not captured when the grid was built, and editing it re-derives, a corrected tier in the customer grid moves the order into a different band in the joined one.

Cross-filtering: the path back up

Derivation runs one way. A derived grid reads its source and never writes to it, which is what makes a chain of them safe to reason about. Cross-filtering is the single deliberate path back up: clicking a row in a summary panel filters the grid it summarises.

const byRep = createGrid(panel, {
  source: {
    mode: 'derived', from: main, groupBy: 'rep', refresh: 'live',
    crossFilter: true,
    select: { total: { of: 'amount', fn: 'sum' } },
  },
  columns: [{ field: 'rep' }, { field: 'total' }],
});

byRep.on('row:clicked', (e) => byRep.crossFilter.toggle(e.key));

The event is row:clicked, not row:click. A handler bound to the wrong name subscribes without error and never fires, so this example is executed on every build to keep the name honest: it wires the same handler to a stand-in grid, emits the event, and checks the click reached the cross-filter.

// A stand-in for the grid's event bus and cross-filter, so the wiring above can
// be executed here without a DOM. The names are the product's own.
const listeners = {};
const filtered = [];
const grid = {
  on: (name, fn) => { (listeners[name] = listeners[name] || []).push(fn); },
  emit: (name, e) => { for (const fn of listeners[name] || []) fn(e); },
  crossFilter: { toggle: (key) => { filtered.push(key); } },
};

// The line from the example, verbatim in its event name.
grid.on('row:clicked', (e) => { if (e.key) grid.crossFilter.toggle(e.key); });

// A click on a rep row. The wrong name - 'row:click' - would reach no handler,
// and this block would produce '' instead of the key.
grid.emit('row:clicked', { key: 'EMEA' });
grid.emit('row:click', { key: 'US' });

return filtered.join(',');

A panel does not filter itself. The grid pushing the filter leaves its own condition out when it reads the source back. Without that, clicking one rep would collapse the panel to that single row and strand the reader with nothing else to click. It is the same rule that keeps a header histogram showing every bar after you click one (facets), applied between grids instead of within one.

Several panels compose. Each leaves out only its own condition, so two panels over different columns narrow each other while both stay whole: pick a rep and the region panel shows that rep's regions, pick a region and the rep panel shows that region's reps.

It needs a memory source. Leaving a panel's own condition out means asking the source for every row that survives the other filters, which a source holding one page cannot answer. Over a remote, paged or stream source the read falls back to the ordinary filtered rows (narrowed by the very column the panel asked to be excluded from) and the panel collapses to the row that was clicked. It warns when it does. Exclude the originating panel on the server instead.

It is an ordinary filter. The condition goes through the source's filters.set, so it undoes, rides in a saved view, and appears in whatever filter UI the grid already has. There is no second filter model beside the real one.

The remote request

Your fetch receives one object and returns { rows, total }.

grid.updates

Control over an incoming feed: hold it, let it through, and see what the batching is actually saving you. Pausing does not drop anything: held changes keep merging, so a long pause costs one entry per changed row rather than one per update.

grid.updates.pause();                // hold the feed; it keeps arriving and merging
grid.updates.stats();                // { pending, queued, coalesced, flushes, ... }
grid.updates.flush();                // apply what is waiting, stay paused
grid.updates.resume();               // apply everything and go live again

coalesced is the number a batching strategy is actually bought with: rows that arrived more than once in a window and were written once. A feed where it stays at zero is not being coalesced, whatever the interval says.

The log is bounded two ways, because an entry is not a fixed size, one carrying a single changed cell and one carrying a fifty-thousand-row batch both count as one.

Applying changes

rows.apply(change) applies immediately and returns what happened; rows.queue(change) batches into the next flush and returns a promise. Both take the same shape: add, update, remove, and an optional at insert position.

An update is a patch, not a replacement. Fields absent from the update are untouched, so a delta from a websocket or a save response can be applied as it arrives without reading the row back first. Coalescing merges fields too: {price} and {volume} arriving as separate messages inside one window both survive.

grid.rows.apply({ update: [{ id: 'R1', price: 42 }] });
// every other field on R1 is left alone

Rows that cannot be applied are reported, not thrown. A batch of a thousand containing three bad rows applies the other 997 and lists the three.

const result = grid.rows.apply({ update: [ ... ] });
result.rejected;  // [{ operation, id, reason }]

stats() reports held against heldLimit: what the log is carrying now, against what it will carry. rows is a lifetime total of everything that ever arrived and says nothing about memory; these two do. Raise logRows for a deeper scrubber on a grid you have measured, and lower it on a feed of very wide rows.

stats().dropped is not lost data. It counts change-log entries trimmed to stay inside heldLimit - the scrubber losing depth on the oldest history - never an update the grid failed to apply. Every row it counts was already applied before its log entry aged out.