Sources and pushdown: JavaScript Data Grid Guide

With retainSource: false alone the saving is only the store's own copy of the object references, not the objects themselves: the source layer and the grid config still hold the array, so whoever handed the grid its rows keeps them alive. dropSourceRows releases those references too, so once the caller lets go the objects can be collected and the grid becomes the sole holder of the data. Reach for it on a large, read-mostly grid where you can let go of the source array and do not depend on caller identity through rows.data().

The request also carries groupBy, groupPath, pivotBy, totals and your own context, so a server that can group and aggregate does that work instead of the browser. Blocks are fetched as the viewport reaches them and cached; changing the sort or the filter invalidates the cache and re-queries.

The filters your callback receives is the same condition tree documented above. You are not handed an opaque object to reverse-engineer, it is the published format, and the same shape you would have written by hand.

Everything is off unless declared. An adapter that declares nothing still works: the grid fetches and does all the work itself. That is the safe direction to be wrong in. Declaring an operator the engine does not really apply is the unsafe one, because the grid will trust it and stop checking.

Splitting a filter is not symmetric. An and group narrows with each condition, so the supported conjuncts go to the engine and the rest stay behind: the engine returns a superset and the grid narrows it. An or group widens with each branch, so pushing only the supported branches would return fewer rows than the filter allows, and the grid cannot recover rows that were never fetched. A disjunction that is not fully supported therefore stays whole on the client. The same asymmetry governs column pruning in the facet path.

Residual work needs the whole result. When anything is left over, the source stops asking for windows and asks for everything, applies the remainder, and pages from what it holds. Filtering a window on the client is not a slower route to the right answer, it is a fast route to a wrong one: the rows that belong on page one may sit on page nine, and the total is whatever the engine happened to count.

from is any FROM expression, so read_parquet('s3://bucket/*.parquet') is as valid as a table name. Values are bound through prepared statements; a connection without prepare is used only for unfiltered queries, because interpolating a user's filter into SQL is the one thing worse than not filtering at all.

Column and table names are checked against an identifier pattern rather than escaped, and a name that fails is refused. Integers past the safe range are kept as strings instead of being rounded into a plausible lie.

A timestamp or date filter is bound through a typed placeholder - "ts" >= CAST(? AS TIMESTAMP) - because a prepared statement binds an ISO string as VARCHAR and DuckDB will not compare that with a TIMESTAMP, TIMESTAMP WITH TIME ZONE or DATE column. The adapter reads the column types once from the engine (DESCRIBE) and falls back to the grid column's declared type, so both a typed and an untyped grid column over a timestamp filter correctly, including a time window. Send instants ending in Z, as the grid's date filter does: the cast is the engine's, and its zone rules apply (see the reference for the TIMESTAMPTZ and naive-string cases).

demo/duckdb.html runs this against a Parquet file of several million readings with no server involved, including a time-window filter on its TIMESTAMP column.

Measured against a 1.5 MB Parquet file on GitHub Pages: without LOAD httpfs;, DuckDB-Wasm 1.32.0's default HTTP path issued zero Range requests and read the whole file - 100% - for every query, chip filters included. With it loaded on the connection first, the same query issued 25 Range requests and read 30% of the file. DuckDB-Wasm 1.29.0 read 119% of the file on the same test (its ranges overlapped), so treat the exact percentages as a version's behaviour, not a fixed promise - re-measure against the DuckDB build you ship.

The file host has to cooperate too: it must answer HEAD, advertise Accept-Ranges: bytes, and answer a range request with 206 Partial Content and a Content-Range header (GitHub Pages does all three). Cross- origin, its CORS policy must also expose Content-Range and Content-Length, or the browser cannot read them back. Any of that missing, and DuckDB falls back to a full read with nothing said.

A file registered with db.registerFileBuffer(...) is always downloaded whole, whatever version is loaded or however the host answers: a buffer has already been fetched in full before DuckDB ever opens it, so there is nothing left to range-read. There is no force_download option - it does not exist in DuckDB-Wasm; registering a buffer is the way to force a whole download.

Measured on a 10 million-row Parquet file served over HTTP range requests, in Chrome:

• A page of 100 rows: about 0.1 s.
• count(*) with no filter: about 25 ms and no data read at all - Parquet keeps the row count in its footer and DuckDB answers from there.
• count(*) with a filter: about 1.3 s and about 28 MB read. The predicate has to be evaluated, so the predicate's columns have to be read.

A filtered count is cheaper than fetching the rows to count them yourself. It is not cheap. Anyone who tells you a filtered count over a remote file is free is quoting the unfiltered number.

Sort (cluster) the file on the columns you filter most. Row-group statistics can only prune a group when the values in it are narrow, and that is what sorting makes them. On the same 10M-row file and the same filter, a file sorted on the filtered column pruned 87× more of the read than the unsorted one - the count went from reading tens of megabytes to reading a fraction of the file. This is the one change worth making first, and it costs you nothing at query time.

Write large row groups. ROW_GROUP_SIZE 1000000 roughly halved the cost of a filtered count that could not be pruned, because a row group is the unit of both the statistics and the read. Tiny row groups mean many range requests and much more metadata.

Turn the count off when nobody reads it. duckdbAdapter({ connection, from, count: false }) issues no count statement at all. The grid scrolls open-ended - it discovers the end as you reach it - and the status bar shows what it knows rather than a total. On a dashboard where nobody has ever asked "how many rows match?", that is the whole 1.3 s back.

What each operator becomes. Each keeps its in-memory meaning, so the answer does not change with who evaluates it. withinBbox is ST_CoveredBy(geom, ST_MakeEnvelope(?, ?, ?, ?)) - edges count as inside, which ST_Within would not. withinPolygon is ST_Within of the centroid (of every point, for a MultiPoint), not of the whole shape. intersects is ST_Intersects. withinDistance is the haversine distance from the centroid on the same sphere, written out in SQL: the extension's ST_DWithin_Spheroid returned a non-boolean on DuckDB-Wasm 1.29.0, takes latitude first, and measures on the ellipsoid, up to 0.5% away from the in-memory answer. Every operand is a bound parameter - GeoJSON to ST_GeomFromGeoJSON(?), WKT to ST_GeomFromText(?), WKB to ST_GeomFromHEXWKB(?) - and an operand that cannot be read matches no row and says so once.

Columns and coordinates. DuckDB's Parquet reader turns a GeoParquet column into GEOMETRY by itself once the extension is loaded; a WKB BLOB named by column is read through ST_GeomFromWKB. When the file's metadata declares another CRS, or crs names one ('EPSG:27700' or 27700; it wins over the file), every read goes through ST_Transform(geom, crs, 'EPSG:4326', always_xy := true), so filters, sorting and the rows themselves are in longitude/latitude.

Sorting, grouping, buckets. A geometry column sorts by centroid latitude, then longitude. Grouping the grid by one is refused by name: a shape is not a category. source.aggregate() takes { col, bucket: 'grid', size } as a groupBy key - the centroid binned into cells size degrees square - and hands each cell back as [x, y], covering longitudes [x·size, (x+1)·size) and latitudes [y·size, (y+1)·size).

When the extension does not load, source.duckdb.spatial.load warns once, the spatial operators and buckets leave the adapter's capabilities, and the pushdown source - which waits for the adapter's first use before planning - leaves the operators to the grid. Pass compute to createPushdownSource so they are applied; geometry then arrives as the stored WKB, which the column reads just the same, and sorting by it is refused rather than ordered by its bytes.

What loading costs (measured 2026-09-27, headless Chrome, DuckDB-Wasm 1.29.0 = DuckDB 1.1.1). selectBundle picks the eh build on a page that is not cross-origin isolated, and the extension then comes from extensions.duckdb.org/v1.1.1/wasm_eh/spatial.duckdb_extension.wasm: 22.8 MB (about 5.9 MB zstd on the wire), served with Access-Control-Allow-Origin: *, so no proxy and no COOP/COEP headers are needed. INSTALL does nothing on Wasm (7 ms); LOAD took 0.88-1.07 s, and the first query over the 13-row fixture that returned GeoJSON about 1 s more. The mvp and threads builds have their own files of the same size. Re-measure against the build you ship.

Why the browser needs the small wrapper. An MDConnection is not a DuckDB-Wasm connection: it answers evaluateQuery and evaluatePreparedStatement, not query and prepare. Do not hand the adapter the underlying DuckDB-Wasm connection instead: the client's own README says an AsyncPreparedStatement's query method can deadlock with the MotherDuck extension, and prepare(sql).query(...) is exactly how the adapter binds a filter. The wrapper routes both through the client's evaluate methods, which it serialises per connection.

The token belongs to the host, not the page. A MotherDuck token in page source is a key to the whole account for anyone who opens devtools. Mint a short-lived, read-scoped token on your backend for the signed-in user and hand it to the page at run time, as the example does; refresh it by calling the client's terminateDuckDB() and creating a new connection. For a public page, publish the data as a read-only MotherDuck share in an account that holds nothing else, and mint the token there. On Node the token lives in the process environment (MOTHERDUCK_TOKEN; DuckDB 1.5.5 also accepts motherduck_token) and never in SQL text, which ends up in logs.

What was measured, and what it means for your grid (2026-09-28, Node 22, @duckdb/node-api 1.5.5-r.5 = DuckDB 1.5.5, 50,000 rows):

• Attaching took 0.7-1.8 s (ATTACH 'md:' or ATTACH 'md:lattice_demo', six runs); budget it once per process, not per query.
• The first query (the adapter's DESCRIBE, the first page and the count) took 57-105 ms over three runs; a filtered, sorted page at offset 100 with its count 89-121 ms.
• Three-part names need nothing from you. from is a FROM expression, so lattice_demo.main.products (and even md:lattice_demo.main.products, which DuckDB also resolves) reads as written. Write-back accepts database.schema.table when every part is a plain identifier and was proven with an UPDATE … RETURNING against MotherDuck. A name that needs quoting ("my-db".main.t) or carries md: is not a plain table, so writable: true is refused by name (refusing to mutate … is not a plain writable table) and the source stays read-only. Reading through such a name still works.
• Result size. No limit was reached: reading all 50,000 rows of seven columns in one statement took 0.21-0.85 s. A pushdown grid reads a page at a time, so it never asks for that many.
• Extensions. spatial: true works unchanged: the adapter's INSTALL spatial; LOAD spatial ran on the client, the table's GEOMETRY column was found by DESCRIBE, and a bounding box, a sort by location and GeoJSON output all matched the oracle. The motherduck extension itself is fetched by DuckDB on the first ATTACH, so the machine needs to reach DuckDB's extension repository once.

The data behind these numbers. lattice_demo.main.products is 50,000 synthetic rows (id, name, category, price, qty, created, a VARCHAR[] of tags, lon, lat and a GEOMETRY point), every column a pure function of the row number. test/duckdb-motherduck.test.js holds the recipe and creates the table when it is absent. That test skips, and says why, without MOTHERDUCK_TOKEN and @duckdb/node-api; this package depends on neither.

The browser recipe, run once. The built dist/lattice-grid.esm.js in headless Chrome, @motherduck/wasm-client 1.5.5-r.1 loaded from jsDelivr, the token fetched from the serving process at run time (2026-09-28): the page did not need to be cross-origin isolated (crossOriginIsolated was false); importing the client took 395 ms and isInitialized() 4.6 s, which is the DuckDB-Wasm download and start, once per page. The first page with its count took 165 ms, and a filtered, sorted page at offset 100 with its count 120 ms, its rows and its total (4,998) identical to a hand-written SQL query. A hasAny over the tags came back as arrays and matched (20,000), and spatial: true loaded the extension in the browser, pushed a bounding box (907, matched) and returned GeoJSON. Nothing was left to the grid. This was a one-off check, not a gated test: it needs a token and the network.

One condition, replaced, not stacked. A second pan replaces the first viewport condition and leaves any other filter on the tree exactly where it was - grid.filters.get() after a pan on a grid that already carries an unrelated filter shows both, an and of the unrelated one and the new box, never two boxes. A "reset view" and destroying the chart both remove the condition; a map with nothing to pan - no shapes and no geometry column, so it never leaves its own identity view - never writes one.

Two-way. A matching condition set on the grid from outside - a saved view, a host, a sibling panel filtering the same column - pans and zooms the map to it instead of being overwritten, and is not written straight back: the box the map ends up showing may differ from the one it was handed when the plot's own aspect does not match the box's (the same letterboxing fitted already gives a pack that does not fill its panel), but once settled nothing writes again until the reader, or another `filters.set`, moves the view again.

What withinBbox actually tests is that the row's own bounding box is entirely inside the given box, edges included (ST_CoveredBy under DuckDB) - not, as a first reading of "the map's own condition should not drop a feature at the edge" suggests, the row's centroid. A polygon whose shape straddles the viewport edge is therefore excluded once its condition narrows its own grid, the same as a point outside the box is; a centroid test would have kept it. This is stated here rather than changed, because the operator's own rule (the geometry module) is not this feature's to redefine.

It is off by default and reuses the same whole-result path that residual work already takes: the flag ORs into needsAll, so once the set is held it is ordinary in-memory data and the grid's existing total and statistics kernels reduce over all of it, with no per-stat change. When it is active and within the limits, the whole matching set is covered, so the windowed-statistic warning stays silent - the figure is now honestly whole-dataset.

It is refused loudly, never truncated. A matching set past maxRows or maxBytesEstimate is thrown and surfaced as a source:error with no rows shown, rather than held as a fraction and presented as the whole. A fraction shown as the whole is exactly the silent wrong answer this feature exists to remove, so it is never how the feature fails. For a restAdapter, which cannot compute, this is the only route to a correct whole-dataset statistic.

Off by default, refused loudly. Filtering or sorting a fraction of the result does not lose rows quietly, it returns the wrong rows, so the default throws rather than warns - nothing fails silently () means a wrong answer is never preferred to a visible failure. The fix is an adapter that follows the engine's own paging, or holding the data in memory. allowPartialResults: true is the deliberate opt-out - a caller who accepts the permissive behaviour keeps the warn-once path - and it is never the default. It does not touch the fullDataset memory guard or the no-residual short-return warning, both of which stand regardless.

Why a limit at all. A pushdown source exists so a host does not download ten million rows. Honouring a twinless predicate means holding the whole matching set, so wiring it in without a gate would let one filter function silently convert a windowed grid into a full download - trading one silent surprise for another. Under the limit the predicate really runs and the counts are whole-dataset counts; at or past it the predicate is not applied, the rows it would exclude stay on screen, and one warning names the adapter, the matching-set size, the limit and the way out.

No row total counts as over the limit. The only way to learn the size from an adapter that cannot count is to fetch the set - which is the download being guarded against - so the source refuses rather than guesses its way into it.

A refusal costs nothing extra. Where the predicate alone would force the whole result, the size is learned from a bounded probe that is exactly the window fetch the request would otherwise have made, and that result is used as that fetch. A host over the limit pays what it paid before the gate existed.

Prefer the twin. A predicate registered with a { condition } twin is pushed to the engine, which narrows the fetch itself - no limit applies, nothing is held here, and it works at any size. Reach for whereRowLimit only when what you are testing genuinely cannot be written as a condition. On a paged or remote source the twin is the only route: those hold a block cache indexed by the server's own ranges and cannot run a host function at all, and the grid warns once when you register one without a twin.

Every statistic is classified IDENTICAL (the engine's result equals the grid's own kernel, verified against it on the same data) or MAY-DIFFER (the engine computes it by a method that can differ from the grid's definition - mode returns a value where the grid returns null). The classification drives the docs and build-time provenance, not whether a stat is pushed: that is your choice. weightedQuantile is the one genuine fallback, always client-side, because the engine cannot express the grid's midpoint convention. The published table of every stat, its class and its SQL is generated from one map (STAT_PUSHDOWN) so it cannot drift.

No mixed provenance. An engine figure and a client figure never appear in one result set. Aggregates are pushed only when the filter and the quick search are both fully pushed; a residual filter the engine could not apply, or a quick search its capabilities.quick refuses, forces every aggregate client-side, because an engine number computed over a superset beside a client number over the real set would be wrong-but-plausible. source.lastPlan().aggregates reports, per stat, whether the engine or the grid computed it and the class it was assigned - inspection during the build, not a per-figure runtime marker.

A change is patched, not re-derived. When a row changes in the source, the derived grid updates the groups that row belongs to rather than rebuilding the lot. Five hundred updates against a two hundred thousand row source cost under 300 ms in total. Set refresh to live, manual or a number of milliseconds to change the coalescing; idle is the default and settles to a frame.

manual means the host says when, and rows.load() is how it says it. A manual derived grid never re-derives on its own: the source can filter, edit and tick underneath it and the panel keeps showing what it last derived. Call rows.load() on the derived grid, with no argument, and it re-reads its from there and then and replaces its rows; call it again whenever you want the next reading. A derived grid takes its rows from from, so anything passed to load is not used. The derived grid's own sort and filters stay as they were. Press it as often as you like - the source is left exactly as one press leaves it, and destroying the panel leaves nothing of it attached there.

Derived grids are read-only. There is one copy of the data and it lives in the source. Write there and the derived grid follows: an edit through the source's editing API (edit.setCells, an inline commit, a fill, a paste, an undo, a redo or an optimistic rollback) is announced once per batch, after the per-cell cell:changed events, and the derived grid re-derives once for the batch. A grouped derivation patches the groups the edited rows belong to; a derivation with a where, an unnest, a producer, or an edit to a column the source filters on re-reads in full, at the cost measured above. Use idle when a derived child follows an editable grid. Under live the derivation runs synchronously inside the edit call, so a keystroke's commit on a 200k-row source pays the whole re-read before the editor closes; idle defers it to the next frame and folds a burst of edits into one.

The key comes for free. A derived grid keys on __key, which the source writes onto every row it produces: the group value, the profiled column, or the source row's own key when nothing is grouped. Set rowKey only to override it.