Connect a Data Grid to DuckDB: Parquet in the Browser

A grid over a real analytical engine

DuckDB is a full analytical engine, and this adapter lets the grid drive one without carrying a byte of it. You create the connection and hand it over; the adapter translates each sort, filter and scroll into SQL and reads the result back. The pay-off is that a large analytical dataset stays fast: the filter, the sort and the paging are answered by an engine built for exactly that, over data that never has to be loaded into the grid row by row.

import * as duckdb from '@duckdb/duckdb-wasm';
import { createGrid, createPushdownSource, duckdbAdapter } from '@toclocoinc/lattice-grid';
import * as compute from '@toclocoinc/lattice-grid';

// You create the DuckDB connection; the grid imports no engine.
const db = await makeDuckDb();                 // your duckdb-wasm bootstrap
const connection = await db.connect();

createGrid(el, {
  columns,
  rowKey: 'id',
  source: createPushdownSource({
    adapter: duckdbAdapter({
      connection,                              // the live connection you made
      from: "read_parquet('data/readings.parquet')",
    }),
    compute,
    pageSize: 100,
  }),
});
// A hundred thousand rows, queried in the tab, with no back end. The adapter answers
// the filter tree, a multi-column sort and the page in one query.

In the browser, or on a server

The adapter takes a connection you made, and does not care where it came from. In the browser that is @duckdb/duckdb-wasm, which runs DuckDB compiled to WebAssembly in the tab, so a hundred-thousand-row Parquet file can be queried with no back end at all. On a server it is any DuckDB client, driven through your own endpoint. Either way the adapter is the same code and imports nothing: DuckDB returns Arrow tables and they are read through their own accessors, so the Arrow library never enters the bundle.

What you query is up to the from expression. A table name, a view, or any FROM expression works, which is most of why the engine is worth reaching for: read_parquet('s3://…') is as valid as a table, so the grid can sit over a Parquet scan directly.

duckdbAdapter({ connection, from: 'orders' })                       // a table
duckdbAdapter({ connection, from: 'analytics.daily_totals' })       // a view
duckdbAdapter({ connection, from: "read_parquet('s3://bucket/*.parquet')" }) // a scan
duckdbAdapter({ connection, from: 'orders', fields: ['id', 'total', 'placed'] }) // a projection

Querying a private bucket

One of the best reasons to reach for DuckDB is to point the grid straight at an archive: a bucket of Parquet logs, exports or snapshots, queried in place with no pipeline to load them first. When that bucket is private, the credential to read it is DuckDB's own configuration, not a grid option. You give the connection access with a SECRET before you hand it over, and from then on the grid simply runs the from expression you gave it. It never sees the credential and does nothing with it: there are no headers and no fetch here, because the adapter is talking to a database connection, not making HTTP calls of its own.

DuckDB can take credentials two ways. It can pick up whatever is already in the environment, an instance role or an AWS profile, through the credential chain, or you can supply an explicit key and secret. Either way it is one statement on the connection, after which read_parquet('s3://…') reads the private bucket exactly as it would a public one.

// Credentials are DuckDB's own configuration, set on the connection before
// it reaches the grid. DuckDB reads a private bucket through a SECRET.

// Ambient credentials: pick up the AWS chain already in the environment
// (an instance role, a profile, or the usual environment variables).
await connection.query(`
  CREATE SECRET archive (
    TYPE s3,
    PROVIDER credential_chain,
    REGION 'eu-west-1'
  );
`);

// ...or supply an explicit key, secret and region.
await connection.query(`
  CREATE SECRET archive (
    TYPE s3,
    KEY_ID '${accessKeyId}',
    SECRET '${secretAccessKey}',
    REGION 'eu-west-1'
  );
`);

// Now hand the connection to the grid. It runs your from expression as-is;
// the credential is already on the connection.
duckdbAdapter({
  connection,
  from: "read_parquet('s3://your-bucket/logs/2026/*.parquet')",
});

What pushes down to DuckDB

The DuckDB adapter answers most of the query in the engine:

-- The page the grid asked for.
SELECT *
FROM read_parquet('data/readings.parquet')
WHERE ("region" = ? AND "value" > ?)
ORDER BY "value" DESC
LIMIT 100 OFFSET 200;

-- And the size of the set it was cut from: the same filter, the same bound
-- values, no ORDER BY and no LIMIT. Both statements are sent together.
SELECT count(*)
FROM read_parquet('data/readings.parquet')
WHERE ("region" = ? AND "value" > ?);

The operators it writes are equals and not-equals, the four inequalities, contains, startsWith and endsWith as ILIKE, blank and not-blank, and in for a set. Two details are worth knowing. The count is its own statement rather than a column on the page: it carries the same filter and the same bound values, and it is sent in the same tick as the page query rather than after it, so the grid still gets the page and the size of the set it was cut from without waiting twice. Read it with adapter.countSqlFor(query), the counting counterpart of adapter.sqlFor(query). A grid that never shows a total can skip it entirely with duckdbAdapter({ count: false }), and then the scrollbar is open-ended and discovers the end as the reader reaches it, which suits a grid people scroll and not one where they jump. And DuckDB returns BIGINT and HUGEINT as JavaScript BigInt: every value is normalised on the way out, and a magnitude past the safe integer range is kept as a string rather than silently rounded, because a wrong identifier is worse than an awkward one.

Grouping, totals and statistics over the whole set

By default a group-by, a total or the statistics layer is computed over the rows the grid has loaded. Over a windowed DuckDB connection that means the current page, not the whole matching set, so those figures describe the view in front of you and a dev-mode warning fires whenever one covers only part of the data. That is honest, but it is rarely the figure you want in a footer. There are two ways to get a whole-dataset figure, and they suit different sizes of data.

Pull the whole set in. Turn the full-dataset pull on and the grid asks DuckDB for everything the filter matches, once, and holds it. From then on grouping, subtotals and the statistics layer all compute over the entire set, and browsing is instant because the rows are already here. It is memory-guarded: a set past the configured row or byte ceiling is refused with a visible error rather than truncated, so a partial figure is never presented as the whole. The trade is that the grid is now memory-resident and no longer issues a query per interaction.

createPushdownSource({
  adapter: duckdbAdapter({ connection, from: "read_parquet('data/readings.parquet')" }),
  compute,
  // Pull the whole matching set in once and hold it, so grouping, subtotals
  // and the statistics layer compute over all of it, not the loaded page.
  fullDataset: { enabled: true, maxRows: 250_000 },
});

Push the aggregation down. Declare an aggregate-pushdown policy and the supported statistics are computed by DuckDB over the whole matching set, ungrouped or with a GROUP BY, and only the numbers come back. The dataset stays where it is, which is the mode to reach for when it is too large to hold. The common statistics match the grid's own definitions, because the engine is asked for the same ones: quantiles are the R type 7 definition a spreadsheet agrees with, and a deviation is the sample form. Two differences are documented rather than hidden. The statistical mode is computed by DuckDB's own convention and can disagree with the grid's at the edges, so it stays client-side under the recommended 'engine-if-identical' policy. And a weighted quantile is always computed in the browser, because there is no faithful way to express it in SQL. Every figure's provenance, engine or client and why, is on source.lastPlan().

createPushdownSource({
  adapter: duckdbAdapter({ connection, from: 'readings' }),
  compute,
  // Compute the supported statistics in DuckDB over the whole matching set,
  // ungrouped or with a GROUP BY. 'engine-if-identical' pushes only the stats
  // whose engine result is verified identical to the grid's, keeping mode and
  // the weighted quantile client-side.
  aggregates: { default: 'engine-if-identical' },
});

Residual work still pulls the set in

Separately from aggregation, when a filter, a sort or a quick search cannot be pushed to DuckDB, the grid asks the engine for the whole matching result rather than a window, because filtering a single page locally would put page-one rows on page nine, and it warns once naming what it could not push. If it is ever handed fewer rows than it asked for while residual work is pending, it refuses the result and raises an error rather than working over a fraction and presenting it as the whole. You can watch the split after every query with source.lastPlan().

Filter values are bound, not interpolated

Filter values come from the user, which is the one part of a grid a stranger can steer, so they are sent through prepared statements rather than pasted into SQL. Column identifiers, which cannot be parameterised, are refused unless they are plain identifiers. If a connection has no prepare method, the adapter declines to send the filter rather than interpolate it, and says so once: filters are dropped, never inlined.

New to the adapter? How to query a Parquet file in the browser with DuckDB is the short version: the least code that points a grid at a file and pushes a sort and a filter down to it.

See it running: a hundred thousand rows in DuckDB, no server, which pulls a Parquet file into the browser and groups it so the subtotals and grand total are over all of it. The live pushdown demo keeps the whole-set pull off and shows the SQL for each interaction. For the log-lake shape, triage a day of security events with the top sources computed by DuckDB over the whole matching set, and read how the grid sits over the Parquet logs you already keep in S3. For the whole picture of what an application over DuckDB looks like, building an interactive frontend for DuckDB runs ten million transactions past a grid and prints every statement it produces. If the data you are after lives in Splunk rather than a lake you already query with DuckDB, bringing Splunk data into the grid covers what actually works, exported search results, not an open bucket format. The pushdown guide covers the split rules shared across every adapter, and connect your data lists the other options.