Pushdown sources

A grid over an engine

A pushdown source sits the grid in front of a query engine and lets the two divide the work. When the user sorts, filters or scrolls, the grid turns that into one portable query, an adapter translates it into whatever the engine speaks, and only the rows asked for come back. The grid does not fetch a table and thin it in the browser; the engine does the part it is good at, over data that never has to leave it.

import { createGrid, createPushdownSource, odataAdapter } from '@toclocoinc/lattice-grid';
import * as compute from '@toclocoinc/lattice-grid';

createGrid(el, {
  columns,
  rowKey: 'OrderID',
  source: createPushdownSource({
    adapter: odataAdapter({ url: 'https://services.odata.org/V4/Northwind/Northwind.svc/Orders' }),
    compute,        // the grid's own kernels, for anything the engine did not do
    pageSize: 100,
  }),
});

The grid stays a client library. An adapter takes a connection, a URL or a token that is already yours and imports nothing, so the bundle is byte-for-byte the same whether a page uses this or not. That is the point worth making: the grid can drive a full analytical engine without carrying a byte of one.

An adapter declares what it can do

Every engine speaks a different amount of the grid's query. Some take a full condition tree, some a single search term, some cannot sort at all. So an adapter declares its capabilities, and the grid pushes down only what the adapter claims and does the rest itself. Everything is off until declared, which is the safe direction to be wrong in: an adapter that declares nothing still works, the grid simply fetches and does the whole job.

See the split

Every grid vendor claims server-side data. What none of them show is which half of your query actually reached the server. source.lastPlan() tells you, after every query.

const plan = source.lastPlan();
plan.pushed;    // the query the adapter was given
plan.residual;  // { filters, sort, quick } the grid applied afterwards
plan.unpushed;  // ['filter'], the parts that stayed behind
plan.needsAll;  // true when the whole result had to be fetched

Turn an adapter's declared capabilities down to nothing and the same filter moves from the engine to the browser, with the plan changing to match. It is the honest version of "server-side": you can watch where the work went.

The rules that shape the split

The grid never trades a right answer for a faster one. Three rules keep a partial pushdown correct:

• A conjunction splits; a disjunction does not. An and narrows with each condition, so the supported parts go to the engine and the grid narrows the superset that comes back. An or widens with each branch, so pushing only the supported branches would return fewer rows than the filter allows, and the grid cannot recover rows that were never fetched. An unsupported or stays whole on the client.
• A sort is pushed whole or not at all. Ordering by the first column and fixing the rest locally needs every row anyway.
• Residual work needs the complete result. When anything is left over, the grid asks for everything rather than a window. Filtering a window on the client is not a slower route to the right answer, it is a fast route to a wrong one: the rows belonging on page one may sit on page nine.

The adapters

Four adapters ship, and none of them carries an engine.

odataAdapter

Point it at an entity-set URL and it pushes a full condition tree, multi-column sort, windowing through $top and $skip, the count, and the quick filter through $search where the service supports it. Authenticate with a header, or a custom fetch for a token that refreshes. It follows an @odata.nextLink when the whole result is needed, up to a ceiling of two hundred pages so a filter matching far more than expected fails loudly rather than fetching forever.

restAdapter

For the endpoint you already have. Rename the parameters to whatever it calls them, and declare only the operators it truly applies, because claiming more returns wrong rows while claiming less only costs speed.

restAdapter({
  url: '/api/orders',
  // Rename the parameters to whatever your endpoint already calls them.
  params: { offset: 'skip', limit: 'take', sort: 'orderBy', order: 'dir' },
  // Declare only the comparisons the endpoint genuinely applies.
  operators: ['eq', 'gt', 'lt', 'contains'],
  rows:  (body) => body.results,        // pull the rows out of your shape
  total: (body) => body.meta.totalCount, // and the match count
})

duckdbAdapter

Hand it a DuckDB connection you made, including DuckDB-Wasm running in the browser over a Parquet file, and it answers the whole query: the filter tree, a multi-column sort and the page, with the total in the same round trip through count(*) OVER (). Values bind through prepared statements, and a BIGINT past the safe integer range is kept as a string rather than rounded into a plausible lie.

const source = createPushdownSource({
  adapter: duckdbAdapter({
    connection,   // a DuckDB connection you made; the grid imports no engine
    from: "read_parquet('data/readings.parquet')",
  }),
  compute,
});
// A million rows, queried in the tab, no back end. The adapter answers the
// whole query: the filter tree, a multi-column sort, and the page.

dfqlAdapter

For a DemandFlow entity, taking a personal access token and a tenant. An internal integration rather than a public one.

One thing to get right

The commonest mistake is total. It is the count of everything matching the filter, not the length of the page returned. The grid sizes its scrollbar from it, so returning the page length makes a large result look like a single page. If your adapter can report the real match count, declare total; if it cannot, leave it off and the grid manages without it.

Two live demos: the OData demo drives a public service and shows lastPlan() as you filter, and the DuckDB demo queries a million-row Parquet file in the browser with no server, printing the SQL the adapter generated. Pushdown pairs with derived grids, which compose grids in the browser once the rows arrive.