Elasticsearch and OpenSearch in a JavaScript Data Grid

Reading an index live

elasticsearchAdapter({ url, index }) speaks POST {index}/_search and _count directly, with no client library and no dependency. The index (a name, an alias, or a pattern such as logs-*) is read once for its field mapping unless you supply one yourself. Your key travels in headers or a fetch wrapper; the adapter stores no credential of its own.

import { createGrid, createPushdownSource, elasticsearchAdapter } from '@toclocoinc/lattice-grid';
import * as compute from '@toclocoinc/lattice-grid';

const grid = createGrid(el, {
  columns,
  rowKey: 'event_id',
  source: createPushdownSource({
    adapter: elasticsearchAdapter({
      url: 'https://es.example.com:9200',            // or a proxy of your own in front of it
      index: 'logs-*',
      headers: { Authorization: 'ApiKey ' + key },    // or supply your own fetch for a key that expires
    }),
    compute,
    pageSize: 100,
  }),
});
// A filter becomes a Query DSL bool query, sent as JSON; a sort adds
// missing-value placement, and paging is from/size until the index's own
// result window, then search_after over a point in time.

See it run against a mock of that API, since there is no public, keyless Elasticsearch or OpenSearch cluster this site could publish: reading Elasticsearch live, with the generated Query DSL on screen beside the grid.

What is pushed into the cluster

Every value is sent as JSON inside the query, never built into a query string, so nothing the grid sends can be interpreted as part of the query syntax itself:

term / terms   eq, ne, in, notIn
wildcard       contains, startsWith, endsWith (case_insensitive unless caseSensitive is set)
range          the comparisons, between, notBetween, and time filters
exists         blank, notBlank
must_not       every negated condition

Text folds case with case_insensitive the way the grid's own filter does, unless caseSensitive is set on the condition. Anything the adapter cannot translate stays with the grid and is named in source.lastPlan(), the same split every pushdown adapter reports.

The .keyword rule

Elasticsearch and OpenSearch do not sort or aggregate an analysed text field directly: the adapter reads the index mapping once and compares, sorts and groups a text field through its own keyword sub-field instead. A text field with no keyword sibling refuses a sort or an aggregate on it by name, with the fix named in the refusal, before a request ever reaches the cluster, never a bare 400 from the engine.

// A text field is compared, sorted and grouped through its own keyword
// sub-field, read once from the index mapping:
GET logs-*/_mapping   // -> { message: { type: 'text', fields: { keyword: { type: 'keyword' } } } }

// A text field with no keyword sibling refuses a sort or an aggregate on it
// by name, before anything is sent:
// "message has no .keyword sub-field to sort or aggregate on. Add a keyword
//  sub-field in the index mapping, or read a field that already has one."

Skip the mapping read entirely by declaring the field types yourself:

// Skip the _mapping read: supply the field types directly.
elasticsearchAdapter({
  url, index: 'logs-*',
  mapping: { status: 'keyword', message: 'text', 'message.keyword': 'keyword', bytes: 'long' },
});

Paging past the result window

A page inside the index's own result window (maxResultWindow, 10,000 by default) is one _search with from/size and an exact track_total_hits count in the same round trip. Past the window, the adapter counts with _count and walks with search_after over a point in time, closed as soon as the grid asks a different question rather than left open.

Grouped aggregates in the cluster

source.aggregate() computes sum, average, minimum, maximum, count and an approximate distinct count (cardinality, named as an estimate past 40,000 values) inside the cluster, grouped through nested terms aggregations rather than pulling rows back to reduce in the browser. A grouping level asking for more than maxBuckets groups is refused rather than returned short.

const result = await source.aggregate(
  { filters: currentFilters, sort: [], range: null, groupBy: ['service'] },
  [{ id: 'errors', col: 'status', fn: 'count' }, { id: 'p99', col: 'duration', fn: 'max' }],
);
// sum, avg, min, max, count and cardinality (an estimate past 40,000 distinct
// values) all run in the cluster, grouped through nested terms aggregations.
// A level asking for more than maxBuckets groups is refused rather than
// returned short.

Where the credential lives

Production: a proxy in front of the cluster (recommended). Your application authenticates the visitor its own way; a small backend holds the cluster API key, forwards only the search endpoints for one index pattern, and adds Access-Control-Allow-* for your own origin. The grid points at the proxy and never sees a key at all, the same shape the Splunk guide's reference proxy uses:

import { createServer } from 'node:http';

const TARGET = 'https://es.internal:9200';
const KEY = process.env.ES_API_KEY;
const ALLOWED = /^\/logs-\*\/(_search|_count)$/;

createServer(async (req, res) => {
  if (!ALLOWED.test(req.url)) { res.writeHead(404).end(); return; }
  res.setHeader('Access-Control-Allow-Origin', 'https://app.example.com');
  res.setHeader('Access-Control-Allow-Methods', 'POST');
  const chunks = [];
  for await (const chunk of req) chunks.push(chunk);
  const upstream = await fetch(TARGET + req.url, {
    method: 'POST',
    headers: { Authorization: 'ApiKey ' + KEY, 'Content-Type': 'application/json' },
    body: Buffer.concat(chunks),
  });
  res.writeHead(upstream.status);
  res.end(Buffer.from(await upstream.arrayBuffer()));
}).listen(8080);
// No key reaches the browser: the proxy forwards only _search and _count for
// one index pattern and adds the credential itself.

Trying it out: your own key in the page. Fine for a developer pointed at a development cluster. Never ship a shared key to visitors: a key grants everything its role can do, across every index it can see, to anyone who reads the page. The adapter holds no credential of its own by design: a key or a fetch wrapper travels only where the host application puts it.

OpenSearch compatibility

engine: 'opensearch' selects OpenSearch's own point-in-time endpoints for paging past the result window; everything else, the Query DSL, the .keyword rule, the aggregations, is identical between the two engines. OpenSearch needs an explicit tiebreaker field (one unique per document) to walk with search_after; Elasticsearch does not, and uses _shard_doc when none is given.

elasticsearchAdapter({
  url: 'https://opensearch.example.com:9200',
  index: 'logs-*',
  engine: 'opensearch',   // its own point-in-time endpoints, used past the result window
  tiebreaker: 'event_id', // OpenSearch needs one to walk with search_after; Elasticsearch does not
});

CORS, reading the cluster directly

A page reading Elasticsearch or OpenSearch directly, rather than through a proxy, needs the cluster's own cross-origin setting to allow the page's origin (http.cors.enabled and http.cors.allow-origin in elasticsearch.yml, or the equivalent OpenSearch setting); a proxy in front of the cluster sidesteps this entirely, since the browser then talks to your own origin.

See a mock of the API queried live with the generated DSL on screen: reading Elasticsearch live. The connect your data page lists every source and adapter the grid ships, and the pushdown guide covers the split every adapter reports in full.