Splunk Data in a JavaScript Data Grid

Reading a search live

splunkAdapter speaks Splunk's own search REST API: it dispatches a search job for the base SPL you give it, turns the grid's filter into an SPL search expression, pages through the job's results, and cancels the job the moment the grid asks something else. A filter on the time column becomes the job's own earliest_time/latest_time window instead of widening the search. Your token travels in headers or in a fetch wrapper; the grid stores no credential of its own.

import { createGrid, createPushdownSource, splunkAdapter } from '@toclocoinc/lattice-grid';
import * as compute from '@toclocoinc/lattice-grid';

const grid = createGrid(el, {
  columns,
  rowKey: 'event_id',
  source: createPushdownSource({
    adapter: splunkAdapter({
      url: 'https://splunk.example.com:8089',        // the management endpoint, not Splunk Web
      search: 'index=security sourcetype=firewall',
      earliest: '-24h',
      headers: { Authorization: 'Bearer ' + token },  // or supply your own fetch for a token that expires
    }),
    compute,
    pageSize: 100,
  }),
});
// A filter or a sort becomes SPL; a filter on the time column narrows the
// search's own earliest_time/latest_time window instead of widening the query.

See it run against a mock of that API, since there is no public Splunk instance or credential this site could publish: reading Splunk live.

Export mode, capped

mode: 'export' reads Splunk's streaming export endpoint to the end instead of paging a job, useful for a long search whose rows you want as they are produced. It has no offset and no count of its own, so set a ceiling with maxResultRows or the stream reads an unbounded search all the way through. A capped stream stops there and raises source.splunk.truncated by name, the same warning a capped blocking search already raises, so a reader knows the count in front of them is a floor, not the whole answer.

const capped = splunkAdapter({
  url: 'https://splunk.example.com:8089',
  search: 'index=security',
  mode: 'export',       // streams the export endpoint to the end, no offset, no count
  maxResultRows: 50000, // stop the stream here instead of reading an unbounded search
  headers: { Authorization: 'Bearer ' + token },
});
// A capped stream raises source.splunk.truncated by name, the same key a
// capped blocking search already raises, so a reader knows the total is a floor.

A finished blocking-mode job is released the moment its last page is read, rather than left to sit on a search-artifact slot until Splunk's own TTL reclaims it, the failure mode that exhausts a read-only role's quota fastest. releaseJobs: false keeps a job open instead, if your instance wants that. A refusal that is not a permissions failure, an exhausted artifact quota or a malformed search, now carries Splunk's own explanation (messages[].text out of the response body) in both the thrown error and a named warning, rather than leaving only an HTTP status to act on.

Where the credential lives

Production: a proxy in front of Splunk (recommended). Your application authenticates the visitor its own way; a small backend holds the Splunk service token, forwards only the search endpoints (/services/search/v2/jobs, /results, /export and a DELETE to cancel a job), restricts index= to what that visitor may see, rate-limits, and adds Access-Control-Allow-* for your own origin. The grid points at the proxy and never sees a token at all:

const adapter = splunkAdapter({
  url: 'https://app.example.com/splunk', // your own backend, not Splunk
  search: 'index=security sourcetype=firewall',
  earliest: '-24h',
});
// No token in this page: the proxy behind that URL adds one before the
// request reaches Splunk.

A reference proxy for that shape, forwarding exactly those routes with a token from an environment variable and refusing anything else:

import { createServer } from 'node:http';

const TARGET = 'https://splunk.internal:8089';
const TOKEN = process.env.SPLUNK_TOKEN;
const YOUR_INDEX = 'security'; // whatever this deployment's users may see
const ALLOWED = /^\/services\/search\/v2\/jobs(\/[^/]+)?(\/(results|export))?$/;

createServer(async (req, res) => {
  const path = req.url.split('?')[0];
  if (!ALLOWED.test(path)) { res.writeHead(404).end(); return; }
  res.setHeader('Access-Control-Allow-Origin', 'https://app.example.com');
  res.setHeader('Access-Control-Allow-Methods', 'GET, POST, DELETE');
  const chunks = [];
  for await (const chunk of req) chunks.push(chunk);
  let body = Buffer.concat(chunks).toString();
  if (req.method === 'POST') body = body.replace(/index=\S+/, 'index=' + YOUR_INDEX);
  const upstream = await fetch(TARGET + req.url, {
    method: req.method,
    headers: { Authorization: 'Bearer ' + TOKEN },
    body: req.method === 'GET' ? undefined : body,
  });
  res.writeHead(upstream.status);
  res.end(Buffer.from(await upstream.arrayBuffer()));
}).listen(8080);

The same shape runs equally well as a small Lambda behind CloudFront, if that suits your infrastructure better than a standalone process.

Internal tools: a per-user session key. Behind your own login, each user signs into Splunk with their own account and the page holds their own short-lived session key in memory, in headers (Authorization: Splunk <key>) or in a fetch wrapper that refreshes it. What leaks if the page is compromised is that one visitor's own access, and it expires on its own. Splunk's management port must allow the page's origin, either through its own cross-origin sharing setting or by sitting behind the proxy above instead.

const key = await fetch(url + '/services/auth/login', {
  method: 'POST',
  body: 'username=' + user + '&password=' + pass + '&output_mode=json',
}).then((r) => r.json()).then((j) => j.sessionKey);

const adapter = splunkAdapter({
  url,
  search: 'index=security sourcetype=firewall',
  headers: { Authorization: 'Splunk ' + key },
});

Trying it out: your own token in the page. Fine for a developer pointed at a development instance. Never ship a shared token to visitors: a token grants everything its role can do, across every index it can see, to anyone who reads the page. The first example on this page, with a bearer token in headers, is exactly this shape.

The adapter holds no credential of its own by design: a token or a session key travels only in the headers or fetch wrapper the host application supplies, in whichever of the three shapes above fits how that page is used.

Working from an export instead

Without direct REST access to the instance, the grid still works from exported search results: run the search you want, take the results out as CSV, JSON or NDJSON, and load them.

Exporting a search result is something Splunk already does well, in a shape the grid already reads, and it can be automated so nobody exports anything by hand.

Export a search as CSV, JSON, XML or raw

Three ways to get a search result out of Splunk in a shape the grid reads, all producing the same rows.

From the UI. Run the search, then Export from the results toolbar and choose CSV, XML, JSON or raw. CSV is the plainest match for a grid.

From the CLI. splunk search takes an -output flag naming the same formats:

splunk search "search index=security sourcetype=firewall earliest=-24h" \
  -output csv -maxout 0 > firewall-24h.csv

From the REST API. Create a search job, then read its results with output_mode set to csv, json, or one of Splunk's other output modes, over ordinary HTTPS:

https://<host>:<mport>/services/search/jobs/{search_id}/results?output_mode=csv

The REST route is the one worth automating: a scheduled job that lands a fresh export on a schedule, with no person exporting anything by hand.

Landing an export in the grid

A CSV export goes through grid.import, the same pipeline behind the grid's own "Import rows from CSV" affordance: parse, infer each column's type, preview the mapping, then apply.

import { createGrid } from '@toclocoinc/lattice-grid';

const grid = createGrid(el, { columns, rowKey: 'id', import: true });

// The CSV a Splunk export produced: from the UI's "Export Results", the CLI's
// -output csv, or the REST results endpoint below. grid.import infers each
// column's type and previews the mapping before anything lands.
const text = await file.text();
grid.import.apply(grid.import.preview(text));

A JSON or NDJSON export goes through createUrlSource, which reads a JSON array whole or streams an NDJSON file line by line, rendering the first rows before the rest has finished arriving:

import { createGrid, createUrlSource } from '@toclocoinc/lattice-grid';

// A REST results export saved as JSON, or fetched directly:
// GET /services/search/jobs/{search_id}/results?output_mode=json
const grid = createGrid(el, {
  columns, rowKey: 'id',
  source: createUrlSource('/exports/splunk-results.json'),
});

Routing events to S3, then querying them with DuckDB

Splunk's Ingest Actions can route event data to an S3 (or S3-compatible) destination as it arrives, as raw text, NDJSON or JSON. What lands there is ordinary files in a bucket you control, so the DuckDB adapter reads it directly, the same way it reads any other Parquet, CSV or NDJSON in S3:

import { createGrid, createPushdownSource, duckdbAdapter } from '@toclocoinc/lattice-grid';
import * as compute from '@toclocoinc/lattice-grid';

// Splunk's Ingest Actions can route event data to an S3 destination as raw
// text, NDJSON or JSON, entirely separate from its own indexed buckets. Point
// DuckDB straight at that landing prefix; the grid never touches Splunk.
const grid = createGrid(el, {
  columns,
  rowKey: 'event_id',
  source: createPushdownSource({
    adapter: duckdbAdapter({
      connection,
      from: "read_ndjson_auto('s3://your-bucket/splunk-events/*.json')",
    }),
    compute,
    pageSize: 200,
  }),
});

One thing to be clear about: this route works because Ingest Actions writes an open format to a destination you own. Pointing the grid at Splunk's own storage is not something it does.

See a lake queried the same way: bringing the Parquet security logs you already keep in S3 into the grid, and triage a day of security events live. The DuckDB adapter guide covers credentials, pushdown and whole-set statistics in full, and connect your data lists every source and adapter the grid ships.