Streaming source

Rows as they arrive

The streaming source is for data that comes in over time rather than in one response: a query that streams its result, a socket, or an async generator you write. Instead of waiting for the whole set, the grid renders rows as chunks arrive, so a reader sees the first rows almost immediately and the table fills in beneath them. You give it an open function that returns an iterator of chunks; the grid does the rest.

createGrid(el, {
  columns,
  rowKey: 'id',
  source: {
    mode: 'stream',
    // open returns an iterator of chunks; the grid awaits next()
    async *open({ sort, filters, quick, context, signal }) {
      for await (const chunk of query(filters, { signal })) {
        yield { rows: chunk };          // rendered as it arrives
      }
    },
    promoteToMemoryBelow: 250000,        // the default
  },
});

The rules that keep it smooth

This is the most performance-sensitive of the sources, and its behaviour is specific and deliberate:

• The first chunk renders immediately. Time to first row is the headline metric, so the first chunk is never deferred to an idle frame.
• One repaint per frame, never one per chunk. Chunks are applied as they arrive and the render is coalesced to the next animation frame, so a fast stream produces sixty repaints a second rather than one per message.
• Arriving rows are merged, not re-sorted. With a sort or filter active, each new row is binary-searched into the existing order and spliced in, rather than re-sorting the whole set on every chunk.
• Scroll position is preserved on append. The source reports how many rows landed above the viewport so the grid can compensate, instead of yanking the reader around as rows arrive above them.
• Backpressure is free. The grid awaits the iterator's next(), so a producer that wants genuine flow control gets it from the iterator protocol, and the source buffers to the next idle frame when chunks outrun the frame budget.

It promotes to memory when done

When the stream ends and the final count is under promoteToMemoryBelow, default 250,000, the source promotes itself to a memory source. The benefit is that once all the rows are here, subsequent sorts and filters are local and instant rather than re-opening the stream. You get progressive load while the data is arriving and a full in-memory grid once it has, with no switch to write yourself.

See it running: the streaming source, with rows landing as they are found. For the full set of options, start from connect your data.