Live Data Grid Updates at High Rates

grid.updates.pause();
grid.updates.resume();          // apply everything held
grid.updates.flush();           // apply what is waiting, stay paused

grid.updates.stats();
// { paused, pending, queued, coalesced, coalescedTotal, rows, dropped, flushes, span }

grid.updates.log({ since: Date.now() - 60000 });   // what arrived, in order

Pausing is a button, not a guess. Inferring it from whether the user looks busy sounds friendlier and is wrong in both directions: too broad and a mouse resting on the grid freezes the feed until someone reloads, too narrow and rows jump the instant somebody stops moving in order to read. An explicit control has no heuristic to get wrong and no invisible state to explain.

Merging continues while paused, so a long pause costs one entry per changed row rather than one per update. Forty updates to one row are one row of work when play is pressed, and coalesced is the thirty-nine, which is the number nobody could see before.

Bounding the rows themselves is separate. The log bounds change history; a streaming source also needs to bound row retention, or a grid left up overnight holds every row it was ever sent. Set source.maxRows and the stream becomes a sliding window, dropping the oldest as new ones arrive and reporting how many it let go through evicted on the progress report.

const grid = createGrid(el, {
  columns,
  source: {
    mode: 'stream',
    open,
    maxAge: 5 * 60 * 1000,   // keep five minutes of rows
    ageBy: 'ts',             // ...aged by this column; omit for arrival time
    maxRows: 20000,          // ...and never more than this many, whichever bites first
  },
});

createChart({
  grid, container, type: 'line', x: 'ts', y: { col: 'value', fn: 'avg' },
  // The x domain is the last five minutes ending *now*, so the chart
  // keeps scrolling left even while the feed is silent.
  axis: { x: { window: { kind: 'time', span: 5 * 60 * 1000 } } },
});

A count window drifts, and it freezes. maxRows equals “the last five minutes” only while the feed rate is steady: a burst silently shrinks the window to two minutes, a quiet spell stretches it to twenty, and the x axis changes span under the reader. Worse, when the feed goes quiet nothing is evicted and the chart stops moving, even though time is still passing - and the silence is usually the thing worth seeing. maxAge is a span of wall clock, so it means the same thing whatever the feed is doing.

An empty reduction is a gap, not a zero. fn: 'avg' above - and sum, mean, min, max, first and last beside it - read as null when a bucket carries no rows to reduce, so a producer that has stopped sending is drawn as a break in the line rather than a value dropping to zero, which would read as a real observation nobody made. count and countValues are the deliberate exception: they are already honest at zero, a tally of rows or of values actually present, so reach for countValues when the reading you actually want is “how many arrived” and zero has to be drawn as zero rather than as a gap.

Two bounds, one eviction path. maxAge and maxRows are independent and compose: both are applied on the same pass and whichever bites first is simply the one that drops rows. Neither is silently ignored. Age eviction reuses the count bound’s machinery outright, so evicted on the progress report and the stream:evicted event carry age evictions exactly as they always carried count evictions - an existing “dropped off the back of the window” readout keeps working with nothing changed.

The span is a bound, not a guillotine. A row lives a little past the span before it goes, and two things add to that. First the eviction slack, ten per cent of the span - exactly the overshoot maxRows already allows on its count - so the row permutation is rebuilt once per block rather than once per arriving row. Second, when the feed is idle, up to one tick of the eviction timer, which runs at a quarter of the span clamped to between 50 ms and one second. The real ceiling is therefore about span × 1.1 + tick, and because the tick has a floor it is proportionally larger the shorter the window: about 10% over at a five-minute window, around 1.25× at ten seconds, and as much as ~1.35× at three. That is the deliberate price of an idle grid that costs no CPU, and it is why the number to reach for is the window you want the reader to see rather than a hard retention limit. The chart’s domain is exact either way - it ends at now - so the extra rows sit off the left edge rather than being drawn.

Which clock: ageBy, or arrival. Given ageBy - a column id, a dotted path, or a function of the row returning a Date, epoch milliseconds or an ISO string - the window follows the data’s own clock, so it means what the producer means. That also inherits the producer’s clock skew: if their clock runs five minutes fast, their rows live five minutes longer than yours. Omit ageBy and rows age from arrival time, when the row reached the source. Arrival time needs no timestamp column and cannot be skewed, but it is not event time - a row delayed in transit is treated as young. A synthetic or metric feed usually wants arrival; a log or event feed usually wants ageBy. A row whose time value cannot be read is never aged out: dropping data because a timestamp was malformed is the worse failure.

Out of order is handled, not reordered. With ageBy the row’s clock need not be monotonic in arrival order, so eviction scans the window rather than walking the head; a late row that is already older than the span is dropped on the same pass it arrived on and counted as evicted, rather than being painted and then withdrawn a moment later. Nothing is re-sorted: a row’s position is still arrival order, only its retention is decided by its time.

The chart axis rolls independently. The axis takes the same WindowSpec vocabulary rolling statistics use - window: { kind: 'time', span } - and its domain ends at now rather than at the newest point, which is what makes the chart keep scrolling with zero new rows. It works with or without maxAge on the source; set both to the same span and the retained data and the drawn domain agree. Only kind: 'time' applies to an axis: a count window over a chart is the source’s maxRows, and kind: 'count' is refused with a warning rather than quietly given a second meaning. The x column has to be continuous and carry wall-clock times - a banded or categorical axis has no domain to roll.

A producer whose clock runs ahead. A reading stamped slightly ahead of the viewer’s clock carries the end of the domain forward with it, so the newest mark is drawn. How far is bounded: a quarter of the span (15 s on a 60 s window). A reading further ahead than that is treated as a producer whose clock is wrong - it does not move the window, it is not drawn, and the chart warns once, naming the column, how many readings were left out and how far ahead they were. Without the bound, one device two minutes fast moved a one-minute window past every other device’s recent readings and the chart drew a single dot. If every reading in the window is that far ahead the chart shows its empty state, with the same warning. chart.data().windowed counts the readings dropped at either edge of the window. The fix for the warning is the producer’s clock, not a wider window.

The other edge: a producer that has simply stopped. The case above is a clock running fast; the opposite is a feed that has gone quiet for longer than the window - every reading is older than the span, so every mark would fall to the left of the domain, off the plot, while the axes and legend keep drawing as if the chart were healthy. Rather than draw that, the chart shows its empty state and warns once per chart instance, naming the span and how old the newest reading actually is, so a dead feed reads as “no data” rather than as a chart that quietly stopped moving. Two charts bound to the same stale column each get their own warning - the key is scoped to the chart, not just the column, so a dashboard of tiled charts sharing one timestamp column does not lose the second warning to the first.

Idle costs nothing. Both halves advance on a plain interval - a quarter of the window, clamped to between 50 ms and one second - and never on an animation frame. The source’s wake returns after a single number comparison unless a row is actually due, and the chart’s wake does nothing at all when the document is hidden or the chart is detached. Both timers are cleared on destroy. Measured over a five-minute window holding 50,000 rows with no feed at all, the window’s CPU cost is inside the run-to-run noise of the same source with no bound set: under 0.04% of one core (bench/idle-window.mjs).

The log keeps the raw sequence, not the merged one. Merging is right for applying a backlog quickly and wrong for looking at what happened, because the intermediate states are exactly what a time scrubber would move between. It survives the flush, pending is what is waiting, the log is what happened, and it is capped, so a grid paused over lunch holds the recent past and reports how much it dropped rather than taking the tab with it.