developer guide

Statistics and shadow columns

What the grid knows about its own numbers, and how they have moved since the page loaded.

Reductions in the totals row

Any numeric column can carry a statistical reduction in its total: total: 'median', 'p95', 'stddev', 'iqr' and more than twenty others. Every figure is computed over the filtered rows, and quantiles use the R type 7 definition a spreadsheet's PERCENTILE agrees with.

grid.statistics

The same figures, and the ones a column at a time cannot answer, are on grid.statistics: a full column profile, any single reduction, two-column figures like correlation and regression, and whole-series measures such as volatility and drawdown over a date-ordered column.

grid.statistics.profile('margin');
// { rows, present, missing, distinct, min, max, mean, median,
//   q1, q3, iqr, stddev, outliers, histogram }

grid.statistics.reduce('margin', 'p95');       // any registered kernel
grid.statistics.correlation('spend', 'revenue'); // Pearson's r
grid.statistics.regression('spend', 'revenue');  // { slope, intercept, r2, n }
grid.statistics.series('price', { by: 'date' }); // volatility, drawdown, ...

The statistics panel

The end-user half of profile() is a docked tool panel: a column picker, the twelve one-pass figures and a histogram of the column's shape, all following the filters. It also shows Shape (skewness, kurtosis, Jarque-Bera), Robust (trimmed and winsorized means, MAD), Concentration (Gini, HHI, entropy) and Capability where a column declares a spec. A section whose reductions all return null is left out rather than shown as a column of dashes.

toolPanel: { side: 'left', panels: ['columns', 'statistics'] }

Shadow columns

A shadow column is declared against another column and maintained by the grid. It has no field in the data, and it is not a computed column either, because its value depends on what happened before. It is a real column throughout: sortable, filterable, groupable, exportable, saveable into a view, which is what makes "every row repriced more than twice, most-changed first" one gesture.

columns: [
  { field: 'price', type: 'number' },
  { id: 'moved',  title: 'Change',  shadow: { of: 'price', kind: 'delta' } },
  { id: 'rank',   title: 'Rank',    shadow: { of: 'price', kind: 'rank' } },
  { id: 'movers', title: 'Places',  shadow: { of: 'price', kind: 'rankChange' } },
  { id: 'run',    title: 'Streak',  shadow: 'streak' }, // shorthand: the column beside it
]

The sixteen kinds

Kind	Value
`updates`	How many times the value has changed. Arrival is not a change.
`updatedAt`	When it last changed, as a Date.
`sinceUpdate`	Milliseconds since it last changed.
`delta`	Current value minus the baseline.
`deltaPercent`	The same as a percentage; a change from nothing reads null.
`rate`	Change per second, from the last two readings.
`history`	The recent readings, oldest first; `depth` sets how many.
`firstValue`	The baseline itself.
`streak`	Consecutive moves in one direction, signed; it resets on a turn.
`rank`	Competition rank, largest first; ties share the better rank.
`rankAsc`	The same ranking from the other end.
`rankChange`	Places climbed since the baseline; the "top movers" column.
`percentile`	The share of rows at or below this one, 0 to 100.
`quartile`	1 to 4, agreeing with percentile.
`zScore`	Deviations from the mean; no spread reads null.
`shareOfTotal`	The value over the column's total, as a percentage.

Shadow columns follow the data live. As values arrive or change - a streaming price, an edit, a new row - the grid keeps every shadow current, so the change count climbs, the delta moves and the rank re-sorts in front of the user without a redraw of your own. A live trading book shows the whole set moving at once.

Positional kinds rank over every tracked row, not the filtered set, so "the top ten movers" does not depend on what happens to be on screen; pass scope: 'filtered' to rank within the filters instead. grid.statistics.rebase(col) is "mark all": it moves every baseline to now, so the deltas and the rank change start again from the current values.

Process capability

Declare a tolerance on a column and the capability figures, a control chart and any out-of-tolerance rule all read the same limits. Cp and Cpk use short-term variation; Pp and Ppk use the overall deviation, and the gap between them is the point: Cpk well above Ppk means the process drifted.

columns: [{ field: 'mm', type: 'number',
  spec: { lower: 9.5, upper: 10.8, target: 10 } }]

grid.statistics.capability('mm');
// { cp, cpk, pp, ppk, sigmaWithin, sigmaOverall, outOfSpec, defectRate, violations }

Confidence intervals

Every figure the grid reports is an estimate from the rows in front of it. interval() says how tightly that estimate is pinned down: the range a mean is likely to sit in, with its margin of error, rather than a single number read to more decimal places than the data earns. Ask for a proportion instead with kind: 'proportion', and process capability can report an interval on Cpk too, which is what turns "the process is capable" into "capable, on this many parts".

grid.statistics.interval('margin');
// { mean, lower, upper, margin, n, confidence }  a 95% interval by default

grid.statistics.interval('passed', { kind: 'proportion' });
// { proportion, lower, upper, n, confidence }

grid.statistics.interval('margin', { confidence: 0.99 }); // a wider, surer interval

It reads the rows the filters leave, so the interval narrows as the grid narrows: it describes the population on screen, not the whole table. It is a range, not a verdict; the grid states how well the figure is known and leaves the judgement to you.

Distribution-driven formatting

The same knowledge drives conditional formatting that describes the data rather than a threshold you typed: an { op: 'outlier' } rule flags the statistical extremes, and a colour scale with from: 'quantile' spans a percentile range so a single large value does not flatten it. See the data-driven rules and quantile scales demos.

See the statistics overview and its live demos.