Pulse

Pulse is a self-describing, high-performance tabular data processing engine. It ships as a Go library (github.com/frankbardon/pulse) and as a single CLI binary (bin/pulse). Every .pulse file carries its own schema in the header, so consumers (programs, agents, and humans) can discover what a file contains without an external catalog.

The library is the primary deliverable. The CLI is a thin adapter that exposes the same operations on the command line, and an embedded MCP server (pulse mcp) exposes them to LLM agents.

Where to go from here

LLM-facing surface

LLM agents do not read this site. Pulse exposes a Model Context Protocol server (pulse mcp) and ships 19 embedded skills under skills/ that LLMs load on demand via the pulse_skills_list and pulse_skills_get tools. The skill voice is MCP-only (tool calls, JSON payloads). This site is the human-facing counterpart — same engine, different idiom.

See How LLMs Use Pulse for a short pointer table.

Source of truth

The authoritative architectural contract for Pulse lives in the repository’s CLAUDE.md. When this site and CLAUDE.md disagree, CLAUDE.md wins; please open an issue.

• Repository: https://github.com/frankbardon/pulse
• Hosted docs: https://frankbardon.github.io/pulse/

Installation

Audience: new users who want a working pulse binary on their PATH.

This page walks through installing Pulse, the prerequisites it needs, and how to verify the install. Pulse is distributed as a single static Go binary; there is no installer, no daemon, and no config file.

LLM agents using MCP: see the session-bootstrap skill via pulse_skills_get — it covers session bootstrap rather than local install.

Prerequisites

go.mod is the source of truth for the supported Go version; if it drifts from this page the go.mod value wins.

Install with go install

The fastest path on a developer machine:

go install github.com/frankbardon/pulse/cmd/pulse@latest

This drops a pulse binary at $(go env GOBIN) (typically ~/go/bin). Make sure that directory is on your PATH.

Pin a specific release by replacing @latest with a tag:

go install github.com/frankbardon/pulse/cmd/pulse@v0.2.0

Build from source

The same binary, built reproducibly from a checkout:

git clone https://github.com/frankbardon/pulse.git
cd pulse
make build
# Binary at ./bin/pulse

The Makefile is documented in CLAUDE.md → Build / Env; the relevant targets are make build, make test, make lint, and make cover.

Configure the data directory

Pulse reads and writes .pulse files under a base directory called PULSE_DATA_DIR. Most commands accept absolute paths and will work without it, but pulse mcp requires the variable so the MCP server can enumerate cohorts:

export PULSE_DATA_DIR=/var/data/pulse

The repo Makefile auto-loads a .env file from the repo root, so you can also drop PULSE_DATA_DIR=... there for local development.

PULSE_DATA_DIR is the only required environment variable. See Flag Reference for the full list of CLI flags and environment knobs.

Verify

pulse --version
pulse --json | head -20

pulse --json prints the root manifest — the full self-description of commands, components, field types, and embedded skills. If you see a top-level format_version: "1.0" envelope, the install is working.

Where to go next

• New to the file format and vocabulary? Your First Cohort
• Want a quick map of every command? CLI Tour
• Embedding Pulse in a Go program? Go API Overview
• Wiring Pulse into an MCP-aware client? pulse mcp

Your First Cohort

Audience: new CLI users. This is a five-minute tour: import a CSV, inspect the resulting .pulse file, run an aggregation, and export the result back.

LLM agents using MCP: the equivalent tour for an agent is the session-bootstrap skill, fetched via pulse_skills_get. That skill speaks in tool calls and JSON payloads; this page speaks in shell commands.

1. Pick a CSV

For this walkthrough we’ll assume a file called sales.csv with columns like:

order_id,region,product,units,revenue,sold_on
1,west,widget,3,29.97,2024-01-04
2,east,gadget,1,19.99,2024-01-04
3,west,widget,7,69.93,2024-01-05
...

Any CSV with a header row works. Pulse also imports TSV, NDJSON, JSON-array, Parquet, Arrow IPC, and Excel — see Flag Reference for per-format flags.

2. Import to a .pulse file

pulse import csv --input sales.csv --output sales.pulse

Pulse samples up to 500 rows by default to infer a schema (you can change that with --sample-rows). Each column gets a typed binary representation and, if it looks like a low-cardinality string, a categorical dictionary.

Want to control the schema explicitly? Generate a template, edit it, and re-import:

# Editable schema template
pulse import schema-template sales.csv > sales.schema.json

# Edit sales.schema.json — set types, add descriptions
# Then import with the schema
pulse import csv --input sales.csv --schema sales.schema.json --output sales.pulse

See Field Types for the type catalog and Dictionary Blocks for how categoricals are encoded.

3. Inspect

The .pulse file is fully self-describing. Read it back:

pulse cohort inspect sales.pulse

Output is a table of fields, their types, and the description string stored in the header. Add --json for the structured envelope, or --full-dict to print every categorical entry instead of truncating after 100.

pulse cohort inspect sales.pulse --json

The envelope is documented in pulse cohort inspect.

4. Validate a request before running it

Pulse separates validation from execution. Write a tiny request file:

{
  "cohort": {"filename": "sales.pulse"},
  "groups": [{"type": "GROUP_CATEGORY", "field": "region"}],
  "aggregations": [
    {"type": "AGG_COUNT", "field": "order_id", "label": "orders"},
    {"type": "AGG_SUM", "field": "revenue", "label": "total_revenue"}
  ]
}

Save it as request.json, then check whether it makes sense against the cohort’s schema:

pulse api predict --request request.json

You’ll see Valid: true, the schema’s field count, and any warnings (e.g., aggregating something numeric on a categorical field). Predict never reads record data, so it’s safe to iterate on a request without touching a multi-GB cohort.

See pulse api predict and the tool-predict skill for the full predict loop.

5. Execute

pulse api process --request request.json --json

The response is wrapped in the standard envelope (format_version, data, errors, warnings). data carries the result rows and a metadata block with total_rows and filtered_rows.

If your result is large, swap --json for --stream to receive rows as NDJSON, one line at a time — useful for pipelines that don’t want to buffer the whole result. See Streaming & ProcessStream for which request shapes actually stream end-to-end inside the engine vs which buffer.

6. Export

You’re done with the .pulse file? Export to whatever your downstream tool understands:

pulse export csv     --input sales.pulse --output sales.out.csv
pulse export parquet --input sales.pulse --output sales.out.parquet
pulse export excel   --input sales.pulse --output sales.out.xlsx

To skip the intermediate .pulse entirely and convert in one shot, use pulse convert source.csv target.parquet — see the top-level README for the full convert recipe.

What you didn’t see

• Compose: batch multiple requests in one call — pulse api compose.
• Sample / Facet: cheap read-only probes — api sample, api facet.
• Window / Feature / Test operators: pull from the skill pack (window-design, feature-engineering, statistical-testing) via pulse skills show <name>.

For a full map of the CLI, see the CLI Tour.

CLI Tour

Audience: anyone who wants a map of every pulse subcommand before diving into per-command details.

This page is a one-liner index of the CLI tree. Each row links to its detailed chapter where applicable; commands that are minor variants of each other (per-format import/export leaves, per-leaf shard maintenance commands) are listed compactly.

LLM agents using MCP: there is no equivalent skill — agents drive Pulse through MCP tools, not the CLI. Start at the session-bootstrap skill instead.

Top-level groups

pulse [--json] [--slim]
├── import      Tabular → .pulse (csv, tsv, ndjson, jsonarray, parquet, arrow, excel, auto)
├── export      .pulse  → tabular (same format set)
├── convert     Tabular → tabular, with .pulse as the transparent middle
├── cohort      Inspect or filter an existing .pulse file
├── api         Processing operations (process, process-chain, compose, predict, sample, facet)
├── shard       Build and maintain shard archives
├── synth       Generate synthetic cohorts (from-schema, from-profile)
├── profile     Capture a statistical profile of a cohort
├── skills      Read the embedded LLM skill pack
├── examples    Search and fetch the embedded runnable request library
├── errors      Look up an error code's message and recovery fixups
└── mcp         Run the Model Context Protocol server over stdio

Bare pulse --json prints the self-describing root manifest — commands, components, field types, and skill metadata in one envelope. Pass --slim to drop prose descriptions for size-sensitive clients.

End-to-end worked tour

A complete read-aggregate cycle, one command per stage:

# 1. Import a CSV into a managed .pulse handle (TTL sidecar tracked).
pulse import auto sales.csv --handle sales --ttl 30d

# 2. Read the cohort's schema — types, descriptions, dictionaries.
pulse cohort inspect sales.pulse

# 3. Validate a request against the schema without running it.
cat > req.json <<'EOF'
{
  "cohort": {"filename": "sales.pulse"},
  "groups":       [{"type": "GROUP_CATEGORY", "field": "region"}],
  "aggregations": [{"type": "AGG_SUM",        "field": "revenue", "label": "total"}]
}
EOF
pulse api predict --request req.json --json | jq '.data | {valid, streamable, defaults_applied}'

# 4. Execute the same request.
pulse api process --request req.json --json

api predict reads only the header and schema, so it stays cheap even on multi-GB cohorts; the streamable / streamable_reasons fields tell you whether the request will run through the single-pass streaming path or be buffered. See Performance Notes.

Pipeline order

Inside api process, operators run in a fixed sequence:

Load -> Features -> Filter -> Attributes -> Group -> Aggregate -> Windows -> Sort -> Output

Features run before filterers (so derived columns are addressable as filter, group, attribute, and window inputs); windows run after aggregation, on the post-aggregate row set.

API operations

The “processing facade” — these are the operations exposed via the Go library API and the MCP tool set.

api process-chain accepts a ChainRequest whose stages each see the previous stage’s rows as their input cohort. Mergeable-only at v1 — non-mergeable stages fail with PULSE_CHAIN_NOT_MERGEABLE so callers can fall back to per-stage api process calls.

Cohort lifecycle

Import / export / convert

pulse import <format> and pulse export <format> share the same flag shape per format (--input, --output, --schema for import). Supported formats today:

csv · tsv · ndjson · jsonarray · parquet · arrow · excel

Each format has a per-leaf command (e.g. pulse import csv). Run pulse import --help or pulse export --help for the full list.

pulse import auto SOURCE auto-detects the source format and converts into the managed .pulse pool under PULSE_IMPORTS_DIR, with a TTL sidecar (--ttl 7d by default, pin to opt out). Use pulse import list to see managed handles and pulse import drop HANDLE to remove one.

pulse convert SOURCE TARGET chains import + export with no intermediate file unless --keep-pulse PATH is passed. Format is auto-detected from extensions.

Shard archives

A .pulse path may resolve to either a single-file cohort or a shard archive (uncompressed Zip64 carrying one canonical _schema.pulse entry plus N standalone shard payloads). Every facade method (Process, Compose, Sample, Facet, Inspect, Predict, ProcessStream) operates transparently on the union of shards.

The archive.pulse#shard.pulse anchor opens one shard as a one-shard cohort. See Internals → Managing Shard Archives.

Synthetic data

Self-description & LLM surface

Cross-cutting flags

Most leaves accept --json (envelope output), --no-defaults (turn off smart operator-type inference — see api predict → Smart defaults), and --echo-request (include the normalized request on envelope.request). Full list: Flag Reference.

The single environment variable to know is PULSE_DATA_DIR — see Installation.

Analysis Guide

A reference to the analysis this instance offers, rendered from the guidance metadata Pulse ships with every operator: what each one is for, the questions it answers, when to use something else, and the terms it relies on.

• Operator catalog: one page per operator category.
• Reading your results: how to read each output field, and the counts beside every result.
• Glossary: plain-language definitions of the terms the guidance uses.
• Skills: every skill as an agent reads it.

Operator catalog

The operators this instance offers, one page per category. Each page opens with a summary table and then a detail block per operator.

• Aggregators (33)
• Attributes (16)
• Filterers (12)
• Groupers (9)
• Window operators (11)
• Feature operators (9)
• Statistical tests (26)
• Regressions (3)
• Matrix operators (8)
• Overlays (44)
• Synth distributions (15)

Aggregators

The aggregators this instance offers: what each is for, the questions it answers and when to reach for something else. Operators are sorted by name; each name links to its detail block below.

Operators

AGG_AVERAGE

Average of a numeric field, over all rows or per group.

Level: basic

Also known as: mean, arithmetic mean

Questions it answers:

• What is the average order value?
• What is the typical rating in each region?

Use cases by domain:

• survey: Mean satisfaction score per segment.
• ops: Average order value by sales channel.
• science: Mean measurement per treatment arm.

Assumptions:

• Missing values are skipped: the average is over rows that have a value. A group where no row has a value reads 0, not empty, so check n in the components before reporting it.

Use something else:

• AGG_MEDIAN when the field is skewed or has extreme values and you want the typical row.
• AGG_WEIGHTED_MEAN when rows carry weights.
• AGG_WELFORD when you also need the spread around the average.

Glossary: mean, median, outlier, skew

Skill: op-agg-average

AGG_CI_LOWER

Lower end of a confidence interval for the mean of a numeric field, showing how precisely the average is pinned down.

Level: intermediate

Questions it answers:

• How precisely do we know the average satisfaction in each segment?
• What is the lowest average order value the data are consistent with at 95% confidence?

Use cases by domain:

• survey: Error bars on a mean score per segment.
• ops: Range for average handling time per team.
• science: Interval for a mean measurement per arm.

Assumptions:

• Rows are independent draws, unweighted.
• Normal critical value (1.96 at 95%), not Student’s t, so small groups, and strongly skewed fields even well past 30 rows, get an interval that is too narrow.
• Empty (NaN) when fewer than two rows have a value.

Use something else:

• AGG_CI_UPPER when you want the upper end of the same interval.
• TEST_WELCH when you want to test whether two group means differ.
• AGG_PERCENTILE when you want the range individual rows fall in, not the mean.

Glossary: confidence-interval, mean, standard-error, sample-size, independence

Skill: op-agg-ci-lower

AGG_CI_UPPER

Upper end of a confidence interval for the mean of a numeric field, showing how precisely the average is pinned down.

Level: intermediate

Questions it answers:

• What is the highest average wait time the data are consistent with at 95% confidence?
• How wide is the uncertainty around each region’s mean score?

Use cases by domain:

• survey: Error bars on a mean score per segment.
• ops: Upper end of the 95% interval for average delivery time per carrier: the true average can still lie above it.
• science: Interval for a mean measurement per arm.

Assumptions:

• Rows are independent draws, unweighted.
• Normal critical value (1.96 at 95%), not Student’s t, so small groups, and strongly skewed fields even well past 30 rows, get an interval that is too narrow.
• Empty (NaN) when fewer than two rows have a value.

Use something else:

• AGG_CI_LOWER when you want the lower end of the same interval.
• TEST_WELCH when you want to test whether two group means differ.

Glossary: confidence-interval, mean, standard-error, sample-size, independence

Skill: op-agg-ci-upper

AGG_COUNT

Number of rows that have a value in a field, over all rows or per group.

Level: basic

Questions it answers:

• How many responses did each region return?
• How many orders were placed through each channel?

Use cases by domain:

• survey: Number of answered questionnaires per segment.
• ops: Order count per store per day.
• science: Number of recorded measurements per treatment arm.

Assumptions:

• Rows with a missing value in the field are not counted.

Use something else:

• AGG_NULL_COUNT when you want the rows where the field is empty.
• AGG_DISTINCT_COUNT when you want how many different values appear, not how many rows.
• GROUP_CATEGORY when you want a count for every value of the field: group by it, then count.
• AGG_FREQUENCY when you want how many rows hold one particular value, such as the Yes answers.

Glossary: missing-value, sample-size

Skill: op-agg-count

AGG_DISTINCT_COUNT

Number of different values a field takes, over all rows or per group.

Level: basic

Questions it answers:

• How many different customers ordered this month?
• Does the respondent ID appear once per row, or are there duplicates?

Use cases by domain:

• survey: Number of distinct respondents per wave.
• ops: Unique customers per store.
• science: Number of distinct subjects per site.

Assumptions:

• Missing values are not counted as a value.

Use something else:

• AGG_COUNT when you want how many rows there are, not how many different values.
• GROUP_CATEGORY when you want how many rows hold each value: group by the field, then count.
• AGG_SET_DISTINCT_VALUES when the field is a multi-select and each combination is the unit.

Glossary: missing-value

Skill: op-agg-distinct-count

AGG_DISTINCT_SUM

Total of a numeric field counting each key once, such as a respondent’s weight summed once per respondent.

Level: intermediate

Questions it answers:

• What is the weighted base when each respondent appears on several rows?
• What is the total contract value when each contract repeats on every line item?

Use cases by domain:

• survey: Weighted base of a segment in a cohort with one row per answer.
• ops: Total customer credit limit when each customer has many orders.

Assumptions:

• The value is the same on every row of a key: when it differs, the first value seen is kept.
• A row missing either the value or the key adds nothing.

Use something else:

• AGG_SUM when every row is its own unit and should be added.
• AGG_DISTINCT_COUNT when you want how many keys there are, not a total.

Glossary: missing-value, weighting

Skill: op-agg-distinct-sum

AGG_FREQUENCY

How many rows hold one chosen value of a field, such as the Yes answers, over all rows or per group.

Level: basic

Questions it answers:

• How many respondents answered Yes in each region?
• How many orders were returned each day?

Use cases by domain:

• survey: Number of Yes answers to a question per segment.
• ops: Orders with the Returned status per store per day.
• science: Positive results per treatment arm.

Assumptions:

• Missing values are skipped: they are neither counted nor part of the base the share in the components divides by.
• The value is matched as a row filter matches it: a category label, otherwise a number; a value no row holds counts 0, so check its spelling.
• The result equals keeping only that value with a row filter, then counting rows.

Use something else:

• AGG_MODE_COUNT when you want how many rows share the field’s most common value, whatever it is.
• GROUP_CATEGORY when you want a count for every value of the field: group by it, then count rows.
• FILTER_INCLUDE when you want rows holding any of several values: keep them, then count rows.
• AGG_SET_FREQUENCY when the field is a multi-select.

Glossary: missing-value

Skill: op-agg-frequency

AGG_KURTOSIS

How heavy a numeric field’s tails are next to a normal distribution: positive when extreme values are more common.

Level: advanced

Questions it answers:

• Do transaction amounts have more extreme values than a bell curve would give?
• Are response times prone to rare, very long waits?

Use cases by domain:

• ops: Spotting fields where rare extreme values dominate risk.
• science: Checking a measurement’s tails before trusting a mean-based test.

Assumptions:

• Missing values are skipped.
• Population excess kurtosis g2 (dividing by n, minus 3): a normal shape scores 0, and at small n it differs from the adjusted G2 most packages print.
• It is 0 when n is 0 or 1 or every value is the same.

Use something else:

• AGG_SKEWNESS when you want to know whether the field is lopsided.
• TEST_SHAPIRO_WILK when you want a test of whether the field follows a normal shape.

Glossary: kurtosis, normal-distribution, outlier

Skill: op-agg-kurtosis

AGG_MAE

Average size of a prediction’s miss in the actual’s own units, comparing an actual column with a prediction column row by row.

Level: intermediate

Also known as: mean absolute error

Questions it answers:

• On average, how many units off is the sales model on the held-out weeks?
• Is the forecast miss larger for some stores than for others?

Use cases by domain:

• survey: Average miss of a predicted score against the observed score, in scale points.
• ops: Average holdout miss of a demand forecast, in units.
• science: Average absolute error of a calibration model on the validation samples.

Assumptions:

• A row is used only when both the actual and the prediction have a value.
• Train and holdout rows are separated with a filter or a range grouper; the operator itself never splits rows.

Use something else:

• AGG_RMSE when large misses should count for more than small ones.
• AGG_MAPE when you want the miss as a share of the actual value.

Glossary: residual, overfitting, missing-value

Skill: op-agg-mae

AGG_MAPE

Average size of a prediction’s miss as a share of the actual value, comparing an actual column with a prediction column row by row.

Level: intermediate

Also known as: mean absolute percentage error

Questions it answers:

• On the held-out weeks, how far off is the sales model, as a share of actual sales?
• Is the forecast’s percentage miss larger in some regions than in others?

Use cases by domain:

• survey: Percentage miss of a model-predicted response rate against the observed rate per wave.
• ops: Holdout forecast accuracy of weekly sales, as a share of actual sales.
• science: Relative error of a calibration model on the validation samples.

Assumptions:

• A row is used only when both the actual and the prediction have a value; a row whose actual is exactly 0 is left out and counted in components n_zero_actual.
• The result is a fraction (0.12 means 12%), not a percentage.
• Train and holdout rows are separated with a filter or a range grouper; the operator itself never splits rows.

Use something else:

• AGG_MAE when actual values can be 0 or close to 0, where a share is undefined or explodes.
• AGG_RMSE when large misses should count for more than small ones.

Glossary: residual, overfitting, missing-value

Skill: op-agg-mape

AGG_MAX

Largest value of a numeric or date field, over all rows or per group.

Level: basic

Questions it answers:

• What is the latest order date in each region?
• Is any order value implausibly large?

Use cases by domain:

• survey: Highest household size reported, to catch typing errors.
• ops: Largest single order per sales channel.
• science: Peak reading per run.

Assumptions:

• Missing values are skipped; a group where no row has a value reads 0, not empty, so check n in the components before reporting it.
• It rests on one row, so a single bad record sets it.

Use something else:

• AGG_MIN when you want the smallest value.
• AGG_PERCENTILE when one extreme row would mislead and you want a high value that ignores it.
• AGG_RANGE when you want the distance between smallest and largest.

Glossary: outlier, percentile

Skill: op-agg-max

AGG_MEDIAN

Middle value of a numeric field once sorted: the typical row, not pulled by a few extreme values.

Level: basic

Questions it answers:

• What is the typical household income in each region?
• What is the typical delivery time when a few deliveries take weeks?

Use cases by domain:

• survey: Typical income per segment when a few incomes are very large.
• ops: Typical order value when a few bulk orders inflate the average.
• science: Typical response time per condition.

Assumptions:

• Missing values are skipped; a group where no row has a value reads 0, not empty, so check n in the components before reporting it.
• With an even number of values it is the halfway point between the two middle values.

Use something else:

• AGG_PERCENTILE when you want a value other than the middle, such as the 90th percentile.
• AGG_AVERAGE when the field is roughly symmetric and you want a figure that combines across groups.
• AGG_MODE when the field holds categories rather than numbers.

Glossary: median, mean, outlier, percentile, skew

Skill: op-agg-median

AGG_MIN

Smallest value of a numeric or date field, over all rows or per group.

Level: basic

Questions it answers:

• What is the earliest order date in each region?
• Are there negative or impossible values in the age field?

Use cases by domain:

• survey: Lowest age reported, to catch out-of-range answers.
• ops: First order date per customer segment.
• science: Lowest reading per sensor, to spot faulty sensors.

Assumptions:

• Missing values are skipped; a group where no row has a value reads 0, not empty, so check n in the components before reporting it.
• It rests on one row, so a single bad record sets it.

Use something else:

• AGG_MAX when you want the largest value.
• AGG_PERCENTILE when one extreme row would mislead and you want a low value that ignores it.
• AGG_RANGE when you want the distance between smallest and largest.

Glossary: outlier, percentile

Skill: op-agg-min

AGG_MODE

Most common value of a field, over all rows or per group.

Level: basic

Questions it answers:

• What is the most common answer to the preferred-brand question?
• Which product is ordered most often in each store?

Use cases by domain:

• survey: Most chosen option per question and segment.
• ops: Most frequent payment method per channel.
• science: Most common category of outcome per group.

Assumptions:

• Missing values are skipped; a group where no row has a value reads 0, not empty, so check n in the components before reporting it.
• When several values share the top count, the smallest wins (for categories, the first in the dictionary); tie_count in the components shows it happened.

Use something else:

• GROUP_CATEGORY when you want the count of every value, not only the top one: group by the field, then count.
• AGG_MEDIAN when the field is numeric and you want its middle value.
• AGG_FREQUENCY when you want how many rows hold one particular value, such as the Yes answers.
• AGG_SET_FREQUENCY when the field is a multi-select.

Glossary: median

Skill: op-agg-mode

AGG_MODE_COUNT

How many rows share a field’s most common value, over all rows or per group.

Level: basic

Questions it answers:

• How many respondents gave the most common answer?
• How many orders used the most popular payment method?
• Grouped by the same field, how many rows fall in each category?

Use cases by domain:

• survey: Size of the largest answer group per question and segment.
• ops: Orders carrying the most common status per day.
• science: Size of the most common outcome category per treatment arm.

Assumptions:

• Missing values are skipped, so only rows that have a value are counted.
• Grouped by the same field, each group holds one value, so the result is that group’s row count.
• When several values tie for most common the count is the same; the components name the smallest tied value as mode_value.

Use something else:

• GROUP_CATEGORY when you want a count for every value of the field: group by it, then count rows.
• capability:facet when you want per-value counts for several fields in one call.
• AGG_MODE when you want which value is most common, not how many rows hold it.
• AGG_FREQUENCY when you want how many rows hold one particular value, such as the Yes answers.
• AGG_DISTINCT_COUNT when you want how many different values there are.
• AGG_SET_FREQUENCY when the field is a multi-select.

Glossary: missing-value

Skill: op-agg-mode-count

AGG_NULL_COUNT

Number of rows where a field has no value, over all rows or per group.

Level: basic

Questions it answers:

• How many respondents skipped the income question?
• Which regions have the most orders missing a delivery date?

Use cases by domain:

• survey: Item non-response per question and per segment.
• ops: Records missing a required field, per data source.
• science: Missing measurements per site before an analysis.

Assumptions:

• Only a stored null counts: an empty selection in a multi-select field is a value, not a null.

Use something else:

• AGG_COUNT when you want the rows that do have a value.
• FILTER_NULL when you want to keep or drop the rows with no value.

Glossary: missing-value

Skill: op-agg-null-count

AGG_PERCENTILE

Value of a numeric field below which a chosen share of rows fall, such as the 90th percentile of delivery time.

Level: basic

Questions it answers:

• How long do the slowest 10% of deliveries take in each region?
• What income marks the top quarter of respondents?

Use cases by domain:

• survey: Quartiles of household income per segment.
• ops: 95th-percentile response time per service.
• science: Reference range (2.5th to 97.5th percentile) of a measurement.

Assumptions:

• Missing values are skipped.
• Linear interpolation between the two nearest sorted values, so the result may be a value no row holds.

Use something else:

• AGG_MEDIAN when you want the 50th percentile.
• ATTR_PERCENTILE when you want each row’s own percentile position.
• GROUP_QUANTILE when you want rows split into equal-sized bands.

Glossary: percentile, median, outlier

Skill: op-agg-percentile

AGG_RANGE

Distance between the smallest and largest value of a numeric field, over all rows or per group.

Level: basic

Questions it answers:

• How far apart are the cheapest and most expensive orders?
• How widely do delivery times span in each depot?

Use cases by domain:

• survey: Span of ages within each segment.
• ops: Spread of daily sales between the slowest and busiest store.
• science: Span of readings across repeat measurements.

Assumptions:

• Missing values are skipped; a group where no row has a value reads 0, not empty, so check n in the components before reporting it.
• It rests on the two most extreme rows, so one outlier can stretch it a lot.

Use something else:

• AGG_STDDEV when you want a spread measure that uses every row, not only the two extremes.
• AGG_PERCENTILE when you want the spread of the middle of the data, ignoring extremes.

Glossary: outlier, standard-deviation

Skill: op-agg-range

AGG_RATIO

Total of one field divided by the total of another, over all rows or per group, such as revenue per order.

Level: basic

Questions it answers:

• What is revenue per visit in each channel?
• What share of budgeted hours were actually worked per team?

Use cases by domain:

• survey: Completes per invitation sent, per wave.
• ops: Revenue per order, or cost per unit, by region.
• science: Events per person-year of follow-up per arm.

Assumptions:

• A row missing either field is left out of both totals.
• Empty (NaN) when the denominator total is 0.

Use something else:

• ATTR_FORMULA when you want the average of each row’s own ratio: compute it per row (guarding a 0 denominator), then average it.
• AGG_WEIGHTED_MEAN when you want an average where some rows count more than others.

Glossary: mean

Skill: op-agg-ratio

AGG_RMSE

Typical size of a prediction’s miss in the actual’s own units, with large misses counting more: the root of the average squared miss.

Level: intermediate

Also known as: root mean squared error

Questions it answers:

• How many units off is the sales model on the held-out weeks, with big misses weighing more?
• Which of two models misses less on the same rows?

Use cases by domain:

• survey: Error of a predicted satisfaction score against the observed score, in scale points.
• ops: Holdout error of a handling-time model, in minutes.
• science: Prediction error of a dose-response model on the validation subjects.

Assumptions:

• A row is used only when both the actual and the prediction have a value.
• Each miss is squared before averaging, so a few large misses can dominate it.
• Train and holdout rows are separated with a filter or a range grouper; the operator itself never splits rows.

Use something else:

• AGG_MAE when every unit of miss should count the same, so a few large misses do not dominate.
• AGG_MAPE when you want the miss as a share of the actual value.

Glossary: residual, overfitting, missing-value

Skill: op-agg-rmse

AGG_SET_CARDINALITY_AVG

For a multi-select field, the average number of options chosen per row.

Level: basic

Questions it answers:

• How many brands does a respondent recognise on average?
• How many features does a typical customer have turned on?

Use cases by domain:

• survey: Average number of answers to a select-all-that-apply question, per segment.
• ops: Average number of products per customer bundle.

Assumptions:

• Rows with an empty selection count as zero and pull the average down; null rows are skipped.

Use something else:

• AGG_SET_CARDINALITY_SUM when you want the total number of selections.
• AGG_SET_FREQUENCY when you want which options were chosen.

Glossary: mean, missing-value

Skill: op-agg-set-cardinality-avg

AGG_SET_CARDINALITY_SUM

For a multi-select field, the total number of options chosen across all rows.

Level: basic

Questions it answers:

• How many brand mentions did the survey collect in total?
• How many add-ons were sold across all orders?

Use cases by domain:

• survey: Total mentions for a select-all-that-apply question, as a base for share of mentions.
• ops: Total add-ons attached to orders per channel.

Assumptions:

• Null rows are skipped; an empty selection adds zero.

Use something else:

• AGG_SET_CARDINALITY_AVG when you want the number per row.
• AGG_SET_FREQUENCY when you want the count for each option.

Skill: op-agg-set-cardinality-sum

AGG_SET_DISTINCT_VALUES

For a multi-select field, how many different combinations of options appear.

Level: basic

Questions it answers:

• How many different combinations of channels do customers use?
• How many different combinations of brands were picked (an empty answer counts as one)?

Use cases by domain:

• survey: Variety of answer patterns to a select-all-that-apply question.
• ops: Number of distinct product bundles sold.

Assumptions:

• Each combination is one value: choosing A and B differs from choosing A alone.
• An empty selection counts as one combination; null rows are skipped.

Use something else:

• AGG_SET_FREQUENCY when you want the count of each single option.
• AGG_SET_UNION when you want every option chosen by anyone.
• GROUP_SET_VALUE when you want how common each combination is: a count alone cannot show whether one dominates.

Skill: op-agg-set-distinct-values

AGG_SET_FREQUENCY

For a multi-select field, how many rows chose each option, over all rows or per group.

Level: basic

Questions it answers:

• How many respondents selected each brand they have heard of?
• How many customers use each payment method?

Use cases by domain:

• survey: Counts per option of a select-all-that-apply question.
• ops: Customers per enabled feature.

Assumptions:

• A row with several options is counted once under each, so the counts can add up to more than the rows.

Use something else:

• GROUP_CATEGORY when the field holds one value per row: group by it, then count.
• AGG_SET_DISTINCT_VALUES when you want how many different combinations were chosen.

Skill: op-agg-set-frequency

AGG_SET_INTERSECTION

For a multi-select field, the options chosen by every row, over all rows or per group.

Level: basic

Questions it answers:

• Which brands did every respondent in this segment recognise?
• Which features does every customer on a plan use?

Use cases by domain:

• survey: Options recognised by the whole segment.
• ops: Features common to every account on a plan.

Assumptions:

• Null rows are skipped, but one row with an empty selection empties the result.

Use something else:

• AGG_SET_UNION when you want every option chosen by anyone.
• AGG_SET_FREQUENCY when you want how many rows chose each option.

Skill: op-agg-set-intersection

AGG_SET_UNION

For a multi-select field, every option chosen by at least one row, over all rows or per group.

Level: basic

Questions it answers:

• Which brands were mentioned by anyone in this segment?
• Which features does at least one customer in each plan use?

Use cases by domain:

• survey: Options that received any mention per segment.
• ops: Features used by anyone on each plan.

Assumptions:

• Null rows are skipped.

Use something else:

• AGG_SET_INTERSECTION when you want the options every row chose.
• AGG_SET_FREQUENCY when you want how many rows chose each option.

Skill: op-agg-set-union

AGG_SKEWNESS

How lopsided a numeric field is: positive when a long tail runs to high values, negative when it runs to low ones.

Level: intermediate

Questions it answers:

• Are incomes in each region bunched low with a few very high ones?
• Is the delivery-time distribution symmetric or dragged out by slow orders?

Use cases by domain:

• survey: Whether a rating scale piles up at one end per segment.
• ops: Whether order values have a long tail of large orders.
• science: Checking a measurement’s shape before choosing a test.

Assumptions:

• Missing values are skipped.
• Population moment coefficient g1 (dividing by n): at small n it runs smaller in size than the adjusted G1 most packages print.
• It is 0 when n is 0 or 1 or every value is the same.

Use something else:

• AGG_KURTOSIS when you want to know whether the tails are heavy rather than lopsided.
• TEST_SHAPIRO_WILK when you want a test of whether the field follows a normal shape.
• AGG_MEDIAN when you want the typical value of a skewed field.

Glossary: skew, outlier, median, normal-distribution

Skill: op-agg-skewness

AGG_STDDEV

Typical distance of a numeric field’s values from their average, in the field’s own units, over all rows or per group.

Level: basic

Also known as: standard deviation, sd

Questions it answers:

• How much do delivery times vary around the average in each region?
• Which product line’s order values vary the most in absolute terms (divide by each line’s average to compare relative consistency)?

Use cases by domain:

• survey: How widely satisfaction scores spread within each segment.
• ops: Day-to-day variability of order volume per store.
• science: Spread of a measurement within each treatment arm.

Assumptions:

• Missing values are skipped.
• Population form: the squared distances are averaged over n, not n - 1.

Use something else:

• AGG_CI_LOWER when you want how precisely the average itself is known.
• AGG_WELFORD when you want the sample form (dividing by n - 1) together with the mean and n.
• AGG_PERCENTILE when the field has extreme values and you want a spread they cannot drag.

Glossary: standard-deviation, variance, mean, outlier

Skill: op-agg-stddev

AGG_SUM

Total of a numeric field, over all rows or per group.

Level: basic

Questions it answers:

• What is total revenue by region?
• How many units did each product line sell in total?

Use cases by domain:

• survey: Total of a per-respondent weight to get the weighted base of each segment.
• ops: Revenue per sales channel per month.
• science: Total dose delivered per treatment arm.

Assumptions:

• Missing values are skipped: they add nothing to the total.

Use something else:

• AGG_AVERAGE when you want the typical value per row rather than the total.
• AGG_WEIGHTED_MEAN when rows carry weights and you want a weighted average.
• AGG_DISTINCT_SUM when a value repeats on several rows of the same key and must be added once per key.
• WIN_RUNNING_SUM when you want the total so far, row by row.

Glossary: missing-value

Skill: op-agg-sum

AGG_VARIANCE

Spread of a numeric field as the average squared distance from its mean, in squared units, over all rows or per group.

Level: intermediate

Questions it answers:

• How much does the order value vary within each channel, in squared units?
• Which sites show the most variable readings?

Use cases by domain:

• ops: Variability of daily demand per product, for a safety-stock formula.
• science: Variance of a measurement across every unit of a fully measured batch.

Assumptions:

• Missing values are skipped.
• Population form: the squared distances are averaged over n, not n - 1.

Use something else:

• AGG_STDDEV when you want the spread in the field’s own units, which is easier to read.
• AGG_WELFORD when you want the sample variance (dividing by n - 1) with the mean and n, as pooling and study-size planning expect.

Glossary: variance, standard-deviation, mean

Skill: op-agg-variance

AGG_WEIGHTED_MEAN

Average of a numeric field where each row counts in proportion to a weight field, such as a survey weight.

Level: intermediate

Questions it answers:

• What is the weighted average satisfaction per region?
• What is the average price per unit when each order counts by its quantity?

Use cases by domain:

• survey: Population-weighted mean score per segment.
• ops: Volume-weighted average price per product.
• science: Precision-weighted mean of repeated measurements.

Assumptions:

• Rows missing the value or the weight, or with weight 0, are left out of the average; when no row is left (or the weights add up to 0) it reads 0, not empty, so check sum_weights in the components.
• Negative weights are not refused and can make the result unstable.

Use something else:

• AGG_AVERAGE when every row should count equally.
• AGG_RATIO when you want one total divided by another.

Glossary: weighting, mean, effective-sample-size

Skill: op-agg-weighted-mean

AGG_WELFORD

Mean, sample variance and row count of a numeric field in one result: what a comparison of group means needs.

Level: intermediate

Questions it answers:

• What are the mean, variance and size of each cell, so cells can be tested against each other?
• How do the average and spread of scores compare across segments?

Use cases by domain:

• survey: Per-cell mean score, variance and base feeding a t-test overlay on a crosstab.
• science: Per-arm summary statistics for a two-sample comparison.

Assumptions:

• Missing values are skipped; only plain integer and float fields are accepted.
• The variance divides by n - 1 and is 0 when fewer than two rows have a value.

Use something else:

• AGG_AVERAGE when you want only the average.
• AGG_STDDEV when you want the population spread (dividing by n) on its own.
• TEST_WELCH when you want the comparison itself on raw rows.

Glossary: mean, variance, sample-size, t-statistic

Skill: op-agg-welford

AGG_ZSCORE

Population mean and standard deviation of a numeric field, for standardizing it; the value itself is always 0.

Level: intermediate

Questions it answers:

• What centre and scale standardize this field in each group?
• How many standard deviations does the group’s last row (in row order) sit from the group’s average, that row included?

Use cases by domain:

• ops: How far the group’s last row, in a fixed row order, sits from the group’s centre.
• science: Group centre and scale for standardizing a measurement by hand.

Assumptions:

• Missing values are skipped; the spread is the population form (dividing by n).
• The last row is the last in row order, not the latest by date, and it is part of the mean and spread it is compared with, which damps its own score; for a reading against its past use OVERLAY_ZSCORE_VS_ROLLING.
• Not streamable: the whole group is read before the result is ready.

Use something else:

• ATTR_ZSCORE when you want a z-score on every row.
• OVERLAY_ZSCORE_VS_TOTAL when you want each group’s value against all groups.
• AGG_STDDEV when you want only the spread.

Glossary: z-score, mean, standard-deviation

Skill: op-agg-zscore

Attributes

The attributes this instance offers: what each is for, the questions it answers and when to reach for something else. Operators are sorted by name; each name links to its detail block below.

Operators

ATTR_CODE_IN

Adds a 1 / 0 column saying whether each row’s code is one of a listed set, keeping every row in the base.

Level: basic

Questions it answers:

• What share of all respondents gave a top-two-box answer (codes 4 or 5)?
• Which orders carry one of the priority status codes, as a column to average by region?

Use cases by domain:

• survey: Top-box flag to average as a share of the whole weighted base: non-answers stay in the denominator unless FILTER_NULL drops them first.
• ops: Flag for orders whose status code is in a chosen group.

Assumptions:

• On a categorical field a code is a dictionary label; one absent from the dictionary matches nothing.
• A row with a missing value reads 0, the same as a row with another code.

Use something else:

• FILTER_INCLUDE when you want to keep only the rows with those codes.
• AGG_FREQUENCY when you want how often a single value occurs.
• ATTR_SET_HAS when the field is a multi-select.

Glossary: missing-value

Skill: op-attr-code-in

ATTR_DATE_PART

Adds a calendar part of a date or timestamp to every row, such as the year, the month, a year-month like 202403 or the hour.

Level: basic

Questions it answers:

• Which month of the year does each order fall in, so seasons can be compared across years?
• What year-month label should each response carry for a monthly breakdown?

Use cases by domain:

• survey: Tag each response with its fieldwork year-month.
• ops: Month-of-year column for comparing seasonal order volume.
• science: Collection year per sample for a by-year breakdown.

Assumptions:

• Reads a date field (epoch days) or a datetime field; a datetime is read on the local clock of its time zone, and the hour part needs a datetime.
• The part is an encoded number (YYYYMM for year_month), so arithmetic on it is meaningless.
• A missing date reads 0.

Use something else:

• GROUP_DATE when you only want rows grouped by day, week, month or year.
• FEAT_DATE_FEATURES when you want several calendar columns at once, including weekday.

Glossary: missing-value

Skill: op-attr-date-part

ATTR_FACTOR_SCORE

Adds to every row its score on one factor behind several related numeric fields: an estimate of the hidden trait they measure together.

Level: advanced

Also known as: factor score, regression factor score, Thurstone score, Bartlett score, latent score

Questions it answers:

• What is each respondent’s score on the satisfaction factor behind these rating questions?
• Can I score every row on the two constructs a factor analysis found, to compare segments on them?

Use cases by domain:

• survey: Score each respondent on the constructs an attitude battery measures, then crosstab or test the scores by segment.
• science: Estimate each subject’s standing on a latent trait from several noisy indicators before a later model.

Assumptions:

• The factors come from one fit over every row that passed the filters (or the subset fit_on selects), never per group; every filtered row is scored.
• The fit is MAT_FACTOR’s with the same settings, so factor numbers and directions match its loadings: read them before naming a score.
• A row missing any of the fields is left out of the fit and gets no score (listwise deletion).
• There must be more complete rows than fields, the factor count must be identified, and no field may be constant or a sum of the others; otherwise every score is empty, with a warning.

Use something else:

• MAT_FACTOR when you want how many factors there are and which fields load on each.
• ATTR_PC_SCORE when you want a summary of all the fields’ variance rather than a common trait.

Glossary: factor, loading, correlation, z-score, listwise-deletion

Skill: op-attr-factor-score

ATTR_FORMULA

Adds a number to every row computed from the row’s own fields with an expression, such as price * qty.

Level: intermediate

Questions it answers:

• What is each order’s line total from its price and quantity?
• Which rows meet a combined rule, as a 0/1 column for counting?

Use cases by domain:

• survey: Sum of several rating items into one score per respondent.
• ops: Margin per order from revenue and cost.
• science: Body-mass index from recorded weight and height.

Assumptions:

• A missing field enters the expression as nil, and an operator that cannot take nil fails the request. A guard such as x ?? 0 turns each missing value into a real 0, which pulls averages down, so prefer dropping those rows with FILTER_NULL or pick a fallback that means something.
• True / false results become 1 / 0.
• It cannot read another attribute’s column from the same request.

Use something else:

• FILTER_EXPRESSION when you want to keep or drop rows by a rule rather than add a column.
• FEAT_LOG when you want the logarithm of a field.

Glossary: missing-value

Skill: op-attr-formula

ATTR_MAHALANOBIS

Adds to every row how unusual its mix of several numeric fields is, allowing for how those fields usually vary together.

Level: advanced

Also known as: mahalanobis distance, multivariate outlier score, d-squared

Questions it answers:

• Which respondents gave an unusual pattern of answers across the whole battery, even if no single answer is extreme?
• Which transactions have an odd combination of amount, frequency and basket size?

Use cases by domain:

• survey: Flag straight-liners and odd answer patterns across a rating battery before analysis.
• ops: Screen orders whose size, discount and delivery time together look unlike the rest.
• science: Find samples whose set of measurements is unusual as a whole.

Assumptions:

• The typical row and the spread come from one fit over every row that passed the filters (or the subset fit_on selects), never per group; every filtered row is scored.
• A row missing any of the fields is left out of the fit and gets no score (listwise deletion).
• The p output reads the score against a chi-square curve, which holds only when the fields are roughly jointly normal.
• There must be more complete rows than fields, and no field may be constant or a sum of the others; otherwise every score is empty, with a warning.

Use something else:

• ATTR_ZSCORE when only one field matters.
• TEST_HOTELLING_T2 when you want how far apart groups are, not single rows.

Glossary: distance, outlier, covariance, mean, listwise-deletion, chi-square, p-value, normal-distribution, degrees-of-freedom

Skill: op-attr-mahalanobis

ATTR_NORMALIZED

Rescales a numeric field to 0 to 1 on every row: 0 is the smallest value, 1 the largest.

Level: basic

Questions it answers:

• How can fields on very different scales be put on one 0-to-1 scale before combining them?
• Where does each row’s value sit between the lowest and highest seen?

Use cases by domain:

• survey: Rescale items to 0-1 by their observed lowest and highest answers, before comparing their spread.
• ops: Scale each site’s load between its observed minimum and maximum.
• harness: Bound an input to 0..1 before handing it to a scoring model.

Assumptions:

• Computed over every row that passed the filters, before grouping: a group never re-centres it.
• It uses the OBSERVED lowest and highest values, not a scale’s endpoints: if nobody answered 1 on a 1-5 item, 2 maps to 0, so the same raw score lands at different positions across items, filtered subsets or waves.
• A missing value reads 0, the same as the minimum, as does every row when all values are equal; in a composite that pulls the score down.

Use something else:

• ATTR_PERCENTILE when the field has extreme values that would squeeze the rest of the range.
• ATTR_ZSCORE when you want distance from the mean in standard deviations.
• ATTR_FORMULA when you want a composite on the scales’ fixed endpoints (1-5, 0-10), such as (x - 1) / 4.

Glossary: outlier, missing-value

Skill: op-attr-normalized

ATTR_PC_SCORE

Adds to every row its score on one principal component of several related numeric fields: one number standing in for what they share.

Level: advanced

Also known as: principal component score, pc score, component score, index score

Questions it answers:

• What is each respondent’s score on the main dimension behind these rating questions?
• Can I replace these five correlated measures with one summary column for a later model?

Use cases by domain:

• survey: Score each respondent on the leading dimension of an attitude battery, to segment or chart by it.
• science: Reduce many correlated measurements of each specimen to a few component columns before further analysis.

Assumptions:

• The components come from one fit over every row that passed the filters (or the subset fit_on selects), never per group; every filtered row is scored.
• A component’s direction (which end is high) follows MAT_PCA’s rule, not the meaning of the fields: read the loadings before naming a score.
• A row missing any of the fields is left out of the fit and gets no score (listwise deletion).
• There must be more complete rows than fields, and no field may be constant or a sum of the others; otherwise every score is empty, with a warning.

Use something else:

• MAT_PCA when you want how many dimensions there are and which fields load on each.
• ATTR_MAHALANOBIS when you want how unusual a row is across the fields.

Glossary: principal-component, eigenvalue, loading, correlation, covariance, z-score, listwise-deletion

Skill: op-attr-pc-score

ATTR_PERCENTILE

Adds to every row its percentile rank, rank / n * 100: the share of rows at or below it when values are untied.

Level: basic

Questions it answers:

• Where does each store’s revenue rank among all stores, as a percentage?
• Which respondents are in the top tenth for spend?

Use cases by domain:

• survey: Percentile of each respondent’s household income.
• ops: Percentile rank of each order’s delivery time.
• science: Percentile of each sample’s reading within the batch.

Assumptions:

• Computed over every row that passed the filters, before grouping: a group never re-centres it.
• Tied values do not share a percentile: each tied row takes its own rank in an arbitrary order, so equal values can read tens of points apart.
• Rows tied at a cut-off are split across it arbitrarily and can change between runs; cut on the value (AGG_PERCENTILE, then a filter) for a reproducible top tenth.
• Not streamable: every value is sorted before the first row is ready.

Use something else:

• AGG_PERCENTILE when you want the value at a chosen percentile, such as the 90th.
• WIN_RANK when you want ranks within partitions, or tied values to share a rank.
• GROUP_QUANTILE when you want rows split into equal-sized bands.

Glossary: percentile, rank, ties

Skill: op-attr-percentile

ATTR_REG_FITTED

Adds to every row the value a straight-line model predicts for its target from its predictors.

Level: advanced

Questions it answers:

• What order value would we expect for each customer given their visits and tenure?
• Which stores sell above or below what their size and footfall predict?

Use cases by domain:

• survey: Expected satisfaction per respondent from their attribute ratings.
• ops: Expected handling time per ticket from queue length and agent tenure.
• science: Expected response per subject from dose and body weight.

Assumptions:

• The model is fitted to the rows that passed the filters and have the target and every predictor (listwise deletion).
• Each ATTR_REG_* slot fits its own straight-line model; the request’s regressions are not reused.
• A prediction reflects association in these rows, never what would happen if a predictor were changed.
• A row missing any predictor reads 0, which is not a prediction.

Use something else:

• REG_OLS when you want the coefficients, R-squared and p-values of the model.
• ATTR_REG_RESIDUAL when you want how far each row sits from its prediction.
• REG_GLM when the target is yes/no or a count.

Glossary: regression-coefficient, overfitting, listwise-deletion

Skill: op-attr-reg-fitted

ATTR_REG_LEVERAGE

Adds to every row its leverage: how unusual its predictor values are, and so how hard it can pull a straight-line fit.

Level: advanced

Questions it answers:

• Which rows have predictor values so far from the rest that they could steer the model?
• Which customers have predictor values extreme enough that they could pull the fitted line (read their residuals to see whether they do)?

Use cases by domain:

• ops: Find the few very large accounts that could dominate a revenue model; read their residuals before concluding they do.
• science: Screen subjects with extreme dose or weight before trusting a fit.

Assumptions:

• The model is fitted to the rows that passed the filters and have the target and every predictor (listwise deletion).
• Each ATTR_REG_* slot fits its own straight-line model; the request’s regressions are not reused.
• Unpenalized least squares only: any penalty is refused.
• It looks at the predictors only, never at the target.

Use something else:

• ATTR_REG_RESIDUAL when you want rows the model predicts badly.
• REG_OLS when you want the model itself.

Glossary: outlier, residual, listwise-deletion

Skill: op-attr-reg-leverage

ATTR_REG_PREDICT

Adds to every row the value a model fitted once, elsewhere in the request, predicts for it from the row’s own predictor values.

Level: advanced

Questions it answers:

• How close are the model’s predictions on weeks it was not fitted to (a holdout)?
• How much of each week’s sales does the TV term of the fitted model account for?

Use cases by domain:

• survey: Score this wave’s respondents with the satisfaction model fitted on last wave.
• ops: Predict handling time for new tickets from a model fitted on last quarter’s.
• science: Predict the held-out subjects’ response from the dose model fitted on the rest.

Assumptions:

• The model is fitted once, on the fit request’s own rows, before any slot runs; scoring never refits.
• The scoring rows must carry every predictor column the fit used, built the same way (same features); an absent one-hot column is the reference level.
• A row missing any predictor the score reads gets no value (null), never 0.
• A prediction reflects association in the fitted rows, never what would happen if a predictor were changed.

Use something else:

• ATTR_REG_FITTED when the model should be fitted to these same rows inside the slot.
• REG_OLS when you want the coefficients, their standard errors and p-values.

Glossary: regression-coefficient, overfitting, link-function, missing-value

Skill: op-attr-reg-predict

ATTR_REG_RESIDUAL

Adds to every row its residual: the actual target value minus the value a straight-line model predicts for it.

Level: advanced

Questions it answers:

• Which stores sell far more or less than their size and footfall predict?
• Do the model’s misses grow with the predicted value, or bend in a curve?

Use cases by domain:

• survey: Respondents far more or less satisfied than their ratings predict.
• ops: Tickets that took far longer than queue length and tenure predict.
• science: Check a dose-response fit for curvature or growing spread.

Assumptions:

• The model is fitted to the rows that passed the filters and have the target and every predictor (listwise deletion).
• Each ATTR_REG_* slot fits its own straight-line model; the request’s regressions are not reused.
• A row missing the target or any predictor reads 0, the same as a row the model fits exactly.

Use something else:

• REG_OLS when you want the model’s summary (coefficients, R-squared).
• ATTR_REG_LEVERAGE when you want how unusual each row’s predictor values are.
• ATTR_ZSCORE when you want how far a value is from the mean, with no model.

Glossary: residual, heteroscedasticity, outlier, listwise-deletion

Skill: op-attr-reg-residual

ATTR_SET_HAS

Adds a 1 / 0 column saying whether each row’s multi-select field includes one named option.

Level: basic

Questions it answers:

• Which respondents ticked ‘price’ among their reasons, as a column to cross with other answers?
• Which tickets carry the ‘urgent’ tag?

Use cases by domain:

• survey: Flag for one option of a multiple-choice question, to average as a share among those who answered: drop missing answers with FILTER_NULL first, since they read 0.
• ops: Flag for orders tagged with a given promotion.

Assumptions:

• The option must be in the field’s dictionary; an unknown one is refused.
• A row with a missing value reads 0, the same as a row that did not choose the option.

Use something else:

• FILTER_SET_CONTAINS_ANY when you want to keep only rows that include any of several options.
• AGG_SET_FREQUENCY when you want how often each option was chosen.

Glossary: missing-value

Skill: op-attr-set-has

ATTR_SET_POPCOUNT

Adds to every row the number of options its multi-select field has selected.

Level: basic

Questions it answers:

• How many reasons did each respondent tick?
• Which customers selected three or more interests?

Use cases by domain:

• survey: Number of brands each respondent is aware of.
• ops: Number of tags on each support ticket.

Assumptions:

• An empty selection reads 0, and so does a row with a missing value.

Use something else:

• AGG_SET_CARDINALITY_AVG when you want the average number of selections per group.
• ATTR_SET_HAS when you want whether one particular option was chosen.

Glossary: missing-value

Skill: op-attr-set-popcount

ATTR_TSCORE

Adds to every row its T-score: the z-score rescaled so the mean is 50 and one standard deviation is 10.

Level: intermediate

Questions it answers:

• How does each candidate’s test result compare with the group, on a 50-centred scale?
• Which respondents score more than one standard deviation above average?

Use cases by domain:

• survey: Report standardized scale scores on a 50-centred scale, where negative values are rare.
• science: Express assessment scores on a mean-50, SD-10 scale relative to the rows analysed (not to a published norm group).

Assumptions:

• Computed over every row that passed the filters, before grouping: a group never re-centres it.
• Uses the population standard deviation (dividing by n).
• A missing value reads 50, as does every row when all values are equal.

Use something else:

• ATTR_ZSCORE when you want plain standard-deviation units centred on 0.
• ATTR_PERCENTILE when you want each row’s percentile rank (the share at or below it when values are untied).

Glossary: z-score, standard-deviation, mean

Skill: op-attr-tscore

ATTR_ZSCORE

Adds to every row its z-score: how many standard deviations the row’s value sits above or below the mean.

Level: intermediate

Also known as: standard score, standardized score

Questions it answers:

• Which orders are unusually large compared with all orders?
• How do scores measured on different scales compare once put on one footing?

Use cases by domain:

• survey: Put rating items with different spreads on a common scale before combining them.
• ops: Flag deliveries far slower than typical for review.
• science: Standardize a measurement before comparing it with another.

Assumptions:

• Computed over every row that passed the filters, before grouping: a group never re-centres it.
• Uses the population standard deviation (dividing by n).
• A missing value reads 0, as does every row when all values are equal.

Use something else:

• AGG_ZSCORE when you want the centre and spread of each group, not a score per row.
• ATTR_PERCENTILE when the field is skewed or has extreme values and you want a position they cannot distort.
• OVERLAY_ZSCORE_VS_TOTAL when you want each group’s value against all groups.

Glossary: z-score, standard-deviation, mean, outlier

Skill: op-attr-zscore

Filterers

The filterers this instance offers: what each is for, the questions it answers and when to reach for something else. Operators are sorted by name; each name links to its detail block below.

Operators

FILTER_DATE_RANGES

Keeps only the rows whose date falls inside one of a set of named date ranges, such as custom fiscal quarters.

Level: basic

Questions it answers:

• What happened during the two campaign periods only?
• What are the figures for the first half of our fiscal year?

Use cases by domain:

• survey: Keep interviews from the two fieldwork windows of this wave.
• ops: Keep orders placed during the holiday trading periods.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• Both ends of each range are inclusive; a range with no start or no end is open on that side.
• The ranges may not overlap; overlapping or duplicate ranges are refused, not merged.
• A date-and-time field is matched by the day it falls on.
• A row whose field is missing is dropped.

Use something else:

• GROUP_DATE_RANGES when you want each row labelled with its range instead of dropping the rest.
• GROUP_DATE when you want plain calendar periods such as months.
• FILTER_RANGE when the field is a number rather than a date.

Glossary: missing-value

Skill: op-filter-date-ranges

FILTER_EXCLUDE

Drops the rows whose field holds one of the listed values and keeps the rest, such as removing test accounts.

Level: basic

Questions it answers:

• What are the totals once internal test accounts are taken out?
• How do the results change without the store that was closed for refit?

Use cases by domain:

• survey: Drop the “Don’t know” answers before working out shares.
• ops: Drop orders flagged as internal or test.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• Several filters in one request all have to keep a row for it to stay: they combine as AND, never OR.
• A row whose field is missing is kept, since it does not hold a listed value.
• On a category field each listed value must be a known label, and an unknown label is an error. On a number or date field the values are read as numbers, and one that no row holds simply matches nothing.

Use something else:

• FILTER_INCLUDE when you want to keep only the listed values.
• FILTER_NULL when you want to drop the rows where the field is missing.
• FILTER_SET_CONTAINS_NONE when the field is a multi-select answer.

Glossary: missing-value

Skill: op-filter-exclude

FILTER_EXPRESSION

Keeps the rows for which a written yes-or-no condition is true, for rules the other filters cannot state, such as two fields together.

Level: intermediate

Questions it answers:

• Which customers spent over 500 and joined this year?
• Which records have an end date before their start date?

Use cases by domain:

• survey: Keep respondents who rated the brand 9 or 10 and bought in the last month.
• ops: Keep orders where the shipped quantity is below the ordered quantity.
• harness: Express a user’s free-form condition as one filter.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• The condition must give true or false for every row; any other result is an error.
• When a missing value makes the condition unknown, such as a missing number compared with 5, the row is dropped.
• A missing value compared with == is false and with != is true; write x ?? 0 to fill it first.

Use something else:

• FILTER_INCLUDE when you only need to keep a list of values.
• FILTER_RANGE when you only need a low and a high limit.
• FILTER_NULL when you only need rows where a field is or is not missing.

Glossary: missing-value

Skill: op-filter-expression

FILTER_FALSE

Keeps only the rows where a yes/no field is no, such as lapsed accounts or unfinished surveys.

Level: basic

Questions it answers:

• What do the customers who did not renew have in common?
• How many interviews were left unfinished, by region?

Use cases by domain:

• survey: Keep respondents who did not complete the survey.
• ops: Keep orders not yet shipped.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• By default the field must be a yes/no field; a truthy option treats 0, an empty text and a missing value as no. On a category field it tests the label text, so a label such as “No” is not no; use FILTER_INCLUDE on the no label instead.
• A missing value is dropped by default but kept under the truthy option, where it counts as no.

Use something else:

• FILTER_TRUE when you want the rows where the field is yes.
• FILTER_NULL when you want only the rows where the field is missing.

Glossary: missing-value

Skill: op-filter-false

FILTER_INCLUDE

Keeps only the rows whose field holds one of the listed values, such as two regions or three product codes.

Level: basic

Questions it answers:

• What do the figures look like for the North and South regions only?
• How did customers on the two premium plans answer?

Use cases by domain:

• survey: Keep only respondents from the three markets in this wave’s report.
• ops: Keep only orders from the stores in one district.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• Several filters in one request all have to keep a row for it to stay: they combine as AND, never OR.
• A row whose field is missing is dropped.
• On a category field each listed value must be a known label, and an unknown label is an error. On a number or date field the values are read as numbers, and one that no row holds simply matches nothing.

Use something else:

• FILTER_EXCLUDE when you want to drop the listed values and keep everything else.
• FILTER_RANGE when the field is numeric and you want everything between two limits.
• FILTER_SET_CONTAINS_ANY when the field is a multi-select answer.
• AGG_FREQUENCY when you only want how many rows hold one value, not the rows themselves.

Glossary: missing-value

Skill: op-filter-include

FILTER_NULL

Keeps either the rows where a field is missing or the rows where it is present, to find gaps or set them aside.

Level: basic

Questions it answers:

• Who skipped the income question, so their other answers can be compared with those who answered?
• What are the averages over only the rows that have a value?

Use cases by domain:

• survey: Look at who skipped a question to see whether they differ from those who answered.
• ops: Find orders with no delivery date recorded.
• science: Set aside samples with no measurement before comparing groups.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• Missing means no recorded value; a 0, an empty text or an empty multi-select answer is a value, not missing.
• Dropping rows with gaps can bias the result when the gaps are not random.

Use something else:

• AGG_NULL_COUNT when you only want to count the missing values, not filter on them.
• FILTER_EXCLUDE when you want to drop particular values, not missing ones.

Glossary: missing-value, listwise-deletion

Skill: op-filter-null

FILTER_RANGE

Keeps only the rows whose number lies between a low and a high limit, both limits included.

Level: basic

Questions it answers:

• What do adults aged 18 to 34 say?
• What is the average order once impossible values above 10,000 are left out?

Use cases by domain:

• survey: Keep respondents aged 18 to 34.
• ops: Keep delivery times between 0 and 72 hours, dropping clearly wrong entries.
• science: Keep readings inside the instrument’s working range.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• Several filters in one request all have to keep a row for it to stay: they combine as AND, never OR.
• Both limits are inclusive: a value equal to the low or the high limit is kept.
• On a date field the limits are day numbers counted from 1 January 1970, not written dates.
• A row whose field is missing is dropped.

Use something else:

• FILTER_EXPRESSION when you need an open limit, such as strictly above a value.
• FILTER_DATE_RANGES when the field is a date and you want named periods.
• GROUP_RANGE when you want to split the values into bands rather than keep one.

Glossary: outlier, missing-value

Skill: op-filter-range

FILTER_SET_CONTAINS_ALL

Keeps the rows whose multi-select answer includes every listed option, other options allowed, such as people who use both A and B.

Level: basic

Questions it answers:

• How do people who use both our app and our website rate us?
• Which tickets carry both the urgent and the billing tags?

Use cases by domain:

• survey: Keep respondents who bought both brands in the last month.
• ops: Keep orders that included both a phone and a case.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• Each listed value must be one of the field’s known labels; an unknown label is an error, not an empty match.
• An empty list of options keeps every row that has an answer.
• A row whose field is missing is dropped.

Use something else:

• FILTER_SET_CONTAINS_ANY when one of the listed options is enough.
• FILTER_SET_EQUALS when the row must hold exactly the listed options and nothing else.

Glossary: missing-value

Skill: op-filter-set-contains-all

FILTER_SET_CONTAINS_ANY

Keeps the rows whose multi-select answer includes at least one of the listed options, such as anyone who uses brand A or B.

Level: basic

Questions it answers:

• What do people who use either of our two apps think of the service?
• Which customers paid by any card at least once?

Use cases by domain:

• survey: Keep respondents aware of at least one of the two new brands.
• ops: Keep tickets tagged with any of the billing tags.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• Each listed value must be one of the field’s known labels; an unknown label is an error, not an empty match.
• An empty list of options keeps no rows.
• A row whose field is missing is dropped.

Use something else:

• FILTER_SET_CONTAINS_ALL when the row must include every listed option.
• FILTER_SET_CONTAINS_NONE when the row must include none of the listed options.
• FILTER_INCLUDE when the field holds one value per row, not several.

Glossary: missing-value

Skill: op-filter-set-contains-any

FILTER_SET_CONTAINS_NONE

Keeps the rows whose multi-select answer includes none of the listed options, such as people who use neither A nor B.

Level: basic

Questions it answers:

• What do people who use none of the competitor apps think?
• Which orders had no discount code applied?

Use cases by domain:

• survey: Keep respondents unaware of every competitor brand.
• ops: Keep tickets with none of the escalation tags.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• Each listed value must be one of the field’s known labels; an unknown label is an error, not an empty match.
• A row whose field is missing is kept, since it does not hold a listed value.

Use something else:

• FILTER_SET_CONTAINS_ANY when you want rows that include one of the options.
• FILTER_EXCLUDE when the field holds one value per row, not several.

Glossary: missing-value

Skill: op-filter-set-contains-none

FILTER_SET_EQUALS

Keeps the rows whose multi-select answer is exactly the listed options, no more and no fewer, such as people who use only A.

Level: basic

Questions it answers:

• How many customers use our app and nothing else?
• Which respondents picked exactly these two reasons?

Use cases by domain:

• survey: Keep the respondents loyal to one brand only.
• ops: Keep orders paid with exactly one card and no other method.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• Each listed value must be one of the field’s known labels; an unknown label is an error, not an empty match.
• An empty list keeps the rows that picked nothing, which differs from a missing answer.
• A row whose field is missing is dropped.

Use something else:

• FILTER_SET_CONTAINS_ALL when other options are allowed alongside the listed ones.
• GROUP_SET_VALUE when you want to see every combination with its count.

Glossary: missing-value

Skill: op-filter-set-equals

FILTER_TRUE

Keeps only the rows where a yes/no field is yes, such as active accounts or completed surveys.

Level: basic

Questions it answers:

• What do only the completed interviews say?
• What are the figures for active subscribers?

Use cases by domain:

• survey: Keep respondents who passed the attention check.
• ops: Keep orders flagged as delivered.

Assumptions:

• Runs before attributes, groups and aggregations, so every figure in the result comes from the rows it keeps.
• By default the field must be a yes/no field; a truthy option treats any non-zero number and any non-empty text as yes. On a category field it tests the label text, so a label such as “No” counts as yes; filter a Yes/No category with FILTER_INCLUDE on the yes label instead.
• A row whose field is missing is dropped.

Use something else:

• FILTER_FALSE when you want the rows where the field is no.
• FILTER_EXPRESSION when the condition involves more than one field.
• GROUP_CATEGORY when you want yes and no side by side as groups, not one of them.

Glossary: missing-value

Skill: op-filter-true

Groupers

The groupers this instance offers: what each is for, the questions it answers and when to reach for something else. Operators are sorted by name; each name links to its detail block below.

Operators

GROUP_CATEGORY

Splits the rows into one group per distinct value of a field, such as one group per region or per answer option.

Level: basic

Questions it answers:

• What is the average spend in each region?
• How many respondents chose each answer?

Use cases by domain:

• survey: Satisfaction score by age band or by brand.
• ops: Orders and revenue per store.
• science: Mean reading per treatment arm.

Assumptions:

• Buckets are built from the rows the filters kept; each aggregation is then worked out once per bucket.
• A row whose field is missing lands in no bucket; it is counted as missing, not as a bucket of its own.
• Groups appear in alphabetical order unless an include list names them, which also sets their order and drops the rest.
• A field with very many distinct values makes very many groups; narrow it with a filter first.

Use something else:

• GROUP_RANGE when the field is a number you want in bands, not one group per value.
• GROUP_DATE when the field is a date and you want months or quarters.
• GROUP_SET_PER_ELEMENT when the field is a multi-select answer.

Glossary: missing-value

Skill: op-group-category

GROUP_DATE

Splits the rows by calendar period of a date, such as month, quarter, ISO week or weekday, to see a measure over time.

Level: basic

Questions it answers:

• How many orders came in each month?
• Which day of the week gets the most complaints?

Use cases by domain:

• survey: Satisfaction by fieldwork month.
• ops: Weekly order volume, or revenue by fiscal quarter.
• science: Daily count of observations.

Assumptions:

• Buckets are built from the rows the filters kept; each aggregation is then worked out once per bucket.
• Named with no period it groups by month; picked by default for a date field with no grouper type, it groups by day.
• Weeks are ISO weeks, Monday to Sunday, numbered within the ISO year.
• A fiscal offset applies to years and quarters only; a fiscal year is named for the calendar year it ends in.
• A date-and-time field is grouped by the day it falls on.
• Only buckets that hold at least one row appear; an empty band or period is left out, not shown as zero.
• A row whose field is missing lands in no bucket; it is counted as missing, not as a bucket of its own.

Use something else:

• GROUP_DATE_RANGES when your periods are custom, such as campaign windows.
• OVERLAY_INDEX_VS_PRIOR when you want each period against the one before it.
• WIN_RUNNING_SUM when you want a running total down the periods.

Glossary: missing-value

Skill: op-group-date

GROUP_DATE_RANGES

Labels each row with the named date range its date falls in, such as custom fiscal quarters or before and after a launch.

Level: basic

Questions it answers:

• How did sales compare before, during and after the campaign?
• What are the totals for each of our custom fiscal quarters?

Use cases by domain:

• survey: Scores per fieldwork wave, where waves have irregular dates.
• ops: Revenue per promotion period.

Assumptions:

• Buckets are built from the rows the filters kept; each aggregation is then worked out once per bucket.
• Both ends of each range are inclusive; the ranges may not overlap.
• A row outside every range goes to an unmatched group (named “unmatched” unless you rename it), not dropped.
• A date-and-time field is matched by the day it falls on.
• A row whose field is missing lands in no bucket; it is counted as missing, not as a bucket of its own.

Use something else:

• GROUP_DATE when plain calendar months, quarters or weeks are enough.
• FILTER_DATE_RANGES when you want to keep only the rows inside the ranges.

Glossary: missing-value

Skill: op-group-date-ranges

GROUP_KMEANS

Finds k segments of rows that are alike across several numbers at once, such as customer types by spend, visits and tenure.

Level: advanced

Questions it answers:

• What natural customer segments do spend, visits and tenure suggest?
• Which respondent types stand out across the rating battery?

Use cases by domain:

• survey: Attitudinal segments from a battery of ratings.
• ops: Store or customer types from several operating figures.
• science: Groups of samples with similar measurement profiles.

Assumptions:

• Buckets are built from the rows the filters kept; each aggregation is then worked out once per bucket.
• Segments are found, not given: each row joins the nearest segment centre, and the centres are chosen to keep rows close to their own.
• Numbers are put on a common scale first by default, so no field dominates because of its units.
• Segments are named cluster_1 to cluster_k from largest to smallest and come back in that order; the same data and seed give the same names.
• It needs every row before it can name any segment, so it cannot stream.
• A row missing any of the numbers lands in no segment; it is counted as missing.

Use something else:

• GROUP_CATEGORY when you already know the groups.
• GROUP_QUANTILE when you want bands of one number.
• ATTR_MAHALANOBIS when you want how unusual each row is.

Glossary: centroid, z-score

Skill: op-group-kmeans

GROUP_QUANTILE

Splits the rows into equal-sized groups by rank of a number, such as quartiles or deciles of spend, to compare top and bottom.

Level: intermediate

Questions it answers:

• How do the top 25% of spenders differ from the bottom 25%?
• What is the average basket in each decile of customer value?

Use cases by domain:

• survey: Satisfaction by income quartile.
• ops: Return rate by decile of order value.
• science: Outcome by quartile of exposure.

Assumptions:

• Buckets are built from the rows the filters kept; each aggregation is then worked out once per bucket.
• Groups are cut by rank among the rows with a value, so each holds as close to the same count as possible; under a row weight each holds as close to the same total weight as possible, a heavy row is never split, and counts stay row counts.
• Equal values can fall in two neighbouring groups when a cut lands among them.
• Groups are named Q1 to Q4 for quartiles, D1 to D10 for deciles, P1 to P100 for percentiles and B1, B2 and so on otherwise; Q1 is the lowest.
• Groups come back in text order of their names, so deciles read D1, D10, D2, … D9; sort them by number before reading a trend.
• It needs every row before it can cut, so it cannot stream.
• A row whose field is missing lands in no bucket; it is counted as missing, not as a bucket of its own.

Use something else:

• GROUP_RANGE when you want bands of a fixed width, such as every 10 years of age.
• ATTR_PERCENTILE when you want each row’s rank as a number, not a group.
• AGG_PERCENTILE when you want the cut-off values themselves.

Glossary: percentile, rank, ties

Skill: op-group-quantile

GROUP_RANGE

Splits a number into bands of equal width, such as ages 20-30 and 30-40, to see how values spread or to compare bands.

Level: basic

Questions it answers:

• How many customers fall in each 10-year age band?
• What does the spread of delivery times look like in 1-hour bands?

Use cases by domain:

• survey: Respondents per age band.
• ops: Orders per 50-unit order-value band, as a histogram.
• science: Count of readings per band of temperature.

Assumptions:

• Buckets are built from the rows the filters kept; each aggregation is then worked out once per bucket.
• Each band is named low-high and holds values from its low edge up to, but not including, its high edge.
• Bands start at multiples of the width, counted from 0; with no width given it is 1, or 10 when picked by default for a numeric field.
• Rows come back in text order of the band names (100-150 before 50-100); sort by the low edge and add the missing bands before charting a histogram.
• Only buckets that hold at least one row appear; an empty band or period is left out, not shown as zero.
• A row whose field is missing lands in no bucket; it is counted as missing, not as a bucket of its own.

Use something else:

• GROUP_QUANTILE when you want groups of equal size rather than equal width.
• GROUP_ROUNDED when you want each band named by its lower edge alone.
• FILTER_RANGE when you want to keep one band, not split into all of them.

Glossary: missing-value

Skill: op-group-range

GROUP_ROUNDED

Groups a number by rounding it down to a multiple of a step, such as 23 and 27 both to 20, naming each group by that multiple.

Level: basic

Questions it answers:

• How many orders are there at each price point, to the nearest lower 10?
• What is the count of scores in each block of 5 points?

Use cases by domain:

• survey: Respondents per 5-point block of a 0-100 score.
• ops: Orders per 10-unit price step, keyed by the step’s start.
• science: Readings binned to a whole-number step.

Assumptions:

• Buckets are built from the rows the filters kept; each aggregation is then worked out once per bucket.
• Values are rounded down, not to the nearest: 19 with a step of 10 goes to 10, and -3 goes to -10.
• It makes the same groups as equal-width bands of the same step; only the names differ.
• Only buckets that hold at least one row appear; an empty band or period is left out, not shown as zero.
• A row whose field is missing lands in no bucket; it is counted as missing, not as a bucket of its own.

Use something else:

• GROUP_RANGE when you want the band’s both edges in its name.
• GROUP_QUANTILE when you want groups of equal size.

Glossary: missing-value

Skill: op-group-rounded

GROUP_SET_PER_ELEMENT

Splits a multi-select answer into one group per option, counting each row once in every option it picked.

Level: intermediate

Questions it answers:

• How many respondents picked each brand they are aware of?
• What is the average rating among users of each feature?

Use cases by domain:

• survey: Awareness count per brand from a pick-all-that-apply question.
• ops: Tickets per tag, where a ticket can carry several tags.

Assumptions:

• Buckets are built from the rows the filters kept; each aggregation is then worked out once per bucket.
• A row that picked three options is counted in three groups, so group counts add up to more than the number of rows.
• A row that picked nothing lands in no group, and neither does a missing answer.

Use something else:

• GROUP_SET_VALUE when you want each exact combination of options as one group.
• AGG_SET_FREQUENCY when you only want the count per option, without other figures.

Glossary: missing-value

Skill: op-group-set-per-element

GROUP_SET_VALUE

Groups the rows by the exact combination of options picked in a multi-select answer, such as A only, A and B, or none.

Level: intermediate

Questions it answers:

• Which combinations of payment methods do customers use, and how common is each?
• How many respondents use our app only, versus our app and a competitor’s?

Use cases by domain:

• survey: Brand-repertoire groups: which sets of brands people buy together.
• ops: Orders per combination of add-ons.

Assumptions:

• Buckets are built from the rows the filters kept; each aggregation is then worked out once per bucket.
• Each row lands in exactly one group, named by its picked options joined with |.
• A row that picked nothing gets its own group with an empty name, which differs from a missing answer.
• A row whose field is missing lands in no bucket; it is counted as missing, not as a bucket of its own.

Use something else:

• GROUP_SET_PER_ELEMENT when you want one group per option, counting a row in each it picked.
• FILTER_SET_EQUALS when you want to keep one combination rather than see them all.

Glossary: missing-value

Skill: op-group-set-value

Window operators

The window operators this instance offers: what each is for, the questions it answers and when to reach for something else. Operators are sorted by name; each name links to its detail block below.

Operators

WIN_DELTA

Adds the change from a set number of rows earlier in the field’s own units, such as orders this month minus last month.

Level: basic

Questions it answers:

• By how many orders did each month go up or down on the month before?
• How many points did satisfaction move between waves?

Use cases by domain:

• survey: Change in a brand’s awareness, in percentage points, from the last wave.
• ops: Week-on-week change in tickets opened per queue.
• science: Change in each subject’s reading since the previous visit.

Assumptions:

• Runs on the result’s rows after grouping: one row per group, or one per filtered record when nothing is grouped.
• Computed separately within each partition_by slice, in order_by order; with no partition the whole result is one series.
• Steps are rows, not calendar periods: a period with no row is skipped, not filled, so one row back can span two periods.
• The first rows of each partition, and any row where either value is missing, read null; an earlier value of 0 is a real change.

Use something else:

• WIN_PCT_CHANGE when you want the change relative to the earlier value.
• OVERLAY_DELTA_VS_PRIOR when you want each period’s change from the one before as a decoration on the result.

Glossary: percentage-point, baseline, missing-value

Skill: op-win-delta

WIN_DENSE_RANK

Ranks rows within each partition in the chosen order; rows with equal values share a rank and the next follows on: 1, 2, 2, 3.

Level: basic

Questions it answers:

• Which price tier is each product in, counting equal prices as one tier?
• With scores ordered desc, how many distinct scores sit above each respondent’s (the dense rank minus 1)?

Use cases by domain:

• survey: Position of each rating level from the top, with equal ratings sharing one position.
• ops: Tier number of each store by sales, equal sales sharing a tier.
• science: Order of distinct dose levels within each trial arm.

Assumptions:

• Runs on the result’s rows after grouping: one row per group, or one per filtered record when nothing is grouped.
• Rank 1 goes to the first row in order_by order: the smallest value, or the largest when the key is desc.
• Rows tie only when equal on every order_by key.
• Rows with a missing order_by value sort last, in both directions.

Use something else:

• WIN_RANK when the rank should skip past equal values, as in sports standings.
• WIN_ROW_NUMBER when every row needs its own number even when values are equal.

Glossary: rank, ties

Skill: op-win-dense-rank

WIN_EWMA

Adds a smoothed series: each new value counts for a fixed share w, the previous smoothed level for 1 - w, so older values fade.

Level: intermediate

Also known as: exponential smoothing, exponentially weighted moving average

Questions it answers:

• What is the underlying level of daily demand once day-to-day noise is damped?
• Is the smoothed defect rate drifting up over recent batches?

Use cases by domain:

• survey: Smoothed tracking score that reacts to recent waves more than old ones.
• ops: Smoothed daily demand for a stock forecast.
• science: Smoothed sensor reading that follows slow drift and damps spikes.

Assumptions:

• Runs on the result’s rows after grouping: one row per group, or one per filtered record when nothing is grouped.
• Computed separately within each partition_by slice, in order_by order; with no partition the whole result is one series.
• Steps are rows, not calendar periods: a period with no row is skipped, not filled, so one row back can span two periods.
• The share w is set directly (params.alpha, above 0 and at most 1) or as its complement (params.decay = 1 - w, at least 0 and below 1); it seeds from the partition’s first value present.
• A missing value reads null and the smoothing carries over it.

Use something else:

• WIN_MOVING_AVG when you want every row in a fixed window to count equally.
• OVERLAY_ZSCORE_VS_ROLLING when you want how unusual each period is against its recent run.

Glossary: mean, missing-value

Skill: op-win-ewma

WIN_LAG

Adds to every row the value of a field from a set number of rows earlier in the ordered series, such as last month’s sales.

Level: basic

Questions it answers:

• What were each store’s sales in the month before, on the same row as this month’s?
• What score did each respondent give in the previous wave?

Use cases by domain:

• survey: Previous wave’s score beside the current one for each tracked brand.
• ops: Last week’s order count beside this week’s, per warehouse.
• science: The previous reading beside each measurement in a time series.

Assumptions:

• Runs on the result’s rows after grouping: one row per group, or one per filtered record when nothing is grouped.
• Computed separately within each partition_by slice, in order_by order; with no partition the whole result is one series.
• Steps are rows, not calendar periods: a period with no row is skipped, not filled, so one row back can span two periods.
• The first rows of each partition have no earlier row and read null, or the default when one is set.

Use something else:

• WIN_DELTA when you want the difference from the earlier row, not its value.
• WIN_LEAD when you want the value from a later row.
• OVERLAY_INDEX_VS_PRIOR when you want each period against the period before it as an index.

Glossary: missing-value

Skill: op-win-lag

WIN_LEAD

Adds to every row the value of a field from a set number of rows later in the ordered series, such as next month’s sales.

Level: basic

Questions it answers:

• What did each customer spend in the following month, beside this month’s spend?
• What reading came next after each measurement?

Use cases by domain:

• survey: Next wave’s answer beside the current one, to see who changed their mind.
• ops: Next shipment date beside each order, to measure the wait.
• science: The following reading beside each measurement in a series.

Assumptions:

• Runs on the result’s rows after grouping: one row per group, or one per filtered record when nothing is grouped.
• Computed separately within each partition_by slice, in order_by order; with no partition the whole result is one series.
• Steps are rows, not calendar periods: a period with no row is skipped, not filled, so one row back can span two periods.
• The last rows of each partition have no later row and read null, or the default when one is set.

Use something else:

• WIN_LAG when you want the value from an earlier row.
• WIN_DELTA when you want how much the field changed between rows.

Glossary: missing-value

Skill: op-win-lead

WIN_MOVING_AVG

Adds a moving average: the mean of a field over a fixed number of neighbouring rows, smoothing short-term swings in a series.

Level: basic

Also known as: rolling mean, simple moving average

Questions it answers:

• What is the trailing seven-day average of daily orders?
• Is the trend in weekly sign-ups rising once the week-to-week noise is smoothed out?

Use cases by domain:

• survey: Three-wave average of a tracking score to steady small-sample waves.
• ops: Seven-day average of daily orders.
• science: Smooth a noisy sensor series before reading its trend.

Assumptions:

• Runs on the result’s rows after grouping: one row per group, or one per filtered record when nothing is grouped.
• Computed separately within each partition_by slice, in order_by order; with no partition the whole result is one series.
• Steps are rows, not calendar periods: a period with no row is skipped, not filled, so one row back can span two periods.
• The frame must be bounded on both sides; near a partition’s edges it holds fewer rows.
• Missing values are skipped, so the figure covers the values present; a frame with no value reads null.

Use something else:

• WIN_RUNNING_AVG when you want the average of everything up to this row.
• WIN_EWMA when you want recent rows to count more than older ones.
• OVERLAY_INDEX_VS_ROLLING_MEAN when you want each period against its recent average.

Glossary: rolling-mean, mean, missing-value

Skill: op-win-moving-avg

WIN_PCT_CHANGE

Adds the change from a set number of rows earlier as a fraction of the earlier value: 0.05 is a 5% rise on last month.

Level: basic

Questions it answers:

• By what percentage did each month’s revenue grow on the month before?
• Which regions grew fastest relative to their own previous quarter?

Use cases by domain:

• survey: Relative change in sample size per wave.
• ops: Month-on-month growth in revenue per product line.
• science: Relative change in a measurement since the previous visit.

Assumptions:

• Runs on the result’s rows after grouping: one row per group, or one per filtered record when nothing is grouped.
• Computed separately within each partition_by slice, in order_by order; with no partition the whole result is one series.
• Steps are rows, not calendar periods: a period with no row is skipped, not filled, so one row back can span two periods.
• The first rows of each partition, any row where either value is missing, and any row whose earlier value is 0 read null.

Use something else:

• WIN_DELTA when the earlier value can be 0, near 0 or negative, or the field is itself a percentage.
• OVERLAY_YOY when you want the same year-ago period as the comparison.
• OVERLAY_INDEX_VS_PRIOR when you want each period as an index value against the one before.

Glossary: baseline, percentage-point, index-value, missing-value

Skill: op-win-pct-change

WIN_RANK

Ranks rows within each partition in the chosen order; rows with equal values share a rank and the next skips: 1, 2, 2, 4.

Level: basic

Questions it answers:

• Where does each store rank on revenue within its region?
• Which product came first in each month by units sold?

Use cases by domain:

• survey: Rank of each brand on preference within every market.
• ops: Rank of each warehouse on on-time delivery within its region.
• science: Rank of each site’s yield within each season.

Assumptions:

• Runs on the result’s rows after grouping: one row per group, or one per filtered record when nothing is grouped.
• Rank 1 goes to the first row in order_by order: the smallest value, or the largest when the key is desc.
• Rows tie only when equal on every order_by key.
• Rows with a missing order_by value sort last, in both directions.

Use something else:

• WIN_DENSE_RANK when equal values should share a rank with no gap after them.
• WIN_ROW_NUMBER when every row needs its own number even when values are equal.
• ATTR_PERCENTILE when you want each record’s position as a percentage of all records.

Glossary: rank, ties

Skill: op-win-rank

WIN_ROW_NUMBER

Numbers the rows 1, 2, 3 within each partition in the chosen order, for picking the first few rows of every group.

Level: basic

Questions it answers:

• Which are the top three products by revenue in each region?
• What is each customer’s first order, by date?

Use cases by domain:

• survey: Number each respondent’s answers in the order given.
• ops: Top five stores by sales within each region.
• harness: Number result rows to page through them in a fixed order.

Assumptions:

• Runs on the result’s rows after grouping: one row per group, or one per filtered record when nothing is grouped.
• Row 1 goes to the first row in order_by order: the smallest value, or the largest when the key is desc, so set desc for a top-N.
• Rows with equal order_by values still get different numbers, in an arbitrary but repeatable order.
• Rows with a missing order_by value sort last, in both directions.

Use something else:

• WIN_RANK when rows with equal values should share a position.
• WIN_DENSE_RANK when you want ranks without gaps after equal values.

Glossary: rank, ties

Skill: op-win-row-number

WIN_RUNNING_AVG

Adds the average of a field over the rows so far in the ordered series, such as the average order value to date.

Level: basic

Questions it answers:

• What is the average monthly revenue for the year so far, at each month?
• How has the cumulative average score settled as more waves came in?

Use cases by domain:

• survey: Average satisfaction to date across fieldwork days.
• ops: Average handling time to date, month by month.
• science: Cumulative mean of repeated measurements as each one is added.

Assumptions:

• Runs on the result’s rows after grouping: one row per group, or one per filtered record when nothing is grouped.
• Computed separately within each partition_by slice, in order_by order; with no partition the whole result is one series.
• The frame sets which rows count: no preceding bound with following 0 gives the average from the start to this row.
• Missing values are skipped, so the figure covers the values present; a frame with no value reads null.

Use something else:

• WIN_MOVING_AVG when you want the average of a fixed number of recent rows.
• AGG_AVERAGE when you want one average per group, not a running one.

Glossary: mean, missing-value

Skill: op-win-running-avg

WIN_RUNNING_SUM

Adds a running total of a field down the ordered rows, such as revenue for the year to date.

Level: basic

Questions it answers:

• What is the year-to-date revenue at the end of each month?
• How many responses had come in by each day of fieldwork?

Use cases by domain:

• survey: Cumulative completes per day against the fieldwork quota.
• ops: Year-to-date sales per region, month by month.
• science: Cumulative dose received by each visit.

Assumptions:

• Runs on the result’s rows after grouping: one row per group, or one per filtered record when nothing is grouped.
• Computed separately within each partition_by slice, in order_by order; with no partition the whole result is one series.
• The frame sets which rows add up: no preceding bound with following 0 gives the total from the start to this row.
• Missing values are skipped, so the figure covers the values present; a frame with no value reads null.

Use something else:

• AGG_SUM when you want one total per group, not a running one.
• WIN_RUNNING_AVG when you want the running average instead of the total.

Glossary: missing-value

Skill: op-win-running-sum

Feature operators

The feature operators this instance offers: what each is for, the questions it answers and when to reach for something else. Operators are sorted by name; each name links to its detail block below.

Operators

FEAT_BUCKETIZE

Puts each value into an ordered bin, using cut points you give or bins holding about equal numbers of records, and adds the bin number.

Level: basic

Questions it answers:

• Which income band is each customer in?
• Which spend decile (tenth of orders ranked by spend) does each order fall in?

Use cases by domain:

• survey: Age bands for each respondent, from fixed cut points.
• ops: Order value tenths, as a column later steps can filter on.
• science: Exposure bands for each subject, for a banded analysis.

Assumptions:

• Runs on every record before filters, so a filter can use the new column but never narrows what the feature sees.
• Bin 0 is the lowest; a value equal to a cut point goes in the bin below it.
• Equal-count bins use cut points from every record of the cohort, and repeated values can leave bins uneven. A missing input value gives a missing output.

Use something else:

• GROUP_RANGE when you want results grouped by fixed-width bands.
• GROUP_QUANTILE when you want results grouped by equal-count bins.

Glossary: percentile, missing-value

Skill: op-feat-bucketize

FEAT_DATE_FEATURES

Splits a date into year, month, day, day of week and quarter columns, so each part can be filtered, grouped or modelled.

Level: basic

Questions it answers:

• Do orders differ by day of the week?
• Is there a seasonal pattern by month across several years?

Use cases by domain:

• survey: Quarter and weekday of each interview, to check fieldwork timing.
• ops: Weekday and month of each order, as model inputs for demand.
• science: Month of each sample, to look for seasonal effects.

Assumptions:

• Runs on every record before filters, so a filter can use the new column but never narrows what the feature sees.
• Accepts a date or datetime field; a datetime is read on the local clock of its time zone and adds an hour column. Day of week is 0 for Sunday through 6 for Saturday.
• A missing date gives missing values in every column.

Use something else:

• ATTR_DATE_PART when you need just one part of the date.
• GROUP_DATE when you want results grouped by day, month or year.

Glossary: missing-value

Skill: op-feat-date-features

FEAT_FREQUENCY_ENCODE

Replaces each category with the share of records that carry it, one number that says how common the category is.

Level: intermediate

Questions it answers:

• How common is each customer’s city, as a single number a model can use?
• Which records belong to rare product codes?

Use cases by domain:

• survey: How common each respondent’s free-text employer code is.
• ops: How common each SKU is, to flag rare items.
• science: How common each species label is in the sample.

Assumptions:

• Runs on every record before filters, so a filter can use the new column but never narrows what the feature sees.
• Its figures come from every record of the cohort, before any filter: filtering afterwards does not recompute them.
• The share is out of records with a category; records missing it read missing and are left out of the total.

Use something else:

• GROUP_CATEGORY when you want the count table itself, one row per category: group by the field, then count.
• FEAT_ONE_HOT when the field has few categories and each should be its own column.

Glossary: missing-value

Skill: op-feat-frequency-encode

FEAT_LOG

Adds a column holding the natural log of 1 + the value, which pulls in a long tail of large values so a skewed field is easier to model.

Level: intermediate

Questions it answers:

• Can I tame the long tail of customer spend before fitting a model on it?
• How does income look on a log-like scale, where for large values doubling counts about the same everywhere?

Use cases by domain:

• survey: Log of reported household income before using it as a predictor.
• ops: Log of order value, where a few huge orders dwarf the rest.
• science: Log of a concentration that spans several orders of magnitude.

Assumptions:

• Runs on every record before filters, so a filter can use the new column but never narrows what the feature sees.
• Computes ln(1 + x), not ln(x): 0 maps to 0, and values between -1 and 0 map below 0.
• A value of -1 or less has no log and reads missing, with no error; A missing input value gives a missing output.

Use something else:

• FEAT_SQRT when the tail is only moderate and a gentler squeeze is enough.
• ATTR_ZSCORE when you want each value in standard units from the mean, not a squeezed scale.
• FEAT_BUCKETIZE when you want ordered bins instead of a continuous transform.

Glossary: skew, missing-value

Skill: op-feat-log

FEAT_ONE_HOT

Turns a category field into one 0/1 column per category, so a model or a sum can treat each category as its own yes/no.

Level: basic

Also known as: dummy coding, dummy variables

Questions it answers:

• How do I feed region into a regression as separate yes/no predictors?
• Which rows are in each plan tier, as columns I can sum?

Use cases by domain:

• survey: One yes/no column per answer to a single-choice question.
• ops: One column per warehouse for a model of delivery time.
• science: One column per treatment arm in a design table.

Assumptions:

• Runs on every record before filters, so a filter can use the new column but never narrows what the feature sees.
• Columns come from the field’s dictionary, so a category with no rows still gets an all-zero column.
• A missing category reads 0 in every column; there is no separate unknown column.
• In a model with an intercept, leave one category’s column out as the reference: all of them together are perfectly collinear and the fit is refused.

Use something else:

• FEAT_FREQUENCY_ENCODE when the field has hundreds of categories and one column each is too many.
• GROUP_CATEGORY when you want how many rows fall in each category: group by the field, then count.

Glossary: missing-value, multicollinearity

Skill: op-feat-one-hot

FEAT_POLY

Adds columns holding the value squared, cubed and so on up to a chosen power, so a straight-line model can bend into a curve.

Level: advanced

Questions it answers:

• Does satisfaction rise with tenure and then level off?
• Is the effect of temperature on yield curved rather than straight?

Use cases by domain:

• survey: Age and age squared as predictors, for an effect that peaks in mid-life.
• ops: Order size and its square, for a cost curve that steepens.
• science: Dose, dose squared and dose cubed for a curved dose-response fit.

Assumptions:

• Runs on every record before filters, so a filter can use the new column but never narrows what the feature sees.
• Emits powers 2 up to the degree (at most 10); the power-1 term is the original column, which you add yourself.
• Raw powers grow fast and move together; centre or standardise the field first. A missing input value gives a missing output.

Use something else:

• REG_OLS when you want to fit the curve itself, not just build its columns.
• ATTR_FORMULA when you want one custom transform of the value.

Glossary: regression-coefficient, multicollinearity, missing-value

Skill: op-feat-poly

FEAT_SQRT

Adds a column holding the square root of the value, a gentler squeeze than a log for counts with a moderate long tail.

Level: intermediate

Questions it answers:

• Can I soften the few very large complaint counts without a full log?
• What does the field look like with its big values pulled in a little?

Use cases by domain:

• survey: Square root of the number of open-text mentions per respondent.
• ops: Square root of daily ticket counts before modelling them.
• science: Square root of event counts, a common step for count data.

Assumptions:

• Runs on every record before filters, so a filter can use the new column but never narrows what the feature sees.
• A negative value has no real square root and reads missing, with no error; A missing input value gives a missing output.

Use something else:

• FEAT_LOG when the tail is very long and needs a stronger squeeze.
• ATTR_ZSCORE when you want each value in standard units from the mean.

Glossary: missing-value

Skill: op-feat-sqrt

FEAT_TARGET_ENCODE

Replaces each category with the average outcome of the records in that category, optionally pulled toward the overall average.

Level: advanced

Questions it answers:

• How do I give a model a single number for each of a thousand postcodes?
• What is the average sale price of each product category, on every record?

Use cases by domain:

• survey: Average satisfaction of each respondent’s employer, as a predictor.
• ops: Average delivery delay of each carrier route, on every shipment.
• science: Average response of each batch from earlier or held-out runs, as a covariate (an average that includes a row’s own response leaks it).

Assumptions:

• Runs on every record before filters, so a filter can use the new column but never narrows what the feature sees.
• Its figures come from every record of the cohort, before any filter: filtering afterwards does not recompute them.
• It reads no split column: placing FEAT_TRAIN_TEST_SPLIT first changes nothing, and test rows and each row’s own outcome still feed every average.
• Smoothing s gives (n * category average + s * overall average) / (n + s); 0 means no pull.

Use something else:

• AGG_AVERAGE when you want to report each category’s average outcome.
• FEAT_ONE_HOT when the field has few categories and each should be its own column.
• FEAT_FREQUENCY_ENCODE when you want how common each category is, without using the outcome.

Glossary: mean, overfitting, missing-value

Skill: op-feat-target-encode

FEAT_TRAIN_TEST_SPLIT

Labels every record train (0), validation (1) or test (2) by a seeded shuffle, so a model can be checked on records it never learned from.

Level: intermediate

Questions it answers:

• How do I hold back 20% of records to test a model on?
• Can I split records so each class keeps its share in train and test?

Use cases by domain:

• survey: Hold out a share of respondents to check a scoring model.
• ops: Train, validation and test sets for a churn model.
• harness: A repeatable split so two runs compare models on the same rows.

Assumptions:

• Runs on every record before filters, so a filter can use the new column but never narrows what the feature sees.
• Each share is the ratio times the record count, rounded; with stratify it is applied within each category.
• The same seed on the same records in the same order gives the same labels; adding or reordering records reshuffles them.
• The split does not isolate other features: frequency encoding, target encoding and equal-count bucketing in the same request still learn from test rows, so build those from the train rows in a separate request.

Use something else:

• capability:sample when you just want a random subset of rows to look at.
• GROUP_CATEGORY when you want to compare groups that already exist.

Glossary: overfitting

Skill: op-feat-train-test-split

Statistical tests

The statistical tests this instance offers: what each is for, the questions it answers and when to reach for something else. Operators are sorted by name; each name links to its detail block below.

Operators

TEST_ANOVA_F

Checks whether the average of a numeric measure differs across three or more groups.

Level: intermediate

Also known as: anova, one-way anova, aov

Questions it answers:

• Does average spend differ across regions?
• Do the treatment arms produce different average outcomes?

Use cases by domain:

• survey: Compare mean satisfaction across age bands.
• ops: Compare average delivery time across warehouses.
• science: Compare mean yield across fertiliser treatments.

Assumptions:

• Rows are independent of each other, within and across groups.
• The measure is roughly normally distributed within each group.
• Groups have similar variances; TEST_BROWN_FORSYTHE checks this.
• It is an overall test: it says whether any group average stands apart, not which one.

Use something else:

• TEST_ANOVA_WELCH when the groups have clearly unequal spread.
• TEST_KRUSKAL_WALLIS when the measure is heavily skewed or only ordered.
• TEST_ANOVA_RM when the same subjects are measured in every group.
• TEST_WELCH when there are only two groups.
• TEST_TUKEY_HSD when you need to know which pairs of groups differ.

Follow up with:

• TEST_TUKEY_HSD when the groups differ and you want to know which pairs of groups do.
• OVERLAY_PAIRWISE_WELCH_T when the groups are the rows of a crosstab of averages and you want every pair tested.

Glossary: effect-size, eta-squared, f-statistic, homogeneity-of-variance, normal-distribution, null-hypothesis, omega-squared, p-value, post-hoc-test, variance

Skill: op-test-anova-f

TEST_ANOVA_RM

Checks whether the average differs across conditions when every subject is measured under each condition.

Level: advanced

Also known as: repeated measures anova, within-subjects anova

Questions it answers:

• Do the same panel members rate the three ad concepts differently on average?
• Does each patient’s score change across the baseline, mid-point and final visits?

Use cases by domain:

• survey: Compare each respondent’s ratings of several brands asked in one survey.
• ops: Compare each store’s sales across three promotion periods.
• science: Compare each subject’s response across repeated measurement sessions.

Assumptions:

• The same subjects (SubjectField) are measured once under every condition; subjects missing a condition are dropped.
• Subjects are independent of each other.
• The measure is roughly normally distributed within each condition.
• Sphericity: differences between each pair of conditions are about equally variable. No correction is applied, so violations make the p-value too small.

Use something else:

• TEST_ANOVA_F when each group holds different, unrelated subjects.
• TEST_PAIRED_T when there are only two conditions.

Glossary: f-statistic, independence, normal-distribution, p-value, partial-eta-squared, repeated-measures, sphericity

Skill: op-test-anova-rm

TEST_ANOVA_WELCH

Checks whether the average of a numeric measure differs across groups when the groups may have unequal spread.

Level: intermediate

Also known as: welch's anova, oneway.test

Questions it answers:

• Does average response time differ across regions whose variability is very different?
• Do the store formats differ in average basket size, given some formats are far more variable?

Use cases by domain:

• survey: Compare mean scores across segments of very different sizes and spreads.
• ops: Compare average processing time across sites with uneven variability.
• science: Compare treatment means when the spread grows with the dose.

Assumptions:

• Rows are independent of each other, within and across groups.
• The measure is roughly normally distributed within each group.
• Groups may have unequal variances; each group needs some spread (a constant group is refused).
• It is an overall test: it says whether any group average stands apart, not which one.

Use something else:

• TEST_KRUSKAL_WALLIS when the measure is heavily skewed or only ordered.
• TEST_ANOVA_RM when the same subjects are measured in every group.
• TEST_WELCH when there are only two groups, or you want pairwise follow-ups: run TEST_WELCH per pair and set multiplicity on the request (for example holm) to adjust their p-values for multiple comparisons.
• TEST_ANOVA_F when spreads are similar and you want Tukey’s pairwise follow-up.

Follow up with:

• OVERLAY_PAIRWISE_WELCH_T when the groups differ and you want each pair tested without assuming equal spread.
• TEST_TUKEY_HSD when the groups differ and their spreads turn out similar after all.

Glossary: degrees-of-freedom, f-statistic, homogeneity-of-variance, independence, multiple-comparisons, normal-distribution, omega-squared, p-value, variance

Skill: op-test-anova-welch

TEST_BARTLETT_SPHERICITY

Checks whether a set of numeric fields are related at all, the first check before summarising them as a few underlying dimensions.

Level: advanced

Also known as: bartlett sphericity test, bartlett's test of sphericity, factorability test

Questions it answers:

• Do the ten satisfaction items move together enough to be worth reducing to a few scores?
• Are these sensor readings related at all, or is each one independent noise?

Use cases by domain:

• survey: Check a rating battery is factorable before a principal component analysis or a reliability score.
• science: Check a set of measurements share variation before reducing them to components.

Assumptions:

• Rows are independent of each other.
• The fields are roughly jointly normal.
• A row with any field missing is dropped from the test (listwise).
• With many rows almost any correlation is significant, so read the size of the correlations too.

Use something else:

• MAT_CORRELATION when you want how strongly each pair is related.
• MAT_PCA when you want the dimensions themselves.

Glossary: chi-square, correlation, degrees-of-freedom, independence, listwise-deletion, normal-distribution, p-value, principal-component, sample-size, test-statistic

Skill: op-test-bartlett-sphericity

TEST_BOX_M

Checks whether several groups share one spread pattern across a set of numeric fields, the assumption behind comparing average profiles.

Level: advanced

Also known as: box's m, box's m test, equal covariance test

Questions it answers:

• Before comparing segment profiles, do the segments have similar spread across the metrics?
• Is it safe to pool the two groups’ covariance for a multivariate comparison?

Use cases by domain:

• survey: Check segments share one covariance across a rating battery before a profile comparison.
• science: Check treatment arms share one covariance across outcomes before a MANOVA.

Assumptions:

• Rows are independent of each other, within and across the groups.
• The fields are roughly jointly normal within each group; the test is very sensitive to departures, so a small p can reflect non-normality rather than unequal spread.
• A row with any field missing is dropped from the test (listwise).
• Every group has more complete rows than fields.

Use something else:

• TEST_BROWN_FORSYTHE when there is only one numeric field.
• TEST_MANOVA when you want whether the average profiles differ.

Glossary: chi-square, covariance, degrees-of-freedom, f-statistic, homogeneity-of-variance, independence, listwise-deletion, normal-distribution, p-value, sample-size, test-statistic

Skill: op-test-box-m

TEST_BROWN_FORSYTHE

Checks whether the spread of a numeric field differs across groups: a robust test of equal variances.

Level: advanced

Also known as: levene's test, median-centred levene test

Questions it answers:

• Is delivery time more variable at some warehouses than at others?
• How different is the spread of the measure across groups?

Use cases by domain:

• survey: Check whether some segments answer far more variably than others.
• ops: Compare the consistency of processing times across sites.
• science: Check whether measurement spread differs across treatment groups.

Assumptions:

• Rows are independent of each other, within and across groups.
• It measures distance from each group’s median, so it stays reliable when the data are not normal.
• Very small groups make the medians unstable.
• A large p-value is not evidence that the spreads are equal; many analysts simply use Welch’s tests rather than pre-testing.

Use something else:

• TEST_ANOVA_WELCH when you want to compare the group averages themselves.
• TEST_SHAPIRO_WILK when you want to check whether a measure is bell-shaped.

Follow up with:

• TEST_ANOVA_WELCH when the spreads differ and you still want to compare group averages.

Glossary: f-statistic, homogeneity-of-variance, independence, median, p-value, variance

Skill: op-test-brown-forsythe

TEST_CHISQ

Checks whether two categorical fields are associated by comparing a cross-tabulation with the counts expected if unrelated.

Level: intermediate

Also known as: chi square, chi-square test, pearson chi-square, chisq.test

Questions it answers:

• Is preferred channel associated with age band?
• Is the mix of plan types different across regions?

Use cases by domain:

• survey: Check whether brand preference varies by region.
• ops: Check whether ticket category is associated with the support channel.
• science: Check whether outcome category depends on treatment arm.

Assumptions:

• Each row is counted once and rows are independent of each other.
• It works on raw counts, never on percentages or averages.
• Expected counts should mostly be 5 or more (Cochran’s rule: none below 1 and no more than a fifth below 5); below that the p-value is unreliable, and Pulse warns when any cell is under 5.
• No continuity correction is applied.

Use something else:

• TEST_FISHER_EXACT when the table is 2x2 and some expected counts are small.
• TEST_PROP_Z when you compare a yes/no rate between exactly two groups and want a confidence interval.
• TEST_SPEARMAN_R when both fields are numeric.
• OVERLAY_CHISQ_VS_POP when you compare one subgroup’s mix with the whole population.

Follow up with:

• OVERLAY_FISHER_EXACT_CELL when the table shows an association and you want to see which cells drive it.
• OVERLAY_CHISQ_ROW when you want to see which rows depart from the overall column mix.

Glossary: chi-square, continuity-correction, cramers-v, cross-tabulation, degrees-of-freedom, independence, p-value, phi

Skill: op-test-chisq

TEST_FISHER_EXACT

Exact test of whether two yes/no fields are associated, built for small 2x2 tables.

Level: intermediate

Also known as: fisher's exact test, fisher.test

Questions it answers:

• In a small pilot, did the treated group recover more often than the control group?
• With only a few dozen responses, is opting in associated with the sign-up channel?

Use cases by domain:

• survey: Compare a yes/no answer between two small subgroups.
• ops: Compare a rare failure rate between two production lines.
• science: Compare a binary outcome between two small treatment arms.

Assumptions:

• Rows are independent of each other, and each row is counted once.
• Strictly 2x2: both fields must have exactly two levels.
• Row and column totals are treated as fixed, which makes its p-values tend to run a little large, so it misses real associations somewhat more often than it needs to.
• The two-sided p-value sums every table no more likely than the one observed; other tools sometimes double a one-sided tail and disagree slightly.

Use something else:

• TEST_CHISQ when either field has more than two categories.
• TEST_PROP_Z when counts are large and you want the difference in rates with a confidence interval.

Glossary: cross-tabulation, exact-test, independence, odds-ratio, p-value, two-tailed

Skill: op-test-fisher-exact

TEST_HOTELLING_T2

Checks whether several numeric fields, taken together as one profile, differ from a target, between two groups, or before vs after.

Level: advanced

Also known as: hotelling t2, hotelling's t-squared, multivariate t test

Questions it answers:

• Do the two app versions differ on the whole set of engagement metrics at once?
• Is our average satisfaction profile across five attributes different from the benchmark profile?
• Did the same customers’ ratings shift as a whole between the before and after surveys?

Use cases by domain:

• survey: Compare two segments on a battery of rating items in one test.
• ops: Compare two sites on throughput, wait time and error rate together.
• science: Compare treatment and control on several outcomes without one test per outcome.

Assumptions:

• Rows are independent of each other, within and across the groups.
• The fields are roughly jointly normal; with many rows the test tolerates departures.
• The two-sample form assumes both groups share one covariance across the fields unless variance is set to unequal.
• A row with any field missing is dropped from the test (listwise).
• There are more complete rows than fields.

Use something else:

• TEST_T when there is only one numeric field.
• TEST_WELCH when you want each field compared on its own.

Glossary: covariance, degrees-of-freedom, f-statistic, independence, listwise-deletion, mean, normal-distribution, p-value, sample-size, test-statistic

Skill: op-test-hotelling-t2

TEST_KENDALL_TAU

Measures how often pairs of rows agree in order on two numeric fields: a rank-based link suited to small samples and ties.

Level: intermediate

Also known as: kendall, kendall's tau, tau-b

Questions it answers:

• In a small panel, do judges who score one entry higher also score the other higher?
• Do two coarse rating scales tend to agree?

Use cases by domain:

• survey: Relate two short rating scales with many tied answers.
• ops: Check whether two priority rankings agree.
• science: Relate two measures in a small study.

Assumptions:

• Pairs of values are independent of each other.
• It detects monotonic links; ties in either field are corrected for (tau-b).
• The p-value is a two-sided normal approximation.

Use something else:

• TEST_SPEARMAN_R when the data are large; its cost grows with the square of the row count.
• TEST_PEARSON_R when you want the strength of a straight-line link.
• TEST_TREND when you want to know whether one ordered series keeps rising or falling.

Follow up with:

• REG_OLS when the link looks roughly straight and you want to predict one field from the other.

Glossary: correlation, independence, kendall-tau, monotonic-trend, p-value, rank, ties, two-tailed

Skill: op-test-kendall-tau

TEST_KRUSKAL_WALLIS

Rank-based check of whether values tend to be larger in some groups than in others, across two or more groups.

Level: intermediate

Also known as: kruskal-wallis h test, kruskal.test

Questions it answers:

• Do satisfaction ratings tend to differ across the four regions?
• Do skewed repair times differ across product lines?

Use cases by domain:

• survey: Compare 1-5 ratings across several customer segments.
• ops: Compare skewed wait times across branches.
• science: Compare a non-normal outcome across several treatment groups.

Assumptions:

• Rows are independent of each other, within and across groups.
• It tests whether values tend to be larger in some groups; it reads as a difference in medians only when all groups have the same shape.
• It is an overall test: to find which groups differ, follow up with pairwise Mann-Whitney tests and set multiplicity on the request to adjust them for multiple comparisons.
• The p-value comes from a chi-square approximation that is shaky for very small groups.

Use something else:

• TEST_ANOVA_WELCH when the data are roughly normal and you want to compare averages.
• TEST_MANN_WHITNEY_U when there are only two groups.
• TEST_ANOVA_RM when the same subjects are measured in every group and the measure is roughly normal (Pulse has no Friedman test for skewed repeated measures; for two conditions use TEST_WILCOXON_SR).

Follow up with:

• TEST_MANN_WHITNEY_U when the groups differ and you want a rank-based comparison of each pair, read with multiple-comparison caution.

Glossary: chi-square, epsilon-squared, independence, median, multiple-comparisons, non-parametric, p-value, rank, ties

Skill: op-test-kruskal-wallis

TEST_KS

Checks whether a numeric field’s values follow the same distribution in two groups, comparing their whole shape.

Level: intermediate

Also known as: kolmogorov-smirnov test, ks test, ks.test

Questions it answers:

• Do order values in the two regions have the same overall distribution?
• Has the shape of response times changed between the old and new system?

Use cases by domain:

• survey: Compare the full spread of scores between two survey waves.
• ops: Detect a change in the shape of latency between two releases.
• science: Compare measurement distributions between two instruments.

Assumptions:

• Rows are independent of each other, within and across the two groups.
• It compares two observed groups; it is not a normality test with parameters estimated from the data (that needs the Lilliefors correction).
• With few rows per group it has low power; with very large groups even trivial differences in shape are flagged.
• The p-value is a two-sided large-sample approximation; many tied values make it conservative.

Use something else:

• TEST_SHAPIRO_WILK when you want to check one field against the normal distribution.
• TEST_MANN_WHITNEY_U when you only care whether one group’s values tend to be larger.
• OVERLAY_KS_VS_POP when you compare a subgroup with the whole population.

Glossary: goodness-of-fit, independence, non-parametric, p-value, statistical-power, statistical-significance, test-statistic, ties, two-tailed

Skill: op-test-ks

TEST_MANN_WHITNEY_U

Rank-based check of whether values in one of two groups tend to be larger than in the other.

Level: intermediate

Also known as: mann-whitney, wilcoxon rank-sum test

Questions it answers:

• Do customers on the new plan tend to rate the service higher?
• Do response times at one site tend to run longer than at the other?

Use cases by domain:

• survey: Compare 1-5 ratings between two segments.
• ops: Compare skewed resolution times between two support teams.
• science: Compare a non-normal outcome between treatment and control.

Assumptions:

• Rows are independent of each other, within and across the two groups.
• It tests whether a value from one group tends to exceed one from the other; it reads as a difference in medians only when both groups have the same shape.
• The p-value is a two-sided large-sample approximation with tie and continuity corrections, not an exact value.

Use something else:

• TEST_WILCOXON_SR when the same subjects are measured twice.
• TEST_KRUSKAL_WALLIS when there are three or more groups.
• TEST_WELCH when the data are roughly normal and you want the difference in averages.
• TEST_KS when you care about any difference in the shape of the two distributions.

Glossary: continuity-correction, independence, median, non-parametric, p-value, rank, rank-biserial, test-statistic, ties, two-tailed

Skill: op-test-mann-whitney-u

TEST_MANOVA

Checks whether several numeric fields, taken together as one profile, differ across two or more groups.

Level: advanced

Also known as: manova, multivariate anova, pillai's trace, wilks' lambda

Questions it answers:

• Do our customer segments differ on the whole set of spend, visit and basket metrics at once?
• Do the three store formats differ on their combined service ratings?

Use cases by domain:

• survey: Compare several segments on a battery of rating items in one test.
• ops: Compare several sites on throughput, wait time and error rate together.
• science: Compare several treatment arms on more than one outcome without one ANOVA per outcome.

Assumptions:

• Rows are independent of each other, within and across the groups.
• The fields are roughly jointly normal within each group; with many rows the test tolerates departures.
• The groups share one covariance across the fields; Pillai’s trace, the headline, is the most forgiving when they do not.
• A row with any field missing is dropped from the test (listwise).
• There are at least as many complete rows as fields plus groups.

Use something else:

• TEST_ANOVA_F when there is only one numeric field.
• TEST_HOTELLING_T2 when there are only two groups and you want the T² figure.

Glossary: covariance, degrees-of-freedom, eigenvalue, f-statistic, independence, listwise-deletion, mean, normal-distribution, p-value, sample-size, test-statistic

Skill: op-test-manova

TEST_MARDIA

Checks whether a set of numeric fields looks jointly normal, through their combined skew and tail weight.

Level: advanced

Also known as: mardia's test, multivariate normality test, multivariate skewness and kurtosis

Questions it answers:

• Before comparing segment profiles, are the metrics close enough to jointly normal?
• Are the distance-based outlier flags safe to read as chi-square p-values?

Use cases by domain:

• survey: Check a rating battery’s joint shape before a profile comparison or a factor model.
• science: Check the joint normality assumption across outcomes before a MANOVA.

Assumptions:

• Rows are independent of each other.
• A row with any field missing is dropped from the test (listwise).
• There are more complete rows than fields; below 20 rows the skewness test uses a small-sample correction.
• With many rows even trivial departures are flagged; with few it misses real ones, so look at the shape too.

Use something else:

• TEST_SHAPIRO_WILK when there is only one numeric field and you want a sharper normality test.
• TEST_BOX_M when you want whether the groups share one covariance.

Glossary: chi-square, covariance, degrees-of-freedom, independence, kurtosis, listwise-deletion, normal-distribution, p-value, sample-size, skew, test-statistic

Skill: op-test-mardia

TEST_PAIRED_T

Checks whether the average change between two measurements of the same rows, such as before and after, differs from zero.

Level: intermediate

Also known as: paired t test, paired-samples t test, dependent t test

Questions it answers:

• On average, did customers’ spend differ between the period before and after the loyalty programme started?
• Do patients’ scores differ between the first and second visit?

Use cases by domain:

• survey: Compare each respondent’s rating of two products in the same questionnaire.
• ops: Compare each store’s weekly sales before and after a layout change.
• science: Compare each subject’s measurement before and after treatment.

Assumptions:

• Each row pairs two measurements of the same subject, held side by side in Field and Field2.
• Pairs are independent of each other.
• The per-row differences are roughly normally distributed; the two fields themselves need not be.
• Rows missing either value are dropped. The p-value is two-sided.

Use something else:

• TEST_WELCH when the two sets of values come from different, unrelated subjects.
• TEST_WILCOXON_SR when the per-row differences are skewed or have extreme values.
• TEST_ANOVA_RM when each subject is measured under three or more conditions.

Glossary: cohens-d, independence, normal-distribution, p-value, paired-data, t-statistic, two-tailed

Skill: op-test-paired-t

TEST_PEARSON_R

Measures how strongly two numeric fields rise and fall together along a straight line.

Level: intermediate

Also known as: pearson, pearson correlation, pearson's r