Statistical tests

The statistical tests this instance offers: what each is for, the questions it answers and when to reach for something else. Operators are sorted by name; each name links to its detail block below.

Operators

TEST_ANOVA_F

Checks whether the average of a numeric measure differs across three or more groups.

Level: intermediate

Also known as: anova, one-way anova, aov

Questions it answers:

• Does average spend differ across regions?
• Do the treatment arms produce different average outcomes?

Use cases by domain:

• survey: Compare mean satisfaction across age bands.
• ops: Compare average delivery time across warehouses.
• science: Compare mean yield across fertiliser treatments.

Assumptions:

• Rows are independent of each other, within and across groups.
• The measure is roughly normally distributed within each group.
• Groups have similar variances; TEST_BROWN_FORSYTHE checks this.
• It is an overall test: it says whether any group average stands apart, not which one.

Use something else:

• TEST_ANOVA_WELCH when the groups have clearly unequal spread.
• TEST_KRUSKAL_WALLIS when the measure is heavily skewed or only ordered.
• TEST_ANOVA_RM when the same subjects are measured in every group.
• TEST_WELCH when there are only two groups.
• TEST_TUKEY_HSD when you need to know which pairs of groups differ.

Follow up with:

• TEST_TUKEY_HSD when the groups differ and you want to know which pairs of groups do.
• OVERLAY_PAIRWISE_WELCH_T when the groups are the rows of a crosstab of averages and you want every pair tested.

Glossary: effect-size, eta-squared, f-statistic, homogeneity-of-variance, normal-distribution, null-hypothesis, omega-squared, p-value, post-hoc-test, variance

Skill: op-test-anova-f

TEST_ANOVA_RM

Checks whether the average differs across conditions when every subject is measured under each condition.

Level: advanced

Also known as: repeated measures anova, within-subjects anova

Questions it answers:

• Do the same panel members rate the three ad concepts differently on average?
• Does each patient’s score change across the baseline, mid-point and final visits?

Use cases by domain:

• survey: Compare each respondent’s ratings of several brands asked in one survey.
• ops: Compare each store’s sales across three promotion periods.
• science: Compare each subject’s response across repeated measurement sessions.

Assumptions:

• The same subjects (SubjectField) are measured once under every condition; subjects missing a condition are dropped.
• Subjects are independent of each other.
• The measure is roughly normally distributed within each condition.
• Sphericity: differences between each pair of conditions are about equally variable. No correction is applied, so violations make the p-value too small.

Use something else:

• TEST_ANOVA_F when each group holds different, unrelated subjects.
• TEST_PAIRED_T when there are only two conditions.

Glossary: f-statistic, independence, normal-distribution, p-value, partial-eta-squared, repeated-measures, sphericity

Skill: op-test-anova-rm

TEST_ANOVA_WELCH

Checks whether the average of a numeric measure differs across groups when the groups may have unequal spread.

Level: intermediate

Also known as: welch's anova, oneway.test

Questions it answers:

• Does average response time differ across regions whose variability is very different?
• Do the store formats differ in average basket size, given some formats are far more variable?

Use cases by domain:

• survey: Compare mean scores across segments of very different sizes and spreads.
• ops: Compare average processing time across sites with uneven variability.
• science: Compare treatment means when the spread grows with the dose.

Assumptions:

• Rows are independent of each other, within and across groups.
• The measure is roughly normally distributed within each group.
• Groups may have unequal variances; each group needs some spread (a constant group is refused).
• It is an overall test: it says whether any group average stands apart, not which one.

Use something else:

• TEST_KRUSKAL_WALLIS when the measure is heavily skewed or only ordered.
• TEST_ANOVA_RM when the same subjects are measured in every group.
• TEST_WELCH when there are only two groups, or you want pairwise follow-ups: run TEST_WELCH per pair and set multiplicity on the request (for example holm) to adjust their p-values for multiple comparisons.
• TEST_ANOVA_F when spreads are similar and you want Tukey’s pairwise follow-up.

Follow up with:

• OVERLAY_PAIRWISE_WELCH_T when the groups differ and you want each pair tested without assuming equal spread.
• TEST_TUKEY_HSD when the groups differ and their spreads turn out similar after all.

Glossary: degrees-of-freedom, f-statistic, homogeneity-of-variance, independence, multiple-comparisons, normal-distribution, omega-squared, p-value, variance

Skill: op-test-anova-welch

TEST_BARTLETT_SPHERICITY

Checks whether a set of numeric fields are related at all, the first check before summarising them as a few underlying dimensions.

Level: advanced

Also known as: bartlett sphericity test, bartlett's test of sphericity, factorability test

Questions it answers:

• Do the ten satisfaction items move together enough to be worth reducing to a few scores?
• Are these sensor readings related at all, or is each one independent noise?

Use cases by domain:

• survey: Check a rating battery is factorable before a principal component analysis or a reliability score.
• science: Check a set of measurements share variation before reducing them to components.

Assumptions:

• Rows are independent of each other.
• The fields are roughly jointly normal.
• A row with any field missing is dropped from the test (listwise).
• With many rows almost any correlation is significant, so read the size of the correlations too.

Use something else:

• MAT_CORRELATION when you want how strongly each pair is related.
• MAT_PCA when you want the dimensions themselves.

Glossary: chi-square, correlation, degrees-of-freedom, independence, listwise-deletion, normal-distribution, p-value, principal-component, sample-size, test-statistic

Skill: op-test-bartlett-sphericity

TEST_BOX_M

Checks whether several groups share one spread pattern across a set of numeric fields, the assumption behind comparing average profiles.

Level: advanced

Also known as: box's m, box's m test, equal covariance test

Questions it answers:

• Before comparing segment profiles, do the segments have similar spread across the metrics?
• Is it safe to pool the two groups’ covariance for a multivariate comparison?

Use cases by domain:

• survey: Check segments share one covariance across a rating battery before a profile comparison.
• science: Check treatment arms share one covariance across outcomes before a MANOVA.

Assumptions:

• Rows are independent of each other, within and across the groups.
• The fields are roughly jointly normal within each group; the test is very sensitive to departures, so a small p can reflect non-normality rather than unequal spread.
• A row with any field missing is dropped from the test (listwise).
• Every group has more complete rows than fields.

Use something else:

• TEST_BROWN_FORSYTHE when there is only one numeric field.
• TEST_MANOVA when you want whether the average profiles differ.

Glossary: chi-square, covariance, degrees-of-freedom, f-statistic, homogeneity-of-variance, independence, listwise-deletion, normal-distribution, p-value, sample-size, test-statistic

Skill: op-test-box-m

TEST_BROWN_FORSYTHE

Checks whether the spread of a numeric field differs across groups: a robust test of equal variances.

Level: advanced

Also known as: levene's test, median-centred levene test

Questions it answers:

• Is delivery time more variable at some warehouses than at others?
• How different is the spread of the measure across groups?

Use cases by domain:

• survey: Check whether some segments answer far more variably than others.
• ops: Compare the consistency of processing times across sites.
• science: Check whether measurement spread differs across treatment groups.

Assumptions:

• Rows are independent of each other, within and across groups.
• It measures distance from each group’s median, so it stays reliable when the data are not normal.
• Very small groups make the medians unstable.
• A large p-value is not evidence that the spreads are equal; many analysts simply use Welch’s tests rather than pre-testing.

Use something else:

• TEST_ANOVA_WELCH when you want to compare the group averages themselves.
• TEST_SHAPIRO_WILK when you want to check whether a measure is bell-shaped.

Follow up with:

• TEST_ANOVA_WELCH when the spreads differ and you still want to compare group averages.

Glossary: f-statistic, homogeneity-of-variance, independence, median, p-value, variance

Skill: op-test-brown-forsythe

TEST_CHISQ

Checks whether two categorical fields are associated by comparing a cross-tabulation with the counts expected if unrelated.

Level: intermediate

Also known as: chi square, chi-square test, pearson chi-square, chisq.test

Questions it answers:

• Is preferred channel associated with age band?
• Is the mix of plan types different across regions?

Use cases by domain:

• survey: Check whether brand preference varies by region.
• ops: Check whether ticket category is associated with the support channel.
• science: Check whether outcome category depends on treatment arm.

Assumptions:

• Each row is counted once and rows are independent of each other.
• It works on raw counts, never on percentages or averages.
• Expected counts should mostly be 5 or more (Cochran’s rule: none below 1 and no more than a fifth below 5); below that the p-value is unreliable, and Pulse warns when any cell is under 5.
• No continuity correction is applied.

Use something else:

• TEST_FISHER_EXACT when the table is 2x2 and some expected counts are small.
• TEST_PROP_Z when you compare a yes/no rate between exactly two groups and want a confidence interval.
• TEST_SPEARMAN_R when both fields are numeric.
• OVERLAY_CHISQ_VS_POP when you compare one subgroup’s mix with the whole population.

Follow up with:

• OVERLAY_FISHER_EXACT_CELL when the table shows an association and you want to see which cells drive it.
• OVERLAY_CHISQ_ROW when you want to see which rows depart from the overall column mix.

Glossary: chi-square, continuity-correction, cramers-v, cross-tabulation, degrees-of-freedom, independence, p-value, phi

Skill: op-test-chisq

TEST_FISHER_EXACT

Exact test of whether two yes/no fields are associated, built for small 2x2 tables.

Level: intermediate

Also known as: fisher's exact test, fisher.test

Questions it answers:

• In a small pilot, did the treated group recover more often than the control group?
• With only a few dozen responses, is opting in associated with the sign-up channel?

Use cases by domain:

• survey: Compare a yes/no answer between two small subgroups.
• ops: Compare a rare failure rate between two production lines.
• science: Compare a binary outcome between two small treatment arms.

Assumptions:

• Rows are independent of each other, and each row is counted once.
• Strictly 2x2: both fields must have exactly two levels.
• Row and column totals are treated as fixed, which makes its p-values tend to run a little large, so it misses real associations somewhat more often than it needs to.
• The two-sided p-value sums every table no more likely than the one observed; other tools sometimes double a one-sided tail and disagree slightly.

Use something else:

• TEST_CHISQ when either field has more than two categories.
• TEST_PROP_Z when counts are large and you want the difference in rates with a confidence interval.

Glossary: cross-tabulation, exact-test, independence, odds-ratio, p-value, two-tailed

Skill: op-test-fisher-exact

TEST_HOTELLING_T2

Checks whether several numeric fields, taken together as one profile, differ from a target, between two groups, or before vs after.

Level: advanced

Also known as: hotelling t2, hotelling's t-squared, multivariate t test

Questions it answers:

• Do the two app versions differ on the whole set of engagement metrics at once?
• Is our average satisfaction profile across five attributes different from the benchmark profile?
• Did the same customers’ ratings shift as a whole between the before and after surveys?

Use cases by domain:

• survey: Compare two segments on a battery of rating items in one test.
• ops: Compare two sites on throughput, wait time and error rate together.
• science: Compare treatment and control on several outcomes without one test per outcome.

Assumptions:

• Rows are independent of each other, within and across the groups.
• The fields are roughly jointly normal; with many rows the test tolerates departures.
• The two-sample form assumes both groups share one covariance across the fields unless variance is set to unequal.
• A row with any field missing is dropped from the test (listwise).
• There are more complete rows than fields.

Use something else:

• TEST_T when there is only one numeric field.
• TEST_WELCH when you want each field compared on its own.

Glossary: covariance, degrees-of-freedom, f-statistic, independence, listwise-deletion, mean, normal-distribution, p-value, sample-size, test-statistic

Skill: op-test-hotelling-t2

TEST_KENDALL_TAU

Measures how often pairs of rows agree in order on two numeric fields: a rank-based link suited to small samples and ties.

Level: intermediate

Also known as: kendall, kendall's tau, tau-b

Questions it answers:

• In a small panel, do judges who score one entry higher also score the other higher?
• Do two coarse rating scales tend to agree?

Use cases by domain:

• survey: Relate two short rating scales with many tied answers.
• ops: Check whether two priority rankings agree.
• science: Relate two measures in a small study.

Assumptions:

• Pairs of values are independent of each other.
• It detects monotonic links; ties in either field are corrected for (tau-b).
• The p-value is a two-sided normal approximation.

Use something else:

• TEST_SPEARMAN_R when the data are large; its cost grows with the square of the row count.
• TEST_PEARSON_R when you want the strength of a straight-line link.
• TEST_TREND when you want to know whether one ordered series keeps rising or falling.

Follow up with:

• REG_OLS when the link looks roughly straight and you want to predict one field from the other.

Glossary: correlation, independence, kendall-tau, monotonic-trend, p-value, rank, ties, two-tailed

Skill: op-test-kendall-tau

TEST_KRUSKAL_WALLIS

Rank-based check of whether values tend to be larger in some groups than in others, across two or more groups.

Level: intermediate

Also known as: kruskal-wallis h test, kruskal.test

Questions it answers:

• Do satisfaction ratings tend to differ across the four regions?
• Do skewed repair times differ across product lines?

Use cases by domain:

• survey: Compare 1-5 ratings across several customer segments.
• ops: Compare skewed wait times across branches.
• science: Compare a non-normal outcome across several treatment groups.

Assumptions:

• Rows are independent of each other, within and across groups.
• It tests whether values tend to be larger in some groups; it reads as a difference in medians only when all groups have the same shape.
• It is an overall test: to find which groups differ, follow up with pairwise Mann-Whitney tests and set multiplicity on the request to adjust them for multiple comparisons.
• The p-value comes from a chi-square approximation that is shaky for very small groups.

Use something else:

• TEST_ANOVA_WELCH when the data are roughly normal and you want to compare averages.
• TEST_MANN_WHITNEY_U when there are only two groups.
• TEST_ANOVA_RM when the same subjects are measured in every group and the measure is roughly normal (Pulse has no Friedman test for skewed repeated measures; for two conditions use TEST_WILCOXON_SR).

Follow up with:

• TEST_MANN_WHITNEY_U when the groups differ and you want a rank-based comparison of each pair, read with multiple-comparison caution.

Glossary: chi-square, epsilon-squared, independence, median, multiple-comparisons, non-parametric, p-value, rank, ties

Skill: op-test-kruskal-wallis

TEST_KS

Checks whether a numeric field’s values follow the same distribution in two groups, comparing their whole shape.

Level: intermediate

Also known as: kolmogorov-smirnov test, ks test, ks.test

Questions it answers:

• Do order values in the two regions have the same overall distribution?
• Has the shape of response times changed between the old and new system?

Use cases by domain:

• survey: Compare the full spread of scores between two survey waves.
• ops: Detect a change in the shape of latency between two releases.
• science: Compare measurement distributions between two instruments.

Assumptions:

• Rows are independent of each other, within and across the two groups.
• It compares two observed groups; it is not a normality test with parameters estimated from the data (that needs the Lilliefors correction).
• With few rows per group it has low power; with very large groups even trivial differences in shape are flagged.
• The p-value is a two-sided large-sample approximation; many tied values make it conservative.

Use something else:

• TEST_SHAPIRO_WILK when you want to check one field against the normal distribution.
• TEST_MANN_WHITNEY_U when you only care whether one group’s values tend to be larger.
• OVERLAY_KS_VS_POP when you compare a subgroup with the whole population.

Glossary: goodness-of-fit, independence, non-parametric, p-value, statistical-power, statistical-significance, test-statistic, ties, two-tailed

Skill: op-test-ks

TEST_MANN_WHITNEY_U

Rank-based check of whether values in one of two groups tend to be larger than in the other.

Level: intermediate

Also known as: mann-whitney, wilcoxon rank-sum test

Questions it answers:

• Do customers on the new plan tend to rate the service higher?
• Do response times at one site tend to run longer than at the other?

Use cases by domain:

• survey: Compare 1-5 ratings between two segments.
• ops: Compare skewed resolution times between two support teams.
• science: Compare a non-normal outcome between treatment and control.

Assumptions:

• Rows are independent of each other, within and across the two groups.
• It tests whether a value from one group tends to exceed one from the other; it reads as a difference in medians only when both groups have the same shape.
• The p-value is a two-sided large-sample approximation with tie and continuity corrections, not an exact value.

Use something else:

• TEST_WILCOXON_SR when the same subjects are measured twice.
• TEST_KRUSKAL_WALLIS when there are three or more groups.
• TEST_WELCH when the data are roughly normal and you want the difference in averages.
• TEST_KS when you care about any difference in the shape of the two distributions.

Glossary: continuity-correction, independence, median, non-parametric, p-value, rank, rank-biserial, test-statistic, ties, two-tailed

Skill: op-test-mann-whitney-u

TEST_MANOVA

Checks whether several numeric fields, taken together as one profile, differ across two or more groups.

Level: advanced

Also known as: manova, multivariate anova, pillai's trace, wilks' lambda

Questions it answers:

• Do our customer segments differ on the whole set of spend, visit and basket metrics at once?
• Do the three store formats differ on their combined service ratings?

Use cases by domain:

• survey: Compare several segments on a battery of rating items in one test.
• ops: Compare several sites on throughput, wait time and error rate together.
• science: Compare several treatment arms on more than one outcome without one ANOVA per outcome.

Assumptions:

• Rows are independent of each other, within and across the groups.
• The fields are roughly jointly normal within each group; with many rows the test tolerates departures.
• The groups share one covariance across the fields; Pillai’s trace, the headline, is the most forgiving when they do not.
• A row with any field missing is dropped from the test (listwise).
• There are at least as many complete rows as fields plus groups.

Use something else:

• TEST_ANOVA_F when there is only one numeric field.
• TEST_HOTELLING_T2 when there are only two groups and you want the T² figure.

Glossary: covariance, degrees-of-freedom, eigenvalue, f-statistic, independence, listwise-deletion, mean, normal-distribution, p-value, sample-size, test-statistic

Skill: op-test-manova

TEST_MARDIA

Checks whether a set of numeric fields looks jointly normal, through their combined skew and tail weight.

Level: advanced

Also known as: mardia's test, multivariate normality test, multivariate skewness and kurtosis

Questions it answers:

• Before comparing segment profiles, are the metrics close enough to jointly normal?
• Are the distance-based outlier flags safe to read as chi-square p-values?

Use cases by domain:

• survey: Check a rating battery’s joint shape before a profile comparison or a factor model.
• science: Check the joint normality assumption across outcomes before a MANOVA.

Assumptions:

• Rows are independent of each other.
• A row with any field missing is dropped from the test (listwise).
• There are more complete rows than fields; below 20 rows the skewness test uses a small-sample correction.
• With many rows even trivial departures are flagged; with few it misses real ones, so look at the shape too.

Use something else:

• TEST_SHAPIRO_WILK when there is only one numeric field and you want a sharper normality test.
• TEST_BOX_M when you want whether the groups share one covariance.

Glossary: chi-square, covariance, degrees-of-freedom, independence, kurtosis, listwise-deletion, normal-distribution, p-value, sample-size, skew, test-statistic

Skill: op-test-mardia

TEST_PAIRED_T

Checks whether the average change between two measurements of the same rows, such as before and after, differs from zero.

Level: intermediate

Also known as: paired t test, paired-samples t test, dependent t test

Questions it answers:

• On average, did customers’ spend differ between the period before and after the loyalty programme started?
• Do patients’ scores differ between the first and second visit?

Use cases by domain:

• survey: Compare each respondent’s rating of two products in the same questionnaire.
• ops: Compare each store’s weekly sales before and after a layout change.
• science: Compare each subject’s measurement before and after treatment.

Assumptions:

• Each row pairs two measurements of the same subject, held side by side in Field and Field2.
• Pairs are independent of each other.
• The per-row differences are roughly normally distributed; the two fields themselves need not be.
• Rows missing either value are dropped. The p-value is two-sided.

Use something else:

• TEST_WELCH when the two sets of values come from different, unrelated subjects.
• TEST_WILCOXON_SR when the per-row differences are skewed or have extreme values.
• TEST_ANOVA_RM when each subject is measured under three or more conditions.

Glossary: cohens-d, independence, normal-distribution, p-value, paired-data, t-statistic, two-tailed

Skill: op-test-paired-t

TEST_PEARSON_R

Measures how strongly two numeric fields rise and fall together along a straight line.

Level: intermediate

Also known as: pearson, pearson correlation, pearson's r

Questions it answers:

• Do customers who spend more also visit more often?
• Does study time go with higher test scores?

Use cases by domain:

• survey: Check whether satisfaction tracks likelihood to recommend.
• ops: See whether ad spend moves with weekly revenue.
• science: Relate dose to measured response.

Assumptions:

• The relationship is roughly a straight line.
• Pairs of values are independent of each other.
• A few extreme values can dominate the result.
• The p-value assumes both fields are roughly normal; with small samples that matters most.
• r is the covariance of the two fields (details.covariance, dividing by n - 1) divided by both sample standard deviations (the square roots of details.variance_x and variance_y, also n - 1), which makes it unit-free; AGG_STDDEV divides by n, so it does not reproduce r.

Use something else:

• TEST_SPEARMAN_R when the link is consistent but curved, or the values are ranks.
• TEST_KENDALL_TAU when the sample is small or has many tied values.
• TEST_CHISQ when both fields are categories.
• REG_OLS when you want to predict one field from several others.
• TEST_TREND when you want to know whether one ordered series keeps rising or falling.

Follow up with:

• REG_OLS when you want to predict one field from the other, or add more predictors.

Glossary: confidence-interval, correlation, covariance, independence, normal-distribution, outlier, p-value, r-squared, standard-deviation

Skill: op-test-pearson-r

TEST_PROP_Z

Checks whether the rate of one outcome, such as conversion, differs between two groups.

Level: basic

Also known as: two-proportion z test, prop.test

Questions it answers:

• Does the new checkout convert at a different rate from the old one?
• Is the share of promoters different between the two regions?

Use cases by domain:

• survey: Compare the share answering yes between two segments.
• ops: Compare conversion rates between two experiment arms.
• science: Compare response rates between treatment and control.

Assumptions:

• Rows are independent of each other, within and across the two groups.
• Each group needs at least 10 successes and 10 failures for the normal approximation.
• The p-value is two-sided; the confidence interval on the rate difference is the simple Wald interval, which is rough near 0% or 100%.

Use something else:

• TEST_FISHER_EXACT when some groups have only a handful of successes or failures.
• TEST_CHISQ when there are more than two groups or more than two outcomes.
• TEST_WELCH when the outcome is a numeric measure rather than a yes/no rate.

Glossary: cohens-h, confidence-interval, independence, normal-distribution, p-value, two-tailed, z-score

Skill: op-test-prop-z

TEST_SHAPIRO_WILK

Checks whether a numeric field looks normally distributed, overall or within each group.

Level: intermediate

Also known as: shapiro test, shapiro.test

Questions it answers:

• Is response time roughly bell-shaped, or clearly skewed or heavy-tailed?
• How far do scores in each group depart from a normal shape?

Use cases by domain:

• survey: Describe how far an index score departs from a normal shape, alongside a plot of it.
• ops: Check whether processing times are bell-shaped or heavily skewed.
• science: Check the normality assumption per treatment group.

Assumptions:

• Rows are independent of each other.
• Needs at least 3 values per group; the p-value is only calibrated for 5 to 5000 values, so treat it as advisory below 5 or above 5000 (Pulse warns).
• With few rows it has low power and can miss real departures; with very large samples even trivial departures are flagged, so look at the shape too.
• Pulse computes the Shapiro-Francia form, a close approximation to Shapiro-Wilk; with SplitBy the headline is the group that departs most.

Use something else:

• TEST_KS when you compare the distributions of two groups with each other.
• TEST_BROWN_FORSYTHE when you want to check whether groups have equal spread.

Follow up with:

• TEST_MANN_WHITNEY_U when the shape is far from normal and you planned a two-group t-test.
• TEST_KRUSKAL_WALLIS when the shape is far from normal and you planned an ANOVA across three or more groups.

Glossary: goodness-of-fit, independence, kurtosis, normal-distribution, null-hypothesis, p-value, skew, statistical-power, statistical-significance, test-statistic

Skill: op-test-shapiro-wilk

TEST_SPEARMAN_R

Measures how consistently one numeric field rises (or falls) as the other rises, using ranks, so the link need not be a straight line.

Level: intermediate

Also known as: spearman, spearman's rho, spearman rank correlation

Questions it answers:

• Do higher-ranked products also tend to sell more, even if not in proportion?
• Does satisfaction tend to rise with tenure?

Use cases by domain:

• survey: Relate two rating-scale answers to each other.
• ops: Check whether skewed order size tends to grow with customer age.
• science: Relate dose to a response that levels off.

Assumptions:

• Pairs of values are independent of each other.
• It detects monotonic links (one keeps rising as the other rises, or falls); a U-shaped link can score near zero.
• The p-value uses a t approximation that weakens with small samples or many tied values.

Use something else:

• TEST_PEARSON_R when you want the strength of a straight-line link.
• TEST_KENDALL_TAU when the sample is small or many values are tied.
• TEST_CHISQ when both fields are categories.
• TEST_TREND when you want to know whether one ordered series keeps rising or falling.

Follow up with:

• REG_OLS when the link looks roughly straight and you want to predict one field from the other.

Glossary: correlation, independence, monotonic-trend, outlier, p-value, rank, spearman-rho, ties

Skill: op-test-spearman-r

TEST_T

Checks whether a numeric field’s average differs from a target value, or between two groups (Welch’s version).

Level: intermediate

Also known as: t test, one-sample t test

Questions it answers:

• Is the average order value different from our target of 50?
• Do new and returning customers spend different amounts on average?

Use cases by domain:

• survey: Compare mean satisfaction between two regions, or against a target score.
• ops: Check average handling time against a service-level target.
• science: Compare the mean response of a treatment group with a control group.

Assumptions:

• Rows are independent of each other, within and across groups.
• The two-group version uses Welch’s correction, so the groups need not have equal variances.
• The average is roughly normally distributed: safe for large groups, risky for small skewed ones.
• The p-value is two-sided: it looks for a difference in either direction.

Use something else:

• TEST_PAIRED_T when the same subjects are measured twice, such as before and after.
• TEST_MANN_WHITNEY_U when two groups are compared and the measure is heavily skewed, has extreme values or is only ordered.
• TEST_ANOVA_WELCH when there are three or more groups.
• TEST_PROP_Z when the outcome is a yes/no rate rather than a numeric measure.
• TEST_WELCH when the request is always a two-group comparison and should say so by name.
• TEST_Z_TWO_SAMPLE when both groups are large and reporting conventions call for a z-test.

Glossary: cohens-d, confidence-interval, homogeneity-of-variance, independence, normal-distribution, p-value, t-statistic, two-tailed, variance

Skill: op-test-t

TEST_TREND

Checks whether an ordered series, such as monthly totals, tends to keep rising or keep falling.

Level: intermediate

Also known as: mann-kendall test

Questions it answers:

• Is monthly churn creeping up?
• Has weekly average wait time been falling over the year?

Use cases by domain:

• survey: Check whether wave-on-wave satisfaction keeps moving one way.
• ops: Check whether a smoothed daily error rate is drifting.
• science: Check whether yearly readings show a steady drift.

Assumptions:

• It runs on result rows (a post-test) in OrderBy order, such as a grouped or windowed series.
• Successive points are independent; seasonality or autocorrelation makes the p-value too small.
• It detects a monotonic trend, not a straight-line slope and not which periods differ.
• The p-value is a two-sided normal approximation; treat it as advisory below about 10 points (Pulse warns below 8).

Use something else:

• REG_OLS when you want the size of a straight-line slope.
• TEST_WELCH when you compare the averages of two specific periods.
• TEST_KENDALL_TAU when you relate two numeric fields rather than one series in order.

Glossary: independence, kendall-tau, monotonic-trend, non-parametric, p-value, test-statistic, two-tailed

Skill: op-test-trend

TEST_TUKEY_HSD

After an ANOVA, compares every pair of group averages to find which ones differ, keeping the overall false-alarm rate in check.

Level: advanced

Also known as: tukey hsd, tukey's honest significant difference

Questions it answers:

• Which regions differ from each other in average spend?
• Which treatment arms have different average outcomes?

Use cases by domain:

• survey: Find which age bands differ in mean satisfaction after an overall ANOVA.
• ops: Find which warehouses differ in average delivery time.
• science: Find which fertiliser treatments differ in mean yield.

Assumptions:

• Rows are independent of each other, within and across groups.
• Groups have similar variances, as in the standard ANOVA.
• It reads per-group averages and counts from result rows plus ms_within and df_within from a preceding TEST_ANOVA_F.
• Its p-values are already adjusted for multiple comparisons across every pair (they hold the family-wise error), using the studentized range (Tukey-Kramer for unequal group sizes); a request-level or instance multiplicity correction skips it, so they are never adjusted twice.

Use something else:

• TEST_ANOVA_F when you only need to know whether any group differs.
• TEST_WELCH when groups have clearly unequal spread: run pairwise Welch tests with multiplicity set on the request to adjust for multiple comparisons.
• TEST_MANN_WHITNEY_U when the measure is heavily skewed or only ordered: run pairwise Mann-Whitney tests with multiplicity set on the request to adjust for multiple comparisons.

Glossary: alpha, family-wise-error, homogeneity-of-variance, independence, multiple-comparisons, p-value, post-hoc-test, studentized-range

Skill: op-test-tukey-hsd

TEST_WELCH

Checks whether the average of a numeric field differs between two groups, without assuming they have equal spread.

Level: intermediate

Also known as: welch's t test, unequal variances t test

Questions it answers:

• Do customers in the two pricing arms spend different amounts on average?
• Is average delivery time different between the two depots?

Use cases by domain:

• survey: Compare mean satisfaction between two customer segments.
• ops: Compare average order value between two sales channels.
• science: Compare the mean outcome of treatment and control groups of different sizes.

Assumptions:

• Rows are independent of each other, within and across the two groups.
• Each group’s average is roughly normally distributed; large groups make this safe.
• Unequal variances are allowed: the degrees of freedom are adjusted (Welch-Satterthwaite).
• The p-value is two-sided.

Use something else:

• TEST_PAIRED_T when the same subjects are measured twice, such as before and after.
• TEST_MANN_WHITNEY_U when the measure is heavily skewed, has extreme values or is only ordered.
• TEST_ANOVA_WELCH when there are three or more groups.
• TEST_T when you compare one group’s average with a fixed target value.
• TEST_KS when you care about the whole shape of the two distributions, not just the average.

Glossary: cohens-d, degrees-of-freedom, homogeneity-of-variance, independence, normal-distribution, p-value, t-statistic, two-tailed

Skill: op-test-welch

TEST_WILCOXON_SR

Rank-based check of whether paired before/after values tend to shift in one direction, without assuming a bell curve.

Level: intermediate

Also known as: wilcoxon signed-rank test, signed-rank test

Questions it answers:

• Did each customer’s rating tend to go up after the redesign?
• Do patients’ skewed symptom scores tend to fall between visits?

Use cases by domain:

• survey: Compare each respondent’s ratings of two concepts on a short scale.
• ops: Compare each machine’s skewed downtime before and after maintenance.
• science: Compare each subject’s non-normal measurement before and after treatment.

Assumptions:

• Each row pairs two measurements of the same subject, held side by side in Field and Field2.
• Pairs are independent of each other.
• The differences are roughly symmetric around their centre; the test asks whether that centre is zero.
• Pairs whose values are equal are dropped; the p-value is a two-sided large-sample approximation and needs at least 6 pairs that differ.

Use something else:

• TEST_MANN_WHITNEY_U when the two sets of values come from different, unrelated subjects.
• TEST_PAIRED_T when the per-row differences are roughly normal and you want the average change.

Glossary: continuity-correction, independence, non-parametric, normal-distribution, p-value, paired-data, rank, rank-biserial, test-statistic, ties, two-tailed

Skill: op-test-wilcoxon-sr

TEST_Z_TWO_SAMPLE

Large-sample check of whether two groups differ in average, with the p-value read from the normal distribution.

Level: intermediate

Also known as: two-sample z test

Questions it answers:

• In a large survey, do the two regions differ in average score?
• Across thousands of sessions, is average basket size different between the two app versions?

Use cases by domain:

• survey: Compare mean ratings between two large respondent groups where reporting conventions call for z.
• ops: Compare average spend between two high-traffic storefronts.
• science: Compare means of two large samples where the t and normal tails agree.

Assumptions:

• Rows are independent of each other, within and across the two groups.
• Both groups are large; with small groups the normal p-value comes out too small.
• Unequal variances are allowed: the standard error is the same as Welch’s t-test.
• The p-value is two-sided.

Use something else:

• TEST_WELCH when either group is small; the t-distribution p-value is more honest there.
• TEST_PROP_Z when the outcome is a yes/no rate.
• TEST_PAIRED_T when the same subjects are measured twice.

Glossary: independence, normal-distribution, p-value, sample-size, standard-error, two-tailed, variance, z-score

Skill: op-test-z-two-sample