Row Weighting

Audience: analysts and embedders running survey or sampled data, where each row stands for a different number of people, accounts or events.

Name a weight column once and every weight-aware figure in the request counts each row as w rows: counts become Σw, sums Σw·x, means Σw·x / Σw, and crosstab cells, margins and shares follow. A request that names no weight produces exactly the output it always did, byte for byte, and format_version stays "1.1".

Where a weight comes from

There are three places to name one. For each slot Pulse takes the first that applies:

1. the slot’s own weight (an explicit null opts that slot out, and resolution stops there);
2. the request’s weight;
3. the engine default, pulse.Options.DefaultWeight;
4. none.

{
  "cohort": {"filename": "survey.pulse"},
  "weight": {"field": "wt", "kind": "probability"},
  "groups": [{"type": "GROUP_CATEGORY", "field": "region"}],
  "aggregations": [
    {"type": "AGG_COUNT", "field": "id", "label": "weighted_n"},
    {"type": "AGG_COUNT", "field": "id", "label": "respondents", "weight": null},
    {"type": "AGG_AVERAGE", "field": "spend", "label": "mean_spend"}
  ]
}

Here weighted_n and mean_spend are weighted and respondents is the raw count of respondents.

A per-slot weight is a field name ("wt"), a {field, kind} object or null. The slot is on aggregations (which covers the crosstab cell and each margin_aggregations entry), tests and post-tests, regressions, attributes, overlays, groups and both crosstab axes. Windows have no slot. Compose and process chains have no top-level weight: each inner request carries its own. Facets are never weighted, and DefaultWeight does not reach them.

From Go:

req.Weight = &types.WeightSpec{Field: "wt"}               // request weight
req.Aggregations[1].Weight = types.NullSlotWeight()       // opt one slot out
req.Aggregations[2].Weight = types.SlotWeightField("w2")  // a different column

p, err := pulse.New(pulse.Options{
    DefaultWeight: &types.WeightSpec{Field: "wt", Kind: types.WeightKindFrequency},
})

types.SlotWeight’s zero value means absent (inherit). NullSlotWeight(), SlotWeightField(f) and SlotWeightOf(spec) build the other two states. The distinction survives JSON, request hashing, MCP and the payload schema.

The weight column must be an unsigned-integer (u4 to u64) or float (f32, f64) field of the cohort. Derived columns are not accepted, but a joined, prefixed name is. Any other type, an empty field or an unknown kind is PROCESSING_CONFIG; an unknown name is SERVICE_VALIDATION. pulse.New checks DefaultWeight up front, but judges its field only on a request where it applies, so an instance default never breaks a cohort that lacks the column for a request that does not use it.

Probability or frequency

Weighted median and percentile follow Hmisc wtd.quantile. Frequency weights give R’s / numpy’s type 7 (linear) on the duplicated rows. Probability weights are first rescaled to sum to the row count (normwt = TRUE), which makes the figure scale-invariant. Cumulative weights within 1e-9 of an integer are snapped to it, so the answer does not depend on float summation order.

Invalid weights

Invalid weights are never coerced:

Each weighted slot reports its exclusions as n_weight_invalid. Each response carries one PULSE_WEIGHT_INVALID_ROWS warning per weight column, with details.by_reason counting null, negative, nan_inf and non_integer_frequency. Under Options.Strict (--strict) that warning is an error.

What a weight applies to

Check the manifest rather than this page: every operator that computes a weighted figure carries weight_kinds — ["frequency", "probability"] when it is weighted under both kinds, ["frequency"] when only frequency weights have a standard weighted form — with weight_aware: true beside it. Extension operators declare weight_aware (they have no weight_kinds; see the last section).

Weighted tests. A weighted moment test uses the frequency formula with its sample size read as Σw under kind: frequency (a weight of 3 is three identical rows) or as Kish’s effective n, n_eff = (Σw)²/Σw², under kind: probability. Means and r are the same under either kind; standard errors and degrees of freedom (which may be fractional) read that effective size. The result’s details.n stays the raw row count; sum_weights and, for probability weights, n_eff appear beside it in the same shape. A group whose n_eff falls below the test’s minimum warns PULSE_WEIGHT_LOW_NEFF (an error under strict mode). A probability-weighted Pearson correlation test whose n_eff is 2 or less has no degrees of freedom left and is refused with PULSE_TEST_INSUFFICIENT_N, its details naming n_eff. This is not design-based (strata / cluster) variance.

The two-proportion z-test reads each group’s rate as Σw of successes over Σw, with the group’s sample size read as above; successes becomes the weighted success total. The χ² independence test builds its table from Σw per cell: under kind: frequency it is the ordinary Pearson test on that table (identical to the expanded rows); under kind: probability the cell proportions are scaled to the table’s n_eff before the ordinary Pearson test, and the expected-count check, Cramér’s V and φ read that scaled table. This is a first-order Kish approximation, not the Rao-Scott correction survey software (survey::svychisq) applies. It matches Rao-Scott only when the weights are unrelated to both classifications; when the weights are constant within each category of one margin (a sample stratified on the row variable, say) it is conservative.

The one-way ANOVA F test credits each group with its own n_eff under probability weights (its own Σw under frequency weights), like the split t and Welch tests, so weights that differ between the groups do not move effective size from one group to another; df_within is the sum of the groups’ n_eff minus the number of groups.

The rank tests take frequency weights only. A row of weight w ranks as w identical rows (a run of tied values shares the mid-rank of its total weight), the tie corrections read those expanded tie sizes and Kendall’s τ-b counts a row pair w_i·w_j times, so each answer equals the unweighted test on the physically expanded rows. details.n and the other row counts stay raw; sum_weights appears beside n. Because every weighted row is itself a tie on the expansion, a frequency-weighted rank test often warns PULSE_TEST_TIES_DOMINATE. Pulse has no probability-weighted rank test: the design-based Mann-Whitney and Kruskal-Wallis tests of survey software (survey::svyranktest) need design information (strata, clusters) Pulse does not carry, and Spearman and Kendall have no standard probability-weighted form. A probability weight is refused with a message naming the kind; switch the weight to kind: frequency (if the weights are replication counts) or set the slot’s weight: null.

Fisher’s exact test, the Kolmogorov-Smirnov test and Brown-Forsythe take frequency weights only, for the same reason. Fisher’s exact test runs on the Σw table (details.contingency reports it); KS steps each empirical CDF by a row’s weight and reads its asymptotic p-value on the Σw group sizes; Brown-Forsythe takes each group’s median of the expanded rows (the frequency weighted median of AGG_MEDIAN) and runs the one-way ANOVA on the weighted absolute deviations. Each equals the unweighted test on the expanded rows. The KS statistic compares the two CDFs only after every row tied at a value is counted on both sides, which is also the unweighted rule (ties inside one sample no longer inflate D).

Weighted regressions. Ordinary least squares fits weighted least squares: the coefficients minimise Σw·(y − ŷ)² and are the same under either kind. Standard errors, degrees of freedom (N* − p − 1, possibly fractional), p-values, adjusted R² and the residual standard error use the effective size N* — Σw under kind: frequency, Kish’s n_eff under kind: probability. Under frequency weights the fit equals the one on the expanded rows. R’s lm(weights = w) gives the same coefficients but counts the rows, not Σw or n_eff, for its residual degrees of freedom, so its standard errors differ. Ridge, lasso and elastic net scale the penalty by Σw, so their coefficients do not change when every weight is multiplied by the same constant, and they match glmnet(weights = w) under frequency weights. Their standard errors stay plug-in approximations, now on N*. Bayesian linear regression takes frequency weights only: X’WX, X’Wy and Σw enter the conjugate posterior, which equals the posterior on the expanded rows. n_obs stays the raw row count. A weighted result adds sum_weights and, for probability weights, n_eff. A probability-weighted fit whose n_eff is at or below the predictor count plus one warns PULSE_WEIGHT_LOW_NEFF: its residual degrees of freedom are zero or negative, so the standard errors, p-values, residual standard error and adjusted R² are reported as null (the coefficients and R² stay defined). A regression with resample or selection refuses any weight, because neither has a standard weighted form.

The GLM runs iteratively reweighted least squares with the row weights as prior weights on w* = w·N*/Σw, so its fit is R’s glm(weights = w*): the coefficients are the same under either kind, the standard errors come from (XᵀWX)⁻¹ and the deviance and null deviance sum w·d. Under frequency weights that is glm(weights = w) and equals the fit on the expanded rows. The dispersion stays fixed at 1 for every family, gamma included, so treat gamma standard errors with the same caution as unweighted ones. The regression attributes refit ordinary least squares with the slot’s weight and accept both kinds: the fitted value comes from the weighted coefficients, the residual is the raw y − ŷ, and the leverage is R’s hatvalues() on the weighted lm, the diagonal of W½X(XᵀWX)⁻¹XᵀW½. A row whose weight is zero or invalid stays out of the refit and gets leverage 0.

Weighted confidence bounds. AGG_CI_LOWER and AGG_CI_UPPER are weight-aware aggregators under either kind. The bound is the weighted mean ∓ z·√(s²/N*): s² is the sample variance on w* = w·N*/Σw, N* is Σw under kind: frequency (the bound on the expanded rows) or Kish’s n_eff under kind: probability, and z is the same normal critical value the unweighted bound uses (qnorm(1 − α/2), reported as t_critical), not a t on N* − 1. They stay mergeable, so they run on every streaming, shard and parallel-decode path.

Weighted scores and quantile buckets. ATTR_ZSCORE and ATTR_TSCORE score each row against the weighted mean and the weighted population standard deviation √(Σw(x − mean)²/Σw), so frequency and probability weights give the same scores. A row whose weight is zero or invalid adds nothing to the mean and spread but is still scored. GROUP_QUANTILE cuts its buckets at the weighted order statistics, by the same rule as the weighted percentile (probability weights rescaled to the row count, as in Hmisc wtd.quantile(normwt = TRUE)): each bucket holds as close to an equal share of the weight as whole rows allow. A heavy row is never split, so it lands whole in the higher bucket when it spans a cut. Bucket count, total_n and crosstab counts stay raw row counts. A row with no usable weight is bucketed by its value. The slot weight on a groups[i] or crosstab-axis quantile grouper opts it in or out like any other slot.

Weighted k-means. GROUP_KMEANS takes a frequency weight only: a row of weight w is w coincident points, so the k-means++ draws, the centroids and the within-cluster sums of squares are those of the expanded rows, and a cluster’s size is its total weight (count stays the raw row count). A row with a zero or invalid weight is assigned to its nearest centroid but does not move the fit. A probability weight is refused permanently (PULSE_WEIGHT_UNSUPPORTED, with details.reason).

Refused operators. The permanently refused row has no standard weighted form (or, for the post-tests, Tukey HSD and the trend test, reads aggregated result rows that carry no row weights), so it is refused rather than silently computed unweighted beside weighted figures. To run one of those operators in a weighted request, give its slot "weight": null; on an instance with a DefaultWeight, each such slot needs that opt-out. PULSE_WEIGHT_UNSUPPORTED says why in its details: reason for a permanent refusal, alternative when a weighted twin exists (the pairwise two-means z points at OVERLAY_PAIRWISE_WEIGHTED_TWO_MEANS_Z), and kind with supported_kinds: ["frequency"] when a frequency-only operator meets a probability weight. A weight-aware aggregator over a decimal128 field is also PULSE_WEIGHT_UNSUPPORTED, because the decimal path has no weighted form.

What the effective sample size does and does not do. Every weighted test, regression, confidence bound and inferential overlay reads one effective size: Σw for frequency weights, Kish’s n_eff for probability weights. Kish’s n_eff corrects for unequal weights only, and assumes the weights are unrelated to the outcome and to the comparison; for slopes, correlations, group contrasts and tables it is a rule of thumb. It knows nothing of strata, clusters or a finite population, so on a complex survey design the p-values and intervals are narrower than a design-based analysis (R survey, Stata svy) would give. Use Pulse’s figures to screen and describe, and a design-based tool when the design matters. In every weighted result, read n_eff (or sum_weights under frequency weights) as the sample size, never n.

pulse predict reports the outcome for every slot under data.weights[]: {slot, operator, field, kind, status, source}. status is applied, skipped_not_weight_aware, opted_out or none. source is slot, request, options or none. Predict refuses exactly what the run would refuse.

AGG_WEIGHTED_MEAN is the weighted AGG_AVERAGE. Its params.weight_field is shorthand for a slot weight with kind: probability and is now optional: without it, the slot inherits the request weight or the default.

Crosstabs and the unweighted base

A weighted crosstab recomputes cells, row, column and grand margins, and every normalization from the weighted rows, on both execution arms. CellCounts and the margin counts stay raw row counts. The standard survey table shows a weighted figure beside an unweighted base, and an auxiliary margin aggregation that opts out gives you that base:

{
  "weight": {"field": "wt"},
  "crosstab": {
    "rows":    [{"type": "GROUP_CATEGORY", "field": "region"}],
    "columns": [{"type": "GROUP_CATEGORY", "field": "segment"}],
    "cell":    {"type": "AGG_COUNT", "field": "id", "label": "weighted_n"},
    "margins": {"rows": true, "columns": true, "grand": true},
    "margin_aggregations": [
      {"type": "AGG_COUNT", "field": "id", "label": "base", "weight": null}
    ]
  }
}

A record reaches an auxiliary margin only if it reached a cell. So the base counts exactly the respondents the weighted cells beside it counted.

Overlays

Share and index overlays (share of row / column / total, index vs margin, total, prior, baseline, rolling mean, sibling, reference) read the host’s figures. On a weighted host they are therefore weighted, with no setting needed. The pairwise two-means z and probit t overlays refuse when a weight reaches the overlay itself: its own weight, the request’s, or the default. Its "weight": null opts it out. A host weighted only by its own slot weight does not trigger the refusal. OVERLAY_PAIRWISE_WEIGHTED_TWO_MEANS_Z is the weighted twin, built for weighted cells: its host is an AGG_WEIGHTED_MEAN cell or a weighted AGG_AVERAGE cell. It accepts both weight kinds, and its required n_basis param (weights for Σw, kish for n_eff) decides the sample size. weights needs frequency weights: on a cell weighted under kind: probability (an AGG_WEIGHTED_MEAN weight_field included) the weight sum is not a sample size, so n_basis: "weights" is refused with PROCESSING_CONFIG; use kish there.

The mean-comparison overlays (OVERLAY_T_CELL, OVERLAY_Z_CELL, OVERLAY_T_VS_REF, OVERLAY_Z_VS_REF, OVERLAY_PAIRWISE_WELCH_T) run weighted under both kinds. A host cell is weighted when its components carry sum_weights, whatever set the weight — including the cell’s own slot weight. Such a cell’s sample size is Σw (frequency) or n_eff (probability), never its raw row count, and its variance is recomputed from m2 on the rescaled weights; degrees of freedom may be fractional. The layer summary’s parameters add sum_weights (and n_eff under probability) over the cells the layer read. Before this release a host weighted only by its own slot weight ran these overlays on raw row counts, which gave standard errors that were too small; that was a bug and is fixed. On a weighted host the pairwise Welch t refuses an n_source that reads raw row counts (cell_n_unweighted, row_margin_n, column_margin_n), and cell_weight_sum under probability, with PROCESSING_CONFIG.

The χ² and proportion overlays also run weighted under both kinds. A weighted host’s cells and margins are weight sums (Σw). Under frequency the χ² overlays run Pearson’s test on the Σw table, which equals the test on the expanded rows. Under probability the table (for OVERLAY_CHISQ_MATRIX), each row (_ROW), each column (_COL) or the target table (_VS_REF) is first scaled to its Kish n_eff. This is a first-order Kish approximation, not the Rao-Scott correction survey packages apply, so expect it to differ from svychisq. The low-expected-count warning reads the scaled table. The proportion overlays read p̂ = Σw of the successes over Σw of the base, with the base’s n_eff (or Σw under frequency) as the sample size, never the payload’s Σw under probability. On a weighted host the pairwise proportion z omits n_source to read that sample size. Every unweighted count (raw rows, n_within, the distinct-key modes) is refused with PROCESSING_CONFIG, and cell_weight_sum / cell_value_weighted are accepted under frequency only. This is a change for requests that used cell_weight_sum over an AGG_WEIGHTED_MEAN cell with params.weight_field, which is probability. The Fisher exact cell overlay runs on the Σw table under frequency only. A probability weight on its host cell is refused with PULSE_WEIGHT_UNSUPPORTED, even when the overlay itself carries no weight.

Under probability the χ² and Compose proportion overlays read each host’s n_eff from its components. A host built with components disabled (disable_components, --no-components or Options.DisableComponents) has none to read, so those overlays are refused on it with PROCESSING_CONFIG naming the host, in predict and at run time. Before, they read Σw as the sample size without a warning. Turn components back on for that host, or use a frequency weight, where Σw is the sample size.

Reading weighted components

A weighted slot’s components add three floor keys beside {n, n_null}:

• sum_weights: Σw over the rows that contributed;
• n_eff: Kish effective sample size, (Σw)² / Σw², for probability weights only;
• n_weight_invalid: rows excluded for an invalid weight.

They appear only on weighted slots, so sum_weights being absent is the per-slot sign that a figure is unweighted. n, n_null, CellCounts, margin counts and grouper total_n stay raw counts. For the effective sample size of a probability-weighted figure, use n_eff, not n. A figure that is undefined under the weights (0/0 when every weight in a cell is zero) is written as null on the wire. See JSON Schema for both rules.

SPSS weights

A cohort imported from a .sav file with WEIGHT BY keeps that variable in its metadata sidecar. pulse cohort inspect, pulse_inspect and pulse_predict then report suggested_weight {field, source: "spss_sidecar", kind: "probability"}. The suggestion is never applied: name the field as weight yourself. SPSS WEIGHT BY replicates cases, so kind: "frequency" is usually the faithful reading. See cohort inspect.

Feature profiles

Row weighting is the feature capability:weighting. A profile that omits it removes the whole weight surface: every weight key is refused as unknown, the schema and manifest drop it, and pulse.New refuses Options.DefaultWeight with PULSE_FEATURE_PROFILE_DEPENDENCY. AGG_WEIGHTED_MEAN’s own weight_field keeps working. See Feature Profiles.

Extension operators

Aggregators, attributes and tests registered through pulse.Options.Extensions declare WeightAware and read the row weight with extend.Record.Weight(). A WeightAware extension runs under both weight kinds: there is no per-kind declaration, so an extension with no probability-weighted form reads the kind off its slot spec and refuses itself. Undeclared ones are skipped or refused (PULSE_EXTENSION_NOT_WEIGHT_AWARE). The recipe is in Extension Points.