Adding a Filterer

Embedding Pulse, not contributing to it? This recipe adds a built-in in internal/processing/, which embedders cannot import. To add your own operator, implement the matching contract in the public extend package instead — see Extension Points.

Audience: Pulse internals contributors adding a new FILTER_* operator — a per-record predicate that decides whether the row participates in the downstream aggregation / grouper / crosstab.

The recipe mirrors the aggregator recipe; the filterer-specific moving parts are the predicate factory, the filter registry, and the universal floor for Response.Components.Filterers.

1. Declare the type constant

Add the new constant to types/types.go and the slice returned by types.AllFiltererTypes():

const (
    // ... existing constants ...
    FILTER_REGEX FiltererType = "FILTER_REGEX"
)

func AllFiltererTypes() []FiltererType {
    return []FiltererType{
        // ... existing entries, alphabetised ...
        FILTER_REGEX,
    }
}

2. Implement and register

Implement the filterer in internal/processing/. Each filterer is a factory that returns a FiltererBuilder — register it in filtererRegistry (internal/processing/registry.go).

The builder produces a per-record FilterFunc that returns (keep bool, err error). The streaming Process path invokes the filter func once per row; the buffered path invokes it once per materialised record.

3. Tests

Add tests in internal/processing/filterer_test.go before the implementation. Cover both the include and exclude branches, the null-handling contract, and any error path.

4. Declare the capability metadata

Add a row to internal/descriptor/capabilities_filterers.go with the filterer’s params, accepted field types, and the ComponentSchema that follows. TestManifestOperatorsComplete enforces a capability row per registered filterer.

5. Declare the ComponentSchema (Response.Components contract)

Every registered filterer MUST declare a ComponentSchema on its capability row, with a Mergeability class (Mergeable / Partial / None).

In v1 every built-in filterer is Mergeable with an empty operator-specific keys slice — the orchestrator fills the universal floor {n_in, n_out, n_null_input} from the filter pass’s record-walker counters, and no per-filter extras are emitted today:

{
    Name:            string(types.FILTER_REGEX),
    Category:        "filterer",
    Description:     "Keep records whose field matches the supplied regular expression.",
    AcceptsTypes:    stringFieldTypes,
    ComponentSchema: filterSchema(Mergeable),
},

The filterSchema helper prepends the universal floor automatically — do not list those keys in your extra slice. The interface MetaFilterer (internal/processing) exists for parity with MetaAggregator / MetaGrouper and leaves room for future per-filter specifics (n_below / n_above for FILTER_RANGE, per-value n for FILTER_INCLUDE / FILTER_EXCLUDE). When a built-in eventually implements it, declare the extra keys in filterSchema(...) and add the Components() method on the filterer struct plus a compile-time sentinel:

var _ MetaFilterer = (*regexFilterer)(nil)

The full Response.Components contract — universal floor semantics, streaming behaviour, parity overlay reads — lives in the response-components skill; extension-side parity lives in Extension Points.

6. Update the aggregation-design skill

Add a section in the filtering portion of skills/aggregation-design.md covering the new filterer’s semantics, parameter shape, and the null- input contract.

7. Update CLAUDE.md

There is no registered-filterer count or list in CLAUDE.md to update — CLAUDE.md never hardcodes registered counts; the manifest is the source of truth. The Update Demand operator row (.claude/reference/update-demand.md) requires instead the atomic skill skills/op-filter-<kebab>.md, the capability declaration in internal/descriptor/capabilities_filterers.go — and an internal/examples/<dir>/*.json example whose _meta.operators names the operator (TestEveryOperatorHasAnExampleTag). Edit CLAUDE.md only if the operator introduces a contract it states directly, and mind TestClaudeMdSizeBudget — long-form prose belongs in .claude/reference/.

8. Run the gates

go test ./internal/skills/ -run TestSkillsCoverAllComponents
go test ./internal/descriptor/ -run TestManifestOperatorsComplete
go test ./internal/processing/ -run TestFilter

The Update Demand row for filterers covers all of these in one PR; see The Update Demand.