Pulse

Pulse is a self-describing, high-performance tabular data processing engine. It ships as a Go library (github.com/frankbardon/pulse) and as a single CLI binary (bin/pulse). Every .pulse file carries its own schema in the header, so consumers (programs, agents, and humans) can discover what a file contains without an external catalog.

The library is the primary deliverable. The CLI is a thin adapter that exposes the same operations on the command line, and an embedded MCP server (pulse mcp) exposes them to LLM agents.

Where to go from here

LLM-facing surface

LLM agents do not read this site. Pulse exposes a Model Context Protocol server (pulse mcp) and ships 19 embedded skills under skills/ that LLMs load on demand via the pulse_skills_list and pulse_skills_get tools. The skill voice is MCP-only (tool calls, JSON payloads). This site is the human-facing counterpart — same engine, different idiom.

See How LLMs Use Pulse for a short pointer table.

Source of truth

The authoritative architectural contract for Pulse lives in the repository’s CLAUDE.md. When this site and CLAUDE.md disagree, CLAUDE.md wins; please open an issue.

• Repository: https://github.com/frankbardon/pulse
• Hosted docs: https://frankbardon.github.io/pulse/

Installation

Audience: new users who want a working pulse binary on their PATH.

This page walks through installing Pulse, the prerequisites it needs, and how to verify the install. Pulse is distributed as a single static Go binary; there is no installer, no daemon, and no config file.

LLM agents using MCP: see the getting-started skill via pulse_skills_get — it covers session bootstrap rather than local install.

Prerequisites

go.mod is the source of truth for the supported Go version; if it drifts from this page the go.mod value wins.

Install with go install

The fastest path on a developer machine:

go install github.com/frankbardon/pulse/cmd/pulse@latest

This drops a pulse binary at $(go env GOBIN) (typically ~/go/bin). Make sure that directory is on your PATH.

Pin a specific release by replacing @latest with a tag:

go install github.com/frankbardon/pulse/cmd/pulse@v0.2.0

Build from source

The same binary, built reproducibly from a checkout:

git clone https://github.com/frankbardon/pulse.git
cd pulse
make build
# Binary at ./bin/pulse

The Makefile is documented in CLAUDE.md → Build / Dev / Test Workflow; the relevant targets are make build, make test, make lint, and make cover.

Configure the data directory

Pulse reads and writes .pulse files under a base directory called PULSE_DATA_DIR. Most commands accept absolute paths and will work without it, but pulse mcp requires the variable so the MCP server can enumerate cohorts:

export PULSE_DATA_DIR=/var/data/pulse

The repo Makefile auto-loads a .env file from the repo root, so you can also drop PULSE_DATA_DIR=... there for local development.

PULSE_DATA_DIR is the only required environment variable. See Flag Reference for the full list of CLI flags and environment knobs.

Verify

pulse --version
pulse --json | head -20

pulse --json prints the root manifest — the full self-description of commands, components, field types, and embedded skills. If you see a top-level format_version: "1.0" envelope, the install is working.

Where to go next

• New to the file format and vocabulary? Your First Cohort
• Want a quick map of every command? CLI Tour
• Embedding Pulse in a Go program? Go API Overview
• Wiring Pulse into an MCP-aware client? pulse mcp

Your First Cohort

Audience: new CLI users. This is a five-minute tour: import a CSV, inspect the resulting .pulse file, run an aggregation, and export the result back.

LLM agents using MCP: the equivalent tour for an agent is the getting-started skill, fetched via pulse_skills_get. That skill speaks in tool calls and JSON payloads; this page speaks in shell commands.

1. Pick a CSV

For this walkthrough we’ll assume a file called sales.csv with columns like:

order_id,region,product,units,revenue,sold_on
1,west,widget,3,29.97,2024-01-04
2,east,gadget,1,19.99,2024-01-04
3,west,widget,7,69.93,2024-01-05
...

Any CSV with a header row works. Pulse also imports TSV, NDJSON, JSON-array, Parquet, Arrow IPC, and Excel — see Flag Reference for per-format flags.

2. Import to a .pulse file

pulse import csv --input sales.csv --output sales.pulse

Pulse samples up to 500 rows by default to infer a schema (you can change that with --sample-rows). Each column gets a typed binary representation and, if it looks like a low-cardinality string, a categorical dictionary.

Want to control the schema explicitly? Generate a template, edit it, and re-import:

# Editable schema template
pulse import schema-template sales.csv > sales.schema.json

# Edit sales.schema.json — set types, add descriptions
# Then import with the schema
pulse import csv --input sales.csv --schema sales.schema.json --output sales.pulse

See Field Types for the type catalog and Dictionary Blocks for how categoricals are encoded.

3. Inspect

The .pulse file is fully self-describing. Read it back:

pulse cohort inspect sales.pulse

Output is a table of fields, their types, and the description string stored in the header. Add --json for the structured envelope, or --full-dict to print every categorical entry instead of truncating after 100.

pulse cohort inspect sales.pulse --json

The envelope is documented in pulse cohort inspect.

4. Validate a request before running it

Pulse separates validation from execution. Write a tiny request file:

{
  "cohort": {"filename": "sales.pulse"},
  "groups": [{"type": "GROUP_CATEGORY", "field": "region"}],
  "aggregations": [
    {"type": "AGG_COUNT", "field": "order_id", "label": "orders"},
    {"type": "AGG_SUM", "field": "revenue", "label": "total_revenue"}
  ]
}

Save it as request.json, then check whether it makes sense against the cohort’s schema:

pulse api predict --request request.json

You’ll see Valid: true, the schema’s field count, and any warnings (e.g., aggregating something numeric on a categorical field). Predict never reads record data, so it’s safe to iterate on a request without touching a multi-GB cohort.

See pulse api predict and the debugging-with-predict skill for the full predict loop.

5. Execute

pulse api process --request request.json --json

The response is wrapped in the standard envelope (format_version, data, errors, warnings). data carries the result rows and a metadata block with total_rows and filtered_rows.

If your result is large, swap --json for --stream to receive rows as NDJSON, one line at a time — useful for pipelines that don’t want to buffer the whole result. See Streaming & ProcessStream for which request shapes actually stream end-to-end inside the engine vs which buffer.

6. Export

You’re done with the .pulse file? Export to whatever your downstream tool understands:

pulse export csv     --input sales.pulse --output sales.out.csv
pulse export parquet --input sales.pulse --output sales.out.parquet
pulse export excel   --input sales.pulse --output sales.out.xlsx

To skip the intermediate .pulse entirely and convert in one shot, use pulse convert source.csv target.parquet — see the top-level README for the full convert recipe.

What you didn’t see

• Compose: batch multiple requests in one call — pulse api compose.
• Sample / Facet: cheap read-only probes — api sample, api facet.
• Window / Feature / Test operators: pull from the skill pack (window-operations, feature-engineering, statistical-testing) via pulse skills show <name>.

For a full map of the CLI, see the CLI Tour.

CLI Tour

Audience: anyone who wants a map of every pulse subcommand before diving into per-command details.

This page is a one-liner index of the CLI tree. Each row links to its detailed chapter where applicable; commands that are minor variants of each other (per-format import/export leaves, per-leaf shard maintenance commands) are listed compactly.

LLM agents using MCP: there is no equivalent skill — agents drive Pulse through MCP tools, not the CLI. Start at the getting-started skill instead.

Top-level groups

pulse [--json] [--slim]
├── import      Tabular → .pulse (csv, tsv, ndjson, jsonarray, parquet, arrow, excel, auto)
├── export      .pulse  → tabular (same format set)
├── convert     Tabular → tabular, with .pulse as the transparent middle
├── cohort      Inspect or filter an existing .pulse file
├── api         Processing operations (process, process-chain, compose, predict, sample, facet)
├── shard       Build and maintain shard archives
├── synth       Generate synthetic cohorts (from-schema, from-profile)
├── profile     Capture a statistical profile of a cohort
├── skills      Read the embedded LLM skill pack
├── examples    Search and fetch the embedded runnable request library
├── errors      Look up an error code's message and recovery fixups
└── mcp         Run the Model Context Protocol server over stdio

Bare pulse --json prints the self-describing root manifest — commands, components, field types, and skill metadata in one envelope. Pass --slim to drop prose descriptions for size-sensitive clients.

End-to-end worked tour

A complete read-aggregate cycle, one command per stage:

# 1. Import a CSV into a managed .pulse handle (TTL sidecar tracked).
pulse import auto sales.csv --handle sales --ttl 30d

# 2. Read the cohort's schema — types, descriptions, dictionaries.
pulse cohort inspect sales.pulse

# 3. Validate a request against the schema without running it.
cat > req.json <<'EOF'
{
  "cohort": {"filename": "sales.pulse"},
  "groups":       [{"type": "GROUP_CATEGORY", "field": "region"}],
  "aggregations": [{"type": "AGG_SUM",        "field": "revenue", "label": "total"}]
}
EOF
pulse api predict --request req.json --json | jq '.data | {valid, streamable, defaults_applied}'

# 4. Execute the same request.
pulse api process --request req.json --json

api predict reads only the header and schema, so it stays cheap even on multi-GB cohorts; the streamable / streamable_reasons fields tell you whether the request will run through the single-pass streaming path or be buffered. See Performance Notes.

Pipeline order

Inside api process, operators run in a fixed sequence:

Load -> Features -> Filter -> Attributes -> Group -> Aggregate -> Windows -> Sort -> Output

Features run before filterers (so derived columns are addressable as filter, group, attribute, and window inputs); windows run after aggregation, on the post-aggregate row set.

API operations

The “processing facade” — these are the operations exposed via the Go library API and the MCP tool set.

api process-chain accepts a ChainRequest whose stages each see the previous stage’s rows as their input cohort. Mergeable-only at v1 — non-mergeable stages fail with PULSE_CHAIN_NOT_MERGEABLE so callers can fall back to per-stage api process calls.

Cohort lifecycle

Import / export / convert

pulse import <format> and pulse export <format> share the same flag shape per format (--input, --output, --schema for import). Supported formats today:

csv · tsv · ndjson · jsonarray · parquet · arrow · excel

Each format has a per-leaf command (e.g. pulse import csv). Run pulse import --help or pulse export --help for the full list.

pulse import auto SOURCE auto-detects the source format and converts into the managed .pulse pool under PULSE_IMPORTS_DIR, with a TTL sidecar (--ttl 7d by default, pin to opt out). Use pulse import list to see managed handles and pulse import drop HANDLE to remove one.

pulse convert SOURCE TARGET chains import + export with no intermediate file unless --keep-pulse PATH is passed. Format is auto-detected from extensions.

Shard archives

A .pulse path may resolve to either a single-file cohort or a shard archive (uncompressed Zip64 carrying one canonical _schema.pulse entry plus N standalone shard payloads). Every facade method (Process, Compose, Sample, Facet, Inspect, Predict, ProcessStream) operates transparently on the union of shards.

The archive.pulse#shard.pulse anchor opens one shard as a one-shard cohort. See Internals → Managing Shard Archives.

Synthetic data

Self-description & LLM surface

Cross-cutting flags

Most leaves accept --json (envelope output), --no-defaults (turn off smart operator-type inference — see api predict → Smart defaults), and --echo-request (include the normalized request on envelope.request). Full list: Flag Reference.

The single environment variable to know is PULSE_DATA_DIR — see Installation.

pulse api process

Audience: CLI users running a single processing request against a cohort.

pulse api process executes one types.Request against a .pulse file and prints the result. It’s the most-used leaf in the binary.

LLM agents using MCP: the equivalent surface is the pulse_process MCP tool — see skills/session-bootstrap.md and skills/aggregation-design.md for request authoring guidance.

Synopsis

pulse api process --request FILE [--json] [--stream] [--no-defaults]
                                 [--strict] [--echo-request]

Flags

--stream and --json are mutually exclusive in spirit — --stream emits one JSON object per line; --json emits the full envelope.

--strict is the post-execute companion to pulse api predict --strict: predict refuses to declare the request valid in the face of any warning; process refuses to run it.

Request file shape

The request file is a types.Request serialised to JSON. Minimal example:

{
  "cohort": {"filename": "sales.pulse"},
  "aggregations": [
    {"type": "AGG_SUM", "field": "revenue", "label": "total_revenue"}
  ]
}

The full request grammar is one JSON object whose top-level keys mirror types.Request: cohort, filterers, features, attributes, groups, aggregations, windows, sort, tests, post_tests, outputs. A canonical filter-group-aggregate example:

{
  "cohort": {"filename": "data.pulse"},
  "filterers": [
    {"type": "FILTER_INCLUDE", "field": "status", "values": ["active"]}
  ],
  "groups": [
    {"type": "GROUP_CATEGORY", "field": "region"}
  ],
  "aggregations": [
    {"type": "AGG_COUNT",   "field": "id",    "label": "n"},
    {"type": "AGG_AVERAGE", "field": "score", "label": "mean_score"}
  ],
  "outputs": [{"format": "json"}]
}

The full grammar — windows, sort, tests, post-tests — is documented in types.Request; the LLM-facing companion is skills/aggregation-design.md.

Pipeline order

Pulse executes a request in a fixed sequence:

Load -> Features -> Filter -> Attributes -> Group -> Aggregate -> Windows -> Sort -> Output

Features run before filterers, so derived columns are addressable as filter, group, attribute, and window inputs. Windows run after aggregation, on the post-aggregate row set. Request.Sort runs last.

Smart defaults

When an aggregations[] or groups[] slot names a field but omits type, the engine infers the operator from the named field’s schema type at request time. --no-defaults (or pulse.Options{DisableDefaults: true}) turns this off and requires every slot to be source-of-truth. The full defaults table is documented on pulse api predict.

Output

Text mode (default)

Pretty-printed JSON of the Response struct: a data array of result rows plus a metadata block with total_rows, filtered_rows, and cohort_file.

--json

The standard envelope:

{
  "format_version": "1.1",
  "data": {
    "data": [ /* result rows */ ],
    "metadata": { "total_rows": 1000, "filtered_rows": 800, "cohort_file": "sales.pulse" },
    "components": {
      "aggregations": [{"label": "n", "n": 800, "n_null": 0}],
      "filterers":    [{"n_in": 1000, "n_out": 800, "n_null_input": 0}],
      "run":          {"total_records": 1000, "filtered_records": 800, "null_records": 0}
    }
  },
  "errors": [],
  "warnings": []
}

data.components is the additive Response.Components slot documented in CLAUDE.md → Output Format Contract and skills/response-components.md. The slot is omitempty: a run that emits no components-shaped state (a streaming --stream chunk between the first and last on a non-mergeable aggregator, for example) marshals without the components key at all — byte-identical to the pre-0.20 wire shape. format_version stays at "1.0" because the slot is additive.

--stream

NDJSON of result rows, one per line. No envelope, no metadata footer. Pair with pulse api predict ahead of time to confirm Streamable=true; predict-buffered shapes still emit via this path, but they materialise inside the engine first.

Exit codes

Examples

Quick aggregation

cat > req.json <<'EOF'
{
  "cohort": {"filename": "sales.pulse"},
  "aggregations": [{"type": "AGG_COUNT", "field": "id", "label": "n"}]
}
EOF

pulse api process --request req.json

Filter, group, and aggregate

cat > req.json <<'EOF'
{
  "cohort": {"filename": "sales.pulse"},
  "filterers": [{"type": "FILTER_RANGE", "field": "revenue", "values": ["100", "10000"]}],
  "groups":    [{"type": "GROUP_CATEGORY", "field": "region"}],
  "aggregations": [
    {"type": "AGG_COUNT",   "field": "id",      "label": "orders"},
    {"type": "AGG_AVERAGE", "field": "revenue", "label": "avg_rev"}
  ]
}
EOF

pulse api process --request req.json --json

Stream rows into a downstream pipeline

pulse api process --request req.json --stream | \
    jq -c 'select(.avg_rev > 500)'

Related

• pulse api compose — batch of requests in one call
• pulse api predict — validate without executing
• pulse api sample — quick row preview
• Library: pulse.New & Options — the Go-side equivalent of --no-defaults
• Library: Streaming & ProcessStream — what streams vs what buffers

pulse api compose

Audience: CLI users executing a batch of related requests in one call.

pulse api compose runs multiple types.Request entries against one or more cohorts. The whole batch is one ComposedRequest; the engine can run the entries sequentially or in parallel against a bounded worker pool.

LLM agents using MCP: see the pulse_compose MCP tool and the compose-requests skill.

Synopsis

pulse api compose --request FILE [--json] [--stream]
                                  [--parallel N] [--no-fail-fast]
                                  [--no-defaults] [--echo-request]

Flags

Request file shape

{
  "requests": [
    { "cohort": {"filename": "sales.pulse"}, "aggregations": [...] },
    { "cohort": {"filename": "sales.pulse"}, "groups":       [...] },
    { "cohort": {"filename": "ops.pulse"},   "filterers":    [...] }
  ]
}

Each requests[i] is a full types.Request. Slots are independent — they may target different cohorts, use different operators, etc.

When the cohort is a shard archive, each request can target the whole archive (fan-out across every shard) or one shard via the archive.pulse#shard.pulse anchor — Compose preserves slot order regardless:

{
  "requests": [
    {"cohort": {"filename": "Q1_2019.pulse"},                "aggregations": [...]},
    {"cohort": {"filename": "Q1_2019.pulse#20190101.pulse"}, "aggregations": [...]},
    {"cohort": {"filename": "wave_2018.pulse"},              "aggregations": [...]}
  ]
}

Output ordering

Responses come back in input order, regardless of --parallel. A worker that finishes early waits its turn before emitting. So data.responses[i] always corresponds to request.requests[i].

Parallel mode

--parallel N:

• 1 (default) — sequential Compose, equivalent to running each request through pulse api process in a loop.
• 0 — runtime.GOMAXPROCS workers.
• >1 — exactly N workers.

Workers share Pulse’s read-only registries; per-request stateful operators are constructed fresh. See Parallel Compose for full mechanics.

FailFast semantics

With --no-fail-fast unset (the default, fail-fast on):

• The first failing request cancels in-flight siblings.
• The command exits non-zero with the first error.

With --no-fail-fast:

• Every request runs to its own completion (or per-request timeout).
• Errors aggregate into a single SERVICE_INTERNAL error whose details.failed_indices lists the slot indices that failed.
• Successful slots populate data.responses[]; failed slots are null.

Output

--json

{
  "format_version": "1.1",
  "data": {
    "responses": [ /* one Response per slot, in input order */ ],
    "overlays":  [ /* one OverlayLayer per ComposedRequest.overlays spec; omitted when no Compose overlays */ ]
  },
  "errors": [],
  "warnings": []
}

data is a ComposedResponse object (since the v0.21.0 Compose facade lift). Each data.responses[i] is a full Response, so per-slot data.responses[i].components follows the same additive contract as api process. data.overlays[i] carries the post-fold Compose-host overlay layer plus its per-layer warnings[] diagnostics (OverlayWarning{code, message, details}); the slot is omitempty so overlay-free Compose responses are byte-identical to the legacy []*Response wire shape that shipped under format_version: "1.0".

--stream

{"index": 0, "row": { ... }}
{"index": 0, "row": { ... }}
{"index": 1, "row": { ... }}

The index field identifies which slot’s request produced each row.

Exit codes

Examples

Sequential batch

pulse api compose --request batch.json --json

Parallel with 4 workers, aggregated errors

pulse api compose --request batch.json --parallel 4 --no-fail-fast --json

Stream a parallel batch into a downstream consumer

pulse api compose --request batch.json --parallel 4 --stream | \
    jq -c 'select(.index == 2)'

Related

• pulse api process — single-request leaf
• Library: Parallel Compose — Go-side equivalents
• skills/compose-requests.md (LLM) — request composition patterns

pulse cohort inspect

Audience: CLI users reading a .pulse file’s schema without running a query — the human-side counterpart of the inspect library method and the pulse_inspect MCP tool. Defined in internal/cli/cohort.go.

pulse cohort inspect reads only the file’s header and schema — it never reads record data. The operation is constant-time regardless of cohort size.

LLM agents using MCP: see the cohort-schema-design skill and the pulse_inspect tool.

Synopsis

pulse cohort inspect PATH [--json] [--full-dict]

Flags

Output (text mode)

Fields: 7
  order_id              u64                  Stable order identifier
  region                categorical_u8       Sales region label
    dictionary: 4 entries
  product               categorical_u16      Product SKU
    dictionary: 240 entries (truncated)
  units                 u32                  Units sold per line
  revenue               decimal128           Line revenue (precision 18, scale 2)
  sold_on               date                 Date the order shipped
  ...

Dictionaries with > 100 entries are flagged (truncated) — pass --full-dict to print every entry.

Output (--json)

{
  "format_version": "1.1",
  "data": {
    "field_count": 7,
    "fields": [
      {
        "name": "order_id",
        "type": "u64",
        "byte_offset": 0,
        "bit_position": 0,
        "description": "Stable order identifier",
        "description_source": "schema"
      },
      {
        "name": "region",
        "type": "categorical_u8",
        "byte_offset": 8,
        "bit_position": 0,
        "description": "Sales region label",
        "description_source": "schema",
        "dictionary": {
          "total_entries": 4,
          "truncated": false,
          "entries": ["east", "west", "north", "south"]
        }
      }
    ]
  },
  "errors": [],
  "warnings": []
}

Fields with empty descriptions on disk get a synthesised fallback ("Categorical field: <name>" / "Numeric field: <name>"); their description_source is "synthesized" rather than "schema".

Exit codes

Examples

# Human-readable inspect
pulse cohort inspect data.pulse

# Full envelope for programmatic consumers
pulse cohort inspect data.pulse --json

# Show all categorical entries
pulse cohort inspect data.pulse --full-dict --json | jq '.data.fields[] | select(.dictionary)'

Related

• Format → Header Layout
• Format → Schema Block
• Format → Dictionary Blocks
• Library: pulse.Inspect — Go counterpart
• skills/cohort-schema-design.md — LLM-facing schema-design skill

pulse api predict

Audience: CLI users validating a request before running it.

pulse api predict validates a types.Request against a .pulse file’s schema without executing it. It reads only the header and schema — never record data — so it’s a cheap, safe iteration loop against arbitrarily large cohorts.

LLM agents using MCP: see the pulse_predict MCP tool and the debugging-with-predict skill. Predict is the LLM’s primary “would this work?” probe.

Synopsis

pulse api predict --request FILE [--json] [--strict] [--echo-request]

Flags

Structural ban

descriptor/predict.go cannot import service/ or processing/. This is enforced by TestPredictNoExecutionImports. Predict is guaranteed to never touch the executor.

Output (text mode)

Valid: true
Schema: 7 fields
Warning [PULSE_AGG_NOT_MEANINGFUL_FOR_CATEGORICAL]: AGG_AVG on field region (categorical_u8)

Without --strict, that warning would still let the command exit 0. With --strict, the warning becomes an error and the command exits non-zero.

Output (--json)

{
  "format_version": "1.1",
  "data": {
    "valid": true,
    "schema_info": {"field_count": 7},
    "streamable": false,
    "streamable_reasons": [
      "AGG_MEDIAN on field price"
    ],
    "request": { /* the request as predict resolved it, with defaults applied */ }
  },
  "errors":  [],
  "warnings": [
    {"code": "PULSE_AGG_NOT_MEANINGFUL_FOR_CATEGORICAL", "message": "..."}
  ]
}

streamable reports whether the request will execute on the streaming Process path; streamable_reasons lists every gate that forced the buffered path. See Performance Notes for the full streaming/buffered table.

request echoes the request after defaults have been applied so you can see what would actually run. To suppress defaults, run with --no-defaults on the executing leaf (api process, api compose); predict reports defaults_applied regardless.

Smart defaults

When an aggregations[] or groups[] slot names a field but omits type, Pulse infers the operator from the named field’s schema type before running the request. Predict reports the inferred slot under data.defaults_applied so you can echo back what was filled in:

The Nullable flag on a field never changes its default operator — it only controls per-record null-bitmap participation.

Rules: defaults never override an explicit type; they never cross categories (a missing aggregator does not insert a grouper); statistical tests (tests[], post_tests[]) are not defaulted; filter expressions, features, attributes, and windows are out of scope.

--no-defaults on pulse api process / pulse api compose disables the inference pass entirely and forces every slot to be source-of-truth. Predict still reports defaults_applied so the caller can see what would have been filled in.

Exit codes

Examples

Quick validity check

pulse api predict --request req.json

Programmatic check with envelope

pulse api predict --request req.json --json | \
    jq -e '.data.valid == true' >/dev/null && echo "OK"

Strict mode for CI

pulse api predict --request req.json --strict --json

Detect that a request will buffer

pulse api predict --request req.json --json | \
    jq '.data | {streamable, streamable_reasons}'

Common warning codes

The full code-by-code recovery playbook is reachable per-code via the MCP pulse_errors_lookup tool or the pulse errors lookup CODE CLI; see also Troubleshooting.

Related

• pulse api process — executes a validated request
• Library: pulse.Predict — Go counterpart
• Debugging Predict — LLM-side iteration recipe

pulse api sample

Audience: CLI users grabbing a quick peek at a few rows from a cohort — for debugging, sanity-checking an import, or seeding a template request.

pulse api sample returns the first N rows from a .pulse file decoded back to a map of field → value. There is no filter, no aggregation, no transformation — just a typed view of raw rows.

LLM agents using MCP: see the pulse_sample MCP tool. It returns the same shape over the MCP transport.

Synopsis

pulse api sample --input PATH [--count N] [--json]
                              [--labels FIELD=TABLE[:replace|augment]]
                              [--echo-request]

Flags

Output (text mode)

Pretty-printed JSON of the row array:

[
  {
    "order_id": 1,
    "region": "west",
    "product": "widget",
    "units": 3,
    "revenue": "29.97",
    "sold_on": "2024-01-04"
  },
  ...
]

Decimal128 values are serialised as strings to preserve precision.

Output (--json)

{
  "format_version": "1.1",
  "data": [ /* row array */ ],
  "errors": [],
  "warnings": []
}

Exit codes

Examples

# 10 rows
pulse api sample --input sales.pulse

# 100 rows, envelope-wrapped
pulse api sample --input sales.pulse --count 100 --json

# Pipe into jq
pulse api sample --input sales.pulse --count 100 | jq '.[] | .revenue'

When sample is the wrong tool

• For filtered subsets, use pulse api process with a FILTER_* and no aggregation — the result will be one row per matching record.
• For distinct values of a single field, use pulse api facet.
• For schema-only views (types, descriptions, dictionaries), use pulse cohort inspect.

Related

• pulse api facet — distinct values for a single field
• Library: pulse.Sample

pulse api facet

Audience: CLI users enumerating distinct values for a single field — a cheap probe for “what are the regions in this cohort?” without building a full filter — or, in rich mode, a multi-field summary of counts, null tallies, percentiles, histograms, and additive contribution counts.

pulse api facet has two modes:

• Simple mode (--input PATH --field NAME) returns the distinct values of one field. Categorical fields read the dictionary directly (no record scan); non-categorical fields scan records.
• Rich mode (any of --request, multiple --field, --top-k, --percentile, --histogram, --additive, --labels) returns a FacetResult covering every named field — counts, null tallies, optional percentiles and histograms over numerics, and optional additive contribution counts. Prefer rich mode over repeated simple calls when summarising more than one field.

LLM agents using MCP: see the pulse_facet MCP tool for simple mode and pulse_facet_schema for rich mode.

Synopsis

pulse api facet --input PATH --field NAME [--json]
                [--request FILE]
                [--field NAME ...] [--top-k N]
                [--percentile P ...] [--histogram]
                [--histogram-bins N] [--histogram-min X] [--histogram-max X]
                [--additive FIELD ...] [--labels FIELD=TABLE[:replace|augment]]
                [--echo-request]

Flags

Mode dispatch: simple mode runs when exactly one --field is passed without any rich-mode flag. Any other combination switches to rich mode and calls FacetSchema.

Output (text mode)

One value per line:

east
north
south
west

Output (--json)

{
  "format_version": "1.1",
  "data": ["east", "north", "south", "west"],
  "errors": [],
  "warnings": []
}

Performance notes

For columns with very high cardinality on the non-categorical path, expect memory proportional to distinct value count.

Exit codes

Examples

# Read categorical dictionary
pulse api facet --input sales.pulse --field region

# JSON envelope
pulse api facet --input sales.pulse --field region --json

# Pipe into another command
for r in $(pulse api facet --input sales.pulse --field region); do
    echo "Region: $r"
done

Related

• pulse api sample — raw rows preview
• Format: Dictionary Blocks — how categorical dictionaries are encoded
• Library: pulse.Facet

pulse manifest

Audience: CLI users (and orchestration agents) discovering Pulse’s self-description — what commands exist, which aggregators are registered, which field types are supported, and what skills the binary ships with.

The manifest is the bare-pulse invocation with --json. It is deterministic and process-wide: it never depends on cohort data or the filesystem.

LLM agents using MCP: the manifest is also available via the pulse_manifest MCP tool. Agents typically call this once per session and cache the result.

Synopsis

pulse --json [--slim]

(There is no pulse manifest subcommand — the manifest is the root command’s --json output.)

Flags

Manifest shape

From descriptor/manifest.go:

{
  "format_version": "1.1",
  "data": {
    "commands":   [ /* every CLI leaf with a usage line */ ],
    "operators":  [ /* every aggregator / attribute / filterer / grouper / window / feature */ ],
    "tests":      [ /* every tier-1 statistical test */ ],
    "post_tests": [ /* every tier-2 post-test variant */ ],
    "distributions": [ /* every synth distribution kind */ ],
    "errors":     [ /* every registered error code with a description */ ],
    "mcp_tools":  [ /* every MCP tool name + description */ ],
    "field_types":[ /* every .pulse field type */ ],
    "skills":     [ /* every embedded skill with metadata */ ]
  },
  "errors":   [],
  "warnings": []
}

Every list is sorted deterministically (alphabetical or category + alphabetical). The same Pulse binary always emits the same manifest bytes (modulo --slim).

Determinism gates

Several CI tests enforce manifest completeness — see Testing Conventions. Notably:

• TestManifestOperatorsComplete — every registered operator appears in the manifest.
• TestManifestTestsComplete / TestManifestPostTestsComplete — every registered statistical test appears.
• TestManifestDistributionsComplete, TestManifestErrorCodesComplete, TestManifestMCPToolsComplete — same for distributions, error codes, and MCP tools.
• TestManifestStreamableMatchesTypes — every operator’s streamable flag mirrors the per-type method.

When to use the manifest

Exit codes

Examples

Print the manifest

pulse --json | jq '.data | keys'

Slim variant for embedding in an agent’s system prompt

pulse --json --slim > manifest.slim.json

List every aggregator with its emitted type

pulse --json | jq '.data.operators[] | select(.category == "aggregation") | {name, emits_type}'

Confirm a feature operator’s parameters

pulse --json | jq '.data.operators[] | select(.name == "FEAT_BUCKETIZE")'

Related

• How LLMs Use Pulse — the manifest is one of the agent discovery primitives
• Library: pulse.Manifest — Go counterpart
• Internals: Architecture — why the manifest cannot import service/ or processing/

pulse schema

Print the JSON Schema (draft 2020-12) describing every public Pulse request and response payload.

pulse schema

The output is a self-contained JSON Schema document — it carries its own $schema and $id — so it is emitted raw, not wrapped in the standard --json envelope. It is byte-identical to the file published at the schema’s $id (https://frankbardon.github.io/pulse/payload-schema.json) and to the pulse://schema MCP resource.

Typical use — validate a request body offline:

pulse schema > payload-schema.json
# validate ./request.json against payload-schema.json#/$defs/Request
# with any draft-2020-12 validator

See the Payload Contract chapter for the schema’s structure, the three access surfaces, how it stays in sync with the engine, and its v1 strictness boundaries.

pulse synth from-schema

Audience: CLI users generating a synthetic .pulse cohort from a declarative spec — for testing, demos, and bootstrapping fixtures.

pulse synth from-schema reads a JSON synth spec (field-by-field distributions, row count, optional pairwise correlations) and writes a deterministic .pulse file. Same (spec, seed) pair produces a byte-identical output.

LLM agents using MCP: see the pulse_synth MCP tool and the synthetic-data skill — it covers spec authoring, the 12 supported distributions, and constraint patterns.

Synopsis

pulse synth from-schema --spec FILE --output FILE
                        [--rows N] [--seed N] [--json]

Flags

Spec shape (sketch)

{
  "row_count": 10000,
  "fields": [
    {"name": "id",      "type": "u64",            "distribution": "monotonic_from", "from": 1},
    {"name": "region",  "type": "categorical_u8", "distribution": "weighted_categorical",
                         "weights": {"east": 0.4, "west": 0.4, "north": 0.1, "south": 0.1}},
    {"name": "revenue", "type": "f64",            "distribution": "lognormal", "mu": 4.0, "sigma": 0.8},
    {"name": "sold_on", "type": "date",           "distribution": "uniform_date",
                         "from": "2024-01-01", "to": "2024-12-31"}
  ]
}

Full spec grammar (constraints, correlations, regex, …) lives in skills/synthetic-data.md and synth/.

Supported distributions

bernoulli, constant, exponential, lognormal, monotonic_from, normal, pareto, poisson, regex, uniform, uniform_date, weighted_categorical.

The full catalog (with parameters) is in skills/synthetic-data.md and pulse --json | jq '.data.distributions'.

Determinism

Same (spec, seed) → byte-identical output. The seed is a int64; default 0. Use a fixed seed for fixtures and a random seed for load-testing variation.

Output

Text mode

Generated 10000 rows -> sales.pulse (rejected 0)

rejected counts rows that failed user-defined constraints (PULSE_SYNTH_CONSTRAINT_INFEASIBLE when the rejection rate is too high to make progress).

--json

{
  "format_version": "1.1",
  "data": {
    "output_path": "sales.pulse",
    "rows_generated": 10000,
    "rows_rejected": 0,
    "seed": 0
  },
  "errors": [],
  "warnings": []
}

Exit codes

Common error codes

Examples

# Build sales.pulse from a spec
pulse synth from-schema --spec sales.spec.json --output sales.pulse --seed 42

# Override row count without editing the spec
pulse synth from-schema --spec sales.spec.json --output sales.pulse --rows 1000

# Programmatic envelope
pulse synth from-schema --spec sales.spec.json --output sales.pulse --json

Related

• pulse synth from-profile — generate from a captured profile of an existing cohort
• pulse profile create — capture the profile
• skills/synthetic-data.md — full spec grammar and distribution table
• Library: pulse.Synth

pulse synth from-profile

Audience: CLI users generating a synthetic .pulse cohort whose distributions match a real cohort — typically to share a sanitised replica without exposing the underlying rows.

pulse synth from-profile reads a profile JSON captured by pulse profile create and writes a synthetic .pulse file whose per-field distributions and (optional) pairwise correlations follow the profile. The profile retains no individual rows from the source; only summary statistics.

LLM agents using MCP: see the pulse_synth_from_profile MCP tool and the synthetic-data skill.

Synopsis

pulse synth from-profile --profile FILE --output FILE --rows N
                         [--seed N] [--json]

Flags

--rows is required (unlike from-schema, which can pull it from the spec) because the profile does not carry a generation count of its own.

Determinism

Same (profile, seed, rows) triple → byte-identical output. Seeds are int64; default 0.

Profile shape

The profile is a synth.Profile JSON object produced by pulse profile create. It carries per-field type, descriptive statistics, top-K categorical entries (default K = 32), optional pairwise correlations (when --include-correlations was passed at profile-creation time), and a row count.

See pulse profile create for how to capture one, and synth/ for the underlying Go types.

Output

Text mode

Generated 1000 rows -> sales.synth.pulse (rejected 0)

--json

Same envelope shape as synth from-schema.

Exit codes

Examples

# Capture once
pulse profile create --input sales.pulse --output sales.profile.json

# Re-generate any number of times with different seeds
pulse synth from-profile --profile sales.profile.json --output sales.s42.pulse --rows 10000 --seed 42
pulse synth from-profile --profile sales.profile.json --output sales.s43.pulse --rows 10000 --seed 43

Limitations

• Categorical tails: anything past the captured top-K is replaced with a sentinel “other” bucket sized to its observed weight.
• Correlations: pairwise only, and only between numeric fields. The profile capture flag --include-correlations opts in; without it, fields are generated independently.
• Decimal and geo fields: regenerated within the same type family but with synthetic value distributions; downstream uses that depend on exact field values (e.g. joinable identifiers) need the schema-driven path instead.

Related

• pulse profile create
• pulse synth from-schema
• skills/synthetic-data.md — the spec / profile grammar

pulse profile create

Audience: CLI users capturing a statistical profile of an existing cohort — typically to feed into pulse synth from-profile.

pulse profile create reads a .pulse file and writes a JSON profile: per-field type, descriptive statistics, top-K categorical entries, optional pairwise correlations. The profile retains no individual rows from the source.

LLM agents using MCP: see the pulse_profile MCP tool.

Synopsis

pulse profile create --input PATH --output PATH
                     [--top-k N] [--include-stats]
                     [--include-correlations] [--correlation-top-k N]
                     [--sample-limit N] [--json]

Flags

What the profile captures

What the profile does NOT capture

• Individual rows.
• The full categorical dictionary beyond --top-k.
• Correlations unless --include-correlations is set.

This is by design — profiles are intended to be safe to share with parties who shouldn’t see the underlying data.

Output

The profile JSON is always written to --output. With --json, the envelope is also written to stdout (typically piped or jq-d).

Profile schema lives in synth/profile.go and is documented in skills/synthetic-data.md.

Text mode summary

Profiled 50000 rows from sales.pulse -> sales.profile.json

Exit codes

Examples

Minimal profile

pulse profile create --input sales.pulse --output sales.profile.json

Rich profile with correlations

pulse profile create --input sales.pulse --output sales.profile.json \
    --include-stats --include-correlations --top-k 64 --correlation-top-k 32

Sample-limited profile for a huge cohort

pulse profile create --input ops.pulse --output ops.profile.json --sample-limit 1000000

Round-trip with synth

pulse profile create --input sales.pulse --output sales.profile.json
pulse synth from-profile --profile sales.profile.json --output sales.synth.pulse --rows 10000 --seed 1
pulse cohort inspect sales.synth.pulse

Related

• pulse synth from-profile — the consumer of profile JSON
• pulse synth from-schema — the alternative spec-driven path
• skills/synthetic-data.md — full profile and spec grammar
• Library: pulse.Profile

pulse mcp

Audience: operators wiring Pulse into an MCP-aware AI client (Claude Desktop, Claude Code, generic MCP clients).

pulse mcp runs the Model Context Protocol server over stdio. The AI client launches pulse mcp as a subprocess, speaks MCP over its stdio streams, and shuts it down on session close.

LLM agents using MCP: the agent-side guide is the mcp-integration skill — fetch it via pulse_skills_get for the tool catalog and request shapes. This page is for the human setting the server up.

Synopsis

pulse mcp [--data-dir PATH] [--bind-on-open]

The command reads stdin, writes MCP responses on stdout, and writes a one-line startup notice (and any subsequent diagnostics) on stderr.

Flags

--data-dir is required in one of its two forms (env var or flag). The MCP server fails to start otherwise:

data directory required: set PULSE_DATA_DIR or pass --data-dir

–bind-on-open

When a session calls pulse_inspect successfully, the server can register session-scoped tool variants whose JSON Schemas constrain field-name parameters to the cohort’s actual fields. This narrows the LLM’s choices and prevents typos at parameter-binding time.

Default: true. Pass --bind-on-open=false if your client binds tool schemas itself.

The binding logic lives in the SDK-free core mcp/bind.go (the per-session server mutation that consumes it lives in the go-sdk adapter, mcp/gosdk/bind.go); see Adding an MCP tool for the LLM-facing implications.

Wiring it into Claude Desktop

~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "pulse": {
      "command": "/usr/local/bin/pulse",
      "args": ["mcp"],
      "env": {
        "PULSE_DATA_DIR": "/var/data/pulse"
      }
    }
  }
}

Restart the client. The Pulse tools (pulse_manifest, pulse_inspect, pulse_predict, pulse_process, pulse_compose, pulse_sample, pulse_facet, pulse_import, pulse_drop, pulse_imports_list, pulse_examples_search, pulse_examples_get, pulse_errors_lookup, pulse_skills_list, pulse_skills_get) and resources (pulse://*.pulse, pulse-skill://*) appear in the tool/resource list.

Wiring it into Claude Code

~/.claude.json (or per-project .claude.json):

{
  "mcpServers": {
    "pulse": {
      "command": "/usr/local/bin/pulse",
      "args":    ["mcp"],
      "env":     { "PULSE_DATA_DIR": "/var/data/pulse" }
    }
  }
}

The full LLM-side recipe (including resource URIs and the schema binding details) is in Adding an MCP tool and Wiring an MCP client.

Exit codes

pulse mcp is a long-running process. It exits non-zero only on fatal startup failure (missing data dir, transport error). Once serving, an MCP client controls the lifecycle.

Examples

Foreground run for debugging

PULSE_DATA_DIR=/tmp/pulse-data ./bin/pulse mcp
# Stderr: pulse mcp: serving over stdio (data dir: /tmp/pulse-data, bind-on-open: true)

Disable schema binding

PULSE_DATA_DIR=/tmp/pulse-data ./bin/pulse mcp --bind-on-open=false

Inspect what the server registers

# Manifest exposes the MCP tool list
pulse --json | jq '.data.mcp_tools[]'

Related

• How LLMs Use Pulse — the pointer table from this site into the skill pack
• Adding an MCP tool — LLM-side wiring, tool catalog, resource schemes, schema binding
• Deployment — production hardening notes
• Troubleshooting — common MCP failure modes

Flag Reference

Audience: CLI users who want one page that lists every flag and every environment variable in scope across the binary.

The per-command pages list each command’s full flag set; this page is the cross-cutting reference for flags that appear on multiple commands and for the environment variables Pulse reads.

LLM agents using MCP: there is no LLM-facing skill for the CLI surface. Agents go via MCP tools (pulse_process, pulse_inspect, …) — see pulse mcp and Adding an MCP tool.

Global flags

Available on the bare pulse invocation:

Both default to off. pulse --json is the discovery entry point — it emits the manifest documented at pulse manifest.

Environment variables

PULSE_DATA_DIR is the only required PULSE_* environment variable. The Makefile auto-loads a repo-root .env file so you can keep these (and any future env vars) there for development.

When embedding the library, you can bypass the env vars entirely by passing pulse.Options{DataDir: "/path"}, pulse.Options{ImportsDir, ImportTTL, LabelTablesDir, FS: myFs} etc. — see pulse.New & Options.

--json envelope

Almost every leaf command accepts --json, which switches output from human prose to a structured envelope. The envelope shape is fixed and documented in CLAUDE.md → Output Format Contract:

{
  "format_version": "1.1",
  "data":     { /* operation-specific result */ },
  "request":  { /* normalized request, omitted unless --echo-request was passed */ },
  "errors":   [ /* {"code": "...", "message": "...", "details": {...}} */ ],
  "warnings": [ /* same shape */ ]
}

format_version is currently "1.0". errors and warnings are always arrays (never null) so JSON consumers can index without nullable-check overhead. request is opt-in (see --echo-request below); data.components is the additive Response.Components slot documented per leaf — see api process → --json.

Shared per-command flags

Several flags appear on multiple commands with identical semantics.

--no-defaults

Available on: api process, api compose.

Disable the runtime smart-defaults pass that infers operator Type from the named field’s schema type when the caller omits it. Forces the request to be source-of-truth. See pulse.New & Options for the underlying library option.

--stream

Available on: api process, api compose.

Stream result rows as NDJSON (one row per line) instead of buffering the full result. For compose, each line carries an {"index": N, "row": {...}} shape so consumers know which sub-request produced each row. See Streaming & ProcessStream.

--strict

Available on: api process, api predict.

Treat request-validation warnings (e.g. numeric aggregation on a categorical field, low-quality field description) as errors. On api predict this fails validation; on api process it refuses to execute. Useful in CI gates that want the strictest possible validation.

--echo-request

Available on: api process, api process-chain, api compose, api predict, api sample, api facet.

Include the normalized request — smart defaults resolved, label bindings expanded — on envelope.request. Absent (and omitted from JSON) by default so the envelope shape is unchanged for callers that do not need it. Streaming output (--stream) skips the echo because NDJSON has no envelope.

--full-dict

Available on: cohort inspect.

Print full categorical dictionaries instead of truncating after 100 entries. Pair with --json for programmatic consumption.

--strict / --seed / --rows

synth from-schema and synth from-profile use --seed (for deterministic RNG) and --rows (override the spec’s row count). See the per-command pages.

Help

Every command supports --help:

pulse --help
pulse api --help
pulse api process --help
pulse mcp --help

--help output is the urfave/cli v3 default — a usage block, description, flag list, and an examples block where applicable.

Cross-references

Go API Overview

Audience: Go developers embedding Pulse in a binary or a service.

Pulse is library-first. The CLI in cmd/pulse/ is a thin adapter around the package documented here. If you’re reaching for os/exec to shell out to the binary from Go, stop and use the library directly — you’ll skip a process boundary and gain typed responses.

LLM agents using MCP: there is no LLM-facing skill that covers Go embedding directly. Agents speak MCP; this page is for the programs that host them.

Module path

import "github.com/frankbardon/pulse"

Sub-packages you’ll commonly touch:

The internal/ subtree (internal/cli) is exactly that — internal. Don’t import it. The MCP server is public: mcp/ is the SDK-free core and mcp/gosdk/ is the go-sdk adapter (gosdk.Register).

The facade

Construct a Pulse once per process (or per filesystem boundary) and re-use it:

p, err := pulse.New(pulse.Options{
    DataDir: "/var/data/pulse",
})
if err != nil {
    return err
}

The full Options shape (custom afero.Fs, smart-default toggling) is documented at pulse.New & Options.

Public methods

From pulse.go:

Re-exported type aliases let you write pulse.Request instead of types.Request:

type (
    Request         = types.Request
    Response        = types.Response
    ComposedRequest = types.ComposedRequest
    SynthSpec       = synth.Spec
    Profile         = synth.Profile
    // … and so on
)

Minimum viable embed

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/frankbardon/pulse"
    "github.com/frankbardon/pulse/types"
)

func main() {
    ctx := context.Background()

    p, err := pulse.New(pulse.Options{DataDir: "/var/data/pulse"})
    if err != nil {
        log.Fatal(err)
    }

    resp, err := p.Process(ctx, &pulse.Request{
        Cohort: &types.Cohort{Filename: "sales.pulse"},
        Aggregations: []*types.Aggregation{
            {Type: types.AGG_AVERAGE, Field: "revenue", Label: "avg_revenue"},
        },
    })
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(resp.Data)
}

Where to go from here

• pulse.New & Options — full Options reference.
• Custom Filesystems — in-memory testing pattern, custom storage backends.
• Streaming & ProcessStream — pull-based iteration, what streams vs what buffers.
• Parallel Compose — worker pool, fail-fast, per-request timeout.

pulse.New & Options

Audience: Go embedders constructing a Pulse instance.

pulse.New(pulse.Options{...}) is the single entry point. There is no config file, no init function, no global state. Every option is declared in code (or comes from PULSE_DATA_DIR when the field is left empty).

LLM agents using MCP: the MCP server constructs its own Pulse instance from CLI flags. Agents don’t see this surface.

The Options struct

From pulse.go:

type Options struct {
    // DataDir is the base directory for cohort files.
    // Defaults to PULSE_DATA_DIR if empty and FS is not set.
    DataDir string

    // FS is an optional custom filesystem.
    // When set, DataDir is ignored for filesystem construction.
    FS afero.Fs

    // DisableDefaults turns off the smart-defaults pass that infers
    // operator Type from the named field's schema type when the caller
    // omits it. Defaults to false (defaults enabled). Predict still
    // computes and reports DefaultsApplied independently — this flag
    // governs only what the runtime mutates on the live request.
    DisableDefaults bool
}

Field reference

DataDir string

The base directory for .pulse files. Relative cohort paths ({"filename": "data.pulse"}) resolve against this directory.

Example:

p, err := pulse.New(pulse.Options{DataDir: "/var/data/pulse"})

FS afero.Fs

A custom afero.Fs implementation. When set, it fully overrides the filesystem layer — DataDir is unused, and PULSE_DATA_DIR is not consulted. Use this for tests (afero.NewMemMapFs()) or non-local backends (S3-backed afero.Fs, encrypted overlays, …).

Example:

import "github.com/spf13/afero"

p, err := pulse.New(pulse.Options{
    FS: afero.NewMemMapFs(),
})

See Custom Filesystems for in-depth usage and the hermetic-test pattern.

DisableDefaults bool

The runtime smart-defaults pass infers an operator’s Type from the named field’s schema type when the caller omits it (e.g. AGG_SUM on a numeric field defaults appropriately; categorical fields default toward AGG_COUNT). Set DisableDefaults = true to require an explicit Type on every aggregation and grouper — useful when you want the request to be source-of-truth and never be silently re-typed.

This option only governs the runtime mutation. predict independently computes and reports DefaultsApplied in its result envelope, so callers can see what would have been inferred even when defaults are disabled.

CLI parity: pulse api process --no-defaults, pulse api compose --no-defaults.

Defaults at a glance

Re-using a Pulse instance

Pulse is safe for concurrent use across goroutines once constructed. The internal registries are read-only after New; each Process call constructs fresh stateful operators per request, so multiple goroutines can call Process/ProcessStream/Compose in parallel against the same Pulse.

For batch parallelism, prefer ComposeParallel — it shares the read-only registries and bounds concurrency for you.

Tearing down

There is no explicit Close() method on Pulse. The filesystem is a borrowed handle; if you supply a custom FS, the embedder is responsible for any cleanup that FS requires. Streaming consumers should still call RowIter.Close() so that the underlying readers release their buffers.

Custom Filesystems

Audience: Go embedders running Pulse in tests (hermetic, no disk), in cloud-storage-backed environments (S3, GCS, Azure Blob via afero), or behind a custom storage layer.

Pulse routes all file I/O through afero.Fs. Pass any afero.Fs-conformant filesystem to pulse.New(pulse.Options{FS: ...}) and Pulse never touches the OS filesystem directly.

LLM agents using MCP: the MCP server’s filesystem is fixed at startup via PULSE_DATA_DIR or --data-dir. Agents don’t swap filesystems mid-session.

In-memory testing pattern

The single most common reason to override the filesystem is hermetic tests. Use fs.NewMemMap() (which wraps afero.NewMemMapFs() with the right config) or pass the afero filesystem directly:

import (
    "github.com/frankbardon/pulse"
    "github.com/spf13/afero"
)

func TestSomething(t *testing.T) {
    p, err := pulse.New(pulse.Options{FS: afero.NewMemMapFs()})
    if err != nil {
        t.Fatal(err)
    }

    // Write a .pulse file into the in-memory FS, then process it.
    // ...
}

The in-memory FS persists for the life of the FS reference. Create a fresh one per test for isolation.

Custom storage backends

Anything that implements afero.Fs works. Common patterns:

• S3 / GCS / Azure Blob — via community afero adapters (afero/gcsfs, afero/s3).
• Encrypted overlays — wrap a base FS with envelope encryption per file.
• Read-only mounts — afero.NewReadOnlyFs(base) for production cohort serving where mutation is by accident, not policy.

Example with a hypothetical S3 wrapper:

import (
    "github.com/frankbardon/pulse"
    "example.com/myorg/aferos3"
)

func main() {
    s3fs := aferos3.New(aferos3.Config{
        Bucket: "my-pulse-cohorts",
        Region: "us-east-1",
    })
    p, _ := pulse.New(pulse.Options{FS: s3fs})
    // p reads and writes cohort files from S3 transparently.
}

The fs package

The lower-level constructors live in fs/:

You can also bypass pulse.Options entirely and construct a service from a *fs.Config, but the public facade is the intended entry point. pulse.New(pulse.Options{FS: yourFs}) covers every embedding case.

Path resolution

Pulse resolves a Cohort to a path with this rule (see resolveCohortPath in pulse.go):

if cohort.DataDir != "" → "<DataDir>/<Filename>"
else                    → "<Filename>"

The custom FS is then asked to open that path. For an afero.MemMapFs, an absolute-looking path like /var/data/sales.pulse is just a key in the in-memory map — no need to mirror the OS layout.

What custom filesystems do NOT do

• Pulse never falls back to os.Open if the custom FS fails. The custom FS is the only filesystem; if it errors, that error propagates verbatim.
• The MCP server (pulse mcp) currently uses afero.NewOsFs() only. Custom filesystems are a library-side capability today.
• The Go race detector and go test -race work normally with in-memory filesystems; tests can run highly concurrent without fighting over a real directory.

Streaming & ProcessStream

Audience: Go embedders feeding rows into an HTTP response, an NDJSON pipeline, or any consumer that wants result rows one at a time instead of buffering the full set.

pulse.ProcessStream returns a pull-based iterator. The API is stable regardless of whether the underlying request shape streams inside the engine — non-streamable requests return the same iterator, they just buffer once internally before yielding.

LLM agents using MCP: the MCP-side streaming surface is pulse_process with the streaming option. The Streamable predicate is the same on both surfaces.

The iterator API

type RowIter = service.RowIter

// In service:
type RowIter interface {
    Next(ctx context.Context) (Row, bool, error)
    Close() error
    Metadata() *ResponseMetadata
}

type Row = service.Row // map[string]any

Usage:

iter, err := p.ProcessStream(ctx, req)
if err != nil {
    return err
}
defer iter.Close()

for {
    row, ok, err := iter.Next(ctx)
    if err != nil {
        return err
    }
    if !ok {
        break
    }
    // … emit row …
}

meta := iter.Metadata() // available after drain

Metadata() returns the full ResponseMetadata (total rows, filtered rows, cohort file) once the iterator has been drained.

What actually streams

ProcessStream always returns an iterator, but the engine only avoids the buffered intermediate row set for a subset of request shapes. Run pulse api predict (or Predict from the library) and check the Streamable flag in the result:

pred, err := p.Predict(ctx, req)
if !pred.Streamable {
    for _, reason := range pred.StreamableReasons {
        log.Printf("buffered because: %s", reason)
    }
}

The streaming-eligible request shapes are listed in Performance Notes → Streaming path.

The complement — the request shapes that force the buffered path — is at Performance Notes → Buffered path.

Streamable=false doesn’t mean the iterator is broken; it just means rows materialise inside the engine before Next yields them. The output API is identical either way.

CLI parity

pulse api process --stream writes NDJSON to stdout, one row per line. pulse api compose --stream does the same with an index field per row identifying which sub-request produced it.

Cancellation

Every Next call accepts a context. Cancellation propagates to the underlying reader; rows that are already in flight may still be returned before Next returns (_, false, ctx.Err()). Close() releases any reader resources and is safe to call multiple times.

Backpressure

The iterator is pull-based: the engine produces rows only as fast as the consumer calls Next. For HTTP responders that flush periodically, this means you can stream a multi-GB result set through a constant-memory buffer.

For pipelines that want to fan rows out across goroutines, copy each row into your own struct before processing — Row is map[string]any and the engine may re-use the backing data after Next returns. Treat it as borrowed.

Inside the engine

Under the hood, ProcessStream calls one of four orchestrator modes depending on the request shape: single-pass streaming, grouped streaming, two-pass streaming, or the buffered fallback. The choice is made via processing.CanStreamRequest(req, schema), which is the same predicate Predict.Streamable reports — this parity is enforced by TestPredict_Streamable_MatchesRuntime.

If you find a request that predict says is streamable but Next materialises something large, that’s a parity drift and a bug — please report it with the request JSON.

StreamResult[T]

Audience: Go embedders that want a structured streaming shape with header, chunks, terminator, and built-in cancellation semantics.

StreamResult[T] is Pulse’s canonical streaming shape. Every operation that produces incremental data exposes a *Stream variant that returns StreamResult[T]. The shape’s contract — request hash in the header, sequence-numbered chunks, terminator on a separate channel — is generic across operators, so a consumer can write one row-handling loop that drives any streamable Pulse call.

Shape

type StreamResult[T any] struct {
    Header StreamHeader
    Chunks <-chan StreamChunk[T]
    Done   <-chan StreamTerminator
}

type StreamHeader struct {
    RequestHash    string    // From req.Hash()
    EstimatedTotal int64     // Best-effort; -1 if unknown
    StartedAt      time.Time
}

type StreamChunk[T any] struct {
    Sequence int          // Monotonic 0-based
    Data     T
    Progress float64      // 0.0–1.0, or -1 if unknown
}

type StreamTerminator struct {
    CompletedAt time.Time
    TotalRows   int64
    Status      StreamStatus  // Completed | Cancelled | Errored
    Error       error
}

Available variants

• Pulse.ProcessStreamResult(ctx, req) (StreamResult[Row], error) — wraps the existing ProcessStream engine.
• Pulse.SynthStream(ctx, spec, opts) (StreamResult[Row], error) — generates synth respondents and yields one row per chunk.

Receiver pattern

res, err := p.ProcessStreamResult(ctx, req)
if err != nil {
    return err
}

for chunk := range res.Chunks {
    handle(chunk.Sequence, chunk.Data, chunk.Progress)
}

term := <-res.Done
if term.Status != pulse.StreamCompleted {
    return fmt.Errorf("stream %s: %w", term.Status, term.Error)
}
fmt.Printf("delivered %d rows in %s\n",
    term.TotalRows, term.CompletedAt.Sub(res.Header.StartedAt))

The Chunks channel closes when the operation finishes. The Done channel delivers exactly one StreamTerminator and then closes — a receiver that selects on Done sees one value followed by channel-closed.

Backpressure

Chunks carries a 4-deep buffer. Slow consumers slow the producer; the producer never drops chunks. This makes StreamResult safe to hand to a downstream HTTP response writer or SSE relay without worrying about chunk loss under pressure.

Cancellation

Cancelling the context cancels the operation:

ctx, cancel := context.WithCancel(parent)
res, _ := p.ProcessStreamResult(ctx, req)

go func() {
    time.Sleep(5 * time.Second)
    cancel()
}()

for chunk := range res.Chunks { ... }
term := <-res.Done
// term.Status == pulse.StreamCancelled, term.Error == context.Canceled

Errors

A mid-stream error closes Chunks early and delivers a StreamTerminator{Status: StreamErrored, Error: err} on Done. Consumers should always check the terminator status — a closed Chunks channel alone does not distinguish success from error.

Non-streaming variants

The non-streaming entry points (Process, Synth) remain — they are convenience wrappers that drain the stream and return the full result. Use the streaming variants when you need incremental output, progress reporting, or early cancellation; use the buffered variants when you want one materialised result and don’t care about the intermediate chunks.

Relationship to ProcessStream

Pulse.ProcessStream (returning RowIter) remains the lower-level pull-iterator API for callers that want the simpler shape. See Streaming & ProcessStream. ProcessStreamResult adds the structured header, terminator, and cancellation semantics on top of the same engine.

Parallel Compose

Audience: Go embedders running multiple requests concurrently against the same cohort or set of cohorts.

pulse.ComposeParallel fans a ComposedRequest across a bounded worker pool. Workers share the engine’s read-only registries; each Process call constructs fresh stateful operators per request, so concurrent execution is safe.

LLM agents using MCP: the MCP server today exposes pulse_compose as a sequential operation. Parallelism is a library-side capability.

When to use

Order of results is preserved regardless of completion order — a worker that finishes early is held until its slot’s index is the next to emit. So callers can index responses[i] against req.Requests[i] directly.

ComposeOptions

From service/compose_parallel.go, re-exported as pulse.ComposeOptions:

type ComposeOptions struct {
    // MaxWorkers caps concurrent in-flight Process calls. Zero means
    // runtime.GOMAXPROCS; negatives clamp to 1.
    MaxWorkers int

    // PerRequestTimeout, if positive, derives a context.WithTimeout for
    // each request.
    PerRequestTimeout time.Duration

    // FailFast cancels in-flight siblings on the first request error.
    // Defaults to true. Set false to aggregate all errors instead.
    FailFast bool
}

Example

ctx := context.Background()

composed := &pulse.ComposedRequest{
    Requests: []*pulse.Request{req1, req2, req3, req4},
}

resps, err := p.ComposeParallel(ctx, composed, pulse.ComposeOptions{
    MaxWorkers:        4,
    PerRequestTimeout: 30 * time.Second,
    FailFast:          true,
})
if err != nil {
    return err
}

for i, resp := range resps {
    fmt.Printf("request %d: %d rows\n", i, len(resp.Data))
}

FailFast semantics

With FailFast = true (the default):

• The first request to return an error cancels the shared context.
• In-flight siblings observe cancellation via ctx.Err() and return early.
• ComposeParallel returns (nil, theFirstError).

With FailFast = false:

• Every request runs to completion (or its own per-request timeout).
• Errors are aggregated into a single SERVICE_INTERNAL error whose details map carries failed_indices (a list of slot indices that errored).
• Successful slots populate the returned response array; failed slots are nil at their index.

CLI parity

pulse api compose --request batch.json --parallel 4
pulse api compose --request batch.json --parallel 4 --no-fail-fast

--parallel N:

• 1 (default) → sequential Compose.
• 0 → runtime.GOMAXPROCS.
• > 1 → exactly that many workers.

--no-fail-fast mirrors FailFast = false.

Performance considerations

• Each worker performs its own filesystem reads. If your cohort lives on slow remote storage, parallelism amortises latency well; on local SSD the gain is smaller and CPU-bound.
• Streaming aggregations are CPU-friendly — ComposeParallel over a pool of streaming requests scales near-linearly to the worker count.
• Buffered request shapes (window operators, median, …) hold memory per request. Watch MaxWorkers × per_request_peak_memory.
• The internal registries are read-only and shared across workers with no locking; only the per-request operator instances are fresh allocations.

Safety

• Pulse is safe for concurrent use after New.
• Per-request operator state (running sums, dictionaries, sorted buffers) is allocated fresh inside each Process call.
• The afero.Fs you supply must itself be safe for concurrent reads — every shipped backend (OsFs, MemMapFs) is.

Request Hashing

Audience: Go embedders building caches, deduplication layers, or materialization stores on top of Pulse.

Every Pulse request type implements Hash() string — a 32-character hex digest of the request’s canonical JSON form. Same logical request produces the same hash, across processes and across Pulse versions where the request’s semantic meaning is unchanged.

Supported types

The lower-level helper types.CanonicalHash(tag, v) hashes any JSON-serializable value with a caller-chosen namespace tag.

Guarantees

• Deterministic. Same in, same out — every call, every process.
• Round-trip stable. json.Marshal → json.Unmarshal → Hash() returns the original digest.
• Field-order invariant. Struct field order normalised by the canonical encoder; map keys sorted.
• Default-normalising. Explicit zero (Limit: 0) and the omitted equivalent hash identically because the request struct tags carry omitempty.
• Negative-zero collapsing. -0.0 hashes identically to 0.0.
• Type-namespaced. A Request and a ComposedRequest with identical wire bytes never collide — the hash function prefixes a namespace tag per request shape.

Use cases

Cache key

key := req.Hash()
if hit, ok := cache.Get(key); ok {
    return hit
}
resp, err := p.Process(ctx, req)
if err == nil {
    cache.Set(key, resp)
}

Filename suffix

out := fmt.Sprintf("derived_%s.pulse", req.Hash())

Idempotency token

// Reject duplicate work in a queue / job system.
if seen.Add(req.Hash()) == false {
    return errAlreadyQueued
}

Dedup across consumers

Independent processes can build the same Request (programmatically or from JSON) and reach the same hash without coordination. Combined with Deterministic FilterToFile, this is the foundation for shared materialization caches.

Algorithm

1. json.Marshal(v) — produces canonical-ordered struct fields and sorted map keys.
2. Walk the resulting JSON tree, dropping representational variants (collapse -0.0 → 0.0, integer-valued floats keep integer form).
3. Re-emit with sorted map keys and no whitespace.
4. SHA-256(tag + 0x00 + canonical).
5. First 16 bytes as 32-character lowercase hex.

The implementation lives in types/hash.go; tests covering the guarantees above are in types/hash_test.go.

Watch & WatchDir

Audience: Go embedders building reactive systems on top of Pulse — caches that invalidate on source change, UI layers that re-fetch when the data updates, batch jobs that fire on file appearance.

Pulse.Watch and Pulse.WatchDir observe .pulse files for changes and emit ChangeEvent records on a channel. The polling watcher coalesces atomic-write patterns (write-temp + rename) into single events so consumers don’t see spurious intermediate states.

Shape

type ChangeEvent struct {
    Path      string
    Kind      ChangeKind  // Created | Modified | Removed | Renamed
    Hash      string      // SHA-256 hex of leading bytes; empty for Removed
    Timestamp time.Time
}

type ChangeKind int
const (
    ChangeCreated ChangeKind = iota
    ChangeModified
    ChangeRemoved
    ChangeRenamed
)

Default usage

ctx, cancel := context.WithCancel(context.Background())
defer cancel()

// Single-file watch.
ch := p.Watch(ctx, "cohort.pulse")

// Directory watch (recursive, .pulse files only).
ch := p.WatchDir(ctx, "cohorts/", true)

for ev := range ch {
    switch ev.Kind {
    case pulse.ChangeCreated:
        log.Printf("new cohort %s (hash=%s)", ev.Path, ev.Hash)
    case pulse.ChangeModified, pulse.ChangeRenamed:
        invalidateCache(ev.Path)
    case pulse.ChangeRemoved:
        cache.Delete(ev.Path)
    }
}

Cancelling the context closes the channel and releases the watcher.

WatchOptions

type WatchOptions struct {
    PollInterval    time.Duration  // Default 250ms
    CoalesceWindow  time.Duration  // Default 100ms (0 disables coalescing)
    HashPrefixBytes int            // Default 64 KiB (< 0 hashes entire file)
    Recursive       bool           // WatchDir-only
    Suffix          string         // Default ".pulse" for WatchDir, "" for Watch
}

ch := p.WatchDirWithOptions(ctx, "cohorts/", pulse.WatchOptions{
    PollInterval:    1 * time.Second,
    HashPrefixBytes: -1,  // hash full file
    Recursive:       true,
    Suffix:          ".pulse",
})

Tuning notes

• PollInterval — stat-poll cadence. The default 250ms is fine for local filesystems. Network filesystems should raise this to ~30s; on remote storage the stat call dominates over any latency fsnotify-style edge events would save.
• CoalesceWindow — events inside this window collapse to one. Critical for atomic-write patterns (see below).
• HashPrefixBytes — how much of the file’s content participates in the hash. Default 64 KiB covers a .pulse file’s header + schema + first dictionary block, enough to detect any structural change. Use -1 when you need full-file hashing for small files where prefix collisions matter.
• Suffix — filter directory contents to files ending in this suffix. WatchDir defaults to .pulse. Set to "" to watch every file in the directory.

Atomic-write coalescing

Most tools that write .pulse files do so atomically — write to a temp path, then rename into place. Without coalescing, a naive watcher would see two events: a Removed(temp) and a Created(target). WatchDir folds these into a single ChangeRenamed(target) event when:

• Both paths share a directory.
• The temp basename matches a canonical scratch pattern (<target>.tmp, <target>.partial, <target>.swp, ~<target>, hidden dotfile variants, or <target>.NNNN).
• The two raw events fall inside CoalesceWindow of each other.

Pulse.FilterToFileWithRequest uses this pattern internally; a consumer watching the output directory sees a clean ChangeRenamed or ChangeCreated event without intermediate states.

Hash semantics

ChangeEvent.Hash is the SHA-256 hex of the file’s first HashPrefixBytes bytes. The hash is informational — two events with the same Hash value describe identical leading-byte content. The watcher itself uses the hash internally to detect content changes the mtime/size signals miss (e.g. an in-place write that preserves both).

For ChangeRemoved events the Hash field is empty.

First-tick behaviour

The watcher does not pre-seed its state map. Files that already exist when the watcher starts surface as ChangeCreated on the first tick. If you only want subsequent mutations, drain the first batch:

ch := p.WatchDir(ctx, dir, true)
// Drain initial Created events.
first := time.After(500 * time.Millisecond)
draining := true
for draining {
    select {
    case <-ch:
    case <-first:
        draining = false
    }
}
// Now treat subsequent events as actionable.

Limitations

• Polling-based — no fsnotify dependency. The default 250ms cadence is the floor for change visibility; tune PollInterval for your latency budget.
• Symlinks are followed implicitly (the underlying afero.Fs.Stat resolves them) but the watcher does not re-resolve a link target that changes after watch-start.
• The afero abstraction means the watcher works against any custom filesystem (in-memory, encrypted, remote) supplied via pulse.Options.FS.

Deterministic FilterToFile

Audience: Go embedders that materialize filtered subsets of a canonical cohort and want to dedup the work across consumers.

Pulse.FilterToFileWithRequest reads a source .pulse cohort, applies a predicate, and writes the surviving records to a new .pulse file. Three guarantees make the operation dedup-safe:

1. Deterministic output naming — same source + same predicate resolves to the same output path.
2. Atomic write — the engine never leaves a partially-written .pulse file at the target path.
3. Pre-existence dedup — if the deterministic output already exists, the engine returns it without re-running the filter.

Shape

type FilterToFileRequest struct {
    SourcePath string             // Path to the source .pulse file
    Expression string             // FILTER_EXPRESSION-style predicate
    Filterers  []*types.Filterer  // Structured-predicate alternative
    OutputDir  string             // Where to write the result
    OutputName string             // Optional override; default: {srcHash}_{predHash}.pulse
}

type FilterToFileResult struct {
    OutputPath string
    OutputHash string
    RowCount   int64
    ElapsedMs  int64
    Reused     bool   // true when a pre-existing output was returned
}

Exactly one of Expression or Filterers must be set — both empty or both set is a configuration error so the predicate hash is unambiguous.

Basic usage

res, err := p.FilterToFileWithRequest(ctx, &pulse.FilterToFileRequest{
    SourcePath: "cohort-2025-Q3.pulse",
    Expression: "age >= 18 && state in [\"CA\", \"NY\"]",
    OutputDir:  "derived/",
})
if err != nil {
    return err
}
fmt.Printf("%s (%s) %d rows reused=%v in %dms\n",
    res.OutputPath, res.OutputHash, res.RowCount, res.Reused, res.ElapsedMs)

On a fresh OutputDir the first call writes the file and reports Reused: false. A subsequent identical call hits the pre-existence check, reports Reused: true, and skips the filter work.

Predicate shapes

Expression

A string evaluated under the FILTER_EXPRESSION engine — same expr- lang semantics as Request.Filterers[*].Expression:

Expression: "age >= 18 && (region == \"NA\" || region == \"EU\")",

Filterers

A slice of structured types.Filterer entries translated into the equivalent expression and AND-combined. Useful when the predicate is built programmatically:

Filterers: []*types.Filterer{
    {Type: types.FILTER_RANGE, Field: "age", Values: []string{"18", "65"}},
    {Type: types.FILTER_INCLUDE, Field: "state", Values: []string{"CA", "NY"}},
    {Type: types.FILTER_NULL, Field: "email", Values: []string{"is_not_null"}},
},

Supported types: FILTER_EXPRESSION (pass-through), FILTER_INCLUDE, FILTER_EXCLUDE, FILTER_RANGE, FILTER_NULL.

Deterministic output naming

When OutputName is empty, the basename is computed as:

{sourceHash[:16]}_{predicateHash[:16]}.pulse

• sourceHash is the SHA-256 of the source file’s bytes.
• predicateHash is types.CanonicalHash of the predicate payload (Expression or Filterers slice, normalised to the same canonical JSON form used by Request.Hash).

Independent consumers building the same request reach the same path without coordination. Combined with a Watch on OutputDir, this is the foundation for shared materialization caches: many callers ask for the same derived cohort, only the first request pays the filter cost, every subsequent caller hits the pre-existence dedup.

Atomic write

The engine stages writes to OutputDir/.<name>.partial and renames into place only after the filter completes. A failed run removes the temp file and never publishes a partial result at the target path. Combined with the Watch API’s rename coalescing, downstream observers see a single ChangeCreated or ChangeRenamed event for the completed file — no spurious intermediate states.

Manifest annotation

filter_to_file appears in Manifest.Operations with annotations {streamable: false, deterministic: true, expensive: true}. The expensive: true hint is the signal to embedders that this is the operation worth caching aggressively — which the function itself already does via pre-existence dedup, so the typical consumer pattern is to simply call FilterToFileWithRequest repeatedly and trust the result.

See also

• Request Hashing — the hashing primitives underlying the deterministic naming.
• Watch & WatchDir — observe derived-output changes.

Header Layout

Audience: anyone reading or writing .pulse files by hand (forensics, custom readers, debugging a truncated file). The Go library handles all of this for you; this page documents the wire format.

The header is fixed-size: 9 bytes, consisting of an 8-byte magic identifier and a 1-byte format version.

LLM agents using MCP: see the cohort-schema-design skill via pulse_skills_get. It speaks in field-type semantics rather than byte layout; this page covers the bytes.

Constants

These live in encoding/header.go:

Byte layout

Offset  Length  Field
------  ------  -----
0       8       Magic: "PULSE\0\0\0"
8       1       Format version (currently 0x01)
9       —       Schema block begins here

That’s the entire fixed header. The schema block immediately follows; see Schema Block.

Version semantics

The format version is single-byte. The reader at encoding.ReadHeader rejects unknown versions with the ENCODING_INVALID error code:

ENCODING_INVALID: unsupported pulse format version
{"version": <byte>}

This is the fail-loud guard against silently mis-decoding a file written by a future binary that introduced a new field type or layout change. A forward-incompatible change bumps the version; the older reader stops at header parse instead of producing wrong rows.

The current value is 0x01. The envelope format_version ("1.0") that all CLI --json output carries is unrelated — it tracks the JSON output schema, not the binary file format.

Hexdump sanity check

A freshly-written .pulse file starts with:

00000000  50 55 4c 53 45 00 00 00  01  ..  ..  ..  ..  ..
          |P  U  L  S  E  \0 \0 \0|ver| schema starts here

If file path/to/data.pulse reports “data” (rather than something plausible) and the first nine bytes don’t match the above, the file is either truncated or corrupted — see Troubleshooting.

What comes next

The schema block follows the header. Read it as documented in Schema Block; it carries per-field descriptors, inline categorical dictionaries, and decimal/H3 metadata. After the schema, fixed-width records start — see Record Layout.

Field Types

Audience: anyone designing a cohort schema, decoding a .pulse file by hand, or trying to understand which type to pick for a column.

Pulse supports 17 field types, each with a fixed type byte, a fixed (or bit-packed) byte size, and well-defined semantics. The full list, mirrored from CLAUDE.md → All 17 field types:

LLM agents using MCP: see the cohort-schema-design skill via pulse_skills_get — it covers nullability, bit-packing trade-offs, and “which type to pick” with MCP-side examples.

The catalog

The Go source-of-truth for this table is encoding/field_type.go; the FieldType enum’s iota order is the byte-value order above.

Type families

Plain integers and floats

u8, u16, u32, u64, f32, f64. Standard little-endian encoding, full range, no null sentinel. Use these when you know the column never carries a missing value.

Nullable integers

nullable_u8, nullable_u16, nullable_u4, nullable_bool. Each reserves one in-band value (or one in-band bit pattern) to mean “null”. For the byte-sized variants the encoding is straightforward; for the sub-byte variants (nullable_u4, nullable_bool, packed_bool) Pulse packs multiple fields into shared bytes — see Record Layout → Bit-packing.

ByteSize() returns 0 for the bit-packed types because they don’t allocate whole bytes of their own; the schema reader uses BitPosition to locate them within shared bytes.

Date

date is a 32-bit count of days since the Unix epoch. The range is ~5.8 million years on either side of 1970 — effectively unbounded for real data.

Categoricals

categorical_u8, categorical_u16, categorical_u32. Each stores its string-to-ID mapping inline as a dictionary block immediately after the field’s schema entry. Pick the smallest variant that fits your cardinality (Pulse’s import path auto-selects during inference).

Dictionary mechanics are documented in Dictionary Blocks.

Decimal128

decimal128 and nullable_decimal128 are 16-byte fixed-point decimal numbers. Each field carries a per-field (precision, scale) pair written into the schema after the description; precision and scale both top out at 38 (PULSE_DECIMAL_OVERFLOW, PULSE_DECIMAL_PRECISION_LOSS).

Use these for currency and any other column where IEEE-754 rounding is not acceptable. See the financial-cohorts skill for full semantics including banker’s rounding and divide-by-zero policy.

Unknown type bytes

The schema reader rejects unknown FieldType bytes at parse time with ENCODING_INVALID. This is the same fail-loud strategy as the header version check: a file written by a future binary that introduced a new type fails immediately at schema parse, not later during row decode where the corruption could go unnoticed.

What you can do with each type

Schema Block

Audience: anyone decoding a .pulse file by hand or writing a non-Go reader. The schema block follows the 9-byte header and carries one descriptor per column.

From CLAUDE.md, byte-layout invariants for .pulse files, plus the on-disk format documented in encoding/schema.go.

Top-level shape

u16 field_count
field_record × field_count

Each field_record is variable-width (it includes UTF-8 name and description strings, and may include a categorical dictionary or decimal/H3 metadata). The reader walks them sequentially.

Per-field record

In write order — see WriteSchema / ReadSchema in encoding/schema.go:

Order matters: every reader walks these in the listed order, so a malformed record stops the parse with ENCODING_INVALID.

Byte offsets and bit positions

byte_offset is the offset of this field’s first byte within a record. For bit-packed types (packed_bool, nullable_bool, nullable_u4), byte_offset plus bit_position together locate the field’s bits within a byte that may be shared with adjacent fields.

For non-packed types, bit_position is always 0.

Record layout mechanics — including the bit-packing rule, record-size computation, and how the encoder packs adjacent sub-byte fields — are in Record Layout.

Conditional trailers

Two trailers attach only to specific field types:

• decimal128 / nullable_decimal128 get a (precision, scale) pair (u8, u8). Both ≤ 38.
• Categorical types (categorical_u8, categorical_u16, categorical_u32) get a full dictionary block in line — see Dictionary Blocks.

A field with none of the above writes nothing after the description.

Field descriptions

The description string is UTF-8 with a 2-byte length prefix. The import path rejects descriptions longer than 1000 bytes (PULSE_IMPORT_DESCRIPTION_TOO_LONG) and warns on low-quality descriptions (empty, under 10 characters, or generic words like "n/a", "tbd", "unknown", "field", "data", "value", "column") — that warning is PULSE_FIELD_DESCRIPTION_LOW_QUALITY, upgraded to an error under --strict.

When the description is empty, pulse cohort inspect synthesises a fallback string (“Categorical field: ” or “Numeric field: ”) with description_source = "synthesized". The original bytes on disk remain empty.

Reader behaviour

encoding.ReadSchema is intentionally strict:

• Field count limit comes from the u16 prefix (max 65,535 fields).
• Unknown type bytes fail loud (ENCODING_INVALID).
• Truncated records fail loud at the first short read.
• The reader produces a *encoding.Schema with one encoding.Field per record; Schema.Field(name) looks fields up by name.

After the schema block, record data starts at the file’s first byte past the schema. The record layout is documented in Record Layout.

Dictionary Blocks

Audience: anyone decoding categorical fields, sizing a categorical type during import, or chasing a dictionary-overflow error.

Categorical fields (categorical_u8, categorical_u16, categorical_u32) store their string-to-ID mapping inline, immediately after the field’s schema entry. The dictionary is part of the schema block, not the record data.

LLM agents using MCP: the cohort-schema-design skill covers when to pick which categorical width; the import-best-practices skill covers fail-closed semantics on overflow.

On-disk layout

From encoding/dictionary.go:

u32 count
(u16 strlen + utf8 bytes) × count

Sizes are little-endian. Each entry’s ID is its insertion index (0..count-1); ID lookups during decode use the ID found in the record byte(s) and resolve to the string at that index.

Sizing the type

The import path samples the source (--sample-rows, default 500) to estimate cardinality and picks the smallest width that fits. You can also force a width by editing the schema template (pulse import schema-template SOURCE).

Overflow and unbounded errors

AddWithLimit enforces the per-type cap and returns PULSE_IMPORT_CATEGORICAL_OVERFLOW when the source has more distinct values than the dictionary can hold:

{
  "code": "PULSE_IMPORT_CATEGORICAL_OVERFLOW",
  "message": "categorical dictionary overflow: max 256 entries",
  "details": {"max_entries": 256, "value": "the_257th_distinct_string"}
}

The companion code PULSE_IMPORT_CATEGORICAL_UNBOUNDED fires when the import path detects an effectively unbounded categorical column (the schema declared categorical_u32 and the column still grew past the caller-provided guardrails). Both errors halt the import — fail-closed, no partial output.

Recovery options, in order of preference:

1. Re-import with a wider categorical type (categorical_u8 → categorical_u16 → categorical_u32).
2. Drop the categorical encoding (treat the column as a plain string field — but Pulse has no native variable-string type; you’d add a pre-import transform to bucket values).
3. Pre-filter the source to a smaller distinct set and re-import.

Inspect behaviour

pulse cohort inspect --json reports each categorical field’s dictionary entry count and sample values. By default the inline list is capped at 100 entries (DefaultDictionaryLimit); pass --full-dict to print the full dictionary:

pulse cohort inspect data.pulse --full-dict --json

Both forms include a truncated: true|false flag and a total_entries count for programmatic consumers.

Performance notes

Dictionary reads are amortised: the reader allocates one shared byte buffer for all string payloads, then does one string(...) copy per entry. This avoids the “one allocation per entry” overhead that naively reading length-prefixed strings would produce. The dictionary itself is held in memory for the life of the cohort’s schema parse.

For very large dictionaries, the categorical_u32 path is still O(N) to deserialise; if you find yourself near the 32-bit cap, you almost certainly want a different model (a separate lookup table, or a plain integer column with the strings stored externally).

Record Layout

Audience: anyone hand-decoding row data or implementing a non-Go reader. The schema block ends; record data starts immediately after.

Records are fixed-width. Every row in a cohort occupies the same number of bytes, computed from the schema’s field types. Variable-width data (strings) lives in the schema (as categorical dictionaries) or is not directly supported.

LLM agents using MCP: the record byte layout is an implementation detail the MCP surface hides — there is no LLM-facing skill for it. The MCP tools operate on the inspect / process / sample abstractions.

Computing record size

Record size is the sum of FieldType.ByteSize() over all schema fields, plus padding bytes that share bits between sub-byte fields. For non-packed types, ByteSize() returns the obvious value (u32 = 4, f64 = 8, decimal128 = 16); for packed types (packed_bool, nullable_bool, nullable_u4), ByteSize() returns 0 and the field shares a byte with adjacent packed fields.

The writer (encoding/record.go) lays out fields in the order they appear in the schema; the reader walks the same order with the per-field ByteOffset and BitPosition recorded in the schema.

Encoding per type

From WriteFieldValue / ReadFieldValue in encoding/record.go:

Bit-packing

Sub-byte types share whole bytes with their packed neighbours. The schema records both ByteOffset (the shared byte’s offset) and BitPosition (which bit slot within that byte).

• packed_bool — 1 bit (true/false).
• nullable_bool — 2 bits (one null bit, one value bit) for the tri-state encoding.
• nullable_u4 — 5 bits (one null bit, four value bits) for the nullable 4-bit unsigned encoding.

The writer aligns these into shared bytes from low bit to high bit; adjacent packed fields stack into the same byte until the byte is full, after which a new byte begins. ByteSize() == 0 is the schema reader’s signal that a field type shares bytes — non-zero ByteSize fields never share.

Null sentinels

u32, u64, f32, f64, date, decimal128 (non-nullable), and all categoricals are non-nullable — the import path either coerces or rejects rows with missing values (PULSE_IMPORT_ROW_ERROR). Pick the nullable_* variant when you need to preserve the difference between “zero” and “missing”.

Reading a record

The Go decoder lives at encoding.Reader / encoding.ReadRecord(*Schema, []byte). A non-Go reader can follow the same recipe:

1. Compute record size from the schema.
2. Read record_size bytes.
3. For each schema field in declaration order:
   • If ByteSize() > 0, decode the value at the field’s ByteOffset.
   • If ByteSize() == 0, decode the bit slot at (ByteOffset, BitPosition) using the type’s bit-pattern rules.

Forward compatibility

Records carry no type tag — they’re a packed binary blob whose interpretation comes entirely from the schema block. That’s why the file’s format version (in the header) and unknown field-type bytes (in the schema block) both fail loud at parse time: the records themselves cannot self-correct, so the format gates everything before record data is observed.

MCP Integration

Audience: operators wiring Pulse into an MCP-aware AI client (Claude Desktop, Claude Code, Cursor, Zed, custom hosts), and embedders who want to expose Pulse to an LLM agent.

This page is the human-facing guide: what the server does, how to wire it up, what the LLM sees, and how to debug a misbehaving session. Agent-facing guidance ships inside the binary as the mcp-integration skill — fetch it via pulse_skills_get (or pulse skills show mcp-integration).

What pulse mcp is

pulse mcp runs the Pulse library as a Model Context Protocol (MCP) server. The host (Claude Desktop, Claude Code, etc.) launches it as a subprocess, speaks JSON-RPC over its stdio streams, and shuts it down on session close. The LLM sees Pulse as a set of tools (callable functions), resources (browseable URIs), and prompts (canned slash commands).

┌─────────────┐  stdio JSON-RPC  ┌────────────┐  Go calls  ┌─────────────┐
│  AI client  │ ───────────────→ │ pulse mcp  │ ─────────→ │ pulse.Pulse │
│   (host)    │ ←─────────────── │ (this bin) │ ←───────── │  (library)  │
└─────────────┘                  └────────────┘            └─────────────┘
                                       │
                                       └── stderr ─→ host log pane

The server is a thin translator. Every tool wraps a public method on pulse.Pulse; the same code path powers the CLI.

Quickstart

# 1. Build and place on PATH
make build && cp ./bin/pulse /usr/local/bin/

# 2. Pick a data directory
mkdir -p /var/data/pulse

# 3. Wire into your host (see below) and restart it

# 4. From the LLM session, call:
#    pulse_manifest      → cache once
#    pulse_inspect       → open a cohort
#    pulse_predict       → validate a request
#    pulse_process       → execute

Wiring into a host

Claude Desktop

Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "pulse": {
      "command": "/usr/local/bin/pulse",
      "args": ["mcp"],
      "env": {
        "PULSE_DATA_DIR": "/var/data/pulse"
      }
    }
  }
}

Restart Claude Desktop. Pulse tools appear in the tool picker.

Claude Code

claude mcp add pulse --env PULSE_DATA_DIR=/var/data/pulse -- pulse mcp

Or by hand in ~/.claude.json (or per-project .claude.json):

{
  "mcpServers": {
    "pulse": {
      "command": "/usr/local/bin/pulse",
      "args":    ["mcp"],
      "env":     { "PULSE_DATA_DIR": "/var/data/pulse" }
    }
  }
}

Cursor / Zed / generic stdio hosts

Any host that speaks the MCP stdio transport can launch pulse mcp the same way — provide the binary path, the mcp argument, and the PULSE_DATA_DIR env var.

What the LLM sees

Tool surface

Fifteen tools, registered at server start. Names and order match mcp/toolmeta/meta.go.

Resources

Resources are registered once at server start. Files added afterwards do not appear until the server restarts. Listing is cheap because the server only reads header bytes.

Prompts

Hosts that surface prompts as slash commands let users trigger these directly.

Recommended session flow

The default sequence for nearly every user request:

1. pulse_manifest once at session start. No arguments. Cache the payload — it is deterministic for a binary version and carries every fact needed to author a valid request.
2. pulse_import when the user hands the LLM a raw tabular file (CSV/TSV/NDJSON/JSONArray/Parquet/Arrow/Excel). Returns a managed handle that subsequent tools address as if it were a .pulse. Skip when the cohort already exists as a managed handle or .pulse file in PULSE_DATA_DIR.
3. pulse_inspect on the handle (or path). Reads header + schema only — no record bytes — and registers session-scoped, schema-bound variants of the action tools (see below).
4. pulse_predict with the authored request. Validates against the schema, surfaces applied defaults, streamability, and any structural errors / warnings. Each error code carries Fixup metadata so the LLM can repair the request without another round trip.
5. pulse_process to execute. Use pulse_compose to batch multiple requests against the same cohort in one round trip.

Cheaper probes are available without going through pulse_process:

• pulse_sample for a row preview.
• pulse_facet for distinct values of a single field.
• pulse_examples_search / pulse_examples_get to crib a runnable request template from the embedded library.

Managed imports + TTL

pulse_import lets the LLM hand the server any tabular file and address it from then on as if it were a .pulse.

• Convertible formats (csv, tsv, ndjson, jsonarray, parquet, arrow, excel) are imported into $PULSE_DATA_DIR/imports/<handle>.pulse with a sidecar <handle>.pulse.meta.json carrying imported_at, expires_at, ttl_seconds, source path, source format, and row count. result.managed=true.
• Pulse passthroughs (.pulse extension) under PULSE_DATA_DIR are not copied — the server returns the relative path verbatim with managed=false. A .pulse outside PULSE_DATA_DIR is copied into the managed pool.

Source path resolution. Relative source paths resolve against PULSE_DATA_DIR. Absolute paths read from the host filesystem through a separate “source fs.”

Import jail. Absolute source paths are confined to a single directory tree (the jail root). Default: the working directory the MCP server was launched from. Paths that escape the jail (including ..) return PULSE_IMPORT_SOURCE_FORBIDDEN. Override via pulse.Options.ImportSourceJailRoot when embedding.

Sliding TTL. Default lifetime is 7d (overridable via PULSE_IMPORT_TTL, or per-import via the ttl field — accepts Go duration like "24h", day form like "7d", or "pin" for never-expire). Every subsequent inspect/predict/process/sample/facet against the handle slides expires_at forward. The pool self-sweeps on every pulse_import call — no daemon required. Inspect with pulse_imports_list; evict manually with pulse_drop.

Schema-bound enums

After a successful pulse_inspect, the server registers session-scoped variants of the action tools (pulse_process, pulse_predict, pulse_compose, pulse_sample, pulse_facet) whose JSON Schemas embed enum constraints on field-name parameters. The LLM picks field names from a typed list rather than free-texting and discovering on predict that the name was wrong.

What gets constrained on bound pulse_process / pulse_predict / pulse_compose schemas:

Trigger and lifecycle. Binding fires on a successful pulse_inspect. The go-sdk server auto-fires notifications/tools/list_changed on the post-serve AddTool / RemoveTools swap; the host refreshes its tool list and picks up the bound schemas on the next list. Bound tools share names with the global tools — the session-scoped variants override globals for that session.

Limitations.

• Multi-file sessions: the latest inspect wins. Track multiple cohorts client-side.
• No per-element type ↔ field correlation: JSON Schema can’t easily express “if aggregations[i].type == AGG_SUM then aggregations[i].field must be numeric.” Operator–type compatibility lives in the type property description; strict validation remains pulse_predict’s job.
• Transport support: binding mutates the single stdio-session server post-serve (AddTool / RemoveTools auto-emit list_changed), so stdio — the transport pulse mcp ships — honours it. There is no per-session schema override for a shared HTTP server, a documented limitation. The manifest’s accepts_types table is still authoritative, so authoring is never blocked regardless of transport.
• Empty enums omitted: when the cohort has zero fields in a category (e.g. no geo fields), the enum is omitted entirely rather than emitted as [].

Disable binding entirely with --bind-on-open=false.

Configuration

Embedders can override per-instance via pulse.Options{DataDir, ImportsDir, ImportTTL, ImportSourceJailRoot, FS, ImportSourceFS, BindOnOpen} — see pulse.go.

Transport caveats

• Stdio. The default and only transport pulse mcp ships today. Schema bind-on-inspect works on the single stdio session (see Limitations). Stdout is the JSON-RPC channel; stderr is the log channel — never write structured output to stdout outside the protocol.
• Streamable HTTP. Not exposed by the mcp CLI leaf yet. The underlying go-sdk server supports it; embedders build their own go-sdk server and call gosdk.Register(server, p, cfg) (or use mcpserve.Serve), then serve over the transport of their choice. Bind-on-inspect has no per-session override on a shared HTTP server (see Limitations).

Troubleshooting

To see what the server registers without launching the host:

pulse --json | jq '.data.mcp_tools[]'
pulse manifest --json | jq '.data.skills[]'

Skill cross-reference for LLM agents

If you are writing a system prompt for an LLM agent that uses Pulse, point it at these skills rather than at this site:

The agent should call pulse_skills_list once at session start to enumerate the catalog, then pulse_skills_get on demand. The returned text is authoritative; this site does not duplicate it and may lag.

Related

• mcp (CLI leaf) — flag reference and exit codes for the server binary
• Deployment — production hardening notes
• Troubleshooting — non-MCP failure modes

Payload JSON Schema

Pulse publishes a single machine-readable JSON Schema (draft 2020-12) describing every public payload — the request envelopes and the universal --json output envelope. Use it to validate requests before sending them, to generate client types, or to drive editor autocompletion.

Where to get it

The schema is reachable three ways, all backed by the same generator (descriptor.BuildPayloadSchema):

The CLI and MCP surfaces emit byte-identical output to the published file.

Structure

The document is a $defs bundle. The root oneOf lists the entry points:

• Requests — #/$defs/Request (process / predict), ComposedRequest (compose), ChainRequest (process-chain), FacetRequest, SampleRequest.
• Results — #/$defs/Response, ComposedResponse, ChainResponse, FacetResult.
• Envelope — #/$defs/Envelope, the universal --json wrapper. Its data slot is intentionally open: it carries whatever the operation returned (a Response, the manifest, a predict result, an inspect result, …). To validate a wrapped result strictly, validate the unwrapped data value against its own def (e.g. #/$defs/Response).

Example — validate a request body with any draft-2020-12 validator:

pulse schema > payload-schema.json
# then point your validator at  payload-schema.json#/$defs/Request

How it stays in sync

The schema is generated, never hand-maintained, so it cannot silently drift from the engine:

1. Reflection over the Go payload structs — a new or renamed field changes the output.
2. Registry-injected enums — the operator, overlay-kind, and regression discriminants draw their value lists from the same types.All*Types() / AllOverlayKinds() / AllRegressionTypes() registries the engine executes against, so registering a new operator changes the schema.
3. Hand-tuned strict unions for the two shapes reflection cannot express: OverlayRef (at most one arm populated) and OverlayPayload (shape-discriminated scalar / series / matrix).

A golden test (TestPayloadSchemaGolden) pins the output and an enum-parity test (TestPayloadSchema_EnumsMatchRegistry) fails CI on drift; the schema’s format_version is held equal to the envelope’s (TestPayloadSchema_VersionMatchesEnvelope). Regenerate after an intentional payload change with:

go test ./descriptor/ -run TestPayloadSchemaGolden -update

v1 boundaries

The schema is faithful but not maximally strict in two places, by design:

• Operator params (the json.RawMessage slot on aggregations, groupers, overlays, etc.) is an open object. There is no central declarative source for per-operator input parameters — each operator’s param schema lives alongside its processor — so encoding it here would duplicate that surface and rot. Consult the operator’s skill / manifest entry for its accepted params.
• Small closed mode enums (OverlayScope, OverlayShape, CrosstabNormalize, CrosstabShape, LabelMode, MarginAxis) are typed as plain strings rather than enums — they have no registry helper, and a hardcoded list would drift silently.

Cross-slot validation rules that depend on more than one field (e.g. a crosstab requires at least one row and one column, mutually exclusive with top-level groups) are enforced by pulse predict at request time, not by the schema.

Request Example Library

Pulse ships a searchable, embedded catalogue of runnable request JSON files spanning every operator category. They are checked into the repo under examples/, mounted into the binary at compile time via //go:embed, and surfaced through three peer access paths:

What the library contains

Every example is a complete types.Request JSON body — the same shape you hand to pulse_process. Each file is annotated with a structured _meta block describing the example. Pulse’s JSON unmarshaller ignores unknown fields by default, so the _meta block is invisible at execution time; the file remains runnable verbatim.

{
  "_meta": {
    "name": "t_test_one_sample",
    "category": "tests",
    "tags": ["hypothesis-test", "t-test", "tier-1-test", "parametric", "one-sample", "streaming-friendly"],
    "operators": ["AGG_AVERAGE", "AGG_COUNT", "TEST_T"],
    "description": "One-sample t-test comparing revenue mean against the hypothesized mu=100."
  },
  "cohort": {...},
  ...
}

Fetching via pulse_examples_get returns the request body with the _meta block already stripped, so you can pass it straight to pulse_process / pulse_predict.

Searching the library

Three filter dimensions, all optional and combined with AND:

CLI

pulse examples search --query welch                       # find Welch-related examples
pulse examples search --tag time-series --tag tier-2-test # AND tag filter
pulse examples search --category tests --json             # JSON envelope
pulse examples show t_test_one_sample                     # print runnable JSON
pulse examples show t_test_one_sample --json              # full record (with _meta)

MCP

// arguments to pulse_examples_search
{"query": "welch"}
{"tags": ["time-series", "tier-2-test"]}
{"category": "features"}

Go API

p, _ := pulse.New(pulse.Options{DataDir: "/data"})

// Search:
hits := p.ExamplesSearch("welch", []string{"experiment-analysis"}, "")
for _, h := range hits {
    fmt.Println(h.Name, "—", h.Description)
}

// Fetch and run:
ex, ok := p.ExampleGet("t_test_one_sample")
if ok {
    var req pulse.Request
    _ = json.Unmarshal(ex.Body, &req)
    resp, _ := p.Process(ctx, &req)
    _ = resp
}

Tag taxonomy

Tags are curated and validated by a CI gate (TestExamples_TagsFromTaxonomy). The taxonomy spans four dimensions:

The category (directory name) is not repeated in the tags — _meta.category carries that.

Adding a new example

1. Write the request JSON under examples/<category>/. Use existing files as shape templates. Keep cohort.data_dir = ".data" and reference one of the fixture cohorts.
2. Add a _meta block at the top of the file:
   • name — kebab-case-with-underscores, unique across the whole library.
   • category — must match the parent directory.
   • tags — pick 3-6 from the taxonomy above.
   • operators — the list of AGG_* / ATTR_* / FILTER_* / GROUP_* / WIN_* / FEAT_* / TEST_* types appearing in the body, alphabetized and deduped.
   • description — one-sentence, present-tense summary.
3. Re-run go test ./examples/... ./descriptor/... to confirm the new file passes:
   • TestExamples_AllParseAsRequest
   • TestExamples_UniqueNames
   • TestExamples_TagsFromTaxonomy
   • TestExamples_OperatorsMatchBody
   • TestExamples_CategoryMatchesDirectory
   • TestManifestExamplesPopulated
4. The annotation tool at cmd/annotate-examples/ is idempotent and may be re-used; updating its in-source annotations slice and re-running will rewrite the file’s _meta block in canonical form.

Regression Modeling

Pulse exposes regression through a compact, composable surface. Three operators, two orthogonal modifiers, and one upstream feature transform together cover every textbook regression variant. This chapter is the human-facing counterpart to skills/regression-modeling.md; agents should fetch the skill via pulse_skills_get rather than read this page.

Overview

Two spec-level modifiers compose with any of the three:

• Resample ∈ {jackknife, bootstrap} — replaces analytical SE / p-values with resample-based estimates. Forces buffered.
• Selection ∈ {forward, backward, stepwise} — drives AIC- or BIC-based greedy subset search. Requires Criterion. Forces buffered.

One upstream feature operator (FEAT_POLY) extends the linear core to polynomial regression. Per-row attributes (ATTR_REG_FITTED, ATTR_REG_RESIDUAL, ATTR_REG_LEVERAGE) attach per-record diagnostics in the output row stream.

The 13 textbook names → Pulse specs

The Indeed regression taxonomy double-counts (Simple ≡ Linear univariate, Multiple ≡ Multiple Linear) and treats orthogonal wrappers (Jackknife, Stepwise) as families. Pulse does not. The table below maps each textbook name onto the corresponding Pulse spec and links to a runnable example file under examples/regression/.

Streamability matrix

pulse_predict reports per-request streamability on PredictResult.Streamable, mirroring the runtime gate.

Operator reference

REG_OLS

Ordinary least squares with optional regularization.

Modifier compatibility: Resample and Selection may be combined; Selection runs first, Resample re-fits the selected subset.

Error codes: PROCESSING_REGRESSION_RANK_DEFICIENT, PROCESSING_REGRESSION_SINGULAR_GRAM, PROCESSING_REGRESSION_NO_CONVERGE, PROCESSING_REGRESSION_INSUFFICIENT_DATA, PROCESSING_REGRESSION_APPROXIMATE_SE (warning, l1/elasticnet without resample), PROCESSING_REGRESSION_REGULARIZED_SELECTION (warning, penalty + selection), PROCESSING_CONFIG.

REG_GLM

Generalized linear model via iteratively-reweighted least squares.

Always buffered. Setting penalty / alpha / l1_ratio on a REG_GLM spec is rejected with PROCESSING_CONFIG; regularized GLM is reserved for a later phase.

Error codes: PROCESSING_REGRESSION_INVALID_FAMILY, PROCESSING_REGRESSION_INVALID_LINK, PROCESSING_REGRESSION_NO_CONVERGE, PROCESSING_REGRESSION_INSUFFICIENT_DATA, PROCESSING_CONFIG.

REG_BAYES_LINEAR

Bayesian linear regression with a conjugate Normal-Inverse-Gamma prior.

Modifier compatibility: Resample and Selection are rejected for REG_BAYES_LINEAR at spec validation — the posterior already conveys uncertainty via credible intervals, and stepwise feature selection on a Bayesian model is a posterior-based question the conjugate-NIG engine doesn’t support.

Setting penalty / alpha / l1_ratio / family / link on a Bayes spec is rejected with PROCESSING_CONFIG.

Error codes: PROCESSING_REGRESSION_RANK_DEFICIENT, PROCESSING_REGRESSION_INSUFFICIENT_DATA, PROCESSING_CONFIG.

Modifiers

Resample

Layered on top of any base operator (except REG_BAYES_LINEAR).

For l1 / elasticnet OLS, setting Resample is the rigorous answer for standard errors: it suppresses the PROCESSING_REGRESSION_APPROXIMATE_SE warning (the SEs are now resample-based, not plug-in over the active set).

Selection

Layered on top of any base operator (except REG_BAYES_LINEAR).

Requires Criterion ∈ {"aic", "bic"}.

• AIC = -2·logL + 2·k. Lighter penalty; may retain weak predictors at moderate n.
• BIC = -2·logL + log(n)·k. Heavier per-parameter penalty; rejects noise predictors more reliably at moderate n.

SelectedFeatures lists the chosen subset; Coefficients drops non-selected predictors entirely (absence ≠ zero — selection’s contract is stronger). Selection may be combined with Resample: Selection picks the active subset, then Resample replaces SE / p-values on the selected model.

Compositional patterns

Polynomial regression — FEAT_POLY + REG_OLS

Polynomial regression is linear in the coefficients; the non-linearity lives in the feature space. Use FEAT_POLY upstream to materialize x_2, x_3, …, x_<degree> derived columns, then list them alongside the original x in predictors:

{
  "features": [
    {"type": "FEAT_POLY", "field": "x", "label": "x", "params": {"degree": 3}}
  ],
  "regressions": [
    {"type": "REG_OLS", "name": "polyfit", "target": "y",
     "predictors": ["x", "x_2", "x_3"]}
  ]
}

Degree is gated at [2, 10]. Numerical stability is the caller’s responsibility: x^10 overflows f64 once |x| clears a few hundred, and the Gram matrix conditions poorly long before that. Centre or standardize predictors before requesting FEAT_POLY.

Ecological regression — group → regress

“Ecological regression” is a regression fit on aggregated group-level statistics — per-precinct means, per-county sums, per-region rates — rather than individual-level rows. Use pulse_compose with two slots: slot 1 produces per-group means via GROUP_* + AGG_AVERAGE, slot 2 fits REG_OLS over the aggregate output (or, in practice, over a pre-aggregated .pulse file).

The two slots are intentionally independent; Pulse does not pipe slot-1 results into slot-2 as cohort input. Either (a) materialize slot 1’s aggregate as its own .pulse cohort upstream, or (b) treat slot 1 as the audit trail (per-group means visible in the composed response) and run slot 2 over a pre-aggregated fixture.

Caution — the ecological fallacy. A significant group-level slope does not imply an individual-level association. Robinson (1950) showed that ecological correlations and individual correlations can take opposite signs in the same data: a per-state regression of literacy on race might suggest a strong relationship that vanishes (or reverses) at the per-person level. Aggregation collapses within-group variation, leaving only between-group structure that frequently encodes confounders.

When ecological regression is the right tool: aggregate-only data (census output, public-health summary tables); genuinely group-level research questions (“do counties with higher median income have higher turnout?”). When it is the wrong tool: individual-level claims; causal claims. Annotate consumer-facing prose with this caveat; Pulse cannot enforce it.

Robinson, W.S. (1950). “Ecological Correlations and the Behavior of Individuals.” American Sociological Review 15(3): 351–357.

Per-row regression attributes

Three attribute operators emit per-record diagnostics from a fitted regression onto the row stream.

Each attribute references a sibling regression spec by regression_name. See skills/attribute-composition.md for the parameter table.

Error codes

Look up full prose via pulse_errors_lookup or pulse errors lookup CODE.

Worked examples

Every Indeed name has a runnable JSON file under examples/regression/. Fetch via pulse_examples_get or read directly:

• 01_ecological_fallacy.json — per-region aggregation + ecological caveat (#12).
• 02_simple_linear.json — univariate OLS (#1, #3).
• 03_multiple_linear.json — multivariate OLS (#2, #4).
• 04_logistic.json — binary classification (#5).
• 05_ridge.json — l2 penalty (#6).
• 06_lasso.json — l1 penalty (#7).
• 07_polynomial.json — FEAT_POLY + OLS (#8).
• 08_bayesian_linear.json — conjugate NIG (#9).
• 09_jackknife.json — leave-one-out resampling (#10).
• 10_elasticnet.json — combined l1 / l2 penalty (#11).
• 11_stepwise.json — BIC-driven stepwise selection (#13).

Architecture Overview

Source of truth: the canonical architectural contract is CLAUDE.md at the repository root. This chapter restates its design principles for human readers; if the two ever disagree, CLAUDE.md is authoritative.

Pulse is a high-performance, self-describing tabular data processing engine. It ships as a Go library (github.com/frankbardon/pulse) and as a CLI binary (cmd/pulse/). The library is the primary deliverable; the CLI is a thin adapter over it.

Design principles

• Library-first. The pulse.go facade (pulse.New, pulse.Options, pulse.Process, pulse.Compose, pulse.Import, pulse.Export, pulse.Convert, pulse.Inspect, pulse.Predict, pulse.Sample, pulse.Facet) is the public API. The CLI calls the library; it never contains business logic.
• Self-describing. Every .pulse file carries its schema in the header. The descriptor/ package provides manifest, predict, and inspect operations that expose the system’s capabilities and validate requests without executing them.
• Skill-augmented. The skills/ package embeds 19 markdown skill files into the binary via //go:embed. LLM agents (and Nexus, the orchestration layer that consumes Pulse) can call skills.List() and skills.Get(name) at boot time to inject domain-specific guidance into their context.
• Nexus relationship. Pulse is a standalone processing engine. Nexus is the upstream orchestration agent that calls Pulse’s library API or CLI. Pulse has no dependency on Nexus. Nexus discovers Pulse’s capabilities via pulse manifest --json and loads skills from the embedded skill pack.

The next chapter, Package Layout, shows where each of these concerns lives in the source tree.

Package Layout

Source of truth: this tree is mirrored from the “Package layout” section of CLAUDE.md. If the project structure changes, that file is updated first; this page follows.

pulse/
├── cmd/
│   └── pulse/              # CLI binary (the only binary)
├── pulse.go                # Public facade — pulse.New, pulse.Options
├── service/                # Orchestration layer; wires processing to encoding
├── processing/             # Aggregators, attributes, filterers, groupers, windows, features
│   ├── window/             # WIN_* operators (LAG, LEAD, RANK, RUNNING_*, EWMA, ...)
│   └── feature/            # FEAT_* pre-filter feature engineers (LOG, SQRT, BUCKETIZE, ...)
├── encoding/               # Dynamic schema + record codec (.pulse binary format)
├── io/                     # Bidirectional tabular <-> .pulse adapters
│   ├── csv/                # CSV reader + writer
│   ├── tsv/                # TSV reader + writer
│   ├── ndjson/             # NDJSON reader + writer
│   ├── jsonarray/          # JSON-array reader + writer (single top-level array of flat objects)
│   ├── jsonshared/         # Value coercion helpers shared by ndjson and jsonarray
│   ├── arrow/              # Arrow IPC / Feather V2 reader + writer; shared Arrow<->Pulse type maps
│   ├── parquet/            # Parquet reader + writer (delegates type maps to io/arrow)
│   └── excel/              # Excel reader + writer (Excelize)
├── fs/                     # afero-based filesystem abstraction + extension hook
├── errors/                 # Typed error codes (CodedError system)
├── types/                  # Request/response structs (JSON-serializable)
├── descriptor/             # Self-description: manifest, predict, inspect, envelope
├── skills/                 # Embedded markdown skill pack (//go:embed)
│   ├── index.json          # Manifest of all bundled skills
│   └── *.md                # Individual skill files with YAML frontmatter
├── mcp/                    # Public SDK-free MCP core (typed In/Out, reflected schemas, handlers, bind)
│   ├── gosdk/              # go-sdk adapter — the only package importing the MCP SDK; gosdk.Register
│   └── toolmeta/           # Leaf metadata package (tool names + descriptions) consumed by descriptor + core
├── synth/                  # Synthetic data generator (from-schema, from-profile)
├── docs/                   # mdBook source for this site (published to GitHub Pages)
└── internal/
    └── cli/                # CLI internals (descriptor walker, json action)

Extension Points

Audience: Pulse embedders writing Go code that calls pulse.New(pulse.Options{Extensions: ...}) to inject domain-specific operators or expression-runtime extensions. If you are adding a new built-in operator to Pulse itself, see Adding an Aggregator (and its sibling recipes) instead.

The extension surface is Go-native. There is no plugin loader, no .so files, no hot reload. Registration happens once when the embedding binary constructs its Pulse instance, and the registration set is fixed for the lifetime of that instance — restart to change it. Registered extensions are first-class participants in Predict / Inspect / Process / Compose / Manifest: the runtime treats them identically to built-ins, the manifest advertises them, and the schema-bound MCP tools include their names in per-category enums.

The implementation lives in the repository root: extensions.go (public types), extensions_validate.go (name + registration shape checks), extensions_probe.go (factory probe and components parity), extensions_runtime.go (built-in/extension fold into the runtime registry), extensions_snapshot.go (descriptor-side read-only projection), and processing/extensions.go (the runtime overlay the processing layer consults).

When to register vs use a built-in

Use a built-in operator when the semantics already ship with Pulse, or when an ATTR_FORMULA plus a custom ExprFunction covers the behaviour. Use a registered extension when you need:

• An operator that encodes proprietary business logic that should not live in Pulse core (e.g. a domain-specific composite score).
• A closed-form function call from within ATTR_FORMULA / FILTER_EXPRESSION (e.g. rank_familiarity(value, total_pop)).
• A static keyed lookup table that drives multipliers or calibration factors (e.g. per-(study, wave) adjustment).
• Manifest + MCP visibility so LLM agents can discover and invoke your operators alongside Pulse built-ins.

The Extensions struct

The full public surface is a single struct on pulse.Options:

import "github.com/frankbardon/pulse"

ext := pulse.Extensions{
    Aggregators:        []pulse.AggregatorRegistration{...},
    Attributes:         []pulse.AttributeRegistration{...},
    Filterers:          []pulse.FiltererRegistration{...},
    Groupers:           []pulse.GrouperRegistration{...},
    Windows:            []pulse.WindowRegistration{...},
    Features:           []pulse.FeatureRegistration{...},
    Tests:              []pulse.TestRegistration{...},
    SynthDistributions: []pulse.DistributionRegistration{...},

    ExprFunctions: []pulse.ExprFunction{...},
    LookupTables:  map[string]pulse.LookupTable{...},
}

p, err := pulse.New(pulse.Options{
    FS:         myFs,
    Extensions: ext,
})

A zero-value Extensions is the no-op case — pulse.New behaves exactly as the unmodified binary. All registrations are validated together; the first failure short-circuits pulse.New with a typed CodedError.

Naming policy

Embedder registrations MUST use a three-segment namespaced form:

<CATEGORY>_<NAMESPACE>_<NAME>

The regex enforced by extensions_validate.go is:

^(AGG|ATTR|FILTER|GROUP|WIN|FEAT|TEST|SYNTH)_[A-Z][A-Z0-9]+_[A-Z](?:[A-Z0-9_]*[A-Z0-9])?$

Failure modes raised at registration time:

• Name fails the regex → PULSE_EXTENSION_NAME_INVALID.
• Namespace is one of the reserved values BUILTIN, STANDARD, CORE, PULSE → PULSE_EXTENSION_NAME_RESERVED.
• Name collides with a built-in (e.g. registering AGG_COUNT) → PULSE_EXTENSION_NAME_COLLISION.
• The same name appears twice in one pulse.New call → PULSE_EXTENSION_DUPLICATE.

Two embedders in the same process must use disjoint namespaces.

Per-category registration shapes

The eight operator categories plus expression functions and lookup tables cover everything pulse.Options.Extensions accepts. Field-input introspection and component-schema emission attach to operator registrations as additional optional fields — see the dedicated sections below.

Aggregator

{
    Name:        "AGG_ACME_BRAND_SCORE",
    Description: "ACME brand composite (0-100).",
    Factory:     acme.NewBrandScoreAggregator,    // processing.AggregatorFactory
    Streamable:  true,                            // factory MUST return OnlineAggregator
    Accepts:     []encoding.FieldType{encoding.FieldTypeF64},
    Params:      []pulse.ParamMeta{{Name: "weights", JSONType: "array"}},
    ComponentSchema: descriptor.ComponentSchema{ /* see below */ },
    ComponentsFunc:  func(instance processing.Aggregator) (map[string]any, error) { /* ... */ },
}

When Streamable=true, the probe at pulse.New time asserts that the factory’s returned value implements processing.OnlineAggregator. Mismatch surfaces as PULSE_EXTENSION_STREAMABLE_MISMATCH.

Attribute

{
    Name:        "ATTR_ACME_ADJUSTMENT",
    Description: "Per-(study, wave) multiplier.",
    Factory:     newAdjustmentAttribute,           // processing.AttributeFactory
    Mode:        pulse.AttributeModeRowLocal,       // row_local | two_pass | buffered
    Accepts:     []encoding.FieldType{encoding.FieldTypeF64},
    Emits:       pulse.AttributeEmitFloat64,
}

Mode drives streaming-tier validation: row_local requires processing.RowLocalAttribute, two_pass requires processing.TwoPassAttribute, buffered requires only the base processing.AttributeComputer. Attributes do NOT declare a ComponentSchema (see the table in the Component schemas section).

Filterer, Grouper, Window, Feature

All four follow the same envelope: name, description, factory, accepted types, params metadata. FiltererRegistration and GrouperRegistration additionally carry ComponentSchema + ComponentsFunc on the same contract as AggregatorRegistration. Filterers are always row-local streamable; windows always run buffered.

Test (tier-1 / tier-2)

// Tier-1 (folds during streaming aggregation pass):
{
    Name:       "TEST_ACME_PROXY",
    Tier:       pulse.TestTierRow,
    RowFactory: newProxyRowTest,                  // processing.RowTestFactory
    Streamable: true,
}

// Tier-2 (runs over materialised result rows after windows):
{
    Name:        "TEST_ACME_AGGREGATE_CHECK",
    Tier:        pulse.TestTierPost,
    PostFactory: newAggregateCheckPostTest,       // processing.PostTestFactory
}

Exactly one of RowFactory / PostFactory must be non-nil and match Tier. Tier-2 tests always run buffered; Streamable on a tier-2 registration is ignored.

Synth distribution

Reserved for embedders shipping bespoke samplers. The factory shape finalises alongside the synth distribution overlay phase; until then the registration validates name + duplicates and reserves the namespace.

Expression functions

Custom Go functions become callable from ATTR_FORMULA and FILTER_EXPRESSION:

{
    Name:        "rank_familiarity",
    Description: "ACME brand familiarity rank.",
    Signature:   "rank_familiarity(value float64, total_pop bool) float64",
    Fn:          acme.RankFamiliarity,
    Pure:        true,        // declares side-effect-free; reserved for future memoisation
}

Pulse passes Fn to expr-lang’s expr.Function(name, fn). expr-lang accepts typed functions via reflection — func(v float64) float64 and func(args ...any) (any, error) both work. Use the variadic shape when zero-allocation calling matters.

Lookup tables

Static keyed tables exposed via the built-in lookup(table, keys...) function:

LookupTables: map[string]pulse.LookupTable{
    "adjustments": {
        Description: "Per-(study, wave-date) calibration multipliers.",
        // Rows is the simple path — caller-joined composite key.
        Rows: map[string]float64{
            "study_a|2025-01-01": 1.07,
            "study_a|2025-02-01": 1.12,
        },
        // OR — Lookup is the escape hatch:
        // Lookup: func(keys ...string) (float64, bool, error) { ... }
    },
},

Exactly one of Rows / Lookup must be non-nil; validation at pulse.New returns PULSE_EXTENSION_PARAM_INVALID otherwise. Rows-backed tables join keys with | before indexing. The Lookup function-backed path receives the key slice directly — compose keys however you want, perform partial-match fallback, or pull from an external store.

At evaluation time lookup() raises PULSE_LOOKUP_TABLE_UNKNOWN for an unregistered table and PULSE_LOOKUP_MISS for a missing key. Both wrap into PROCESSING_RUNTIME when surfaced through ATTR_FORMULA / FILTER_EXPRESSION — use errors.HasCode(err, errors.PULSE_LOOKUP_MISS) to detect inside the chain.

Component schemas (v0.20.0)

Response.Components is the operator-keyed sibling payload that runs alongside Response.Data. Every aggregator, grouper, and filterer in a request lands a typed map under Response.Components.Aggregations[i].Operator, .Groupers[i].Operator, or .Filterers[i].Operator. Embedder-registered operators participate via two coupled optional fields on AggregatorRegistration, GrouperRegistration, and FiltererRegistration. The universal contract — typed shells, floor keys, additive omitempty shape — lives in skills/response-components.md; this section covers only the extension-side wiring.

Which categories declare a ComponentSchema?

A registration in a category that does not declare a ComponentSchema silently ignores the field if you set it.

Declaration shape

type ComponentSchema struct {
    Keys         []ComponentKey
    Mergeability ComponentsMergeability   // Mergeable | Partial | None
}

type ComponentKey struct {
    Name        string  // snake_case, matches the runtime emission key
    Type        string  // "int" | "float64" | "string" | "map" | "array" | "object"
    Description string  // surfaces in manifest + MCP schema
}

Keys is the canonical declared set. Names use snake_case (matches every other Pulse JSON key — n_null, mode_count, range_min). Type is a JSON-shape declaration that flows through to the manifest and schema-bound MCP tools; Description is the one-liner shown in LLM-bootstrap output.

The orchestrator owns the universal floor keys; the embedder’s ComponentsFunc returns ONLY operator-specific keys. Two conventions for what goes in Keys are tolerated by the probe — either list both floor and operator-specific keys (so a manifest reader sees the full payload shape), or list only operator-specific keys (the convention used by built-ins, where readers compose with the per-category floor table). The runtime contract is the same either way.

Mergeability axis

ComponentsMergeability declares how the components state composes when multiple chunks (streaming) or shards (parallel) flow into a single aggregator:

Choose the axis that matches the math, not the convenience of the registration site. Declaring Mergeable on an operator whose state cannot actually fold via MergeOnline produces silently-wrong components in parallel shard processing — there is no runtime gate for the math, only the streaming-tier wiring.

Two emission paths: ComponentsFunc and MetaAggregator

There are two equivalent ways to surface operator-specific keys at runtime; pick whichever fits your operator type.

ComponentsFunc closure (the registration-level path). Supply a closure on the registration; the runtime wraps the factory return value so the orchestrator can find it:

pulse.AggregatorRegistration{
    Name:    "AGG_ACME_BRAND_SCORE",
    Factory: acme.NewBrandScoreAggregator,
    Streamable: true,
    Accepts: []encoding.FieldType{encoding.FieldTypeF64},
    ComponentSchema: descriptor.ComponentSchema{
        Keys: []descriptor.ComponentKey{
            {Name: "weighted_sum", Type: "float64", Description: "Running weighted sum."},
            {Name: "weights_applied", Type: "int", Description: "Weight multipliers fired."},
        },
        Mergeability: descriptor.Mergeable,
    },
    ComponentsFunc: func(instance processing.Aggregator) (map[string]any, error) {
        a := instance.(*brandScoreAggregator)
        return map[string]any{
            "weighted_sum":    a.WeightedSum(),
            "weights_applied": a.WeightsApplied(),
        }, nil
    },
}

The closure signatures, defined in extensions.go, are:

type AggregatorComponentsFunc func(instance processing.Aggregator)     (map[string]any, error)
type GrouperComponentsFunc    func(instance processing.Grouper)         (map[string]any, error)
type FiltererComponentsFunc   func(instance processing.FiltererBuilder) (map[string]any, error)

The orchestrator invokes each func ONCE after the operator’s terminal pass (post-Aggregate/Finalize for aggregators; post-partition for groupers; post-eval for filterers). Returning (nil, nil) is the canonical signal for “no operator-specific keys; the orchestrator’s universal floor is the entire payload” — the floor-only shape.

MetaAggregator / MetaGrouper / MetaFilterer sibling interfaces (the type-level path). If your operator’s Go type already satisfies the sibling interface, leave ComponentsFunc nil — the runtime detects the interface and skips the wrapping shim:

type MetaAggregator interface {
    Aggregator
    Components() (map[string]any, error)
}

type MetaGrouper interface {
    Grouper
    Components() (map[string]any, error)
}

type MetaFilterer interface {
    FiltererBuilder
    Components() (map[string]any, error)
}

Probe-validation still asserts emitted keys against ComponentSchema.Keys — implementing the interface does not bypass the contract.

Pick the shape that matches your code: implement MetaAggregator on a type you control; use ComponentsFunc when the factory returns a third-party type you cannot extend.

Floor-only registrations

If both ComponentSchema.Keys is empty and ComponentsFunc is nil (and the factory’s return value does NOT implement the sibling interface), the registration is floor-only: the orchestrator emits the universal floor keys for that category and nothing else. This is valid and supported, and it is the most common shape for filterer extensions today — no built-in filterer adds operator-specific keys. For aggregators it suits simple counter-style operators that do not need to surface internal state. The probe does NOT reject this shape.

pulse.AggregatorRegistration{
    Name:    "AGG_ACME_COUNT_NONNEG",
    Factory: newNonNegCountAgg,
    // No ComponentSchema, no ComponentsFunc — Response.Components carries
    // only {"n": ..., "n_null": ...}.
}

Probe-validation parity check

The probe at pulse.New (see extensions_probe.go) exercises each factory once against a minimal synthetic schema, then asserts the components contract for AGG / GROUP / FILTER registrations:

Fetch the Message + Fixup template via pulse errors lookup <CODE> (CLI) or call pulse_errors_lookup from an MCP session.

Probe validation (full surface)

The probe (extensions_probe.go) runs after schema + name validation and before pulse.New returns. For every aggregator, attribute, grouper, and filterer registration it constructs the factory once against a minimal synthetic schema and asserts:

flowchart TD
    A[pulse.New] --> B[validate names + reserved namespaces]
    B --> C[probe each factory]
    C -->|panic / nil return| F1[PULSE_EXTENSION_FACTORY_PANIC]
    C --> D{Streamable declared?}
    D -->|yes, interface missing| F2[PULSE_EXTENSION_STREAMABLE_MISMATCH]
    D -->|no, or interface satisfied| E{Components contract?}
    E -->|emitter, no schema| F3[PULSE_EXTENSION_MISSING_COMPONENT_SCHEMA]
    E -->|emitter, key divergence| F4[PULSE_EXTENSION_COMPONENT_SCHEMA_MISMATCH]
    E -->|all clear| G[snapshot + runtime overlay]

Factory contract for the probe (documented in extensions_probe.go): embedder factories MUST tolerate a nil/empty Schema and a spec carrying only the operator Name. The probe never feeds real records.

Factory panics or nil returns surface as PULSE_EXTENSION_FACTORY_PANIC. Streamability declarations that do not match the returned interface surface as PULSE_EXTENSION_STREAMABLE_MISMATCH. The components-contract failures are listed in the table above.

Manifest visibility and the extensions snapshot

pulse manifest --json and pulse_manifest include a top-level extensions block whenever the host registered anything:

{
  "format_version": "1.1",
  "components": { ... built-ins ... },
  "extensions": {
    "aggregators": [
      {"name": "AGG_ACME_BRAND_SCORE", "namespace": "ACME", "streamable": true, "...": "..."}
    ],
    "expr_functions": [
      {"name": "rank_familiarity", "signature": "rank_familiarity(value float64, total_pop bool) float64"}
    ],
    "lookup_tables": [
      {"name": "adjustments", "has_rows_data": true}
    ]
  }
}

The plumbing that fills that block is the extensions snapshot (extensions_snapshot.go). buildExtensionsSnapshot(ext) translates the public Extensions struct into the read-only descriptor.ExtensionsSnapshot projection. The snapshot is passed into descriptor.PredictOptions.Extensions and into mcp.BindSessionToolsWithExtensions so the descriptor layer stays free of service/ and processing/ imports — the no-execute contract for descriptor/ remains intact, and predict / manifest treat custom operators identically to built-ins.

LLM agents that call pulse_manifest see both the built-in set and the embedder additions in one fetch. The schema-bound MCP tools (after pulse_inspect) also include custom operator names in their enum lists.

FieldInputs hook (buffered-projection introspection)

Every operator registration accepts an optional FieldInputs callback:

type FieldInputsFunc func(raw json.RawMessage) []string

When pulse.Options.ProjectBufferedFields is enabled, the runtime walks each request before opening the streaming iterator and calls processing.NeededFields(req, schema, ext) to compute the set of source fields the operators actually read. Built-in operators are fully introspectable from their spec (Field, Field2, PartitionBy, OrderBy, Target, Predictors, plus expr-AST identifiers for ATTR_FORMULA / FILTER_EXPRESSION). Custom operators registered through this surface are opaque by default — without a FieldInputs hook, the projection extractor widens the retained set to “every field” so the runtime stays correct.

To opt into projection, register FieldInputs and return every schema field the operator reads beyond its spec’s explicit references:

pulse.AggregatorRegistration{
    Name:    "AGG_ACME_BLENDED_SCORE",
    Factory: blendedScoreFactory,
    Params:  []pulse.ParamMeta{{Name: "weight_field", JSONType: "string"}},
    // Spec.Field carries the value field; weight_field names a second
    // numeric field. Both must be in the projected map for the
    // aggregator to compute correctly.
    FieldInputs: func(raw json.RawMessage) []string {
        var p struct {
            WeightField string `json:"weight_field"`
        }
        _ = json.Unmarshal(raw, &p)
        if p.WeightField == "" {
            return nil
        }
        return []string{p.WeightField}
    },
}

Return-value semantics:

• nil or empty slice: no extra fields beyond the spec’s Field.
• Names not present in the schema are silently dropped by NeededFields — return-what-you-read; the extractor filters against the live schema.
• Errors are not part of the signature on purpose — FieldInputs runs on the hot path and should be allocation-free. Anything that needs decoding belongs in the factory.

For filterers the callback receives nil as raw (filterers do not carry a Params block today). Tier-2 post-tests do not decode source records and should leave FieldInputs nil.

The hook is plumbed via buildRuntimeExtensions into processing.ExtensionRegistry.FieldInputs, keyed by StreamabilityKey(category, name). ExtensionRegistry.FieldInputsFor(category, name, raw) returns (inputs, true) when the callback ran successfully and (nil, false) when the operator is custom but has no registered callback — that second case is what triggers the extractor to widen.

The retained set NeededFields returns feeds Schema.BuildDecodePlan. A registration with FieldInputs participates normally — its contributed fields land in the retained set and the plan emits SkipBytes segments for every contiguous unprojected run, so unread byte ranges advance with a single Seek. A registration without FieldInputs widens the retained set to *, producing a full-coverage plan with no SkipBytes segments. Both shapes are correct; only the plan-driven one elides byte ranges.

Streamability contract

Embedders declare streamability at registration time; the runtime trusts that declaration. Probe-validation catches obvious mismatches.

Filterers are always row-local streamable; windows always run buffered.

Migration recipe — pre-processing → registration

Before extensions, the canonical pattern was for an embedder to rewrite the request before submitting it:

// Old: rewrite "adjustment" attribute into a formula with the
// multiplier inlined.
req.Attributes = append(req.Attributes, &types.Attribute{
    Type:       types.ATTR_FORMULA,
    Field:      "score",
    Expression: fmt.Sprintf("score * %f", adjustmentFor(study, wave)),
})

With the extension API the request stays domain-named and the engine resolves the value at runtime:

// New: register the lookup once at startup.
pulse.New(pulse.Options{
    Extensions: pulse.Extensions{
        LookupTables: map[string]pulse.LookupTable{
            "adjustments": {Lookup: acme.LookupAdjustment},
        },
    },
})

// Request stays declarative:
req.Attributes = append(req.Attributes, &types.Attribute{
    Type:       types.ATTR_FORMULA,
    Field:      "score",
    Expression: "score * lookup(\"adjustments\", study, wave_date)",
})

Benefits: the manifest advertises adjustments, the schema-bound MCP tool surfaces it, predict can typecheck the expression, and the lookup runs without pre-processing every request.

Error codes raised by this surface

Fetch the Message + Fixup template for any of these via pulse errors lookup <CODE> (CLI) or pulse_errors_lookup (MCP).

Naming-policy violations and probe failures are also enforced by the gates listed in The Update Demand — any change to an extension registration’s ComponentSchema (adding a registration, renaming a key, changing mergeability) MUST update this page and the CLAUDE.md Update Demand table in the same PR.

Related pages

• Adding an Aggregator — the in-tree recipe for built-in operators; mirrors the same ComponentSchema + mergeability contract described here.
• The Update Demand — enforced gates for extension registration ComponentSchema changes and naming-policy drift.
• skills/response-components.md — universal Response.Components contract (typed shells, floor keys, additive omitempty shape).

Adding an Aggregator

Audience: Pulse internals contributors adding a new AGG_* operator.

This page is a step-by-step recipe. The same content lives in CLAUDE.md → Common Claude Code Workflows → Adding a new aggregator; this is the human-readable mirror.

From CLAUDE.md, Common Claude Code Workflows.

1. Declare the type constant

Add the new constant to types/types.go and the slice returned by types.AllAggregationTypes(). Example, for a hypothetical AGG_GINI:

const (
    // ... existing constants ...
    AGG_GINI AggregationType = "AGG_GINI"
)

func AllAggregationTypes() []AggregationType {
    return []AggregationType{
        // ... existing entries, alphabetised ...
        AGG_GINI,
    }
}

The exhaustiveness tests (TestStreamability_AggregationsKnown and friends) will fail until you add the streamability case in step 4.

2. Implement the aggregator and register it

The operator implementation lives in processing/. Write the factory function (newGini(...) returning the aggregator interface) and register it in aggregatorRegistry in processing/registry.go.

If the aggregator can update one row at a time, also implement the OnlineAggregator interface so it joins the streaming Process path. Sort-based or sum-of-deviation aggregators (like AGG_MEDIAN, AGG_ZSCORE) skip this interface and run in the buffered path.

3. Tests

Tests come first: write them in processing/aggregator_test.go before the implementation, run the suite, confirm they fail informatively, then port the implementation until green. See Testing Conventions.

4. Declare streamability

Add a case for the new type in types/streamability.go:

func (t AggregationType) Streamable() bool {
    switch t {
    // ...
    case AGG_GINI:
        return false // sort-based
    }
}

Add the same row to the table in types/streamability_test.go.

If the aggregator is online, also expect TestRegistryStreamabilityMatchesTypes to compare your OnlineAggregator implementation against the AggregationType.Streamable() return value — they must agree.

5. Update the skill pack

Add a section for the new aggregator in skills/aggregation-design.md. Cover when to use it, what its inputs and outputs look like, and any caveats (sort cost, memory, supported field types).

The CI gate TestSkillsCoverAllComponents parses the skill body for the operator name; the section can live anywhere in the file as long as the name appears.

6. Declare the capability metadata

Add a row to descriptor/capabilities_aggregators.go describing the operator’s params, accepted field types, emitted type, and the streamable hint. TestManifestOperatorsComplete enforces that every registered aggregator has a capability row.

The capability row also carries the operator’s ComponentSchema and Mergeability — both required for every registered aggregator since v0.20.0. The next section walks through both.

6a. Declare the ComponentSchema (Response.Components contract)

Every registered aggregator MUST declare a ComponentSchema on its capability row in descriptor/capabilities_aggregators.go. The schema enumerates the operator-specific keys the aggregator emits into Response.Components.Aggregations[i].Operator and tags the operator with one of three mergeability classes:

The aggSchema helper in capabilities_aggregators.go prepends the universal floor ({"n", "n_null"}) automatically — do not list those keys in your extra slice, the helper adds them:

{
    Name:          string(types.AGG_GINI),
    Category:      "aggregator",
    Description:   "Gini coefficient of the field across the input set.",
    AcceptsTypes:  numericFieldTypesAnalyticsNoDecimal,
    EmitsTypeNote: "scalar float64 in [0, 1]",
    Streamable:    false,
    ComponentSchema: aggSchema(None,
        ComponentKey{Name: "sorted_n", Type: "int", Description: "Number of non-null values seen before the final sort."},
    ),
},

A floor-only aggregator (operator-specific keys empty) is valid — pass no extra arguments and the schema declares only {n, n_null} under the chosen mergeability class. AGG_COUNT is the canonical floor-only operator.

6b. Emit per-operator component values at runtime

Two equivalent paths exist for emitting the operator-specific keys at runtime; pick whichever fits your aggregator type.

Sibling interface (preferred for built-in operators). Implement processing.MetaAggregator on your aggregator type and add a compile-time assertion in processing/aggregator.go:

type giniAggregator struct {
    // ... operator state ...
}

func (g *giniAggregator) Components() (map[string]any, error) {
    return map[string]any{
        "sorted_n": len(g.values),
    }, nil
}

// In processing/aggregator.go's compile-time assertion block:
var _ MetaAggregator = (*giniAggregator)(nil)

The assertion list near the bottom of processing/aggregator.go is the grep-discoverable record of which operators emit components. Add the new entry so interface drift is caught at build time.

ComponentsFunc closure. Used primarily by pulse.Options.Extensions registrations where you cannot extend the aggregator type. The closure receives the constructed aggregator instance and returns the operator-specific keys map. See Extension Points for the full extension recipe and the probe-validation contract (PULSE_EXTENSION_MISSING_COMPONENT_SCHEMA, PULSE_EXTENSION_COMPONENT_SCHEMA_MISMATCH).

In both paths the orchestrator owns the universal floor {n, n_null} unconditionally — your Components() MUST NOT re-emit those keys. Return (nil, nil) to signal “no operator-specific keys; let the orchestrator fill the floor only” — the canonical AGG_COUNT shape.

The full Response.Components contract (per-family typed shape, streaming behaviour by mergeability class, overlay parity reads) lives in the response-components skill — that file remains the authoritative contract document.

6c. Mergeable-aggregator rule

If your aggregator is mergeable, implement MergeableAggregator.Merge(other) and declare it as Mergeable() in AggregationType.Mergeable() (types/types.go) so it composes correctly under parallel decode (service/parallel_reduce.go) and shard reduce (service/shard_reduce.go). Both surfaces fold per-worker / per-shard partials in deterministic index order via mergeShardPartials + finalizeMergedPartial.

An aggregator registered but not Mergeable() silently forces the request down the serial scanIter / shardIter path; both parallel paths gate on processing.CanMergeRequest and fall through cleanly when an entry is not flagged.

Associative + commutative aggregators (count, sum, min, max, frequency, distinct_count, mode) produce byte-equal merge output; Welford-Pébaÿ aggregators (mean, variance, stddev) use Chan-Welford and stay within ULP of serial. If your aggregator’s online state cannot be folded associatively, leave Mergeable() returning false. See Cohort schema design — Parallel decode for the gate composition and observed perf characteristics.

7. CLAUDE.md and registered-component lists

Update CLAUDE.md’s “Current registered components” section with the new aggregator name in the right alphabetised slot. If the operator interacts with categorical fields in a special way, also update descriptor/predict.go’s numericAggregations map.

8. Run the gates

go test ./skills/ -run TestSkillsCoverAllComponents
go test ./descriptor/ -run 'TestManifest|TestPredict'
go test ./processing/ -run TestRegistryStreamability
go test ./...

The full Update Demand row for aggregators says: skill update + capability declaration + CLAUDE.md update + the existing test coverage. All four ride in the same PR. See The Update Demand.

Adding an Attribute

Audience: Pulse internals contributors adding a new ATTR_* operator — a per-record derived value computed from one or more cohort fields (z-score, formula, lookup, etc.).

The recipe is the aggregator recipe with three swaps: a different type constant, a different registry, and a different skill file. The shape is otherwise identical.

1. Declare the type constant

Add the new constant to types/types.go and the slice returned by types.AllAttributeTypes():

const (
    // ... existing constants ...
    ATTR_GINI_BUCKET AttributeType = "ATTR_GINI_BUCKET"
)

func AllAttributeTypes() []AttributeType {
    return []AttributeType{
        // ... existing entries, alphabetised ...
        ATTR_GINI_BUCKET,
    }
}

2. Implement and register

Implement the attribute in processing/. Each attribute is a factory function registered in attributeRegistry (processing/registry.go). Attribute factories return a closure with the signature func(record encoding.RecordView) (any, error).

If the attribute can be evaluated row-at-a-time (the common case), that is all the implementation needs; the streaming Process path invokes the closure on every record.

3. Tests

Write tests first in processing/attribute_test.go. Run the suite, confirm informative failure, port the implementation until green.

4. Declare the capability metadata

Add a row to descriptor/capabilities_attributes.go with the attribute’s params, the field types it accepts as input, the type it emits, and any documentation strings. TestManifestOperatorsComplete enforces a capability row per registered attribute.

5. Update the attribute-composition skill

Add a section in skills/attribute-composition.md covering the new attribute’s params, output column naming convention, and any caveats (NaN propagation, integer underflow, expr-runtime cost). The TestSkillsCoverAllComponents gate parses the skill body for the operator name.

6. Update CLAUDE.md

Bump the count in CLAUDE.md’s “Skill Pack” section (the current registered counts line) so the registered-attribute total reflects the new operator.

7. Run the gates

go test ./skills/ -run TestSkillsCoverAllComponents
go test ./descriptor/ -run TestManifestOperatorsComplete
go test ./processing/ -run TestAttribute

The Update Demand row for attributes covers all of these in one PR; see The Update Demand.

Adding a Filterer

Audience: Pulse internals contributors adding a new FILTER_* operator — a per-record predicate that decides whether the row participates in the downstream aggregation / grouper / crosstab.

The recipe mirrors the aggregator recipe; the filterer-specific moving parts are the predicate factory, the filter registry, and the universal floor for Response.Components.Filterers.

1. Declare the type constant

Add the new constant to types/types.go and the slice returned by types.AllFiltererTypes():

const (
    // ... existing constants ...
    FILTER_REGEX FiltererType = "FILTER_REGEX"
)

func AllFiltererTypes() []FiltererType {
    return []FiltererType{
        // ... existing entries, alphabetised ...
        FILTER_REGEX,
    }
}

2. Implement and register

Implement the filterer in processing/. Each filterer is a factory that returns a FiltererBuilder — register it in filtererRegistry (processing/registry.go).

The builder produces a per-record FilterFunc that returns (keep bool, err error). The streaming Process path invokes the filter func once per row; the buffered path invokes it once per materialised record.

3. Tests

Add tests in processing/filter_test.go before the implementation. Cover both the include and exclude branches, the null-handling contract, and any error path.

4. Declare the capability metadata

Add a row to descriptor/capabilities_filterers.go with the filterer’s params, accepted field types, and the ComponentSchema that follows. TestManifestOperatorsComplete enforces a capability row per registered filterer.

5. Declare the ComponentSchema (Response.Components contract)

Every registered filterer MUST declare a ComponentSchema on its capability row, with a Mergeability class (Mergeable / Partial / None).

In v1 every built-in filterer is Mergeable with an empty operator-specific keys slice — the orchestrator fills the universal floor {n_in, n_out, n_null_input} from the filter pass’s record-walker counters, and no per-filter extras are emitted today:

{
    Name:            string(types.FILTER_REGEX),
    Category:        "filterer",
    Description:     "Keep records whose field matches the supplied regular expression.",
    AcceptsTypes:    stringFieldTypes,
    ComponentSchema: filterSchema(Mergeable),
},

The filterSchema helper prepends the universal floor automatically — do not list those keys in your extra slice. The interface processing.MetaFilterer exists for extension-author parity with MetaAggregator / MetaGrouper and leaves room for future per-filter specifics (n_below / n_above for FILTER_RANGE, per-value n for FILTER_INCLUDE / FILTER_EXCLUDE). When a built-in eventually implements it, declare the extra keys in filterSchema(...) and add the Components() method on the filterer struct plus a compile-time sentinel:

var _ MetaFilterer = (*regexFilterer)(nil)

The full Response.Components contract — universal floor semantics, streaming behaviour, parity overlay reads — lives in the response-components skill; extension-side parity lives in Extension Points.

6. Update the aggregation-guide skill

Add a section in the filtering portion of skills/aggregation-design.md covering the new filterer’s semantics, parameter shape, and the null- input contract.

7. Update CLAUDE.md

Bump the registered-filterer count in CLAUDE.md’s “Skill Pack” section.

8. Run the gates

go test ./skills/ -run TestSkillsCoverAllComponents
go test ./descriptor/ -run TestManifestOperatorsComplete
go test ./processing/ -run TestFilter

The Update Demand row for filterers covers all of these in one PR; see The Update Demand.

Adding a Grouper

Audience: Pulse internals contributors adding a new GROUP_* operator — a bucketing function that maps each record to a group key the orchestrator uses to partition the aggregation.

The recipe mirrors the aggregator recipe; the grouper-specific moving parts are the per-bucket key contract, the grouper registry, and the v0.20.0 ComponentSchema / MetaGrouper emission path.

1. Declare the type constant

Add the new constant to types/types.go and the slice returned by types.AllGroupTypes():

const (
    // ... existing constants ...
    GROUP_DECILE GroupType = "GROUP_DECILE"
)

func AllGroupTypes() []GroupType {
    return []GroupType{
        // ... existing entries, alphabetised ...
        GROUP_DECILE,
    }
}

2. Implement and register

Implement the grouper in processing/. Register the factory in grouperRegistry (processing/registry.go). The interface choice depends on whether the grouper can run in the streaming path:

• Grouper — buffered-only. Group(rows) returns a slice of bucket assignments. Used by quantile-class groupers that need the full sorted view.
• StreamingGrouper / StreamableGrouper — streaming. The per-record KeyFor(record) / KeyForRow(record) returns a bucket key as the iterator walks rows; the orchestrator accumulates per key.
• MultiKeyStreamingGrouper — multi-emit streaming. KeysForRow returns N keys per row; used by set-typed groupers (GROUP_SET_PER_ELEMENT).

3. Tests

Add tests in processing/grouper_test.go (or the per-grouper test file) before the implementation. Cover empty input, single-value input, null-bearing input, the Include filter slot if your grouper honours it, and the streaming-vs-buffered parity assertions where applicable.

4. Declare the capability metadata

Add a row to descriptor/capabilities_groupers.go with the grouper’s params, accepted field types, and the ComponentSchema that follows. TestManifestOperatorsComplete enforces a capability row per registered grouper.

5. Declare the ComponentSchema (Response.Components contract)

Every registered grouper MUST declare a ComponentSchema on its capability row in descriptor/capabilities_groupers.go, tagged with one of three mergeability classes:

The groupSchema helper in capabilities_groupers.go prepends the universal floor ({"total_n", "n_null"}) automatically — do not list those keys in your extra slice:

{
    Name:        string(types.GROUP_DECILE),
    Category:    "grouper",
    Description: "Partition the numeric field into ten equally-sized quantile buckets.",
    AcceptsTypes: numericFieldTypesAnalyticsNoDecimal,
    ComponentSchema: groupSchema(None,
        ComponentKey{Name: "edges", Type: "[]float64", Description: "Decile boundary values (length 9)."},
        ComponentKey{Name: "buckets", Type: "map[string]int", Description: "Per-decile record counts keyed by label."},
    ),
},

6. Emit per-operator component values at runtime

Implement processing.MetaGrouper on the grouper type and add a compile-time assertion in the grouper implementation file:

type decileGrouper struct {
    edges   []float64
    buckets map[string]int
    // ...
}

func (g *decileGrouper) Components() (map[string]any, error) {
    return map[string]any{
        "edges":   g.edges,
        "buckets": g.buckets,
    }, nil
}

var _ MetaGrouper = (*decileGrouper)(nil)

The orchestrator owns the universal floor {total_n, n_null} — the post-filter record walker fills it unconditionally. Your Components() MUST NOT re-emit those keys.

For single-key groupers, total_n equals the sum of bucket counts. For multi-key streaming groupers (GROUP_SET_PER_ELEMENT), the sum of bucket counts exceeds total_n because a single record contributes to multiple buckets — total_n reflects the row count.

The full Response.Components contract for groupers — streaming behaviour by mergeability class, the orchestrator-owned floor, the extension-side parity contract — lives in the response-components skill. Embedder extensions implement the same MetaGrouper interface via ComponentsFunc or directly; see Extension Points.

7. The Include inclusion-list slot

If your grouper honours an Include whitelist of allowed bucket labels, update processing/grouper.go + processing/grouper_set.go so the include filter is applied at key-emission time, not on the output table after the fact. The contract is enforced by the following gates:

• TestGroupCategory_IncludeFiltersLabels
• TestGroupSetValue_IncludeFiltersCompositeKey
• TestGroupSetPerElement_IncludeFilters

8. Update the grouper-design skill

Add a section in skills/grouper-design.md covering the new grouper’s bucket-naming convention, the params it accepts, the streamability classification, and any cardinality bound.

9. Update CLAUDE.md

Bump the registered-grouper count in CLAUDE.md’s “Skill Pack” section.

10. Run the gates

go test ./skills/ -run TestSkillsCoverAllComponents
go test ./descriptor/ -run TestManifestOperatorsComplete
go test ./processing/ -run TestGroup

The Update Demand row for groupers (and the Group.Include slot) covers all of these in one PR; see The Update Demand.

Adding a Window Operator

Audience: Pulse internals contributors adding a new WIN_* operator — a window-frame function that emits a per-row value derived from a sliding or anchored frame around the current record (WIN_LAG, WIN_LEAD, WIN_RANK, WIN_RUNNING_*, WIN_EWMA, …).

The recipe mirrors the aggregator recipe; the window-specific moving parts are the frame contract, the windows registry, and the streamability case.

1. Declare the type constant

Add the new constant to types/types.go and the slice returned by types.AllWindowTypes():

const (
    // ... existing constants ...
    WIN_PERCENT_RANK WindowType = "WIN_PERCENT_RANK"
)

func AllWindowTypes() []WindowType {
    return []WindowType{
        // ... existing entries, alphabetised ...
        WIN_PERCENT_RANK,
    }
}

2. Implement and register

Window operators live under processing/window/. Each file is one operator; register the factory in the package’s init() via the register(types.WIN_X, newX) call shape used by sibling files.

The frame semantics — anchored vs sliding, lookahead vs lookback — are part of the operator’s contract and ride in the operator’s struct parameters; the orchestrator’s window-pass invokes the implementation per row in declared order.

3. Declare streamability

Window operators that need a full forward / backward pass (lead-based, percent-rank style) are not streamable. Add the case in types/streamability.go:

func (t WindowType) Streamable() bool {
    switch t {
    // ...
    case WIN_PERCENT_RANK:
        return false
    }
}

Add the matching row to types/streamability_test.go so TestStreamability_WindowsKnown passes.

4. Capability declaration

Add a row to descriptor/capabilities_windows.go with the operator’s params, accepted field types, and streamable hint. TestManifestOperatorsComplete enforces a capability row per registered window operator.

5. Tests

Add tests in processing/window/<name>_test.go. Cover the empty-frame, single-row, null-bearing, and order-sensitive cases.

6. Update the window-operations skill

Add a section in skills/window-design.md covering the operator’s frame contract, parameter shape, and output column naming. The TestSkillsCoverAllWindowTypes gate enforces presence.

7. Update CLAUDE.md

Bump the registered-window count in CLAUDE.md’s “Skill Pack” section.

8. Run the gates

go test ./skills/ -run TestSkillsCoverAllWindowTypes
go test ./descriptor/ -run TestManifestOperatorsComplete
go test ./processing/window/...
go test ./types/ -run TestStreamability_WindowsKnown

The Update Demand row for windows covers all of these in one PR; see The Update Demand.

Adding a Feature Operator

Audience: Pulse internals contributors adding a new FEAT_* operator — a pre-filter feature engineer that runs before the aggregation / window pass and emits one or more derived columns (FEAT_LOG, FEAT_SQRT, FEAT_BUCKETIZE, …).

The recipe mirrors the aggregator recipe; the feature-specific moving parts are the feature.StreamingComputer interface, the output-label emitter, and the predict-side label projection.

1. Declare the type constant

Add the new constant to types/types.go and the slice returned by types.AllFeatureTypes():

const (
    // ... existing constants ...
    FEAT_BOX_COX FeatureType = "FEAT_BOX_COX"
)

func AllFeatureTypes() []FeatureType {
    return []FeatureType{
        // ... existing entries, alphabetised ...
        FEAT_BOX_COX,
    }
}

2. Implement in processing/feature/

Each feature operator lives in processing/feature/<name>.go. Register via the package’s init() calling register(types.FEAT_X, newX).

If the operator is streaming-eligible, implement the feature.StreamingComputer interface — a three-method shape:

• PrePass(rows) — accumulate any whole-cohort statistics needed (mean, stddev) on a first pass.
• Finalize() — close out the pre-pass, compute coefficients.
• EmitRow(row) — emit the per-row feature value(s) on the second pass.

Operators without whole-cohort statistics skip PrePass and run as single-pass row transforms.

3. Tests

Write tests in processing/feature/<name>_test.go before the implementation. Cover the empty-input, single-row, null-bearing, and boundary cases.

4. Capability declaration

Add a row to descriptor/capabilities_features.go with the operator’s params, accepted field types, and any emit shape. TestManifestOperatorsComplete enforces a row per registered feature.

5. Predict-side label projection

Update descriptor/predict_feature.go:

• Validate the operator’s params (raise the appropriate PROCESSING_CONFIG / SERVICE_VALIDATION error code on invalid input).
• Emit the operator’s output column labels in featureOutputLabels so predict can show the LLM client what columns the request will materialise. TestPredict_Feature enforces parity.

6. Update the feature-engineering skill

Add a section in skills/feature-engineering.md covering the operator’s params and output column naming convention. The TestSkillsCoverAllComponents gate enforces presence by name.

7. Update CLAUDE.md

Bump the registered-feature count in CLAUDE.md’s “Skill Pack” section.

8. Run the gates

go test ./skills/ -run TestSkillsCoverAllComponents
go test ./descriptor/ -run 'TestManifestOperatorsComplete|TestPredict_Feature'
go test ./processing/feature/...

The Update Demand row for feature operators covers all of these in one PR; see The Update Demand.

Adding a Statistical Test

Audience: internals contributors adding a new TEST_* operator — tier-1 (row-stream) or tier-2 (post-test on the materialised result set).

The recipe mirrors the aggregator and feature recipes; the test-specific moving parts are streamability, the test catalog, and the registered-test capability table.

From CLAUDE.md, “Update Demand” rows for statistical tests and tier-2 post-test variants.

1. Decide tier

• Tier 1. Runs against the raw row stream, alongside aggregators. Online-moments tests (TEST_T, TEST_WELCH, TEST_CHISQ, TEST_ANOVA_F) stay in the streaming Process path. Sort-required tests (TEST_KS) force the buffered path.
• Tier 2. Runs after the result set is materialised, in req.PostTests. Always buffered.

2. Declare the type constant

Add to types/types.go:

const (
    // ... existing constants ...
    TEST_GINI_TREND TestType = "TEST_GINI_TREND"
)

Add it to types.AllTestTypes().

3. Implement and register

Tests live in processing/test_*.go. Existing examples to mirror:

• processing/test_t.go — online tier-1 test.
• processing/test_anova.go — tier-1 ANOVA with grouper support.
• processing/test_post.go and processing/test_post_more.go — tier-2 post-tests.
• processing/test_studentized.go — numerical integration utilities (used by TEST_TUKEY_HSD).

Register the test in processing/test.go (the registry construction calls). For tier-2 variants, declare both the base type and the variant identifier the post-test surface uses.

4. Streamability

Add a case in types/streamability.go for the new TestType:

func (t TestType) Streamable() bool {
    switch t {
    // ...
    case TEST_GINI_TREND:
        return false // sort-based
    }
}

Add the matching row in types/streamability_test.go so TestStreamability_TestsKnown passes.

5. Capability declaration

Add a row to descriptor/capabilities_tests.go:

• For a tier-1 test, declare it in the tier-1 catalog (testCapabilities).
• For a tier-2 post-test, declare it in postTestCapabilities.

TestManifestTestsComplete and TestManifestPostTestsComplete enforce that the manifest enumerates every registered test.

6. Skill update

Add an entry to skills/statistical-testing.md under “Operator catalog”. Describe the test’s family, inputs, outputs (statistic, p, df, effect size, …), and any preconditions (PULSE_TEST_* error codes it can raise). For tier-2 variants, also document the variant field shape since the post-test API exposes it.

7. Tests

Use the same TDD pattern as for aggregators. The processing package has rich existing test files to model new cases against: processor_test_pipeline_test.go, test_parametric_test.go, test_nonparametric_test.go, test_post_more_test.go. Add hermetic fixtures that exercise the streaming and buffered paths.

8. Error codes

If your test introduces a new failure mode, add a code to errors/codes.go (mirror the existing PULSE_TEST_* family), register its description row in descriptor/capabilities_errors.go, and document recovery in errors/fixup_metadata.go (codeMetadata Message + Fixups, surfaced per-code via pulse_errors_lookup / pulse errors lookup CODE). See the Adding an Aggregator recipe for the same pattern at the aggregator layer.

9. CLAUDE.md

Update CLAUDE.md’s “Current registered components → statistical tests” line with the new operator. If the test introduces a new preconditions class (e.g. paired sample, repeated measures), also add a sentence describing it in the parent paragraph.

10. Run the gates

go test ./processing/ -run TestType_Streamable
go test ./types/    -run TestStreamability_TestsKnown
go test ./descriptor/ -run TestManifest
go test ./skills/    -run TestSkillsCoverAll
go test ./...

See The Update Demand for the full row that governs statistical-test changes.

Adding a Synth Distribution

Audience: Pulse internals contributors adding a new synthetic-data distribution kind to the synth/ package — Normal, Uniform, Categorical, ZipfMandelbrot, etc.

The synth surface generates .pulse cohorts from either a declared schema (pulse synth from-schema) or a profile sampled from an existing cohort (pulse synth from-profile). Distributions plug into both code paths via the synth.AllDistributions() registry.

1. Implement in synth/

Add the distribution implementation in synth/. Each distribution satisfies the package’s distribution interface (per-field RNG + parameter validation + JSON marshalling). Register it in the synth.AllDistributions() slice so the generator surface enumerates it.

2. Capability declaration

Add a row to descriptor/capabilities_distributions.go so the manifest exposes the new distribution to LLM clients. TestManifestDistributionsComplete enforces a capability row per registered distribution kind.

3. Update the synthetic-data skill

Add an entry under “Supported distributions” in skills/synthetic-data.md covering the parameter shape, the distribution’s family (continuous, discrete, heavy-tailed, categorical-like), and any sampling caveats. TestSkillsCoverAllSynthDistributions enforces presence by name.

4. Update CLAUDE.md

Bump the registered-synth-distribution count in CLAUDE.md’s “Skill Pack” section.

5. Run the gates

go test ./skills/ -run TestSkillsCoverAllSynthDistributions
go test ./descriptor/ -run TestManifestDistributionsComplete
go test ./synth/...

The Update Demand row for synth distributions covers all of these in one PR; see The Update Demand.

Adding an I/O Format

Audience: internals contributors adding a new bidirectional tabular format (a peer to the existing csv/, tsv/, ndjson/, jsonarray/, arrow/, parquet/, excel/ sub-packages).

From CLAUDE.md, Common Claude Code Workflows.

1. Create the sub-package

Each format is a sub-package under io/. Create io/<format>/<format>.go with both a reader and a writer.

The two interfaces to implement live in io/:

// Reader
type Reader interface {
    ReadHeader() ([]string, error)
    ReadRows(ctx context.Context, fn func(row []string) error) error
    Close() error
}

// Writer
type Writer interface {
    WriteHeader(columns []string) error
    WriteRow(values []string) error
    Close() error
}

If the reader needs schema inference (header sample, then full import), also implement io.ResetReader.Reset() so the import job can rewind after sampling.

2. Tests

Add io/<format>/<format>_test.go with the standard round-trip checks: write rows, read them back, verify equality. Hermetic tests should use afero.NewMemMapFs() — see Testing Conventions.

3. Wire it into the CLI

The CLI registers per-format leaves in internal/cli/import.go and internal/cli/export.go. Add the format string to:

• The switch in makeImportReader(format, ...) in import.go.
• The corresponding newWriterForFormat(format, ...) switch in export.go.
• The Commands: slice on ImportCommand() and ExportCommand() in the same files (one importFormatCmd("yourformat") / exportFormatCmd("yourformat") line).

The pulse convert leaf auto-detects format from extension via formatFromExt; add the extension mapping if the new format has a canonical file extension.

4. Schema mapping

If the new format has a native type system (Arrow / Parquet do, CSV does not), share the type map with neighbouring formats via the io/arrow package the way Parquet already does. CSV / TSV / NDJSON / JSON-array share io/jsonshared for value coercion.

5. Skill update

Add or update a skill that points users at the new format. Cohort- schema considerations (field-type round-trip, dictionary behaviour, null markers) belong in skills/cohort-schema-design.md.

If the format adds a CLI flag (e.g. --sheet for Excel), update skills/session-bootstrap.md so TestSkillsCoverAllCliLeaves keeps passing.

6. Convert and orchestration plumbing

Make sure both directions flow through pio.ImportJob and pio.ExportJob. The orchestration layer is format-agnostic; you should not need to touch service/ unless the new format requires special metadata (e.g., Parquet’s per-column statistics).

7. Run the gates

go test ./io/<format>/...
go test ./skills/ -run TestSkillsCoverAll
go test ./...

For format-specific perf, add benchmarks (Benchmark<Format>...) in the sub-package. There’s no required perf gate today, but neighbouring formats have benchmarks you can mirror as a baseline.

Adding a Field Type

Audience: Pulse internals contributors adding a new .pulse field type — a new on-wire encoding the schema block can describe, the record codec can read / write, and the operator surface can accept.

Adding a field type touches more layers than any other recipe — the binary format, the schema reader, the operator-accept tables, the predict routing layer, and the cohort-schema skill. It is also one of the highest-impact extensions: every operator that touches the new type either accepts it via its AcceptsTypes list or is excluded from it explicitly.

1. Declare the FieldType constant

Add the FieldType constant and its ByteSize() method. The byte size is the wire footprint per record; bit-packed types (packed_bool, u4) return 0 and signal sharing via FieldType.IsBitPacked().

2. Schema reader

Wire the type byte into the schema reader case in encoding/. The unknown-type-byte branch in the reader is what surfaces ENCODING_INVALID for an unknown type — when you add the new constant, add the corresponding decode arm.

If the new type carries a dictionary block (categoricals, sets), follow the inline-dictionary contract documented in CLAUDE.md’s “Byte-layout invariants” — the dictionary lives between the schema block and the record data, keyed by the position of the bearing field.

3. Update the operator accept tables

Every operator that should accept the new field type needs its row in descriptor/capabilities_*.go extended. The accept tables are shared slices declared near the top of capabilities_aggregators.go (numericFieldTypes, numericFieldTypesAnalytics, allCohortFieldTypes, setFieldTypes, …) — extend the right slice rather than adding the new type to each operator individually.

4. Predict-side routing

If the new type changes how an operator behaves (categorical-vs-numeric routing, nullable-vs-non-nullable handling), update descriptor/predict.go’s routing tables (numericAggregations, categoricalAggregations, …).

5. Update the cohort-schema-design skill

Add the new type to the “All field types” table in skills/cohort-schema-design.md. TestSkillsCoverAllFieldTypes enforces presence by name.

6. Update CLAUDE.md

The CLAUDE.md “Byte-layout invariants” section enumerates the canonical field-type count and the bit-packed vs dictionary-bearing distinctions. Update it to reflect the new type.

7. Run the gates

go test ./skills/ -run TestSkillsCoverAllFieldTypes
go test ./encoding/...
go test ./descriptor/ -run 'TestManifest|TestPredict'

The Update Demand row for .pulse format changes (header / field type) covers all of these in one PR; see The Update Demand.

Adding an Error Code

Audience: Pulse internals contributors adding a new PULSE_* / SERVICE_* / ENCODING_* / PROCESSING_* / DATA_* / CLI_* error code, or renaming / removing one.

The Pulse error system runs under six domains. Every code carries a typed CodedError, a human-readable Message, and at least one Fixup template explaining how to recover. The full taxonomy is in The Update Demand.

1. Declare the code constant

Add the constant in errors/codes.go and append it to the allCodes slice. Pick the domain prefix:

2. Declare the message + fixup template

Add an entry to the codeMetadata map in errors/fixup_metadata.go. Every code MUST have either:

• A Message plus one or more Fixup templates explaining how to recover (the canonical case), or
• FixupNotApplicable: true when there is truly no remediation the caller can apply (rare — most error codes are recoverable in principle).

TestCodesHaveFixups enforces that every code has either fixups or the explicit FixupNotApplicable: true flag. The fixup templates are surfaced to MCP clients via pulse_errors_lookup and to CLI users via pulse errors lookup CODE — no separate skill-file edit is required.

3. Wire any test that emits the new code

If your change introduces the code as part of a new failure mode, make sure the test that exercises that mode asserts on the code constant rather than the error message.

4. Run the gates

go test ./errors/ -run 'TestCodesHaveFixups|TestErrorsLookup'
go test ./descriptor/ -run 'TestManifestErrorCodesComplete|TestManifest_ErrorCodesSlim'

The Update Demand row for error codes covers all of these in one PR; see The Update Demand.

Adding an MCP Tool

Audience: Pulse internals contributors adding a new tool to the embedded MCP server. The MCP layer is split into the SDK-free core (mcp/), the go-sdk adapter (mcp/gosdk/, the only package importing the MCP SDK), and the leaf metadata package (mcp/toolmeta/).

Each MCP tool wraps one slice of pulse.Pulse and surfaces it over stdio / Streamable HTTP transports. The catalog covers one tool per facade method (pulse_inspect, pulse_predict, pulse_process, pulse_compose, pulse_sample, pulse_facet, pulse_facet_schema, pulse_manifest, pulse_errors_lookup, …) plus the skills / examples / import / label tools, and the two resource schemes (pulse://*.pulse, pulse-skill://*). Treat the manifest (pulse manifest) as the source-of-truth count — never hardcode a number. Adding a new tool means extending the core catalog, registering the metadata, optionally binding a field-aware JSON Schema, and updating the MCP-integration skill.

1. Implement the handler

Implement the new tool handler in the SDK-free core (mcp/handlers.go). The handler is a typed function over *pulse.Pulse: accept the typed In struct, call the facade method, return the typed Out (coded errors surface verbatim as {code, message, details}). Add the tool’s ToolDescriptor to the core catalog (mcp/tools.go) so Tools(cfg) emits it; the go-sdk adapter (mcp/gosdk/) mounts whatever the catalog returns via gosdk.Register.

2. Register tool metadata

Add the tool’s name + description in mcp/toolmeta/meta.go. The mcp/toolmeta package is imported by descriptor/ (which assembles the manifest) and by the core, so this is the leaf-metadata package that lets the descriptor surface the tool without importing the MCP layer or the SDK.

3. Field-name parameters (optional)

If the new tool has field-name parameters (e.g. a field: string argument that takes a cohort field name), add a per-tool JSON Schema builder in mcp/bind.go + an entry in Bind. After pulse_inspect succeeds against a cohort the adapter binds session-scoped variants of every schema-aware tool whose JSON Schema constrains field-name parameters to the inspected cohort’s actual fields. mcp/bind.go is pure (no MCP SDK); the per-session server mutation that consumes the schemas lives in the adapter (mcp/gosdk/bind.go).

Schema-binding parity is enforced by:

• TestMCPSchemaBinding_RemovesInvalidFields
• TestMCPSchemaBinding_AllFieldsInFiltererEnum
• TestMCPSchemaBinding_SampleAndFacetFieldEnum
• TestMCPSchemaBinding_DedupAndSort
• TestMCPSchemaBinding_NilSchema

The transport caveat: bind-on-inspect works on the single stdio session (post-serve AddTool/RemoveTools auto-emits list_changed); there is no per-session override for shared HTTP servers — a documented limitation. See the MCP integration skill for the configuration recipe.

4. Update the session-bootstrap skill

Add a section to skills/session-bootstrap.md covering the new tool’s purpose, request shape, response shape, and (if applicable) the Schema-bound enums it exposes after pulse_inspect. TestSkillsCoverAllMCPTools enforces presence by name.

5. Run the gates

go test ./skills/ -run TestSkillsCoverAllMCPTools
go test ./descriptor/ -run TestManifestMCPToolsComplete
go test ./mcp/ -run TestMCPSchemaBinding

The Update Demand row for MCP tools covers all of these in one PR; see The Update Demand.

Adding a Facet Capability Variant

Audience: Pulse internals contributors extending the pulse.FacetSchema endpoint with a new top-level facet behaviour — a streaming auto-range histogram, a new aggregation kind on numeric fields, a new contribution-style accumulator.

The facet endpoint sits behind descriptor.FacetCapability and runs through service/facet_rich.go. New behaviours land in five files in lockstep — request type, accumulator dispatch, capability flag, validator, MCP JSON Schema builder — plus the facet-design skill.

1. Extend the request / result types

Add the new fields to types.FacetRequest and types.FacetResult in types/facet.go. Keep JSON tags backward-compatible — additive only. Renames or removals trigger a format_version bump per the Output Format Contract; new fields use omitempty and do not bump.

2. Implement per-row accumulation

Add the per-row accumulation in service/facet_rich.go, dispatching off the schema field type via newKindAccumulator. The accumulator contract — Add(value), Finalize(), Result() — runs per-row during the facet pass.

3. Capability flag

Add the capability flag in descriptor/capabilities_facet.go so the manifest exposes the new behaviour to LLM agents. TestManifestFacetCapability enforces parity between the capability block and the runtime surface.

4. Validator

descriptor/facet.go::ValidateFacet runs without importing service / processing (it is structurally no-execute, governed by TestPredictNoExecutionImports). Any new structural rule lands here as a SERVICE_VALIDATION error or an advisory warning.

5. MCP JSON Schema builder

Update the JSON Schema builder in mcp/bind.go — the buildFacetSchemaRequestSchema function — so the LLM sees the new fields in the pulse_facet_schema tool surface. TestMCPSchemaBinding_SampleAndFacetFieldEnum enforces parity.

6. Update the facet-design skill

Update skills/facet-design.md with the new behaviour’s request shape, output shape, and any worked example. When the behaviour warrants a runnable fixture, add it under examples/facet/ and update the example metadata. TestExamples_* enforces the fixture contract.

7. Run the gates

go test ./skills/ ./examples/ ./descriptor/
go test ./service/ -run TestFacet
go test ./mcp/ -run TestMCPSchemaBinding

The Update Demand row for facet-capability changes covers all of these in one PR; see The Update Demand.

Adding a Chain-Stage Predicate

Audience: Pulse internals contributors tightening or relaxing the gate that decides whether an operator is allowed in a ProcessChain stage.

ProcessChain (pulse.ProcessChain, pulse_process_chain, pulse api process-chain) executes a linear pipeline whose stages all pass processing.CanChainRequest. The gate enforces that each stage emits a shape the next stage can consume; v1 admits mergeable scalar- emitting operators only.

1. Edit the runtime gate

Edit processing/chain.go. CanChainRequest calls CanMergeRequest first, then layers chain-specific exclusions (aggregatorEmitsScalar). Add a new exclusion branch when an operator is mergeable but its emit shape would break the synthesised f64 / categorical_u32 schema the next stage expects.

2. Mirror in the predict gate

Mirror the rule in descriptor/chain.go. chainGateOK is the predict-side equivalent — keep them in lockstep. A divergence makes predict pass requests that runtime later rejects, which surfaces as a late SERVICE_VALIDATION rather than as a predict-time advisory.

3. Update the capability surface

Edit descriptor/capabilities_chain.go. processChainCapability() carries the manifest-facing allowlists and RejectionRules strings. After editing, regenerate descriptor/testdata/manifest.json:

go test ./descriptor/ -update -run Golden

Then verify the new hash sticks:

go test ./descriptor/ -run TestGoldensNotHandEdited

4. Tests

Add a failing-gate test in service/chain_test.go and a matching predict test in descriptor/chain_test.go. The two surfaces share the gate contract; covering both prevents the predict / runtime divergence the rule exists to prevent.

5. Allowlist skim

Skim skills/session-bootstrap.md and skills/process-chain.md for any operator allowlist that needs adjustment in prose.

6. Whole-chain overlays

ChainRequest.Overlays []*ChainOverlaySpec is the whole-chain overlay surface — overlays here execute AFTER every stage finalises (NOT between stages). Per-stage overlays continue to ride the universal ChainStage.Request.Overlays []OverlaySpec slot.

• ChainOverlaySpec (in types/chain.go): Name string, Kind OverlayKind, Ref StageRef, Target StageRef, Scope OverlayScope, Params map[string]any.
• StageRef (in types/chain.go): XOR {Index *int, Name string}. Index is a pointer so 0 is meaningful — the canonical “stage 0” call site sets Index = &zero, not Index unset. The downstream validator enforces “exactly one of Index / Name”.
• OverlayRef.Stage (in types/overlay.go) is the same *StageRef — there is exactly one StageRef declaration in the codebase. The legacy OverlayStageRef identifier is a type alias to StageRef and is deprecated.
• ChainResponse.Overlays []*OverlayLayer reuses the universal OverlayLayer wrapper from types/overlay.go (one entry per ChainRequest.Overlays spec in matching index order).

Canonical-hash coverage is data-driven (types/hash.go): the slot’s omitempty tag means overlay-free chain requests hash byte-identically to the overlay-free baseline; populated overlays fold into the hash automatically. Locked by TestChainCanonicalHash_OverlayFreeByteIdentity and TestChainCanonicalHash_OverlaysIncluded.

Whole-chain handler dispatch lives in processing/overlay_chain_dispatch.go. Predict-time validation lives in descriptor/chain_overlay.go — ValidateChain walks ChainRequest.Overlays after the per-stage gate and emits:

• PULSE_OVERLAY_KIND_UNKNOWN — anything outside OVERLAY_INDEX_VS_STAGE / OVERLAY_DELTA_VS_STAGE is rejected today.
• PULSE_OVERLAY_REFERENCE_UNKNOWN / PULSE_OVERLAY_TARGET_UNKNOWN — StageRef resolution failures. Same Index / Name / XOR / latest-stage-default contract the runtime resolveChainStageRef uses.
• PULSE_OVERLAY_CHAIN_STAGE_SHAPE_DIVERGENT — Ref and Target stages produce different host shapes per inferChainStageShape: req.Crosstab != nil ⇒ MATRIX, req.Aggregations + req.Groups ⇒ SERIES, req.Aggregations only ⇒ SCALAR.

Every rejected (Ref, Target) pair also lands in ChainValidationResult.OverlaysSchemaDivergence so LLM planners can budget reshapes without re-parsing envelope details.

7. Run the gates

go test ./service/ -run TestChain
go test ./descriptor/ -run 'TestValidateChain|TestProcessChain'
go test ./descriptor/ -run TestGoldensNotHandEdited

The Update Demand row for ProcessChain capability changes covers all of these in one PR; see The Update Demand.

Adding an afero.Fs Implementation

Audience: Pulse internals contributors and embedders wiring a new filesystem backend (a copy-on-read overlay caching remote objects, a base-path wrapper, any prefix-translating shim) under pulse.Options.FS.

The Pulse iterator engages an mmap fast path when the cohort path ultimately resolves to a real on-disk file. The eligibility probe lives in service/fs_probe.go and is documented inline; the gist: implement the service.RealPather capability interface so the probe can ask your fs “where is this file on disk?” without opening it.

The RealPather contract

// service.RealPather — defined in service/fs_probe.go.
type RealPather interface {
    RealPath(name string) (string, error)
}

The returned path MUST be os.Open-able at the moment of the call, and the bytes it returns MUST be identical to what fs.Open(name).Read would yield. The signature deliberately matches *afero.BasePathFs.RealPath so that wrapper is detected automatically.

Why this matters

service.resolveRealPath is the eligibility probe that decides whether the streaming iterator engages the mmap fast path. Without RealPather, the probe falls through to the *afero.OsFs check and then to the afero.ReadFile slow path.

Failure to implement this interface silently disables the mmap optimisation — no error, no warning, just a regression in scan throughput on cold-cache wide cohorts. The mmap policy and probe order are documented in the Cohort schema design skill (“Iterator mmap policy”); the rationale for omitting an open-and- inspect fallback is inline at service/fs_probe.go.

The regression gate

The countingFs test family in service/ (e.g. TestCountingFs_*) is the regression gate. It wraps an fs and fails the test if Process calls afero.ReadFile on a single-file cohort path when the fs advertises a real path. If you add a new wrapper and the gate flips red, the wrapper is almost certainly missing RealPather.

Lazy-materialising backends

For caches that materialise the file lazily (copy-on-read overlays), advertise RealPath only after the local copy is on disk. Return a non-nil error during the in-flight download window so the probe declines and the iterator falls back to afero.ReadFile for that call.

Hermetic tests that need to exercise the non-mmap path keep using fs.NewMemMap() — MemMapFs does not satisfy RealPather and the probe correctly declines.

Run the gate

go test ./service/ -run TestCountingFs

Managing a Shard Archive

Audience: Pulse internals contributors and embedders who manage a multi-shard .pulse archive — a Zip64 store-only cohort that fans out across N standalone .pulse shards under union semantics.

A shard archive is byte-distinct from a single-file .pulse cohort: the magic-byte dispatch at pulse.Open looks at the first four bytes and chooses single-file vs archive based on PULSE vs PK\x03\x04. Read-side commands (pulse api process, pulse api compose, pulse api sample, pulse api facet, pulse inspect, pulse predict) accept either transparently.

For the union semantics, per-shard cohesion, and the memory multiplier of read paths see the Cohort schema design skill (Sharded cohorts).

1. Create the archive

The first include seeds the canonical schema; remaining includes are validated against it via structural cohesion + the dict prefix rule. Atomic temp + rename:

pulse shard create q1_2019.pulse \
    --include 20190101.pulse \
    --include 20190108.pulse \
    --include 20190115.pulse

2. Append a shard

Validates cohesion + dict prefix, grows the canonical dict if needed (rewriting _schema.pulse before placing the new shard), then in-place appends the payload:

pulse shard add q1_2019.pulse 20190122.pulse

3. List shards

Reads _schema.pulse + central directory, prints basenames + per-shard record counts:

pulse shard list q1_2019.pulse

4. Verify

Re-validates every shard’s header + cohesion against the canonical schema. Useful after manual archive surgery or when a build pipeline appends shards from multiple producers:

pulse shard verify q1_2019.pulse

5. Compact

Reclaims orphan bytes (e.g. after pulse shard remove) and refreshes canonical metadata (aggregate_record_count, shard_count):

pulse shard compact q1_2019.pulse

6. Anchor syntax

Anchor syntax archive.pulse#shard.pulse opens a single shard inside an archive as a one-shard cohort — useful for diagnostics, debugging, and tests that exercise the cohesion path against a known-good shard.

Concurrency caveat

Pulse does not provide writer locking. Two processes running pulse shard add against the same archive race; the last writer wins and the earlier writer’s shard is lost. Sharding is single-writer by design — the caller owns concurrency control (orchestrator coordination, an external advisory lock, or a single-writer architecture).

Implementation surface

For maintainers extending the sharding internals, the surface lives in:

Width overflow on a categorical dictionary grown by an append surfaces as PULSE_SHARD_DICT_WIDTH_OVERFLOW; the stricter prefix-only validator (PULSE_SHARD_DICT_DIVERGENCE) is retained for the pulse shard verify strict path.

Run the gates

go test ./service/ -run TestShardArchive
go test ./encoding/ -run TestShardArchive
go test ./service/ -run TestCohesion

Wiring Pulse into an MCP Client

Audience: Pulse internals contributors and integrators wiring the pulse mcp server into an MCP-aware client (Claude Desktop, Claude Code, any custom MCP host).

Pulse ships an embedded MCP server that exposes the public facade through ten tools (one per facade method plus pulse_facet_schema) and two resource schemes (pulse://*.pulse, pulse-skill://*). Wiring it into a client is a three-step process: build, configure, restart.

1. Build the binary

make build

The resulting bin/pulse must be on the client’s PATH (or referenced by absolute path in the client’s configuration).

2. Configure the client

Add an mcpServers.pulse entry to the client’s configuration file (claude_desktop_config.json for Claude Desktop, ~/.claude.json for Claude Code) running pulse mcp and exporting PULSE_DATA_DIR:

{
  "mcpServers": {
    "pulse": {
      "command": "/abs/path/to/bin/pulse",
      "args": ["mcp"],
      "env": {
        "PULSE_DATA_DIR": "/abs/path/to/your/cohorts"
      }
    }
  }
}

The --bind-on-open flag (default true) controls whether successful pulse_inspect calls trigger registration of session-scoped tool variants whose JSON Schemas constrain field-name parameters to the cohort’s actual fields. Pass --bind-on-open=false for clients that bind themselves:

{
  "command": "/abs/path/to/bin/pulse",
  "args": ["mcp", "--bind-on-open=false"]
}

3. Restart the client

After the client reloads its MCP configuration the Pulse tools (pulse_inspect, pulse_predict, pulse_process, …) and resources (pulse://*.pulse, pulse-skill://*) appear in the tool / resource list.

Schema-bound enums

The full configuration recipe — including the schema-bound enums section that describes the inspect trigger, the multi-file limitation (latest inspect wins), and the transport-support caveat (bind-on-inspect works on the single stdio session via post-serve AddTool / RemoveTools; there is no per-session override for a shared HTTP server) — lives in Adding an MCP tool.

Verification

bin/pulse mcp --help
bin/pulse manifest --json | jq .mcp_tools

If the client supports an MCP tool inspector, point it at the configured server and confirm ten tools and two resource schemes are exposed.

Debugging a Predict Mismatch

Audience: Pulse internals contributors and embedders triaging a case where pulse predict says a request is valid but execution fails, or vice versa.

pulse predict reads only the .pulse header and schema; it never touches record data. That makes the predict path fast and safe to call from an LLM agent, but it also means predict can only catch structural and shape errors — type mismatches, unknown field names, operators that cannot accept a given field type. Anything that depends on the actual record values (e.g. a divide-by-zero in ATTR_FORMULA) is a runtime concern.

The triage sequence

Run predict against the request first:

pulse predict --json < request.json

Read the envelope’s errors and warnings arrays — predict returns both. Errors are structural failures; warnings are advisories that identify low-quality input but do not block execution.

The most common issues:

Resolve the typo / mismatch and re-run predict.

Cross-check against the actual schema

If predict’s error mentions a field that you believe should exist, read the schema with pulse inspect:

pulse inspect --json <file.pulse>

The inspect output lists every field’s name, type, nullability flag, and (if present) description. A mismatch between what you think the schema says and what predict / inspect actually reads almost always traces to a stale cohort file or a request crafted against a different schema generation.

When predict passes but runtime fails

If predict reports the request as valid but execution fails, the bug is in the processing layer — not predict.

descriptor/predict.go has a structural ban on importing service/ and processing/ (TestPredictNoExecutionImports). The predict path runs against encoding.ReadHeader + encoding.ReadSchema only, never against records. A divergence between predict’s verdict and runtime’s verdict is therefore always either:

• A processing-layer bug (the runtime rejects something predict had no way to know was bad), or
• A capability declaration that drifted from the registry (the capability row says the operator accepts a type the runtime rejects, or vice versa).

For the second case, the registry-vs-capability parity gates are:

• TestManifestOperatorsComplete
• TestStreamability_AggregationsKnown
• TestPredict_Streamable_MatchesRuntime
• TestCanStreamRequest_RegressionMatrix

Re-run them locally and inspect any failure — they catch capability drift before it ships.

Streamability mismatch

Predict surfaces a per-slot Streamable flag derived from the per-type Streamable() methods plus schema gates (decimal). The runtime parity check is processing.CanStreamRequest(req, schema). A divergence means either: