Point Lookup & Index Management

Audience: Go embedders resolving a single row (or a small duplicate set) by an exact key value instead of scanning or filtering a whole cohort.

Pulse.Lookup is a keyed point-lookup store: O(1) — flat latency regardless of cohort size — against a prebuilt sidecar index (Pulse.BuildIndex). It only works against a single-file cohort (shard archives are rejected with PULSE_INDEX_UNSUPPORTED_SHARDED; the archive.pulse#shard.pulse anchor is a tested workaround), it is equality-only (no ranges, no expressions), and it always resolves the full key — no prefix or partial-key matching. For anything else (ranges, expressions, unindexed ad hoc queries), use Process with FILTER_INCLUDE / FILTER_EXPRESSION instead.

Build then look up

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/frankbardon/pulse"
    "github.com/frankbardon/pulse/types"
)

func main() {
    ctx := context.Background()

    p, err := pulse.New(pulse.Options{DataDir: "/var/data/pulse"})
    if err != nil {
        log.Fatal(err)
    }

    // Build the sidecar index once — cheap to re-run (idempotent,
    // byte-identical sidecar for an unchanged cohort), but there's no
    // need to call it on every lookup.
    if _, err := p.BuildIndex(ctx, "sales.pulse", []string{"order_id"}); err != nil {
        log.Fatal(err)
    }

    result, err := p.Lookup(ctx, &pulse.LookupRequest{
        Cohort: &types.Cohort{Filename: "sales.pulse"},
        Field:  "order_id",
        Value:  "42",
    })
    if err != nil {
        log.Fatal(err)
    }
    for _, row := range result.Rows {
        fmt.Println(row)
    }
}

BuildIndex returns a *pulse.BuildIndexResult{IndexPath, ManifestPath, Index} — IndexPath is the derived sidecar path, Index is the in-memory *pulse.SidecarIndex that was serialized there, handed back in case you want to inspect it without a round-trip read. Every type inside it has a root spelling: Keys []pulse.SidecarIndexKeySpec (ordered key columns), Buckets []pulse.SidecarIndexBucket whose Entries are pulse.SidecarIndexEntry{Key, RowIDs}, and Fingerprint, a pulse.CohortFingerprint (the SHA-256 of the cohort’s bytes).

Lookup’s single-key convenience path — Field + Value, both strings — is the E1 shape and stays the ergonomic default. Value is always text, the same string-literal convention Filterer.Values uses for FILTER_INCLUDE/FILTER_EXCLUDE.

Composite keys

A multi-column key is an ordered tuple — column order matters end to end, from the BuildIndex call that wrote the sidecar through every Lookup that probes it. Use Keys instead of Field/Value:

if _, err := p.BuildIndex(ctx, "sales.pulse", []string{"region", "date"}); err != nil {
    log.Fatal(err)
}

result, err := p.Lookup(ctx, &pulse.LookupRequest{
    Cohort: &types.Cohort{Filename: "sales.pulse"},
    Keys: []pulse.LookupKey{
        {Field: "region", Value: "EU"},
        {Field: "date", Value: "2026-01-01"},
    },
    ReturnColumns: []string{"order_id", "revenue"},
    Multiplicity:  pulse.LookupMultiplicityAll,
})
if err != nil {
    log.Fatal(err)
}

Keys takes precedence over Field/Value when both are set. {region, date} and {date, region} name the same two logical values but resolve against two different sidecar indexes — build the index with the same column order you’ll query with.

ReturnColumns projects the result down to named fields; empty means every schema field (LookupResult.Rows[i] is a full map[string]any), matching Sample’s no-projection default.

Multiplicity

Multiplicity controls what happens when the matched key resolves to more than one row-id (a duplicate key value in the source cohort):

The mode type and its constants are re-exported on the pulse package, so you never need to import types to set LookupRequest.Multiplicity:

result, err := p.Lookup(ctx, &pulse.LookupRequest{
    Cohort: &types.Cohort{Filename: "sales.pulse"},
    Field:  "region",
    Value:  "EU",
})
if err != nil {
    if errors.HasCode(err, errors.PULSE_LOOKUP_AMBIGUOUS) {
        // region isn't unique — rebuild the index on a real key, or
        // opt into LookupMultiplicityFirst / LookupMultiplicityAll.
    }
    log.Fatal(err)
}

errors.HasCode(err, errors.CODE) (github.com/frankbardon/pulse/errors) is the coded-error check used across the codebase — the same pattern covers PULSE_INDEX_MISSING, PULSE_INDEX_STALE, PULSE_INDEX_UNSUPPORTED_SHARDED, PULSE_INDEX_MANIFEST_INVALID, PULSE_INDEX_MANIFEST_STALE, and PULSE_LOOKUP_NOT_FOUND.

Index lifecycle

Three more facade methods manage sidecars without a lookup:

// Freshness check — size fast-path before an authoritative
// full-content SHA-256 recompute.
verify, err := p.VerifyIndex(ctx, "sales.pulse", []string{"order_id"})
// verify.Fresh, verify.Reason ("stat_mismatch" | "fingerprint_match" |
// "fingerprint_mismatch"), verify.FastPath, verify.ModTimeDrift

// Enumerate every sidecar built against a cohort, with each one's
// ordered key tuple. Served from the keyless manifest when present, so
// it needs no directory listing.
indexes, err := p.ListIndexes(ctx, "sales.pulse")
// []pulse.IndexInfo{IndexPath, Keys, DistinctKeys, IndexedRecords}

// Remove a sidecar. Non-interactive — no confirmation prompt. Prunes
// the manifest entry in the same call.
err = p.DropIndex(ctx, "sales.pulse", []string{"order_id"})

Staleness has three outcomes, not two

• Size mismatch → PULSE_INDEX_STALE immediately, with no hash. This is conclusive rather than merely cheap: the sidecar’s fingerprint covers the whole file, so a file of a different length cannot produce the recorded digest.
• Size and mtime both match → served hash-free. That is the O(1) property; a full hash per lookup would defeat the point of the store.
• Mtime moved while the size held → not staleness. It escalates to the sidecar’s own SHA-256 and refuses only if the content really differs.

That third case is the one worth knowing about. SourceModTime is recorded as Unix nanoseconds; an S3 LastModified is whole seconds, and every copy, restore and cache hop truncates somewhere of its own choosing. Treating drift as proof of mutation refused byte-identical indexes permanently — not transiently — so a cohort read through an object-storage backend could not be looked up at all.

The escalation is memoised per (cohort path, size, mtime), so a backend that never preserves nanoseconds pays one whole-file hash per distinct stat, not one per lookup. The memo assumes exactly what the stat fast-path already assumes (same path, size and mtime means the same bytes), so it adds no freshness gap of its own.

VerifyIndex is the authoritative path: a matching size alone is inconclusive (a same-size rewrite is possible), so it always falls through to a full content-hash recompute before reporting Fresh: true, and it never reads that memo — its whole job is the verdict that does not rest on a stat pair. ModTimeDrift reports that the mtime disagreed and the hash overruled it, on both a fresh and a stale verdict. Run VerifyIndex explicitly wherever you need a stronger guarantee than Lookup gives you for free — before a batch of lookups against a cohort you don’t fully trust, or in a periodic health check.

The one residual gap is unchanged and deliberate: an in-place edit that preserves both size and mtime passes Lookup and is caught only by VerifyIndex.

Discovering which indexes exist

A sidecar’s filename embeds a hash of its ordered key tuple (cohort.pulse.<keyhash>.idx), so the file cannot be opened without already knowing the tuple — and key order is significant, so guessing is a permutation search. On a local disk you could glob <cohort>.*.idx and read each match’s key spec; object storage cannot list a directory, so that escape is unavailable exactly where cohorts are usually served from.

BuildIndex therefore also writes a keyless catalog beside the cohort at a deterministic path — sales.pulse.indexes.json — holding every index’s ordered key tuple with its column types and counts:

{
  "format_version": "1",
  "kind": "pulse.index-manifest",
  "cohort": "sales.pulse",
  "indexes": [
    {
      "index_path": "sales.pulse.42fdb66c2f329c5c.idx",
      "key_hash": "42fdb66c2f329c5c",
      "keys": [
        {"name": "category_id", "type": "u16"},
        {"name": "brand_id", "type": "u32"}
      ],
      "distinct_keys": 480,
      "indexed_records": 50000
    }
  ]
}

ListIndexes reads it first, which is why discovery costs one read of that document plus one existence probe per entry and no read of any sidecar. BuildIndexResult.ManifestPath names the file. Its format_version is the manifest’s own, independent of the .pulse format and the sidecar index format, so a field can be added to it without any rebuild.

Behaviour at the edges:

The manifest is one of Pulse’s own *.json sidecars, so a PULSE_LABEL_TABLES_DIR or PULSE_RANGE_TABLES_DIR pointed at a data directory skips it by suffix.

Keyable-type policy

Not every field type can serve as a lookup key — set_* is rejected (a multi-select bitmask has no single unambiguous equality value; use FILTER_SET via Process instead). See the full allow/reject table in pulse api lookup or skills/cohort-sidecar-index.md.

When to use this vs Process + FILTER_INCLUDE

Reach for Lookup when you have an exact key and want the fastest possible single/few-row read; reach for Process with FILTER_* for anything that isn’t a full-key equality match (ranges, expressions, set_* membership, or querying a column you haven’t built an index for). The performance shape is the load-bearing claim, not any particular absolute number: indexed lookup is flat regardless of cohort size, while an unindexed scan is linear in record count — the gap widens, not narrows, as a cohort grows. Build the index once with BuildIndex; every subsequent Lookup against that key combination pays only the flat O(1) cost.

See also

• pulse api lookup — the CLI leaf for Lookup, plus pulse index build/list/verify/drop for the lifecycle methods above.
• Go API Overview — the rest of the public facade.