Managing a Shard Archive

Audience: Pulse internals contributors and embedders who manage a multi-shard .pulse archive — a Zip64 store-only cohort that fans out across N standalone .pulse shards under union semantics.

A shard archive is byte-distinct from a single-file .pulse cohort: the magic-byte dispatch at pulse.Open looks at the first four bytes and chooses single-file vs archive based on PULSE vs PK\x03\x04. Read-side commands (pulse api process, pulse api compose, pulse api sample, pulse api facet, pulse inspect, pulse predict) accept either transparently.

For the union semantics, per-shard cohesion, and the memory multiplier of read paths see the Cohort schema design skill (Sharded cohorts).

1. Create the archive

The first include seeds the canonical schema; remaining includes are validated against it via structural cohesion + the dict prefix rule. Atomic temp + rename:

pulse shard create q1_2019.pulse \
    --include 20190101.pulse \
    --include 20190108.pulse \
    --include 20190115.pulse

2. Append a shard

Validates cohesion + dict prefix, grows the canonical dict if needed (rewriting _schema.pulse before placing the new shard), then in-place appends the payload:

pulse shard add q1_2019.pulse 20190122.pulse

Go embedders building shards from rows skip the intermediate files: Pulse.NewCohortBuilder with ShardSplit writes a new archive through one CreateShardArchive call, and an anchored target (q1_2019.pulse#20190122.pulse) appends one shard through one AddShard call, with the same cohesion checks and mandatory warnings. See Reading & Building Cohorts.

Set-width auto-widen

Two conditions trigger it, and they compose:

1. The dictionary union outgrows the bitmask. The field is promoted to the narrowest rung that holds the union.
2. The arriving shard declares a different rung. A re-import of a new period infers the rung its own data needs, so a set_u128 shard legitimately meets a set_u64 archive with no dictionary overflow in sight.

Either way the add does not fail. The field is promoted across the canonical _schema.pulse and every shard payload already in the archive, and the merge proceeds. The alternative, refusing, forces a full re-import to reach a layout a mechanical re-stride already produces.

The promotion is only ever wider. A shard arriving at a narrower rung never narrows the archive — narrowing would drop every selection above the target’s ceiling, and it would do so silently, with the record still decoding to a smaller and entirely plausible selection. The arriving shard is promoted to the archive’s rung instead, and it is the only payload rewritten. SetWidening.From != To is the test for “the archive moved”; the warning’s details.archive_widened says the same.

Only the rung dimension relaxes, and it relaxes by order, not by loosening a validator: encoding.PlanSetWidening now runs before encoding.ValidateStructuralCohesion, reconciles the rungs, and cohesion then runs over the reconciled schemas as strictly as ever. That matters because pulse shard verify calls the same validator, and an archive whose shards genuinely carry different strides is a corruption signature that must stay fatal there.

pulse shard create widens on the same conditions. Refusing at seed while widening at add was an asymmetry with no defence: the same two files produced an archive when fed one after the other and an error when fed together. Pulse.CreateShardArchive therefore returns a *CreateShardArchiveResult, not a bare error, for the same reason AddShard returns a result — so the mandatory warning has nowhere to be dropped.

The rewrite is atomic (temp file + fsync + rename), so a failure mid-rewrite leaves the original archive byte-identical and openable at its old rung. It is also never silent: every widen emits a mandatory PULSE_SHARD_SET_WIDENED warning naming the field, the old rung, the new rung and the number of shards rewritten, on the --json envelope’s warnings array and on the text output.

Added 20190122.pulse to q1_2019.pulse (now 4 shard(s))
  WARN   [PULSE_SHARD_SET_WIDENED] set field "media" was widened from set_u64 to set_u128 ...

A union above set_u256 has nowhere left to widen to and still fails with PULSE_SHARD_DICT_WIDTH_OVERFLOW. Categorical widths are untouched — a categorical’s width is fixed at folder creation.

pulse shard verify reports per-set-field width headroom so a widen is foreseeable before you pay for it.

Grouped (0x02) shards

Shards may be deduped cohorts (pulse import --group, pulse dedup). Every shard is decoded with the canonical schema, so an archive has ONE parent-group layout, and create / add share one merge rule (internal/service/shard_groups.go, mergeShard):

1. The arriving shard is flattened to its ungrouped twin and reconciled exactly as an ungrouped shard — set-rung plan, strict field cohesion, categorical union and remap (null placeholders keep their bytes, since they are part of a group entry’s identity).
2. The archive’s layout wins. An ungrouped archive stores a grouped arrival flattened; a grouped archive re-encodes an arrival with other groups (or none) into its own.
3. Group dictionaries union-merge canonical-first: the encoder is seeded with the canonical dictionaries, so stored shards are untouched and only the arrival is renumbered. A union past the u32 index space is PULSE_SHARD_DICT_WIDTH_OVERFLOW.
4. A constant group the arrival disagrees with is promoted to an indexed group across every shard (a whole-archive rewrite); a declared key the arrival violates is refused with PULSE_GROUP_MEMBER_NOT_CONSTANT.
5. Strict cohesion runs over the reconciled grouped schemas.

Layout changes emit the mandatory PULSE_SHARD_GROUPS_REWRITTEN warning (details.reason = incoming_flattened / incoming_regrouped / constant_promoted, details.archive_rewritten) and a regrouped entry in --json data. pulse shard verify refuses a shard whose group layout differs from the canonical one, applies the prefix rule to group dictionaries, and reports group_index_headroom.

3. List shards

Reads _schema.pulse + central directory, prints basenames + per-shard record counts:

pulse shard list q1_2019.pulse

4. Verify

Re-validates every shard’s header + cohesion against the canonical schema. Useful after manual archive surgery or when a build pipeline appends shards from multiple producers:

pulse shard verify q1_2019.pulse

5. Compact

Reclaims orphan bytes (e.g. after pulse shard remove) and refreshes canonical metadata (aggregate_record_count, shard_count):

pulse shard compact q1_2019.pulse

6. Anchor syntax

Anchor syntax archive.pulse#shard.pulse opens a single shard inside an archive as a one-shard cohort — useful for diagnostics, debugging, and tests that exercise the cohesion path against a known-good shard.

Concurrency caveat

Pulse does not provide writer locking. Two processes running pulse shard add against the same archive race; the last writer wins and the earlier writer’s shard is lost. Sharding is single-writer by design — the caller owns concurrency control (orchestrator coordination, an external advisory lock, or a single-writer architecture).

Implementation surface

For maintainers extending the sharding internals, the surface lives in:

Width overflow on a categorical dictionary grown by an append surfaces as PULSE_SHARD_DICT_WIDTH_OVERFLOW; the stricter prefix-only validator (PULSE_SHARD_DICT_DIVERGENCE) is retained for the pulse shard verify strict path. The set-width auto-widen planner is encoding.PlanSetWidening and its executor is internal/service/shard_widen.go over E4-S1’s internal/encoding/widen.go engine; encoding.SetWidthHeadroomFor feeds verify’s headroom report.

Run the gates

go test ./internal/service/ -run 'TestShardArchive|TestShardGroups'
go test ./internal/encoding/ -run TestShardArchive
go test ./internal/encoding/ -run Cohesion