Inspect-Mode Protocol

nexus eval --inspect-mode is the headless JSON-on-stdin/stdout protocol that external eval harnesses (AISI Inspect AI, Braintrust, custom CI tooling) use to drive Nexus. The binary reads one JSON object from stdin, runs a single agent turn (or multi-turn run, capped by max_turns) under the supplied config, and writes one JSON object to stdout.

This page is the durable wire-format reference. It is pinned by the schema-stability snapshot test in pkg/eval/protocol/schema_test.go. PRs that change the wire format must update both that snapshot and the documentation in this file in the same change.

Nexus does not ship a Python shim. A 15-line shim is sketched at the bottom of this page so the interop story is documented; the shim itself is out-of-tree by design (see plan.md, “No Python in this repo”).

Invoking

echo '{"schema":1, ...}' | nexus eval --inspect-mode

Flags:

• --inspect-mode — required. Mutually exclusive with subcommands and positional args; combining them returns an INVALID_REQUEST error.
• --timeout=<duration> — optional. Per-request deadline. When unset, the env var NEXUS_EVAL_INSPECT_TIMEOUT is honored. Default 60s.

The deadline applies to the entire request (engine boot, agent run, shutdown). Crossing it surfaces as a TIMEOUT error code.

Request

Exactly one JSON object on stdin, terminated by EOF:

{
  "schema": 1,
  "config_path": "configs/coding.yaml",
  "config_inline": "<yaml string>",
  "user_input": "explain the build error in main.go",
  "max_turns": 8,
  "metadata": { "case_id": "swe-bench-1234" }
}

Strict parsing applies: unknown fields are rejected with INVALID_REQUEST. This catches typos at the harness boundary instead of silently defaulting fields.

The runner overlays your config to make the run hermetic:

• core.sessions.root is overridden to a temp directory under os.TempDir(). The directory is removed on exit unless NEXUS_EVAL_INSPECT_KEEP_SESSIONS=1 is set (useful for post-hoc journal forensics).
• nexus.io.test is added to plugins.active if absent; visual transports (nexus.io.tui, nexus.io.browser, nexus.io.wails, nexus.io.oneshot) are stripped because the run must be headless.
• plugins.nexus.io.test.inputs is set to [user_input], approval_mode to approve, timeout to 600s. Any other keys you set on nexus.io.test (notably mock_responses) are preserved.

Response

One JSON object on stdout, exit 0 on success:

{
  "schema": 1,
  "session_id": "01HK...",
  "final_assistant_message": "the build error is in line 42",
  "tool_calls": [
    {
      "tool": "shell",
      "args": { "cmd": "go build ./..." },
      "result_summary": "main.go:42: undefined: Foo",
      "duration_ms": 412
    }
  ],
  "tokens": { "input": 6213, "output": 1102 },
  "latency_ms": 18733,
  "metadata": { "case_id": "swe-bench-1234" },
  "error": null
}

Errors

When the request fails, the process exits non-zero and the response’s error field is populated. This redundancy is deliberate: harnesses can key off either signal.

{
  "schema": 1,
  "tool_calls": [],
  "metadata": { "case_id": "..." },
  "error": {
    "code": "CONFIG_LOAD",
    "message": "config load: read /missing.yaml: no such file or directory"
  }
}

Schema versioning

The schema field is the durable contract. Bumping it is a deliberate event:

1. Increment SchemaVersion in pkg/eval/protocol/protocol.go.
2. Update the snapshots in pkg/eval/protocol/schema_test.go.
3. Add a migration note to this page describing what changed and how external harnesses should adapt.

The schema-stability snapshot test enforces this discipline: a drift in field names, types, or order fails CI until the snapshot is updated deliberately.

External harness integration

Nexus does not ship a Python shim. The protocol is the contract; any language with subprocess and a JSON parser can drive it. Here is a minimal Python sketch demonstrating the wire format; copy it out-of-tree into your eval harness as needed.

# example: drive_nexus.py — NOT shipped with Nexus, copy out-of-tree.
import json
import subprocess

def run_nexus(config_path, user_input, *, max_turns=0, metadata=None,
              timeout="60s"):
    req = {
        "schema": 1,
        "config_path": config_path,
        "user_input": user_input,
    }
    if max_turns:
        req["max_turns"] = max_turns
    if metadata:
        req["metadata"] = metadata
    proc = subprocess.run(
        ["nexus", "eval", "--inspect-mode", f"--timeout={timeout}"],
        input=json.dumps(req).encode(),
        capture_output=True,
        check=False,
    )
    resp = json.loads(proc.stdout)
    if resp.get("error"):
        raise RuntimeError(f"{resp['error']['code']}: {resp['error']['message']}")
    return resp

For Inspect AI specifically, write a Solver whose __call__ drives this protocol and emits the final assistant message as the model output; score on the round-tripped metadata.

See also

• Eval Harness Overview — the bigger picture.
• Case Format — the on-disk eval bundle that runs through nexus eval run rather than the inspect protocol.
• pkg/eval/protocol/ — Go source of truth for the wire format.
• pkg/eval/protocol/schema_test.go — snapshot test pinning the format.