Integration Testing

Nexus provides an integration test framework for automated validation of test configurations. Tests run real engines with real LLM calls, verifying end-to-end behavior.

Quick Start

# Run all integration tests
go test -tags integration ./tests/integration/ -v

# Run a specific test
go test -tags integration ./tests/integration/ -run TestMinimal -v

# With timeout (tests make real API calls)
go test -tags integration ./tests/integration/ -timeout 5m -v

Prerequisites: ANTHROPIC_API_KEY must be set in the environment.

Architecture

Three components work together:

1. nexus.io.test — IO plugin that replaces nexus.io.tui in test configs. Feeds scripted inputs, collects all events, handles approvals.
2. pkg/testharness — Go test helper that boots the engine, waits for completion, and provides assertion methods.
3. Semantic judge — Optional Haiku-based LLM judge for evaluating dynamic response content.

test config (YAML with nexus.io.test)
    ↓
Engine boots normally (real plugins, real LLM)
    ↓
Test IO plugin emits scripted io.input events
    ↓
Collects ALL bus events
    ↓
Go test assertions on collected events
    ↓
Optional: LLM-as-judge for semantic validation

Writing Tests

Basic Test

//go:build integration

package integration

import (
    "testing"
    "time"

    "github.com/frankbardon/nexus/pkg/testharness"
)

func TestMyFeature(t *testing.T) {
    h := testharness.New(t, "configs/test-my-feature.yaml",
        testharness.WithTimeout(60*time.Second),
    )
    h.Run()

    h.AssertEventEmitted("io.output")
    h.AssertNoSystemOutput()
}

Test Config

Test configs use nexus.io.test instead of nexus.io.tui:

plugins:
  active:
    - nexus.io.test      # replaces nexus.io.tui
    - nexus.llm.anthropic
    - nexus.agent.react
    # ...

  nexus.io.test:
    inputs:
      - "Hello, who are you?"
    approval_mode: approve
    timeout: 60s

Mock LLM Responses

Most gate and plugin tests don’t need a real LLM. Use mock_responses to inject synthetic responses — no API key, no cost, millisecond execution:

nexus.io.test:
  inputs:
    - "Include the word FORBIDDEN in your response."
  mock_responses:
    - content: "This should never be seen."
  timeout: 15s

Gates fire before mock responses (priority 10 vs 20), so gate behavior is tested accurately. The mock just replaces the expensive LLM call.

Override Config Per Test

Use copyConfig to create a temp config with different inputs or settings:

func TestStopWords(t *testing.T) {
    cfg := copyConfig(t, "configs/test-all-gates.yaml", map[string]any{
        "nexus.io.test": map[string]any{
            "inputs":        []string{"Include FORBIDDEN_WORD in your response."},
            "approval_mode": "approve",
            "timeout":       "30s",
        },
    })

    h := testharness.New(t, cfg, testharness.WithTimeout(45*time.Second))
    h.Run()

    h.AssertSystemOutputContains("Content blocked")
}

Assertion Reference

Tier 1: Deterministic (free, fast, reliable)

Tier 2: Semantic (LLM judge, ~$0.001/assertion)

Semantic assertions require ANTHROPIC_API_KEY. Tests are skipped (not failed) if no judge is configured.

h.AssertOutputSemantic("response recalls the user's earlier question about greetings")

Harness Options

Raw Event Access

For assertions not covered by built-in methods:

for _, e := range h.Events() {
    if e.Type == "llm.response" {
        // inspect e.Payload
    }
}

Build Tags

Integration tests use //go:build integration so they don’t run with go test ./.... They require API keys and make real LLM calls.

# Unit tests only (default)
go test ./...

# Integration tests only
go test -tags integration ./tests/integration/

# Both
go test -tags integration ./...

Testing UI Plugins (Future)

The test IO plugin validates the bus contract shared by all IO plugins (TUI, browser, wails). When integration tests pass, the bus-side behavior is validated.

Transport-specific rendering requires separate test suites per plugin:

These validate that the transport correctly bridges bus events to/from the UI. The test IO plugin serves as a reference for the event flow, ordering, and payload shapes.

Future work: extract an IOContract interface from common subscription/emission patterns so any IO plugin can run a shared contract test suite.