GHOST TRIALS / AI AGENT DOMAIN

Find what breaks.
Prove why.

A second verification domain built on the existing Hunt lifecycle. It runs versioned reference agents, observes tool requests, verifies bounded failures, and checks the next version against saved cases.

Controlled reference milestone. Only checked-in reference agents and in-memory mock tools execute. Arbitrary customer code, production credentials, external services and open network access are not supported.

Supported scope

Select AI Agent on the Hunt page. The initial targets are a deliberately vulnerable reference agent, a corrected version, and a timeout-test version. Tests cover unauthorized tool requests, permission scope, instruction boundaries, task completion, protected state, and adversarial tool output. The tools are local mocks; a denied request is recorded but never applied to a real service.

From Hunt to Ghost Case

Choose a bundled target, verification objective, trial count and wall-time budget.
Ghost explores bounded input and tool-response variants, repeats observed failures, and mutates a nearby input.
Open a trial to inspect parent-observed tool calls, structured output, and versioned assertions.
Open a verified Ghost Case for its saved scenario, seed, evidence digest, reproduction attempts and regression action.
Run the corrected reference version on the same saved case. Passing this case is not a general security guarantee.

What counts as evidence

The parent harness authorizes mock tools, owns simulated state, and records each request and result. The deterministic oracle evaluates forbidden invocations, protected-state changes, and required task completion, with exact event or state references. Execution errors and timeouts do not become verified policy failures. A SHA-256 digest protects the retained record; reads recompute the assertions and flag tampering. Hashes alone do not prove an external execution environment.

Optional Nemotron interpretation

After a trace passes integrity checks, an operator may explicitly request a Nebius Token Factory / Nemotron Super review. It must cite existing trace events and returns an explanation, possible factors, remediation and regression suggestions. The interpretation is separate from, and cannot change, the deterministic outcome. The button remains disabled until server-side credentials, enablement, organization/platform budgets and provider spending controls are ready. No paid inference is required for a local Hunt.

Trust and execution limits

Reference agents run in a subprocess with CPU, memory, output, file-descriptor and wall-time limits. The child has a sanitized environment; mock tools and state live in the parent process. These controls are not a secure sandbox for untrusted software. The Railway service must reject external agent executables until a genuinely isolated runner with non-root execution, filesystem and network restrictions is available. Existing sign-in, role and organization checks protect the API and stored traces.

Reproduce both domains

LOCAL CPU · NO PAID PROVIDER

uv sync --frozen --extra dev
uv run python -m scripts.multidomain_demo --output data/multidomain-demo

This command executes MuJoCo physics, then a reference agent Hunt, evidence verification, failure replay and corrected-agent regression. For the complete contracts and limitations, read docs/AGENT_VERIFICATION.md in the repository.