---
title: "Autoresearch Experiment Loops"
source: https://docs.autohand.ai/guides/teams-and-swarms/autoresearch
---

# Autoresearch Experiment Loops

Use /autoresearch when progress can be measured repeatedly: change one variable, run a benchmark and correctness checks, keep or revert the result, record what happened, and choose the next experiment.

**Availability:** `/autoresearch` is a newer command surface. Run `/help` and confirm it appears before relying on it in a workflow. Update Autohand Code if your installed build does not advertise the command.

**Use a clean branch or dedicated worktree.** The loop instructs Autohand to stage and commit kept experiments, and to revert discarded, failed, or crashed experiments with a hard reset or file checkout to the last kept commit. Commit or move unrelated work before starting, and review the generated benchmark and checks before allowing the loop to run.

## Use it for a measurable objective

Autoresearch is an optimizer, not a general “keep trying” mode. A good objective has one primary numeric metric, a direction, a stable benchmark, correctness checks, an editable scope, and a finite budget.

| Good fit | Metric | Direction | Backpressure check |
|---|---|---|---|
| Reduce unit-test runtime | total_ms | Lower | Full tests and typecheck |
| Shrink a browser bundle | bundle_bytes | Lower | Build plus smoke tests |
| Improve retrieval quality | ndcg_at_10 | Higher | Latency and regression set |
| Reduce query latency | p95_ms | Lower | Result equivalence and migration safety |

Use `/deep-research` when the primary outcome is an evidence-backed answer. Use a normal agent team when the outcome is a multi-part implementation without a stable numeric objective.

## Start interactively or from the CLI

``` bash
# Interactive session
/autoresearch optimize unit test runtime

# Non-interactive aliases
autohand auto-research optimize unit test runtime
autohand autoresearch optimize unit test runtime
```

With only a goal, Autohand infers the metric, unit, direction, benchmark, checks, scope, iteration budget, timeout, and useful sub-agent phases from the repository. It asks concise setup questions only for material gaps, then initializes the session before the first experiment.

### Initialize the contract explicitly

``` bash
/autoresearch reduce unit test runtime \
  --metric total_ms \
  --unit ms \
  --direction lower \
  --measure "node scripts/measure-tests.mjs" \
  --checks "npm run typecheck && npm test" \
  --max-iterations 20 \
  --timeout-ms 600000 \
  --scope src \
  --scope tests \
  --subagent-ideas \
  --subagent-analysis \
  --subagent-finalization
```

When `--metric`, `--unit`, `--direction`, and `--measure` are all present, the command writes the initial benchmark configuration immediately. Repeated `--scope` flags record the intended editable paths or globs in the experiment prompt; they are an instruction boundary, not a filesystem sandbox. Run Autohand in an isolated worktree when enforcement matters.

## Build a deterministic metric contract

The measure command must exit successfully and print a machine-readable metric line:

``` text
METRIC total_ms=1842.6
```

The metric name must exactly match the configured name. Decimal, signed, and scientific-notation values are accepted. Additional benchmark output is preserved as a bounded excerpt, but only the `METRIC <name>=<number>` line drives comparison.

### Example measurement wrapper

``` javascript
// scripts/measure-tests.mjs
import { spawnSync } from 'node:child_process';
import { performance } from 'node:perf_hooks';

const started = performance.now();
const result = spawnSync('npm', ['run', 'test:unit'], {
  stdio: 'inherit',
  env: { ...process.env, CI: '1' }
});
const elapsed = performance.now() - started;

if (result.status !== 0) process.exit(result.status ?? 1);
console.log(`METRIC total_ms=${elapsed.toFixed(2)}`);
```

-   Warm or reset caches consistently; do not let setup differ between runs.
-   Pin datasets, concurrency, environment variables, and hardware where possible.
-   Use repeated samples inside the measure script when variance is high.
-   Keep correctness checks separate from the optimized metric so a faster wrong result cannot win.

## Understand one experiment iteration

1.  Read prior outcomes from `.auto/log.jsonl` and the living context in `.auto/prompt.md`.
2.  Optionally delegate idea generation or surprising-result analysis to sub-agents.
3.  Choose one focused hypothesis and make one bounded change.
4.  Run the before hook, benchmark, after hook, and correctness checks within the configured timeout.
5.  Log the metric, status, description, optional commit, evidence excerpt, hypothesis, learning, and next focus.
6.  Keep an improvement with a reviewable commit or revert a regression to the last kept commit before the next iteration.
7.  Update the living prompt with wins, dead ends, and constraints, then repeat until the budget or stop condition is reached.

The default maximum is 30 iterations and the default benchmark timeout is 600,000 ms. Set smaller budgets first; expand only after the metric is stable and the keep/revert behavior is trustworthy.

## Know the three experiment tools

| Tool | Writes or runs | Important contract |
|---|---|---|
| init_experiment | config.json, measure.sh, prompt.md, optional checks.sh | Requires name, metric name/unit, direction, and measure script |
| run_experiment | Local hooks, benchmark, metric parser, and correctness checks | Returns metric, output, timeout/failure state, and whether checks failed |
| log_experiment | log.jsonl and updates session statistics | Records status and description; can include commit and bounded output |

## Delegate the thinking, not the metric

The sub-agent flags store phase choices in `.auto/config.json` and `.auto/prompt.md`. The lead then uses existing `delegate_task` or `delegate_parallel` tools during those phases.

| Flag | Phase | Good agent choices | Expected output |
|---|---|---|---|
| --subagent-ideas | Before selecting a change | performance-engineer, researcher, language specialist | Ranked hypotheses with affected paths and predicted mechanism |
| --subagent-analysis | When a result is noisy or surprising | data-analyst, debugger, database-optimizer | Variance diagnosis, confounders, and a discriminating next run |
| --subagent-finalization | After useful runs are kept | reviewer, knowledge-synthesizer, technical-writer | Review of kept commits, interactions, risks, and branch grouping |

**The lead keeps benchmark authority.** Sub-agents may propose or analyze experiments, but the configured measure script, correctness checks, and keep/revert rules remain the source of truth.

## Inspect the `.auto/` workspace

| Artifact | Purpose | Review it for |
|---|---|---|
| .auto/config.json | Metric, direction, budget, timeout, and sub-agent phases | Wrong optimization target or unsafe budget |
| .auto/measure.sh | Executable benchmark wrapper | Determinism and exact metric output |
| .auto/checks.sh | Optional correctness backpressure after a passing benchmark | Missing functional or compatibility coverage |
| .auto/prompt.md | Goal, scope, tried ideas, wins, dead ends, and delegation plan | Repeated failed ideas or scope drift |
| .auto/log.jsonl | Append-only experiment history and evidence excerpts | Metric trend, regressions, commits, and confidence |
| .auto/state.json | Active/paused state and iteration count | Unexpected resume behavior |
| .auto/dashboard.html | Self-contained result dashboard | Run comparison and status distribution |
| .auto/finalize.md | Reviewable finalization plan for kept runs | Interactions and follow-up before integration |
| .auto/finalize-branches.json | Structured kept-run branch manifest | Exact commits and suggested branch commands |

## Control, inspect, and finalize a session

``` bash
/autoresearch status
/autoresearch off
/autoresearch export
/autoresearch finalize
/autoresearch clear --yes
```

-   `status` summarizes active state, iterations, metric configuration, and recorded runs.
-   `off` pauses the loop and disables automatic resume without deleting artifacts.
-   `export` writes the self-contained `.auto/dashboard.html`.
-   `finalize` derives a review plan and branch manifest from kept runs. It does not create or switch branches, rewrite history, or cherry-pick changes.
-   `clear --yes` deletes session state and requires explicit confirmation.

The same subcommands are available through `autohand auto-research ...` and `autohand autoresearch ...` for non-interactive use.

Starting again while a paused session has `.auto/prompt.md` resumes persisted context rather than replacing the original goal. Clear the session explicitly before beginning an unrelated objective.

## Add lifecycle and workspace hooks

Autoresearch emits lifecycle events for start, pause, initialization, before/after benchmark execution, run, and log activity. Use them for observability, policy enforcement, or external notifications.

For benchmark-local setup and cleanup, add executable scripts:

``` bash
.auto/hooks/before.sh
.auto/hooks/after.sh
```

Both run with `AUTO_RESEARCH_WORKSPACE` set to the project root and `AUTO_RESEARCH_HOOK` set to `before` or `after`. A failed or timed-out hook is recorded as an experiment failure; do not hide benchmark invalidation inside a successful exit.

## Use the same state from RPC and ACP

JSON-RPC clients can start, inspect, and stop the same persisted session through `autohand.autoresearch.start`, `autohand.autoresearch.status`, and `autohand.autoresearch.stop`. The start payload can supply metric name/unit, direction, measure or checks command/script, timeout, file scope, iteration budget, and sub-agent phase configuration.

ACP sessions advertise `/autoresearch` in command metadata and use the same slash-command path. This keeps the experiment contract consistent across the terminal, editors, and automation clients.

## Failure modes and corrections

### No metric is parsed

Make the benchmark print exactly `METRIC configured_name=<number>`. Check spelling, exit code, working directory, and whether output is written to standard output or standard error.

### Results oscillate

Increase samples inside the measure script, control caches and background load, record variance, and delegate measurement analysis before accepting small improvements.

### Faster runs break behavior

Strengthen `.auto/checks.sh`. A metric improvement with failed correctness checks must not be treated as a win.

### The loop repeats dead ends

Inspect and update `.auto/prompt.md` with the mechanism, evidence, and why the idea failed. Ask idea-generation agents for hypotheses that are materially different, not syntactic variants.

### The working tree becomes hard to review

Reduce experiment scope, require one focused change per iteration, record commit hashes for kept runs, pause the loop, and run `/autoresearch finalize` before further changes.