Autoresearch Experiment Loops

Use it for a measurable objective

Autoresearch is an optimizer, not a general “keep trying” mode. A good objective has one primary numeric metric, a direction, a stable benchmark, correctness checks, an editable scope, and a finite budget.

Use /deep-research when the primary outcome is an evidence-backed answer. Use a normal agent team when the outcome is a multi-part implementation without a stable numeric objective.

Start interactively or from the CLI

# Interactive session
/autoresearch optimize unit test runtime

# Non-interactive aliases
autohand auto-research optimize unit test runtime
autohand autoresearch optimize unit test runtime

With only a goal, Autohand infers the metric, unit, direction, benchmark, checks, scope, iteration budget, timeout, and useful sub-agent phases from the repository. It asks concise setup questions only for material gaps, then initializes the session before the first experiment.

Initialize the contract explicitly

/autoresearch reduce unit test runtime \
  --metric total_ms \
  --unit ms \
  --direction lower \
  --measure "node scripts/measure-tests.mjs" \
  --checks "npm run typecheck && npm test" \
  --max-iterations 20 \
  --timeout-ms 600000 \
  --scope src \
  --scope tests \
  --subagent-ideas \
  --subagent-analysis \
  --subagent-finalization

When --metric, --unit, --direction, and --measure are all present, the command writes the initial benchmark configuration immediately. Repeated --scope flags record the intended editable paths or globs in the experiment prompt; they are an instruction boundary, not a filesystem sandbox. Run Autohand in an isolated worktree when enforcement matters.

Build a deterministic metric contract

The measure command must exit successfully and print a machine-readable metric line:

METRIC total_ms=1842.6

The metric name must exactly match the configured name. Decimal, signed, and scientific-notation values are accepted. Additional benchmark output is preserved as a bounded excerpt, but only the METRIC <name>=<number> line drives comparison.

Example measurement wrapper

// scripts/measure-tests.mjs
import { spawnSync } from 'node:child_process';
import { performance } from 'node:perf_hooks';

const started = performance.now();
const result = spawnSync('npm', ['run', 'test:unit'], {
  stdio: 'inherit',
  env: { ...process.env, CI: '1' }
});
const elapsed = performance.now() - started;

if (result.status !== 0) process.exit(result.status ?? 1);
console.log(`METRIC total_ms=${elapsed.toFixed(2)}`);

• Warm or reset caches consistently; do not let setup differ between runs.
• Pin datasets, concurrency, environment variables, and hardware where possible.
• Use repeated samples inside the measure script when variance is high.
• Keep correctness checks separate from the optimized metric so a faster wrong result cannot win.

Understand one experiment iteration

1. Read prior outcomes from .auto/log.jsonl and the living context in .auto/prompt.md.
2. Optionally delegate idea generation or surprising-result analysis to sub-agents.
3. Choose one focused hypothesis and make one bounded change.
4. Run the before hook, benchmark, after hook, and correctness checks within the configured timeout.
5. Log the metric, status, description, optional commit, evidence excerpt, hypothesis, learning, and next focus.
6. Keep an improvement with a reviewable commit or revert a regression to the last kept commit before the next iteration.
7. Update the living prompt with wins, dead ends, and constraints, then repeat until the budget or stop condition is reached.

The default maximum is 30 iterations and the default benchmark timeout is 600,000 ms. Set smaller budgets first; expand only after the metric is stable and the keep/revert behavior is trustworthy.

Know the three experiment tools

Delegate the thinking, not the metric

The sub-agent flags store phase choices in .auto/config.json and .auto/prompt.md. The lead then uses existing delegate_task or delegate_parallel tools during those phases.

Inspect the .auto/ workspace

Control, inspect, and finalize a session

/autoresearch status
/autoresearch off
/autoresearch export
/autoresearch finalize
/autoresearch clear --yes

• status summarizes active state, iterations, metric configuration, and recorded runs.
• off pauses the loop and disables automatic resume without deleting artifacts.
• export writes the self-contained .auto/dashboard.html.
• finalize derives a review plan and branch manifest from kept runs. It does not create or switch branches, rewrite history, or cherry-pick changes.
• clear --yes deletes session state and requires explicit confirmation.

The same subcommands are available through autohand auto-research ... and autohand autoresearch ... for non-interactive use.

Starting again while a paused session has .auto/prompt.md resumes persisted context rather than replacing the original goal. Clear the session explicitly before beginning an unrelated objective.

Add lifecycle and workspace hooks

Autoresearch emits lifecycle events for start, pause, initialization, before/after benchmark execution, run, and log activity. Use them for observability, policy enforcement, or external notifications.

For benchmark-local setup and cleanup, add executable scripts:

.auto/hooks/before.sh
.auto/hooks/after.sh

Both run with AUTO_RESEARCH_WORKSPACE set to the project root and AUTO_RESEARCH_HOOK set to before or after. A failed or timed-out hook is recorded as an experiment failure; do not hide benchmark invalidation inside a successful exit.

Use the same state from RPC and ACP

JSON-RPC clients can start, inspect, and stop the same persisted session through autohand.autoresearch.start, autohand.autoresearch.status, and autohand.autoresearch.stop. The start payload can supply metric name/unit, direction, measure or checks command/script, timeout, file scope, iteration budget, and sub-agent phase configuration.

ACP sessions advertise /autoresearch in command metadata and use the same slash-command path. This keeps the experiment contract consistent across the terminal, editors, and automation clients.

Failure modes and corrections

No metric is parsed

Make the benchmark print exactly METRIC configured_name=<number>. Check spelling, exit code, working directory, and whether output is written to standard output or standard error.

Results oscillate

Increase samples inside the measure script, control caches and background load, record variance, and delegate measurement analysis before accepting small improvements.

Faster runs break behavior

Strengthen .auto/checks.sh. A metric improvement with failed correctness checks must not be treated as a win.

The loop repeats dead ends

Inspect and update .auto/prompt.md with the mechanism, evidence, and why the idea failed. Ask idea-generation agents for hypotheses that are materially different, not syntactic variants.

The working tree becomes hard to review

Reduce experiment scope, require one focused change per iteration, record commit hashes for kept runs, pause the loop, and run /autoresearch finalize before further changes.