Autonomous SRE Skill

---
name: autonomous-sre
description: Monitor infrastructure, respond to incidents, and execute automated remediation. Use for alert triage, root cause analysis, runbook execution, and maintaining service reliability.
allowed-tools: run_command read_file write_file http_request sql_query
---

# Skill: Autonomous SRE

## Purpose

Monitor infrastructure health, respond to alerts and incidents, perform root cause analysis, and execute remediation runbooks. This skill operates within defined safety boundaries and escalates appropriately when human judgment is required.

## When to use this skill

- Responding to **monitoring alerts** from Datadog, PagerDuty, or similar systems.
- Performing **root cause analysis** for service degradations or outages.
- Executing **predefined runbooks** for known incident types.
- Analyzing **logs, metrics, and traces** to diagnose issues.
- Performing **capacity planning** and scaling operations.

## Inputs

- **Alert details**: severity, affected services, metrics, and timestamps.
- **System context**: architecture diagrams, service dependencies, recent deployments.
- **Runbooks**: documented procedures for known incident types.
- **Access credentials**: API keys for monitoring, cloud providers, and internal systems.

## Out of scope

- Actions that could cause **data loss** without explicit approval.
- Changes to **production databases** without proper backup verification.
- Modifications to **security configurations** (firewalls, IAM policies).
- Customer communication or **external incident disclosure**.

## Conventions

- Follow the **incident severity matrix** for escalation decisions.
- Document all actions taken in the **incident timeline**.
- Use **read-only operations** by default; write operations require explicit approval.
- Apply the **principle of least privilege** when accessing systems.
- Prefer **reversible actions** over irreversible ones.

## Required behavior

1. Acknowledge alerts and gather initial context before taking action.
2. Check for **recent changes** (deployments, config updates, infrastructure changes).
3. Analyze logs and metrics to **identify the root cause**.
4. Execute remediation steps from approved runbooks.
5. Verify the fix and **monitor for recurrence**.
6. Document findings and update runbooks if needed.

## Required artifacts

- **Incident timeline**: chronological log of observations and actions.
- **Root cause analysis**: documented explanation of what went wrong.
- **Remediation record**: what was done to resolve the incident.
- **Follow-up items**: tickets for prevention or improvement.

## Implementation checklist

1. Acknowledge the alert and assess severity.
2. Identify affected services and potential blast radius.
3. Check recent deployments and infrastructure changes.
4. Gather logs, metrics, and traces from affected systems.
5. Formulate hypothesis and identify root cause.
6. Select and execute appropriate runbook.
7. Verify resolution and monitor for stability.
8. Document incident and create follow-up items.

## Verification

The incident is resolved when:

- Affected metrics return to **normal baseline**.
- No new alerts are firing for the affected systems.
- User-facing impact has been **mitigated or eliminated**.
- Stakeholders have been notified of resolution.

Post-incident:

- Incident timeline is complete and accurate.
- Root cause is documented.
- Prevention/improvement tickets are created.

## Safety and escalation

- If an incident is **P0/SEV1** or affects critical customer functionality, immediately page the on-call engineer.
- If remediation requires **data modification or deletion**, get explicit approval first.
- If you are **unsure about the root cause**, escalate rather than guess.
- If an action could make the situation **worse**, pause and consult a human.
- Never bypass **change management processes** for non-emergency fixes.

# Start SRE monitoring daemon
autohand --skill autonomous-sre --daemon

# Respond to an alert
autohand --skill autonomous-sre --prompt "High CPU alert on web-server-01, analyze and remediate"

# Investigate an issue
autohand --skill autonomous-sre --prompt "Users reporting slow page loads. Investigate and identify root cause"

# Execute a runbook
autohand --skill autonomous-sre --prompt "Execute the database failover runbook for postgres-primary"

# Post-incident analysis
autohand --skill autonomous-sre --prompt "Generate post-incident report for yesterday's API outage"