Observability Integration

Overview

Modern observability pipelines need low-level, high-fidelity signals for AI agents to reason about performance and reliability. This guide covers integrating Autohand with your telemetry infrastructure.

The observability layer supports three signal types:

Metrics: Time-series data from application and infrastructure
Logs: Structured and unstructured event records
Traces: Distributed request flows across services

OpenTelemetry integration

OpenTelemetry is the standard for collecting distributed traces, metrics, and logs. Autohand connects to your OTel collector to receive correlated signals.

Collector configuration

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 10s
    send_batch_size: 1000

exporters:
  otlp/autohand:
    endpoint: autohand-sre:4317
    tls:
      insecure: false
      cert_file: /etc/certs/client.crt
      key_file: /etc/certs/client.key

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/autohand]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/autohand]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/autohand]

Autohand receiver setup

# .autohand/observability.yaml
opentelemetry:
  receiver:
    grpc_port: 4317
    http_port: 4318

  processing:
    trace_sampling_rate: 0.1
    metric_aggregation_interval: 30s
    log_parsing:
      format: auto
      structured_fields: true

eBPF for kernel-level observability

eBPF (Extended Berkeley Packet Filter) unlocks deep kernel-level observability without modifying application code. When combined with OpenTelemetry, it provides precise, real-time signals from the application layer down to the kernel.

Capabilities

Network latency measurement at the socket level
System call tracing for I/O bottlenecks
CPU scheduling analysis
Memory allocation patterns
Container and pod network flows

Deployment options

# Deploy eBPF agent with Autohand integration
kubectl apply -f https://autohand.ai/manifests/ebpf-agent.yaml

# Verify agent is running
kubectl get pods -n autohand-system -l app=ebpf-agent

# Check signal collection
autohand sre signals --source ebpf --last 5m

Metrics integration

Connect to your existing metrics platforms for time-series data collection.

Prometheus

# Prometheus integration
metrics:
  prometheus:
    endpoints:
      - url: http://prometheus:9090
        name: primary

    queries:
      - name: error_rate
        expr: "sum(rate(http_requests_total{status=~'5..'}[5m])) / sum(rate(http_requests_total[5m]))"
        threshold: 0.01
        severity: warning

      - name: latency_p99
        expr: "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))"
        threshold: 2.0
        severity: critical

Datadog

# Datadog integration
metrics:
  datadog:
    api_key: ${DATADOG_API_KEY}
    app_key: ${DATADOG_APP_KEY}
    site: datadoghq.com

    monitors:
      - name: high_cpu
        query: "avg:system.cpu.user{env:production} by {host}"
        threshold: 85

      - name: memory_pressure
        query: "avg:system.mem.used{env:production} / avg:system.mem.total{env:production} * 100"
        threshold: 90

Log aggregation

Autohand ingests logs from multiple sources and applies NLP parsing for semantic understanding.

Kubernetes logs

# Kubernetes log collection
logs:
  kubernetes:
    namespaces:
      - production
      - staging

    include:
      - "app=*"

    exclude:
      - "app=health-check"

    parsing:
      format: json
      timestamp_field: "@timestamp"
      message_field: "msg"

CloudWatch logs

# CloudWatch integration
logs:
  cloudwatch:
    region: us-east-1
    log_groups:
      - /aws/eks/production/application
      - /aws/lambda/payment-processor

    filter_pattern: "ERROR OR WARN OR Exception"
    start_time: "-1h"

Alert routing

Configure alert sources to trigger incident response workflows.

PagerDuty

# PagerDuty integration
alerting:
  pagerduty:
    api_token: ${PAGERDUTY_TOKEN}
    service_id: P1234ABC

    routing:
      - priority: P1
        auto_acknowledge: true
        escalation_timeout: 5m

      - priority: P2
        auto_acknowledge: true
        escalation_timeout: 15m

      - priority: P3
        auto_acknowledge: false
        escalation_timeout: 30m

OpsGenie

# OpsGenie integration
alerting:
  opsgenie:
    api_key: ${OPSGENIE_API_KEY}
    team: platform-sre

    priorities:
      P1: critical
      P2: high
      P3: moderate

Signal correlation

Autohand correlates signals across sources to build a complete picture of system behavior.

# Correlation rules
correlation:
  enabled: true

  rules:
    - name: trace_to_metric
      match:
        trace.service: "payment-api"
      enrich_with:
        - metric: http_request_duration_seconds
          labels: ["service", "endpoint"]

    - name: log_to_trace
      match:
        log.level: ERROR
      link_by:
        - trace_id
        - span_id

  time_window: 5m
  max_correlation_depth: 3

Correlated signals enable the reasoning engine to identify root causes that span multiple system layers.

Verification

Verify your observability integration is working correctly:

# Check signal ingestion status
autohand sre status --observability

# Output:
# OpenTelemetry:
#   Traces: 1,234/min (healthy)
#   Metrics: 5,678/min (healthy)
#   Logs: 12,345/min (healthy)
#
# Prometheus:
#   Connected: true
#   Queries active: 12
#
# Datadog:
#   Connected: true
#   Monitors synced: 8
#
# PagerDuty:
#   Connected: true
#   Alerts received (24h): 23