telemetry

Telemetry Skill

Access the CTO observability stack for logs, metrics, and dashboards.

When to Use

• Querying pod logs via Loki
• Checking metrics via Prometheus
• Viewing dashboards in Grafana
• Debugging agent failures
• Monitoring Play workflow health

Stack Overview

Port Forwards (Required for Local Access)

kubectl port-forward svc/prometheus-server -n observability 9090:80
kubectl port-forward svc/loki-gateway -n observability 3100:80
kubectl port-forward svc/grafana -n observability 3000:80

MCP Tools Available

Prometheus Tools

Loki Tools

Grafana Tools

Loki (Logs)

LogQL Basics

# All logs from CTO namespace
{namespace="cto"}

# Filter by pod name
{namespace="cto", pod=~"coderun-.*"}

# Search for errors
{namespace="cto"} |= "error"

# JSON parsing
{namespace="cto"} | json | level="error"

# Regex filter
{namespace="cto"} |~ "tool.*mismatch"

Common Queries for CTO

# All CodeRun pod logs
{namespace="cto", app="coderun"}

# Morgan intake logs
{namespace="cto", pod=~"intake-.*"}

# Play workflow logs
{namespace="cto", pod=~"play-.*"}

# Errors only
{namespace="cto"} |= "error" | json

# Tool inventory issues (A10)
{namespace="cto"} |~ "tool.*(mismatch|missing)"

# MCP initialization failures (A12)
{namespace="cto"} |~ "mcp.*failed"

# Config issues (A11)
{namespace="cto"} |~ "cto-config.*(missing|invalid)"

Via MCP Tool

loki_query(query='{namespace="cto"} |= "error"', limit=100)

Via curl

curl -G "http://localhost:3100/loki/api/v1/query_range" \
  --data-urlencode 'query={namespace="cto"} |= "error"' \
  --data-urlencode 'limit=100' | jq

Prometheus (Metrics)

PromQL Basics

# Current CPU usage
container_cpu_usage_seconds_total{namespace="cto"}

# Memory usage
container_memory_usage_bytes{namespace="cto"}

# Rate of requests
rate(http_requests_total{namespace="cto"}[5m])

# Pod restarts
kube_pod_container_status_restarts_total{namespace="cto"}

Common Queries for CTO

# CodeRun pod count
count(kube_pod_info{namespace="cto", pod=~"coderun-.*"})

# Memory by pod
container_memory_usage_bytes{namespace="cto", container!=""}

# CPU by pod
rate(container_cpu_usage_seconds_total{namespace="cto"}[5m])

# OOM killed containers
kube_pod_container_status_last_terminated_reason{namespace="cto", reason="OOMKilled"}

# Pod restart count
sum(kube_pod_container_status_restarts_total{namespace="cto"}) by (pod)

Via MCP Tool

prometheus_query(query='count(kube_pod_info{namespace="cto"})')

Via curl

curl "http://localhost:9090/api/v1/query" \
  --data-urlencode 'query=kube_pod_info{namespace="cto"}' | jq

Grafana (Dashboards)

Access

• URL: http://localhost:3000
• Default credentials: admin/admin (or configured)

Common Dashboards

Via MCP Tool

grafana_search_dashboards(query="kubernetes")
grafana_get_dashboard(uid="abc123")

kubectl Alternatives

When MCP tools aren't available, use kubectl directly:

Stream Logs

# All CTO pods
kubectl logs -n cto -l app.kubernetes.io/part-of=cto -f --tail=100

# Specific CodeRun
kubectl logs -n cto -l app=coderun -f

# With grep
kubectl logs -n cto -l app=coderun -f | grep -E "error|mismatch|failed"

Get Pod Status

kubectl get pods -n cto -o wide
kubectl describe pod -n cto <pod-name>

Events

kubectl get events -n cto --sort-by='.lastTimestamp'

Healer Integration

Healer uses Loki to watch for patterns:

// From crates/healer/src/scanner.rs
// Patterns that trigger alerts:
"tool\\s+inventory\\s+mismatch"  // A10
"cto-config.*(missing|invalid)" // A11
"mcp.*failed\\s+to\\s+initialize" // A12

Query Healer-Relevant Logs

# All Healer detection patterns
{namespace="cto"} |~ "tool.*mismatch|cto-config.*(missing|invalid)|mcp.*failed"

Troubleshooting

No Logs Appearing

1. Check port forward is running: lsof -i :3100
2. Verify Loki pods: kubectl get pods -n observability -l app=loki
3. Check label: loki_labels() to see available labels

Metrics Not Found

1. Check port forward: lsof -i :9090
2. Verify Prometheus: kubectl get pods -n observability -l app=prometheus
3. List metrics: prometheus_labels() or browse http://localhost:9090/targets

Grafana Not Loading

1. Check port forward: lsof -i :3000
2. Verify pod: kubectl get pods -n observability -l app.kubernetes.io/name=grafana

Reference

• Loki LogQL Documentation
• Prometheus PromQL Documentation
• Grafana Documentation