escalation-and-ownership

Escalation and Ownership

Overview

Escalation and Ownership defines who is responsible for incident response at each stage and when to escalate to more senior or specialized personnel. Clear ownership prevents incidents from falling through the cracks and ensures efficient resolution.

Core Principle: "Every incident must have an owner. Escalation is not failure—it's smart resource allocation."

1. The Ownership Model

Primary On-Call (L1)

• Responsibility: First responder, initial triage
• Authority: Can page L2, declare SEV2+
• Typical Role: Mid-level engineer familiar with the system

Secondary On-Call (L2)

• Responsibility: Subject matter expert, complex debugging
• Authority: Can page L3, escalate to SEV1
• Typical Role: Senior engineer, team lead

Escalation Engineer (L3)

• Responsibility: Cross-team coordination, architectural decisions
• Authority: Can page executives, declare SEV0
• Typical Role: Staff/Principal engineer, architect

Executive On-Call (L4)

• Responsibility: Business decisions, customer communication
• Authority: Final authority on trade-offs
• Typical Role: VP Engineering, CTO

2. When to Escalate

Time-Based Escalation

SEV0: Escalate immediately to L2 + L3
SEV1: Escalate to L2 after 15 minutes if no progress
SEV2: Escalate to L2 after 30 minutes if no progress
SEV3: Escalate to L2 after 2 hours if no progress

Complexity-Based Escalation

Escalate immediately if:

• Root cause involves multiple services
• Requires database schema changes
• Needs vendor support (AWS, GCP)
• Involves security breach or data loss
• Affects enterprise/VIP customers

Skill-Based Escalation

Issue Type → Escalate To
─────────────────────────────
Database performance → DBA team
Network/DNS → Infrastructure team
Payment failures → Payments team
Security alert → Security team

3. The RACI Matrix for Incidents

Responsible, Accountable, Consulted, Informed

4. Escalation Procedures

Step 1: Assess Severity

Use the severity matrix (see severity-levels skill) to determine if escalation is needed.

Step 2: Notify Next Level

# PagerDuty CLI example
pd incident escalate INC-123 --level 2 --reason "Database deadlock requires DBA expertise"

Step 3: Provide Context

When escalating, always include:

• Incident ID and severity
• What you've tried so far
• Current hypothesis
• Why you're escalating (time/complexity/skill)

Step 4: Transfer Ownership

**Escalation Handoff**

From: @alice (L1)
To: @bob (L2 - Database SME)

**Summary**: Payment API returning 500s
**Root Cause Hypothesis**: Connection pool exhaustion
**Attempted**: Restarted app servers (no effect)
**Current State**: 50% of payment requests failing
**Logs**: [link]
**Metrics**: [dashboard link]

@bob you have ownership. Let me know how I can assist.

5. The Incident Commander Role

For SEV0/1 incidents, assign a dedicated Incident Commander (IC).

IC Responsibilities

• Coordinate: Manage the war room, assign tasks
• Communicate: Update stakeholders, run status calls
• Decide: Make trade-off decisions (rollback vs. forward fix)
• Document: Ensure timeline is captured
• Protect: Shield responders from distractions

IC is NOT

• The person fixing the issue (that's the technical lead)
• The most senior person (it's a role, not a rank)
• Responsible for the root cause (they coordinate, not code)

6. On-Call Rotation Best Practices

Rotation Schedule

Week 1: Alice (Primary), Bob (Secondary)
Week 2: Bob (Primary), Carol (Secondary)
Week 3: Carol (Primary), Alice (Secondary)

Handoff Checklist

• Review open incidents from previous week
• Check for scheduled maintenance windows
• Verify PagerDuty/Opsgenie is configured correctly
• Review recent postmortems for new failure modes
• Test pager (send test alert)

On-Call Compensation

• Follow the Sun: Distribute on-call across time zones
• Compensation: Pay on-call stipend + overtime for pages
• Recovery Time: Day off after a SEV0 overnight incident

7. Escalation Antipatterns

❌ The "Hero" Pattern

One person tries to solve everything alone instead of escalating. Fix: Make escalation a positive action, not a sign of weakness.

❌ The "Hot Potato"

Passing the incident around without clear ownership transfer. Fix: Use explicit handoff messages with acknowledgment.

❌ The "Escalation Spam"

Paging everyone immediately without attempting L1 triage. Fix: Follow the escalation ladder; respect the on-call rotation.

❌ The "Silent Escalation"

Escalating without context or documentation. Fix: Always provide a handoff summary.

8. Vendor Escalation

AWS Support

Severity 1 (Critical): < 1 hour response
Severity 2 (Urgent): < 4 hours response
Severity 3 (Normal): < 12 hours response

Escalation: Open case via AWS Console or CLI
aws support create-case --subject "RDS outage" --severity urgent

GCP Support

P1 (Critical): 15-minute response (Premium Support)
P2 (High): 4-hour response
P3 (Medium): 8-hour response

Escalation: Cloud Console → Support → Create Case

Third-Party SaaS (Stripe, Twilio, etc.)

• Check status page first: status.stripe.com
• Use dedicated support channel for enterprise customers
• Escalate via account manager for critical issues

9. Ownership Documentation

Service Ownership Registry

# services.yaml
payment-api:
  owner: payments-team
  primary_oncall: "@payments-oncall"
  escalation_path:
    - L1: "@payments-oncall"
    - L2: "@payments-lead"
    - L3: "@platform-architect"
  runbook: "https://wiki/payments-api-runbook"
  slack_channel: "#team-payments"

CODEOWNERS (GitHub)

# Automatically request reviews and assign ownership
/services/payment-api/ @payments-team
/services/auth/ @security-team
/infrastructure/ @platform-team

10. Escalation and Ownership Checklist

• Clear Ladder: Is the L1 → L2 → L3 escalation path documented?
• Response SLAs: Do we have defined response times for each level?
• Handoff Protocol: Do we use a standard template for ownership transfer?
• IC Training: Have we trained multiple people to be Incident Commanders?
• Vendor Contacts: Do we have escalation contacts for all critical vendors?
• Service Registry: Is every service mapped to an owning team?
• Rotation Health: Is on-call load balanced (no one person > 50% of pages)?

Related Skills

• 41-incident-management/escalation-paths
• 41-incident-management/severity-levels
• 41-incident-management/incident-triage