scenario-testing

Scenario-Driven Testing for AI Code Generation

Core Principle

The Iron Law: "NO FEATURE IS VALIDATED UNTIL A SCENARIO PASSES WITH REAL DEPENDENCIES"

Mocks create false confidence. Only scenarios exercising real systems validate that code works.

The Truth Hierarchy

1. Scenario tests (real system, real data) = truth
2. Unit tests (isolated) = human comfort only
3. Mocks = lies hiding bugs

As stated in the principle: "A test that uses mocks is not testing your system. It's testing your assumptions about how dependencies behave."

When to Use This Skill

• Validating new functionality
• Before declaring work complete
• When tempted to use mocks
• After fixing bugs requiring verification
• Any time you need to prove code works

Required Practices

1. Write Scenarios in .scratch/

• Use any language appropriate to the task
• Exercise the real system end-to-end
• Zero mocks allowed
• Must be in .gitignore (never commit)

2. Promote Patterns to scenarios.jsonl

• Extract recurring scenarios as documented specifications
• One JSON line per scenario
• Include: name, description, given/when/then, validates
• This file IS committed

3. Use Real Dependencies

External APIs must hit actual services (sandbox/test mode acceptable). Mocking any dependency invalidates the scenario.

4. Independence Requirement

Each scenario must run standalone without depending on prior executions. This enables:

• Parallel execution
• Prevents hidden ordering dependencies
• Reliable CI/CD integration

What Makes a Scenario Invalid

A scenario is invalid if it:

• Contains any mocks whatsoever
• Uses fake data instead of real storage
• Depends on another scenario running first
• Never actually executed to verify it passes

Common Violations to Avoid

Reject these rationalizations:

• "Just a quick unit test..." - Unit tests don't validate features
• "Too simple for end-to-end..." - Integration breaks simple things
• "I'll mock for speed..." - Speed doesn't matter if tests lie
• "I don't have API credentials..." - Ask your human partner for real ones

Definition of Done

A feature is complete only when:

1. ✅ A scenario in .scratch/ passes with zero mocks
2. ✅ Real dependencies are exercised
3. ✅ .scratch/ remains in .gitignore
4. ✅ Robust patterns extracted to scenarios.jsonl

Example Workflow

1. Write scenario - Create .scratch/test-user-registration.py
2. Use real dependencies - Hit real database, real auth service (test mode)
3. Run and verify - Execute scenario, confirm it passes
4. Extract pattern - Document in scenarios.jsonl
5. Keep .scratch ignored - Never commit scratch scenarios

Why This Matters

• Unit tests verify isolated logic
• Integration tests verify components work together
• Scenario tests verify the system actually works

Only scenario tests prove your feature delivers value to users.