dev-testing

dev.testing

Choose the right test type for agentic code. Unit tests for deterministic logic, VCR for LLM calls, evals for behavioral quality.

When to Use

Use this skill when:

• Writing tests for code that interacts with LLMs
• Unsure whether to use unit tests or evals
• Implementing EP08+ features with non-deterministic behavior
• A task mentions "test" and involves agent behavior

Invocation

/dev.testing [context]

CRITICAL: Test Execution Rules

NEVER run bun test directly. Use npm scripts:

A global preload in bunfig.toml blocks e2e/evals unless RUN_LIVE_TESTS=1 is set.

Decision Framework

Ask: What am I testing?

Quick Decision Tree

Is the output deterministic (same input → same output)?
├─ YES → Unit test
└─ NO → Does it call an LLM API?
         ├─ YES → VCR integration test (recorded responses)
         └─ NO → Is it about reasoning/decision quality?
                  ├─ YES → TruLens eval
                  └─ NO → Unit test with mocks

Is this about component integration (reachable from entry point)?
├─ YES → Wiring test (tests/integration/wiring/)
└─ NO → Use above decision tree

Red Flags: Use Evals NOT Unit Tests

Stop and use evals if you're testing:

• What an LLM responds (non-deterministic)
• If an agent chose the right tool (behavioral)
• Recommendation quality (subjective judgment)
• Reasoning accuracy (semantic evaluation)
• Subagent selection decisions

Example anti-pattern:

// BAD: Testing LLM content with assertions
expect(response.recommendations[0].action).toBe("Add error handling");

Why? LLM responses vary. Use TruLens to evaluate if recommendations are actionable, not if they match exact strings.

Test Types

1. Unit Tests (tests/unit/)

Use for: Deterministic, pure functions

// tests/unit/parsers/claude-md.test.ts
describe('parseClaudeMd', () => {
  it('extracts build commands', () => {
    const result = parseClaudeMd('## Build\n```bash\nbun build\n```');
    expect(result.buildCommands).toEqual(['bun build']);
  });
});

Run: bun test tests/unit/

2. VCR Integration Tests (tests/integration/)

Use for: Code that calls LLM APIs

Records real API responses for deterministic replay. See vcr-patterns.md.

// tests/integration/agent/analysis.test.ts
describe('Agent Analysis Flow', () => {
  beforeAll(async () => {
    await vcr.load('tests/integration/recordings/analysis.json');
  });

  it('completes analysis phases', async () => {
    const result = await orchestrator.run(task);
    expect(result.phases).toContain('config-analysis');
  });
});

Run: bun test tests/integration/ Record: bun run record (when prompts change)

3. TruLens Evals (tests/evals/)

Use for: Behavioral quality assessment

Runs on release tags only (expensive). See eval-patterns.md.

// tests/evals/behavioral/actionability.eval.ts
import { runAnalysis, evaluateActionability } from '../lib/eval-helpers';
import { goldenScenario } from '../fixtures/golden-scenarios';

test('recommendation actionability meets threshold', async () => {
  const result = await runAnalysis(goldenScenario);
  const score = await evaluateActionability(result.recommendations);
  expect(score).toBeGreaterThanOrEqual(0.7);
});

Run: bun run tests/evals/run-evals.ts (release only)

4. Wiring Tests (tests/integration/wiring/)

Use for: Verifying components are actually integrated into the system

These tests verify the "last mile" - that built components are reachable from user entry points. A component that passes unit tests but is never wired to the system is dead code.

// tests/integration/wiring/cli-components.test.ts
import { glob } from 'glob';
import { getExports, hasImportPath } from './wiring-utils';

describe('CLI Component Wiring', () => {
  it('all exported components have import paths to entry points', () => {
    const componentFiles = glob.sync('src/cli/components/*.tsx');
    for (const file of componentFiles) {
      const exports = getExports(file);
      for (const exp of exports) {
        expect(hasImportPath(exp, 'src/cli.ts')).toBe(true);
      }
    }
  });
});

When to write wiring tests:

• After building ANY user-facing component
• After creating tools that should be registered
• As part of Phase 6 (Integration) in every epic

Run: bun test tests/integration/wiring/

Test Suite Structure

Per ADR-0011:

EP08 Specific Guidance

For ACT Subagents (EP08):

References

• VCR patterns: vcr-patterns.md
• Eval patterns: eval-patterns.md
• ADR-0011: docs/architecture/adr/0011-testing-strategy-for-agentic-components.md
• ADR-0012: docs/architecture/adr/0012-evaluation-framework-for-analysis-quality.md

Constitution Alignment

• III. Causal-First: Evals trace recommendations to evidence
• IV. Mixed-Methods: Unit (quantitative) + evals (qualitative)
• VIII. Compounding Value: Eval baselines enable regression detection
• IX. Agent-Aware: Test types match agent cognitive patterns