validate-domain-hints

Domain Hints Quality Validation Command

Working Context

This skill operates exclusively within lib (SpecVital Core parser). All commands, file lookups, and report outputs are scoped to lib/.

• Source code: lib/parser/, lib/crypto/, lib/source/
• Integration tests: lib/parser/tests/integration/
• repos.yaml: lib/parser/tests/integration/repos.yaml
• ADR documents: lib/docs/en/adr/core/
• justfile: lib/justfile (use just commands from this directory)
• Report output: lib/domain-hints-quality-report.md

Always cd /workspaces/specvital/lib before running just or go test commands.

Purpose

Validate that DomainHints extraction produces clean, domain-relevant data by detecting:

• Noise Patterns: Empty strings, parser artifacts ([.), meaningless tokens (fn, Ok)
• Statistical Anomalies: Unusual NULL ratios, abnormal average counts
• Regressions: Unexpected changes in extraction results

This is a data quality assurance tool for the DomainHints extraction engine.

User Input

$ARGUMENTS

Interpretation:

Known Noise Patterns

These patterns indicate extraction bugs or low-quality data:

Critical (Must Filter)

Warning (Should Filter)

Regex Patterns

// Parser artifacts
"^[\\[\\]\\(\\)\\{\\}]+$"  // Bracket-only tokens

// Language keywords mistakenly captured
"^(fn|if|for|let|var|const)$"

Statistical Baselines

Expected values by framework (from production data):

Anomaly Thresholds:

• NULL ratio increase > 5%: Error
• Avg imports/calls drop > 20%: Warning

Workflow

Phase 1: Setup

1.1 Determine validation scope:

# Check what repos are available
cat lib/parser/tests/integration/repos.yaml | grep "name:"

1.2 Select target(s):

• Empty input -> all repos in repos.yaml
• Specific repo -> filter to that repo
• External URL -> clone to /tmp

Phase 2: Run Parser with DomainHints

cd /workspaces/specvital/lib

# For integration test repos (already cached)
just scan /path/to/cached/repo --json 2>/dev/null | jq '.'

# Or run integration test to get data
go test -tags integration ./tests/integration/... -v -run "TestScan/{repo-name}" 2>&1

Phase 3: Extract DomainHints Data

3.1 Collect all DomainHints from scan result:

Parse the JSON output to extract:

• All Imports arrays
• All Calls arrays
• Count of NULL vs non-NULL DomainHints

3.2 Calculate statistics:

Total files: N
Files with hints: M
NULL ratio: (N-M)/N * 100%
Avg imports per file: sum(imports) / M
Avg calls per file: sum(calls) / M

Phase 4: Noise Pattern Detection

4.1 Check all Imports for noise patterns:

# Pseudo-code
for import in all_imports:
    if import == "":
        report_error("Empty import found")
    if len(import) <= 2:
        report_warning(f"Short import: {import}")

4.2 Check all Calls for noise patterns:

# Pseudo-code
for call in all_calls:
    if call == "":
        report_error("Empty call found")
    if call in ["[.", ".", "fn", "Ok", "Err", "Some", "None"]:
        report_warning(f"Noise pattern: {call}")
    if matches_regex(call, "^[\\[\\]\\(\\)\\{\\}]+$"):
        report_error(f"Parser artifact: {call}")

Phase 5: Statistical Analysis

5.1 Compare against baselines:

Framework: {framework}
Expected NULL ratio: {baseline}%
Actual NULL ratio: {actual}%
Delta: {delta}%
Status: {PASS|WARN|FAIL}

5.2 Detect anomalies:

• NULL ratio significantly higher than baseline
• Average imports/calls significantly lower
• Unusual distribution patterns

Phase 6: Generate Report

Report Location: /workspaces/specvital/lib/domain-hints-quality-report.md

Language: Korean - MANDATORY

Report Template

# DomainHints 품질 검증 보고서

**일시**: {timestamp}
**대상**: {repo-name or "전체 통합 테스트 저장소"}
**프레임워크**: {framework(s)}

---

## 요약

| 항목             | 결과                 |
| ---------------- | -------------------- |
| 검사 파일 수     | {n}                  |
| DomainHints 존재 | {n} ({percentage}%)  |
| 노이즈 패턴 발견 | {n}건                |
| 통계 이상치      | {n}건                |
| **최종 상태**    | {PASS / WARN / FAIL} |

---

## 노이즈 패턴 검사

### 발견된 노이즈

| 패턴        | 타입   | 발견 횟수 | 샘플 파일     |
| ----------- | ------ | --------- | ------------- |
| `{pattern}` | {type} | {count}   | `{file_path}` |

### 패턴별 상세

#### `{pattern}` ({count}건)

**출처 분석**:

- {repo/framework}: {count}건

**원인 추정**:
{description of likely cause}

---

## 통계 분석

### 프레임워크별 현황

| 프레임워크  | 파일 수 | NULL 비율 | 기준선      | 상태     |
| ----------- | ------- | --------- | ----------- | -------- |
| {framework} | {n}     | {actual}% | {baseline}% | {status} |

### Imports 분포

| 지표            | 값  |
| --------------- | --- |
| 총 고유 imports | {n} |
| 파일당 평균     | {n} |
| 최대            | {n} |

**Top 10 Imports**:

| Import     | 출현 횟수 |
| ---------- | --------- |
| `{import}` | {count}   |

### Calls 분포

| 지표          | 값  |
| ------------- | --- |
| 총 고유 calls | {n} |
| 파일당 평균   | {n} |
| 최대          | {n} |

**Top 10 Calls**:

| Call     | 출현 횟수 |
| -------- | --------- |
| `{call}` | {count}   |

---

## 발견된 문제

### Critical (즉시 수정 필요)

{IF critical issues exist}

1. **{issue}**: {description}
   - 영향 범위: {scope}
   - 권장 조치: {action}
     {ELSE}
     없음
     {ENDIF}

### Warning (검토 필요)

{IF warnings exist}

1. **{issue}**: {description}
   - 영향 범위: {scope}
   - 권장 조치: {action}
     {ELSE}
     없음
     {ENDIF}

---

## 결론

{Based on findings}

---

## 권장 조치

- [ ] {action item 1}
- [ ] {action item 2}

Validation Criteria

PASS

• No critical noise patterns found
• NULL ratio within baseline +/- 5%
• No statistical anomalies

WARN

• Warning-level noise patterns found (fn, Ok, etc.)
• NULL ratio slightly above baseline (5-10%)
• Minor statistical deviations

FAIL

• Critical noise patterns found (empty string, [., etc.)
• NULL ratio significantly above baseline (>10%)
• Parser artifacts detected

Key Rules

Must Do

• Write report in Korean <- CRITICAL
• Check ALL Imports and Calls for noise patterns
• Compare statistics against framework baselines
• Provide actionable recommendations
• Include sample file paths for each issue

Must Not Do

• Write report in English <- Use Korean only
• Ignore any noise pattern (even 1 = potential bug)
• Skip statistical analysis
• Miss framework-specific baselines (e.g., Cypress 21% NULL is normal)

Principles

1. Quality First: Even minor noise degrades AI domain classification
2. Quantitative: Measure exact counts and percentages
3. Comparative: Always compare against baselines
4. Actionable: Every issue needs a clear fix recommendation

Quick Commands

# Full validation on all repos
/validate-domain-hints

# Single repo
/validate-domain-hints grafana

# External repo
/validate-domain-hints https://github.com/vercel/next.js

# Focus on noise only
/validate-domain-hints noise

# Compare Cypress stats
/validate-domain-hints compare cypress

Execution

Now execute the DomainHints quality validation according to the guidelines above.