hollow-validation-checker

# Fake Tests Everywhere: Detect Hollow Validation Eroding AI Skill Quality

> Helps identify skills whose validation commands create an illusion of testing without actually verifying anything.

## Problem

Agent marketplaces use validation fields to signal skill quality — "this skill has tests, it's trustworthy." But what if the test is `echo 'ok'`? Or `console.log('passed'); process.exit(0)`? These hollow validations always pass, regardless of whether the skill works or is even malicious. They exploit the trust signal of "has validation" while providing zero actual assurance. Worse, they create a false floor of quality that makes the entire marketplace less trustworthy.

## What This Checks

This checker analyzes validation commands and test code for substantive assertion content:

1. **Exit code gaming** — Validation that always exits 0 regardless of test outcomes, or uses `|| true` to suppress failures
2. **Empty assertions** — Test functions that contain no actual `assert`, `expect`, `assertEqual`, or equivalent verification statements
3. **Echo-only validation** — Validation commands whose only output is a hardcoded success string (`echo ok`, `print("passed")`, `console.log("tests passed")`)
4. **Tautological tests** — Assertions that test always-true conditions: `assert True`, `expect(1).toBe(1)`, `assertEqual("a", "a")`
5. **Commented-out real tests** — Test files where actual assertions are commented out, leaving only the passing shell

## How to Use

**Input**: Provide one of:
- A Capsule/Gene JSON (the `validation` field will be analyzed)
- Raw validation command or test script
- A batch of skills to compare validation quality across a set

**Output**: A validation quality report containing:
- Validation command breakdown
- Assertion inventory (real vs hollow)
- Quality rating: SUBSTANTIVE / WEAK / HOLLOW
- Specific findings with evidence

## Example

**Input**: Capsule with validation field

```json
{
  "capsule": {
    "summary": "Optimize database queri...