dns-debug

DNS Debugging

Diagnose DNS resolution failures using Blocky query logs stored in PostgreSQL. Primary scenario: a user reports a website is broken and you need to determine whether Blocky is blocking a required domain.

Tool

./scripts/blocky.py queries the log_entries table in the Blocky CNPG PostgreSQL cluster via kubectl exec.

Run ./scripts/blocky.py --help for full usage.

Quick Reference

# Search: who queried a domain recently?
./scripts/blocky.py search homedepot -f 24h

# Logs: all queries from a specific client (by IP, name, or VLAN)
./scripts/blocky.py logs -c 192.168.3.40 -f 1h
./scripts/blocky.py logs -c pixel -f 1h

# Blocked: only blocked queries for a client
./scripts/blocky.py blocked -c 192.168.3.40 -f 1h

# Combine filters
./scripts/blocky.py logs -c 192.168.3.40 -d homedepot -f 2h

# VLAN shorthand (lan, iot, kids, guest, work, cameras)
./scripts/blocky.py blocked -c kids -f 24h

# Machine-readable
./scripts/blocky.py -j blocked -c 192.168.3.40

The -c/--client flag accepts: partial IP, partial device name (from reverse DNS), CIDR notation, or VLAN name. All matching is case-insensitive.

Diagnostic Workflow

When a user reports "website X is broken" or "app Y is not working":

1. Identify the device. If the user names a specific device, use -c with the device name to target it directly: ./scripts/blocky.py blocked -c pixel -f 1h. NEVER assume a VLAN based on the app or use case; always confirm or search. If the device is unknown, search for the domain to find it: ./scripts/blocky.py search <domain> -f 24h and pick the client with the most recent last_seen timestamp.

2. Get blocked queries for that client to find the offending domain: ./scripts/blocky.py blocked -c <ip-or-name> -f 1h The blocked domain is often not the main site but a subdomain (API, CDN, auth service).

3. Verify the block by checking the reason field. It indicates which blocklist group matched (e.g., ads, threats, social).

4. Determine the fix (see Remediation below).

5. After pushing the fix, Flux applies the change and Blocky reloads automatically (reloader annotation). Verify resolution: ./scripts/blocky.py logs -c <ip-or-name> -d <domain> -f 5m The domain should now show RESOLVED or CACHED instead of BLOCKED.

Remediation

Option A: Add an allowlist entry (single domain fix)

Add an allowlists section to the blocking config in kubernetes/apps/dns-private/blocky/data/config.yaml. The allowlist group name must match the denylist group that blocked the domain.

blocking:
  denylists:
    ads:
    - https://...
  allowlists:
    ads:
    - |
      alloweddomain.com
      api.someservice.com

Allowlists take precedence over denylists within the same group. A domain present in both the deny and allow list for a group will be allowed.

Option B: Switch to a less aggressive blocklist

If false positives are frequent for a list category, consider switching to a less aggressive variant. Read the current config to identify which list tier is in use, then check upstream for available tiers.

Option C: Remove a blocklist group from a client group

If an entire category is causing problems for a VLAN, remove it from clientGroupsBlock for that VLAN.

Configuration Reference

Read kubernetes/apps/dns-private/blocky/data/config.yaml for current:

• VLAN-to-subnet mappings and which block groups apply to each
• Denylist URLs and their group names
• Any existing allowlist entries

Unimplemented Subcommands

The following subcommands were deferred. If you need one during diagnosis, implement it in ./scripts/blocky.py following the patterns of the existing subcommands, then update this skill file to move it from this list to the Quick Reference section above.

• top-domains: Top queried domains by count. Flags: -f, -c, -l. GROUP BY question_name, ORDER BY count DESC.
• top-blocked: Top blocked domains by count. Same as top-domains but filtered to response_type = 'BLOCKED'. Include the reason column to show which blocklist group matched.
• top-clients: Top clients by query volume. Flags: -f, -l. GROUP BY client_ip, client_name, ORDER BY count DESC.
• slow: Queries exceeding a duration threshold. Flags: -f, -c, --threshold (milliseconds, default 500). Filter duration_ms >= threshold, ORDER BY duration_ms DESC.

Pattern Detection

When adding allowlist entries, check git history for systemic issues:

git log --oneline --invert-grep --author="renovate" \
  -- kubernetes/apps/dns-private/blocky/data/config.yaml

If you see 3+ allowlist additions in a short period, the denylist tier may be too aggressive. Propose downgrading the list tier rather than maintaining a growing allowlist.