qemu-debug-session

Breenix QEMU Debug Session

This skill provides comprehensive QEMU debugging capabilities for Breenix kernel development, including live inspection, detailed logging, and monitor access.

Usage

Invoke this skill when:

• Investigating interrupt handling bugs
• Debugging memory management issues
• Analyzing boot sequence problems
• Tracing hardware interactions
• Inspecting CPU state during failures

Debugging Scenarios

1. Interrupt and Timer Debugging

For interrupt-related issues (context switches, timer behavior, IRQ handling):

BREENIX_QEMU_LOG_PATH=/tmp/breenix-debug.log \
BREENIX_QEMU_DEBUG_FLAGS="int,cpu_reset,guest_errors" \
BREENIX_QEMU_MONITOR=tcp \
cargo run --release --bin qemu-uefi -- -serial stdio -display none

This enables:

• int: Log all interrupts (IRQ delivery, exceptions, traps)
• cpu_reset: Track CPU resets and initialization
• guest_errors: Capture guest OS errors (page faults, invalid ops, etc.)

2. Memory and Page Table Debugging

For memory allocation, paging, or MMU issues:

BREENIX_QEMU_LOG_PATH=/tmp/breenix-memory.log \
BREENIX_QEMU_DEBUG_FLAGS="mmu,guest_errors,page" \
BREENIX_QEMU_MONITOR=tcp \
cargo run --release --bin qemu-uefi -- -serial stdio -display none

This enables:

• mmu: Memory Management Unit operations
• page: Page table walks and TLB operations
• guest_errors: Page faults and access violations

3. Boot Sequence Analysis

For boot hangs, firmware issues, or early initialization problems:

BREENIX_QEMU_DEBUGCON_FILE=/tmp/ovmf-debug.log \
BREENIX_QEMU_LOG_PATH=/tmp/breenix-boot.log \
BREENIX_QEMU_DEBUG_FLAGS="guest_errors,cpu_reset" \
BREENIX_QEMU_MONITOR=tcp \
cargo run --release --bin qemu-uefi -- -serial stdio -display none

This captures:

• OVMF firmware debug output to separate file
• CPU reset events
• Guest errors during boot

4. CPU State Inspection

For register corruption, flag issues, or instruction tracing:

BREENIX_QEMU_LOG_PATH=/tmp/breenix-cpu.log \
BREENIX_QEMU_DEBUG_FLAGS="cpu,in_asm,int,guest_errors" \
BREENIX_QEMU_MONITOR=tcp \
cargo run --release --bin qemu-uefi -- -serial stdio -display none

WARNING: This generates MASSIVE logs (100+ MB/second). Use only for targeted debugging:

• cpu: Dump CPU state after each instruction
• in_asm: Show instruction disassembly
• Only run for short durations!

QEMU Monitor Access

The monitor allows live inspection of the running kernel without stopping execution.

TCP Monitor (Recommended for Development)

BREENIX_QEMU_MONITOR=tcp cargo run --release --bin qemu-uefi

Then in another terminal:

telnet localhost 4444

Stdio Monitor (Interactive)

BREENIX_QEMU_MONITOR=stdio cargo run --release --bin qemu-uefi

WARNING: Stdio monitor mixes with kernel output. Use TCP for cleaner separation.

Monitor Commands

Once connected to the monitor, useful commands:

CPU and Register Inspection

info registers          # Show all CPU registers
info registers -a       # Show all registers including hidden state
info cpus              # List all virtual CPUs
info fpu               # Show FPU registers
info idt               # Show Interrupt Descriptor Table
info gdt               # Show Global Descriptor Table

Memory Inspection

info mem               # Show virtual memory mappings
info tlb               # Show TLB entries
x/10i $rip             # Disassemble 10 instructions at current RIP
x/32xb 0xdeadbeef      # Dump 32 bytes at address in hex
xp/10gx 0xdeadbeef     # Dump 10 8-byte values (physical address)

Interrupt and Timer State

info pic               # Show PIC (legacy interrupt controller) state
info ioapic            # Show I/O APIC state
info lapic             # Show Local APIC state
info qtree             # Show device tree (find timer devices)

Execution Control

stop                   # Pause execution
cont                   # Resume execution
system_reset          # Reset the system
quit                  # Exit QEMU

Available Debug Flags

Set via BREENIX_QEMU_DEBUG_FLAGS (comma-separated):

High-Level Flags

• guest_errors: Guest OS errors (page faults, invalid ops) - Start here
• unimp: Unimplemented device/feature access
• int: Interrupt delivery and exceptions
• cpu_reset: CPU initialization and resets

Memory Flags

• mmu: MMU operations (PT walks, TLB fills)
• page: Page table operations
• pcall: Protected mode call gates

CPU Flags (VERY VERBOSE)

• cpu: Full CPU state after each instruction
• in_asm: Disassembly of executed instructions
• exec: Basic execution trace
• nochain: Disable TB chaining

Device Flags

• ioport: I/O port access (useful for timer/PIC debugging)
• pci: PCI configuration space access

Log Analysis Tips

Finding Interrupt Issues

grep -A5 "exception\|interrupt\|IRQ" /tmp/breenix-debug.log

Look for:

• Unexpected exceptions (e.g., GPF, Page Fault during interrupt handling)
• Missing or duplicate interrupts
• Interrupt delivery to wrong CPU

Detecting Timer Problems

grep -E "APIC|timer|IRQ 0|IRQ 32" /tmp/breenix-debug.log

Look for:

• Timer interrupts not firing at expected rate
• Spurious interrupts
• APIC timer configuration changes

Memory Issues

grep -E "page fault|#PF|CR3|MMU" /tmp/breenix-memory.log

Look for:

• Page faults in kernel space (usually bugs)
• CR3 changes (context switches)
• Invalid page table entries

Boot Hangs

tail -f /tmp/ovmf-debug.log  # Watch firmware output
grep "cpu_reset\|triple fault" /tmp/breenix-boot.log

Look for:

• Triple faults (usually bad IDT or bad interrupt handler)
• Hangs after specific firmware phase
• Repeated resets

Example Debugging Workflow

Scenario: Timer interrupt not firing

1. Start with basic interrupt logging:

BREENIX_QEMU_LOG_PATH=/tmp/debug.log \
BREENIX_QEMU_DEBUG_FLAGS="int,guest_errors" \
BREENIX_QEMU_MONITOR=tcp \
cargo run --release --bin qemu-uefi -- -serial stdio

2. In another terminal, connect to monitor:

telnet localhost 4444

3. Check APIC state:

info lapic

Look for:

• Timer mode (one-shot vs periodic)
• Initial count vs current count
• Whether timer interrupt is masked

4. Check interrupt delivery:

info pic
info ioapic

Verify IRQ 0 (PIT) or IRQ 32 (APIC timer) is not masked.

5. Grep log for interrupt activity:

grep "IRQ.*timer\|exception 32" /tmp/debug.log | less

6. If no interrupts appear, check IDT: In monitor:

info idt

Verify entry 32 (or appropriate vector) has valid handler address.

Environment Variables Reference

Common Pitfalls

1. Stdio monitor + serial output = chaos: Always use BREENIX_QEMU_MONITOR=tcp
2. Forgetting -serial stdio: Monitor doesn't show kernel output; you need both
3. Too much logging: Start with guest_errors,int, not cpu,in_asm
4. Log file grows huge: Check log size; cpu flag can fill GB in seconds
5. Monitor commands fail: Ensure QEMU is still running (kernel panic exits QEMU)

Quick Reference Card

# Start debugging session (interrupt focus)
BREENIX_QEMU_LOG_PATH=/tmp/debug.log \
BREENIX_QEMU_DEBUG_FLAGS="int,guest_errors" \
BREENIX_QEMU_MONITOR=tcp \
cargo run --release --bin qemu-uefi -- -serial stdio

# Connect to monitor
telnet localhost 4444

# Essential monitor commands
info registers         # CPU state
info lapic            # Timer and interrupts
x/10i $rip            # Disassemble at current position
info mem              # Virtual memory map

# Analyze logs
grep -A5 "exception\|IRQ" /tmp/debug.log

Notes

• Debug logging adds overhead; execution will be slower
• TCP monitor allows inspection without pausing execution
• Most bugs are visible with just guest_errors,int logging
• Save logs before QEMU exits; they're not persistent across runs
• GDB stub (BREENIX_GDB=1) is complementary; use for source-level debugging