data-processing

Data Processing

Query, filter, and transform structured data (JSON, YAML, TOML) efficiently from the command line.

Tools

jq Essentials

# Extract single field
jq '.name' package.json

# Extract nested field
jq '.scripts.build' package.json

# Extract from array
jq '.dependencies[0]' package.json

# Extract multiple fields
jq '{name, version}' package.json

# Navigate deeply nested
jq '.data.users[0].profile.email' response.json

# Filter by condition
jq '.users[] | select(.active == true)' data.json

# Transform each element
jq '.users | map({id, name})' data.json

# Count elements
jq '.users | length' data.json

# Raw string output
jq -r '.name' package.json

yq Essentials

# Extract field
yq '.name' config.yaml

# Extract nested
yq '.services.web.image' docker-compose.yml

# List all keys
yq 'keys' config.yaml

# List all service names (Docker Compose)
yq '.services | keys' docker-compose.yml

# Get container images (K8s)
yq '.spec.template.spec.containers[].image' deployment.yaml

# Update value (in-place)
yq -i '.version = "2.0.0"' config.yaml

# TOML to JSON
yq -p toml -o json '.' config.toml

Quick Reference

When to Use

• Reading package.json dependencies
• Parsing Docker Compose configurations
• Analyzing Kubernetes manifests
• Processing GitHub Actions workflows
• Extracting data from API responses
• Filtering large JSON datasets
• Config file manipulation
• Data format conversion

Additional Resources

For complete pattern libraries, load:

• ./references/jq-patterns.md - Arrays, filtering, transformation, aggregation, output formatting
• ./references/yq-patterns.md - Docker Compose, K8s, GitHub Actions, TOML, YAML modification
• ./references/config-files.md - package.json, tsconfig, eslint/prettier patterns