scraping

Scraping

Web scraping techniques using nu-shell (nu) and browser tools. Use this skill for quick data extraction from websites, HTML parsing, API interactions, and dynamic content scraping.

Prerequisites

• nu-shell installed (nu)
• query web plugin installed (for HTML scraping): nu -c "plugin add query web"
• Browser extension enabled (for dynamic content): Enable the browser extension in your agent configuration

Common Tasks

Fetching Web Pages

Use http get to retrieve HTML content:

# Simple GET request
nu -c 'http get https://example.com'

# With headers
nu -c 'http get -H [User-Agent "My Scraper"] https://example.com'

HTML Parsing and Data Extraction

Use the query web plugin to parse HTML and extract data using CSS selectors:

# Extract text from elements
nu -c 'http get https://example.com | query web -q "h1, h2" | str trim'

# Extract attributes
nu -c 'http get https://example.com | query web -a href "a"'

# Parse tables as structured data
nu -c 'http get https://example.com/table-page | query web --as-table ["Column1" "Column2"]'

Browser-Based Scraping for Dynamic Content

For websites requiring JavaScript execution or complex DOM interactions, use browser automation tools.

# Start browser
start-browser

# Navigate to page
navigate-browser --url https://example.com

# Extract data with JavaScript evaluation
evaluate-javascript --code "Array.from(document.querySelectorAll('selector')).map(e => e.textContent)"

# Screenshot for visual inspection
take-screenshot

# Query HTML fragments
query-html-elements --selector ".content"

API Interactions

For JSON APIs, use http get and parse with from json:

# GET JSON API
nu -c 'http get https://api.example.com/data | from json'

# POST requests
nu -c 'http post https://api.example.com/submit -t application/json {key: value}'

Handling Authentication

# Basic auth
nu -c 'http get -u username:password https://api.example.com'

# Bearer token
nu -c 'http get -H [Authorization "Bearer YOUR_TOKEN"] https://api.example.com'

# Custom headers
nu -c 'http get -H [X-API-Key "YOUR_KEY" User-Agent "Scraper"] https://api.example.com'

Rate Limiting and Delays

# Add delays between requests
nu -c '$urls | each { |url| http get $url; sleep 1sec }'

Parallel Processing

# Scrape multiple pages in parallel
nu -c '$urls | par-each { |url| http get $url | query web -q ".data" }'

One-liner Examples

Basic HTML Scraping

# Extract all h1 titles
nu -c 'http get https://example.com | query web -q "h1"'

# Get all links
nu -c 'http get https://example.com | query web -a href "a"'

# Scrape product prices
nu -c 'http get https://store.example.com | query web -q ".price"'

HTML Scraping Example: Hacker News

# Scrape HN front page titles and URLs
nu -c 'http get https://news.ycombinator.com/ | query web -q ".titleline a" | get text | zip (http get https://news.ycombinator.com/ | query web -a href ".titleline a" | get href) | each { |pair| echo $"($pair.0) - ($pair.1)" }'

For static sites like HN, use http get directly. Reserve browser tools for dynamic content requiring JavaScript execution.

GitHub Stars Scraper

# Get star count for a repo
nu -c 'http get https://api.github.com/repos/nushell/nushell | get stargazers_count'

API Data Extraction

# Fetch JSON and extract fields
nu -c 'http get https://api.example.com/users | from json | get -i 0.name'

API Authentication

# Bearer token
nu -c 'http get -H [Authorization "Bearer YOUR_TOKEN"] https://api.example.com/data'

# API key
nu -c 'http get -H [X-API-Key "YOUR_API_KEY"] https://api.example.com/data'

# Basic auth
nu -c 'http get -u username:password https://api.example.com/protected'

Related Skills

• nu-shell: Core nu-shell scripting patterns and commands.

Related Tools

• start-browser: Start Cromite browser via Puppeteer.
• navigate-browser: Navigate to a URL in the browser.
• evaluate-javascript: Evaluate JavaScript code in the active browser tab.
• take-screenshot: Take a screenshot of the active browser tab.
• query-html-elements: Extract HTML elements by CSS selector.
• list-browser-tabs: List all open browser tabs with their titles and URLs.
• close-tab: Close a browser tab by index or title.
• switch-tab: Switch to a specific tab by index.
• refresh-tab: Refresh the current tab.
• current-url: Get the URL of the current active tab.
• page-title: Get the title of the current active tab.
• wait-for-element: Wait for a CSS selector to appear on the page.
• click-element: Click on an element by CSS selector.
• type-text: Type text into an input field.
• extract-text: Extract text content from elements by CSS selector.
• search-web: Perform web searches and extract information from search results.