---
name: scrape-builder
description: Research and document how to scrape a new website, then write a site-specific skill reference file for use with agent-browser. Use when the user wants to add a new website to an existing scraping skill, or build a new scraper workflow from scratch.
allowed-tools: Bash(agent-browser:*)
---

# Scrape Builder Skill

Researches a website's structure and documents it as a reusable skill reference — URL patterns, query parameters, key selectors, and working code snippets.

## Output

The result is a `.md` reference file that can be placed inside an existing skill directory (e.g. `~/.claude/skills/travel-booking/new-site.md`) or used as its own skill.

---

## Step 1 — Understand the Site

Before opening a browser, clarify with the user:
- What is the site URL?
- What flows matter? (search, detail page, booking, results)
- Are there known URL patterns or query params?
- Any login required?

## Step 2 — Explore URL Patterns

Open the site manually and perform the target flow. Watch for URL changes at each step:

```bash
PROXY_URL=$(tc proxy)
agent-browser close
agent-browser --proxy "$PROXY_URL" open "https://example.com"
sleep 3
agent-browser screenshot /workspace/group/step1.png
```

After each meaningful action (search, filter, select dates), capture the URL:

```bash
agent-browser get url
```

**What to look for in URLs:**
- Query parameters that encode search criteria (dates, location, guests, etc.)
- Path segments that identify the resource (hotel slug, city name, etc.)
- Parameters that can be changed to skip UI steps entirely

**Example — discovering Booking.com's URL pattern:**
1. Go to `https://www.booking.com`
2. Search for "Sol Barbados, Magaluf"
3. Click through to the hotel
4. Note URL: `https://www.booking.com/hotel/es/sol-barbados.html`
5. Add dates manually: `?checkin=2026-07-01&checkout=2026-07-07&group_adults=2`
6. Reload — if it works, document the params

## Step 3 — Map Key Selectors

Use snapshot and inspector to identify stable selectors for results, prices, and key data:

```bash
# Snapshot the page for interactive elements
agent-browser snapshot -i

# Snapshot a specific section
agent-browser snapshot -s "main"

# Get text from a suspected element
agent-browser get text "h1"
agent-browser get text "[data-testid='price']"

# Screenshot for visual reference
agent-browser screenshot /workspace/group/page_state.png
```

**Look for:**
- `data-testid` attributes — most stable across deploys
- Semantic elements (`h1`, `h2`, `article`, `[role="listitem"]`)
- Class names with meaningful prefixes (avoid hash-based classes like `sc-abc123`)

Test selectors multiple times across different results to confirm reliability.

## Step 4 — Test Anti-Bot Handling

Check if the site blocks scrapers:
- Does the page load normally or show a CAPTCHA?
- Does a proxy help? (`tc proxy` for residential proxy)
- Are there cookie consent banners to dismiss first?

Note any required setup steps:

```bash
# Common cookie dismissal patterns
agent-browser click "#onetrust-accept-btn-handler"   # OneTrust
agent-browser click "[id*='accept']"                  # Generic accept
agent-browser click "button:has-text('Accept')"       # Text-based
```

## Step 5 — Write the Reference File

Structure the reference file as follows:

```markdown
# {Site Name} Reference

## {Flow Name} URL

Describe the URL pattern with a table of query params:

| Parameter | Description | Example |
|---|---|---|
| `param` | What it does | `value` |

**Example URL:**
\`\`\`
https://example.com/search?q=hotel&checkin=2026-07-01
\`\`\`

## Opening with Proxy

\`\`\`bash
PROXY_URL=$(tc proxy)
agent-browser close
agent-browser --proxy "$PROXY_URL" open "https://example.com/..."
sleep 3
agent-browser screenshot /workspace/group/result.png
\`\`\`

## Cookie/Consent Handling

\`\`\`bash
agent-browser click "#selector"
\`\`\`

## Key Selectors

\`\`\`bash
agent-browser get text "selector"   # what it extracts
\`\`\`

## Tips

- Any timing, retry, or anti-bot notes
```

## Step 6 — Place the File

For an existing skill (e.g. travel-booking):
```bash
cp /workspace/group/new-site.md ~/.claude/skills/travel-booking/new-site.md
```

Or ask the platform admin to add it to the shared skill repository at:
```
/home/admin/treehouse/nanoclaw/container/skills/{skill-name}/{site-name}.md
```

Once added to the repository, all tenants will have access to it on next container start.

---

## Reference File Naming Convention

- Use lowercase kebab-case matching the domain: `booking-com.md`, `google-flights.md`, `airbnb.md`
- One file per site — multiple flows within the same file
- Keep it concise: working examples over prose explanations
