AI Prompts for Web Scraping

A marketer I know built an AI-generated scraper to track competitor prices. It worked perfectly for one day. Then the target site changed a class name, the scraper's selector returned an empty list instead of throwing an error, and for a full week it recorded prices of zero for everything. Nobody noticed until a report showed competitors apparently giving products away. The scraper never crashed — it just quietly lied. This is the failure mode AI prompts for web scraping almost never account for, and this guide fixes it.

Quick-Start (Copy This Right Now)

Here's a scraping prompt that builds in the safeguards from the start:

Write a Python scraper using requests + BeautifulSoup for
[URL]. Requirements:
- Set a realistic User-Agent header
- Raise a clear error if the target element is missing —
  never return empty/zero silently
- Retry with backoff on network errors (max 3)
- Respect a 1-2 second delay between requests
- Validate each scraped field (price is a positive number,
  title is non-empty) and log anything that fails validation
- Save results with a timestamp
Explain which selectors are fragile and might break.

What this does: It turns silent failure into loud failure — the scraper raises errors when the page changes instead of recording garbage — and it validates every field so bad data gets flagged, not stored.

⚡ Pro tip: The line "raise a clear error if the target element is missing" is the single most important instruction in any scraping prompt. Silent empty results are how scrapers lie to you for a week. Fail loud, fail early.

Understanding the Variables

Each requirement in that prompt targets a specific way scrapers fail in the wild.

User-Agent header matters because many sites block or serve different content to requests that don't look like a browser. Without it, you might scrape an error page and not realize it.

Fail-loud-on-missing-element is the core lesson from the intro story. A selector that returns

None

Retry with backoff handles the reality that networks and sites are flaky. A single failed request shouldn't kill a long scrape, but infinite instant retries will get you blocked. Backoff is the balance.

Rate limiting is both courtesy and self-protection. Hammering a site gets your IP blocked and can overload a small server. A short delay keeps you under the radar and is simply the responsible way to scrape.

Field validation is your last line of defense. Even if a selector matches, it might grab the wrong thing. Checking that a price is a positive number catches the case where your selector accidentally matched a "0 reviews" count instead of the price.

⚠️ Common mistake: Assuming a scraper that runs without errors is working correctly. A scraper can run flawlessly while collecting completely wrong data if selectors match the wrong elements or silently return empty. Always validate the actual values, not just that the code executed.

Step-by-Step: Building a Reliable Scraper

Follow this loop for scrapers you can actually trust over time.

Step 1 — Inspect before you prompt. Open the page, view source, and identify the elements you want. Paste a snippet of the actual HTML into your prompt so the model writes selectors for the real structure, not a guess.

Here's the HTML around the price [paste snippet].
Write a selector that targets the price specifically,
and explain why it's more stable than alternatives.

What this does: Real HTML produces real selectors. A described page produces guessed selectors that often miss.

Step 2 — Demand validation on every field. Don't let any value into your dataset unchecked. Prices are positive numbers, dates parse, titles aren't empty.

Step 3 — Build in monitoring. Ask for a summary after each run: how many records scraped, how many failed validation, how many fields were missing. A run that suddenly scrapes 3 records instead of 300 should be obvious immediately.

Step 4 — Handle pagination and dynamic content. If the site loads content with JavaScript, requests + BeautifulSoup won't see it. Tell the model, and it'll suggest a headless browser approach like Playwright instead.

Step 5 — Log fragile selectors. Ask the model to flag which selectors are most likely to break, so you know where to look when the site changes.

⚡ Pro tip: Ask for a post-run summary line: "scraped 297/300 records, 3 failed price validation." This turns a silent multi-day failure into an obvious one-run anomaly you catch the same day.

Pro-Level Variations

For harder scraping jobs, these variations help.

For JavaScript-heavy sites, specify a headless browser:

This site loads products via JavaScript. Use Playwright,
wait for the product grid to load, then extract. Handle
the case where the grid never appears (timeout + error).

What this does: It picks the right tool for dynamic content and handles the timeout case that breaks naive headless scrapers.

For large scrapes, ask for resumability:

Make this scraper resumable: save progress after each page
so a crash at page 400 of 500 doesn't lose everything.

For anti-bot handling, be honest about scope and legality:

Explain what anti-scraping measures this site likely uses
and how to scrape respectfully within its robots.txt and
terms. Suggest an official API if one exists.

A growth analyst I know always asks the last question first, and says the AI has pointed her to an official API more than once — saving her from building a fragile scraper at all. Checking for a sanctioned data source before scraping is both faster and safer.

⚡ Pro tip: Before building any scraper, ask the AI whether the site offers an official API or data export. Scraping is the fallback, not the first choice — an API is stable, sanctioned, and won't break when someone renames a CSS class.

Troubleshooting Common Issues

Scraper returns empty results. Fix: check whether the content loads via JavaScript. If so, you need a headless browser, not requests.

Getting blocked after a while. Fix: add rate limiting, rotate User-Agents, and respect robots.txt. Aggressive scraping gets IPs banned fast.

Data looks wrong but no errors. Fix: this is the silent-failure trap. Add field validation and a post-run summary so wrong data announces itself.

Selectors break weekly. Fix: prefer stable selectors — IDs and data attributes over deeply nested class chains. Ask the model which selectors are most durable.

Your Turn

Take the Quick-Start prompt, paste in the real HTML from the page you're targeting, and build a scraper that fails loud. Then add the post-run summary so you'd catch the "recording zeros for a week" disaster on day one.

The people who run reliable scrapers aren't avoiding failures — they're making failures loud and visible. That's a set of reusable habits, and PromptABCD is where they belong: save your fail-loud scraper template, your validation checklist, and your "check for an API first" prompt, then reuse them every time. The scraper that quietly lied for a week becomes impossible, because loud failure is baked into every prompt you send. That reliability is worth far more than the few minutes it takes to save the template.

It's worth a word on the responsible side of scraping, too, because the best prompt habits and the ethical ones overlap. Rate limiting protects the site you're scraping as much as it protects your IP. Checking robots.txt and terms of service keeps you on the right side of both etiquette and, in some cases, the law. Preferring an official API when one exists is faster, more stable, and sanctioned. When you prompt for these things by default, you end up building scrapers that are both more reliable and more defensible — the two goals point in the same direction more often than people assume.

⚡ Pro tip: Add "respect robots.txt and add a delay between requests" to every scraping prompt as a standing rule. It costs nothing, keeps you from accidentally hammering a small site, and produces scrapers that won't get your IP banned mid-project. Responsible scraping is reliable scraping. One last practical note: sites change, and even the most carefully built scraper will eventually break when a target redesigns its pages. The goal isn't a scraper that never breaks — that's impossible — but one that breaks loudly and obviously the moment the page changes, so you fix it in an hour instead of discovering a week of corrupted data. Every safeguard in this guide serves that single goal. Build for loud failure, and the maintenance burden of scraping drops from constant anxiety to occasional, obvious repairs.

Think of a scraper less as a one-time script and more as a small piece of infrastructure that needs monitoring. The post-run summary is your dashboard, the field validation is your alerting, and the fail-loud behavior is your circuit breaker. Framed that way, the extra requirements in the Quick-Start prompt stop feeling like overhead and start feeling like the minimum any data pipeline deserves. Once you treat scraping as infrastructure, saving your battle-tested prompts becomes obvious rather than optional, because infrastructure is exactly the kind of thing you never want to rebuild from memory under pressure.