How to Write Prompts for Data Extraction

Before: The Weak Prompt

A research analyst spent an entire afternoon manually cross-checking 80 AI-extracted company names from a batch of press releases, because roughly 1 in 6 entries had silently merged two different company names into one plausible-looking but completely wrong entity. That's the kind of failure that happens quietly with weak prompts for data extraction — not a crash, not an error message, just confidently wrong output that looks fine until someone checks it closely against the actual source material.

Her original prompt:

Extract the company names mentioned in this press release.

This produced a list that looked complete and reasonable at a glance, with clean formatting and nothing about it that would suggest a problem to anyone giving it a quick review. The issue only surfaced when she checked against the source text and found cases where the model had combined "Meridian Health Partners" and "Meridian Capital Group" — two separate companies mentioned in the same paragraph — into a single garbled entry, apparently because both names shared the word "Meridian" and appeared close together in the text.

Why It Fails

Vague extraction prompts fail because they don't specify how to handle ambiguity, and press releases are full of it: similar names, abbreviated references, pronouns standing in for previously mentioned entities. Without explicit instructions for handling these cases, the model makes a plausible guess, and plausible guesses in extraction tasks are exactly the failure mode that's hardest to catch, because the output still looks structurally correct and nothing about the formatting signals that anything went wrong.

⚠️ Common mistake: Trusting extraction output without a specific spot-check process, especially for any list longer than what you could quickly compare against the source text yourself in under a minute. The larger the batch, the more tempting it is to skip verification entirely, which is exactly backwards — larger batches have more opportunities for the same silent error to repeat undetected.

After: The Improved Prompt

Here's the rebuilt version the analyst now uses, built specifically to prevent the merging issue and surface uncertainty instead of hiding it:

Extract every distinct company name mentioned in the text below. Treat companies as separate entities even if their names share words (e.g. "Meridian Health Partners" and "Meridian Capital Group" are two different companies, not one).

For each company found:
- Exact name as it appears in the text
- First sentence where it's mentioned (for verification)
- Confidence: high/medium/low (low if the name is abbreviated, implied by pronoun, or ambiguous)

Output as a table. If you're uncertain whether two mentions refer to the same or different entities, flag both with a note rather than merging or guessing.

What this does: the explicit instruction about shared-word names directly targets the exact failure mode that caused the original error, and the confidence field turns an invisible guess into a visible flag the analyst can actually check before trusting the data on anything that matters downstream.

⚡ Pro tip: When you've identified a specific failure pattern from a past extraction, write the fix directly into the prompt as a named example, not just a general instruction. "Don't merge similar names" is vague; "Meridian Health Partners and Meridian Capital Group are different companies" gives the model a concrete anchor it can actually generalize from, rather than an abstract rule it might interpret inconsistently.

Breaking Down Each Element

The "first sentence where mentioned" field does more verification work than it might seem to at first glance. It lets the analyst spot-check extracted entities against source text quickly, without re-reading the entire document, which turns a slow manual audit into a fast scan of just the flagged sentences instead of a full re-read every time she wants to verify the output.

The confidence field is arguably the single most valuable addition. A data analyst at a legal research firm doing similar extraction work said adding a confidence field to her prompts didn't reduce errors directly, but it made errors visible instead of invisible, which is almost more valuable, since it meant she could focus her limited review time on the low-confidence entries instead of spot-checking everything at random and hoping she happened to catch the actual mistakes.

⚡ Pro tip: For any extraction task feeding into a decision or a report, always request a confidence or uncertainty field. It costs almost nothing to add and it transforms your review process from "check everything randomly" to "check the flagged items specifically," which scales much better as your extraction batches grow larger over time.

⚡ Pro tip: Give the model explicit disambiguation examples pulled from your actual domain, not generic ones. If your data regularly includes tricky cases — similar product names, near-duplicate entity names, ambiguous abbreviations — name a real example from your own data rather than a hypothetical one, since the model generalizes much better from a case it recognizes as genuinely representative.

Variations for Different Contexts

For extracting numerical data — prices, dates, quantities — add explicit format normalization instructions. "Dates in original format may vary; normalize all extracted dates to YYYY-MM-DD" prevents a downstream headache when half your extracted dates are in one format and half in another because the source documents themselves were inconsistent. This kind of normalization instruction is cheap to add and expensive to skip, since format inconsistencies tend to surface much later, often after the data's already been imported somewhere.

For extraction from documents with inconsistent structure — scanned forms, varied email formats, unstructured customer feedback — lower your expectations for a single extraction pass and build in an explicit "not found" option for every field. A model forced to fill in a value will sometimes invent something plausible rather than admit the information isn't present in the source text, and an explicit "not found" option gives it permission to be honest about a genuine gap instead.

⚡ Pro tip: For extraction tasks involving any kind of personally identifiable or sensitive information, add an explicit instruction about what to do with fields that are ambiguous or only partially present — don't leave this to the model's default judgment, since the cost of a wrong guess is usually much higher for sensitive data than for something like a company name.

⚠️ Common mistake: Running extraction on a large batch without first testing the prompt against 5-10 known-tricky examples pulled from your actual data. Generic test cases won't surface the specific ambiguity patterns your real documents contain, and the patterns that trip up a prompt are almost always specific to your actual data, not generic edge cases you'd think to test otherwise.

A market research firm extracting product mentions from thousands of online reviews found their biggest recurring issue wasn't ambiguous entity names at all, but inconsistent handling of sarcasm and negation — reviews saying "not bad" being extracted as negative sentiment, or sarcastic praise being read as genuine positive sentiment. Adding a handful of real examples of both patterns directly into their extraction prompt cut this specific error category by more than half, since the model had concrete anchors for what sarcasm and negation actually looked like in their specific review data rather than a generic definition.

Save and Reuse This

The research analyst now keeps a running list of extraction failure patterns she's encountered — the Meridian merge issue, an abbreviation confusion from a different project, a date format inconsistency from another — and checks new extraction prompts against that list before trusting them on a new batch. It's a small habit, but it means she's not rediscovering the same category of error twice, which matters a lot when the alternative is another afternoon of manual cross-checking she'd rather not repeat.

This kind of failure-pattern list gets more valuable over time, not less, because each new project tends to surface at least one new edge case worth adding. After about six months of building hers out, the analyst estimates she catches roughly 80% of likely failure modes just by checking a new prompt against the list before running it on a real batch, rather than finding out the hard way after the extraction is already done and trusted.

It's also worth sharing this list across a team rather than keeping it individual, since different people tend to encounter different failure patterns depending on the kind of data they work with most. A researcher extracting from legal documents and one extracting from customer reviews will build genuinely different lists, and combining them gives everyone a more complete picture of what to watch for than either person would develop alone.

If your team does extraction work regularly, that failure-pattern list is worth turning into a shared template rather than keeping it in one person's head. A tool like PromptABCD is useful here for versioning extraction prompts alongside notes on what specific failure modes each version was built to catch, so the next person doing similar work inherits the lessons instead of learning them the hard way through their own afternoon of manual cross-checking.