How to Write Prompts for Sentiment Analysis

The Problem a Customer Insights Analyst Faced

Why do sentiment analysis tools so often get sarcasm exactly backwards? That's the question a customer insights analyst named Priya kept running into while building prompts for sentiment analysis on a batch of 500 product reviews she needed classified before a quarterly business review. A review that said "oh great, another update that breaks the feature I actually used" got tagged as positive sentiment, because the words "great" and "update" carry positive weight on their own, even though any human reading the full sentence immediately catches the sarcasm without a second thought.

This wasn't a one-off glitch. Roughly 8% of her negative reviews that used sarcasm or backhanded phrasing got misclassified as positive, which quietly skewed her overall sentiment dashboard toward looking healthier than the actual customer feedback supported. Nobody had caught it because the dashboard number looked plausible — just wrong in a way that was invisible without checking individual cases one by one against the source text.

⚠️ Common mistake: Trusting an aggregate sentiment score without ever spot-checking individual classifications, especially for a product category where sarcastic or backhanded complaints are common in how customers actually express frustration in their own words.

The Wrong Approach

Priya's original prompt:

Classify the sentiment of this review as positive, negative, or neutral.

This fails because it gives the model no instruction about how to handle sarcasm, mixed sentiment within a single review, or context clues beyond the literal words used. Sentiment classification done at the word level rather than the intent level consistently misreads sarcasm, since sarcastic statements are specifically constructed to use positive-sounding words to convey a negative meaning, which is exactly the trap a shallow word-counting approach falls into every time.

What this does: produces confident, plausible-looking sentiment labels that are wrong in a specific, predictable, and hard-to-catch way — exactly the kind of error that survives into a dashboard because nothing about the output looks obviously broken to someone glancing at aggregate numbers rather than individual classifications.

The Correct Prompt

Here's what Priya rebuilt her prompt into, directly targeting the sarcasm failure she'd identified:

Classify the overall sentiment of this review as positive, negative, or neutral, based on the reviewer's actual intent, not just the literal words used.

Watch for sarcasm, backhanded compliments, and mixed sentiment (e.g. praising one feature while criticizing another). If sentiment is mixed, classify based on which aspect the reviewer seems to weight more heavily, and note the mixed nature in a one-sentence explanation.

Example of sarcasm to watch for: "Oh great, another update that breaks the feature I actually used" is NEGATIVE despite containing positive words.

Review: [PASTE TEXT]

What this does: naming sarcasm and mixed sentiment explicitly, along with a concrete example of exactly the failure pattern Priya had already found in her own data, gives the model a specific case to generalize from rather than an abstract instruction to "consider context," which tends to produce inconsistent results on its own without a concrete anchor to work from.

⚡ Pro tip: When you find a specific misclassification pattern in your own data, add that exact example to your prompt rather than writing a general rule about it. Concrete examples of edge cases transfer better than abstract descriptions of what to watch for, since the model can pattern-match directly instead of interpreting a rule that might apply differently than you intended.

⚡ Pro tip: If your sentiment analysis feeds into any kind of automated alert or escalation system, weight false negatives (missing genuine negative sentiment) more heavily than false positives in your review process. A missed complaint that never gets addressed is usually more costly than a neutral comment that gets briefly flagged for no reason.

Results and What Changed

After the rebuild, Priya's sarcasm-related misclassification rate dropped from roughly 8% to under 2% on a follow-up sample of 200 reviews. The remaining errors were mostly reviews with genuinely ambiguous tone even to a human reader, which is a much more reasonable failure mode than confidently misreading obvious sarcasm the way her original prompt had been doing at scale across the whole dataset.

A social media manager at a consumer electronics brand ran into a related issue: her sentiment prompt handled sarcasm reasonably well but struggled with mixed reviews — customers who loved the product but hated the customer service experience they'd had when something went wrong. Adding an explicit instruction to output separate sentiment scores per aspect, rather than forcing one overall label, gave her team a far more useful signal than a single blended score that averaged out two genuinely different experiences into a misleading middle ground that didn't represent either aspect accurately.

A market researcher analyzing survey open-text responses found that providing 3-4 real examples of ambiguous sentiment from her own actual dataset, rather than generic examples pulled from a tutorial somewhere, improved her classification consistency more than any wording change to the instructions themselves. Real examples from your actual data almost always outperform generic ones, because they reflect the specific ways your particular audience actually expresses sentiment, which varies more between different customer bases than people tend to expect.

⚡ Pro tip: For product categories where sarcasm or backhanded complaints are common — software, subscriptions, anything customers have ongoing frustration with — assume sarcasm handling needs to be explicit from the start rather than something you add after noticing a problem in your results months later.

⚡ Pro tip: For mixed-sentiment content, consider requesting per-aspect sentiment rather than one overall score whenever your content naturally covers multiple distinct topics. A single blended number often hides more than it reveals about a genuinely two-sided reaction, and it's a signal you can't easily recover once you've already collapsed it down to one label.

How to Apply This to Your Situation

Before trusting any sentiment analysis pipeline for a real decision, manually review 30-50 classifications against the actual text, specifically looking for sarcasm, backhanded compliments, and mixed sentiment — the three failure modes that consistently trip up word-level sentiment reading. This manual review step feels slow compared to just trusting the output, but it's dramatically faster than discovering a skewed dashboard months later after decisions have already been made based on it.

If you find a specific pattern of misclassification, as Priya did, add that exact example into your prompt rather than trying to write a general rule that covers every possible case of that failure type. Specific beats general here, consistently, because the model generalizes more reliably from a concrete case it can pattern-match against than from an abstract instruction it has to interpret fresh each time.

⚠️ Common mistake: Assuming a sentiment analysis prompt that works well for one type of content — say, formal customer support transcripts — will transfer cleanly to a different register, like casual social media comments where sarcasm and slang are far more common and more central to how sentiment actually gets expressed. Each content type tends to need its own calibration pass, even if the underlying prompt structure stays largely the same.

It's also worth checking whether your sentiment categories themselves make sense for your actual use case and business need. A simple positive/negative/neutral scheme works fine for many purposes, but if you're trying to catch specific emotional signals — frustration versus disappointment, for instance, which might warrant different follow-up actions from your team — a finer-grained category scheme captures distinctions a three-way split would flatten out entirely.

Next Steps

Build a small library of real sarcasm and mixed-sentiment examples from your own data as you encounter them, the same way Priya did after her first review. This becomes a genuinely valuable asset over time, since each new example you add makes your prompt more resilient to that entire category of edge case rather than just the one instance you happened to catch during a spot-check.

Set up a recurring review cadence rather than treating the sarcasm fix as a one-time patch you never revisit. Priya now spot-checks 20 random classifications every two weeks against the actual review text, specifically looking for new sarcasm patterns or phrasing her existing examples don't cover. This caught a new pattern within the first month — customers using a specific slang term ironically that her original example set hadn't anticipated at all when she first built it.

There's a broader lesson here that extends past sentiment analysis specifically: any classification task working with human-written text will eventually run into some form of the sarcasm problem, where literal meaning and intended meaning diverge in ways that are obvious to a human but invisible to a system reading words at face value. Building a habit of collecting and feeding back real misclassification examples, rather than treating your first prompt version as final, is what actually keeps accuracy improving over time instead of quietly degrading as your data's language patterns shift.

Once you have a sentiment analysis prompt that reliably handles your specific content's quirks, save it as a versioned template rather than rebuilding your sarcasm-handling instructions from memory on your next project. PromptABCD works well for this, letting you keep your accumulated edge-case examples attached to the prompt itself, so the lessons from Priya's 8% sarcasm error don't have to be relearned by the next analyst who inherits a similar dataset.