Every day, millions of people post reviews, comments, tweets, and feedback.
How do companies analyze what people REALLY think when there are 10,000 reviews to read?
You're about to learn:
• How computers "read" sentiment (happy, angry, sad)
• Why AI sometimes gets it completely wrong
• How to detect when text analysis is biased
By the end, you'll know which methods to trust... and which to question.
You work for a restaurant chain.
Last month, you got 5,000 customer reviews on Yelp, Google, and social media.
Your boss asks: "Are customers happy or unhappy? I need to know by tomorrow."
You CAN'T read 5,000 reviews manually.
So you turn to AI...
But there's a catch: Different AI methods give you DIFFERENT answers.
Which one do you trust?
Which review is POSITIVE? Which is NEGATIVE?
Now let's see how different AI methods analyze these...
Sounds simple, right?
Dictionary method finds:
But wait... read the review again. Is it ACTUALLY positive?
A HUMAN reads this and knows: This person is being sarcastic.
"Disgusting" is used ironically - they mean it was SO GOOD it was almost too much.
But the dictionary method doesn't understand sarcasm.
It just counts words.
So... what's better?
For each review, guess which method will get it RIGHT
Which method gets this RIGHT?
Which method gets this RIGHT?
Which method should you use?
Complete all 4 sections to continue
Look at these two movie reviews:
Run both through a sentiment analyzer. What scores do they get? Which is rated more "positive"?
Most sentiment analyzers were trained on Amazon product reviews and Yelp.
• What KIND of language appears in product reviews?
• Is "brilliantly deconstructs late capitalism" the kind of phrase that appears in Amazon reviews?
• Whose vocabulary is the AI trained on?
Imagine a company uses sentiment analysis to decide which employee feedback to prioritize.
Both employees are giving POSITIVE feedback. Who gets heard? Who gets ignored? What happens when AI is trained on one group's language patterns?
If you were building a sentiment analyzer, what would you do differently to reduce bias?
Ideas to consider: Training data sources, multiple models for different audiences, human oversight, transparency about limitations
💾 Your responses are saved to your learning journal
Guess the sentiment:
Guess the sentiment:
Guess the sentiment:
Beyond positive/negative, text classification is used to detect:
The stakes are HIGH:
• False negatives: Toxic content stays up, harms users
• False positives: Legitimate speech gets censored
This is why human oversight still matters.
Who gets censored?
Research from 2024 found that toxicity detectors disproportionately flag:
Meanwhile, they MISS: Coded hate speech, dog whistles (subtle discriminatory language), sealioning (bad-faith questioning)
What did you find about toxicity detector bias?
Training data matters. If a toxicity detector was trained on flagged comments from 2010-2015, what language patterns did moderators flag back then?
• Whose speech was considered "toxic"?
• Who had the power to decide what's "acceptable"?
• How have norms changed since then?
Scenario: You run content moderation for a social platform. Your AI auto-removes 10,000 comments per day for "toxicity."
But you discover it's removing comments from marginalized groups describing their discrimination.
• What's the harm of false positives in this case?
• What's the harm of false negatives (letting actual toxicity through)?
• How do you balance freedom of expression with user safety?
Design a better moderation system:
• Should AI auto-remove or just flag for human review?
• How would you reduce bias in the training data?
• What role should community input play?
• Should different communities have different moderation standards?
After analyzing thousands of text datasets, here's what works best:
STEP 1: Unsupervised Topic Modeling
Discover themes in your data automatically (no labeling required)
Use: BERTopic or LDA
Output: "There are 15 topics in this dataset"
STEP 2: LLM Labeling
Ask an LLM to label each topic
Use: GPT-4, Claude, etc.
Output: Topic 1 = "Shipping complaints", Topic 2 = "Product quality praise"
STEP 3: Train Supervised Classifier
Now that you know the categories, label 100-500 examples per category. Train a fast, cheap supervised ML model. Use this for future classification at scale.
This is how modern companies do text analysis in 2026.
What gets lost in automation?
You just automated customer support ticket categorization.
What did the AI miss?
What information do humans capture that AI doesn't?
Consider: Emotional urgency, context, frustration, power dynamics
Can sentiment analysis capture all of this?
Imagine your company automates text analysis to:
For each use case: What's the benefit of automation? What's the risk? Who is harmed when the AI gets it wrong?
Create your own "Human-in-the-Loop" policy:
For text analysis in high-stakes situations (hiring, content moderation, etc.): What percentage should be human-reviewed? What triggers should escalate to human review? How do you audit the AI's decisions? When should you NOT use automation at all?
Write a 3-5 sentence policy:
✅ Fast, cheap, transparent
❌ Misses sarcasm, context, negation
Use when: Simple sentiment on large scale
✅ Learns patterns, handles context better
❌ Requires training data, expensive setup
Use when: You have labeled examples, need speed + accuracy
✅ Best accuracy, explains reasoning, zero-shot
❌ Expensive, slower
Use when: Accuracy > cost, need explanations
Topic modeling → LLM labeling → Supervised classifier
The power to analyze millions of texts comes with responsibility.
Use it wisely.
You now know:
✅ How sentiment analysis works (3 methods)
✅ When each method fails
✅ How to detect bias in text classification
✅ The professional workflow for text analysis
Next up: Lesson 7 - Topic Modeling