Micro-Course
Identifying AI Hallucinations
AI agents can navigate websites, extract data, and complete tasks — but sometimes they confidently return information that isn't actually on the screen. These are called hallucinations, and catching them is one of the most critical skills for data evaluators.
In this 10-minute micro-course, you'll learn to spot hallucinations, understand why they happen, and practice classifying them correctly.
Learning Objectives
- Define what an AI hallucination is in the context of web agent evaluation
- Distinguish between silent failures (hallucinations) and declared failures (errors)
- Identify hallucinated data by comparing agent returns against source screenshots
- Correctly classify hallucination severity (high vs. low impact)
Concept
What Is an AI Hallucination?
A hallucination occurs when the AI agent returns information that does not exist on the webpage it was viewing. The agent "thinks" it succeeded, but the data it extracted is fabricated or distorted.
Silent Failure
The agent declares success but returned wrong data. It doesn't know it failed. This is a hallucination.
Declared Failure
The agent throws an error or says it can't complete the task. It knows something went wrong. This is NOT a hallucination.
Types
Common Hallucination Patterns
Here are the most frequent hallucination types you'll encounter:
Fabricated Data
Agent returns names, numbers, or details that don't appear anywhere on the page.
Example: Returns a list of 5 doctors, but only 3 are shown on screen.
Distorted Values
Data exists on the page but the agent returns it slightly wrong.
Example: Price shows $68.10 but agent returns $68.01.
Context Hallucination
Agent fills in information it was never given, like making up a password or username.
Example: Form asks for a name, agent types "John Smith" when no name was provided.
False Verification
Agent claims it clicked a button or completed an action, but the screenshot shows it didn't.
Example: "I clicked Submit" but the form is still unfilled on screen.
Scenario 1
Spot the Hallucination
Context
A user asked the AI agent: "Find the top 3 rated Italian restaurants near downtown Boston."
What the webpage shows:
What the agent returned:
1. Giacomo's Ristorante — 4.6 stars
2. Trattoria Il Panino — 4.4 stars
3. Mare Oyster Bar — 4.5 stars"
How would you classify this workflow?
Scenario 2
Subtle or Severe?
Context
A user asked: "What is the current price of the MacBook Air M3 on the Apple website?"
What the webpage shows:
What the agent returned:
The agent returned $1,099.99 instead of $1,099.00. How do you classify this?
Scenario 3
Tricky One
Context
A user asked: "Fill out the contact form with my information."
The user did NOT provide any personal details in the prompt.
What the agent did:
The agent made up a name and email. What type of hallucination is this?
Reference
Quick Reference: Hallucination Cheat Sheet
Keep this handy while evaluating workflows:
| Type | What Happened | Failure Bucket | Severity |
|---|---|---|---|
| Fabricated Data | Agent returns items not on the page | EXTRACT_DATA_WRONG | Silent Failure — High Impact |
| Distorted Values | Numbers/text slightly off from source | EXTRACT_DATA_WRONG | Silent Failure — High Impact |
| Context Hallucination | Agent invents info not in the prompt | CONTEXT_HALLUCINATION | Silent Failure — High Impact |
| False Verification | Agent claims action succeeded but it didn't | FALSE_VERIFICATION | Silent Failure — High Impact |
| Perception Error | Chain of thought misreads the screen | PERCEPTION_COT | Silent Failure — High Impact |
Complete
Nice work!
Scenarios Correct
Key Takeaways
- Hallucinations are silent failures — the agent won't tell you something is wrong
- Always verify agent returns against the actual screenshot
- Even small data distortions (a decimal point, a swapped digit) count as high impact
- Context hallucinations happen when the agent fabricates information it was never given
- You are the last line of defense for data quality
This micro-course was designed by Anusha Batra as a sample learning experience demonstrating scenario-based instructional design for AI evaluator training.