How to Fact-Check AI Answers Before You Use Them
Short answer: To fact-check an AI answer, pick out the claims that would cause harm if wrong (numbers, dates, section numbers, names), ask the AI for its source, then confirm each one on a primary source such as the official government, regulator or company website. If you cannot find the claim there in a few minutes, do not use it.
AI assistants are wrong in a particular way: fluently, confidently and with the right format. A made-up circular number looks exactly like a real one. That is what makes checking a habit rather than an occasional precaution.
This guide explains why it happens, shows two realistic Indian examples, and gives you a three-minute routine with the official sources to check against. It works for answers from ChatGPT, Gemini, Claude, Copilot or any other assistant.
What is an AI hallucination?
An AI hallucination is a false statement an AI presents as fact, such as a non-existent law section, a fake citation or a wrong figure. The model is not lying; it generates the most plausible-sounding text, and plausible is not the same as true.
OpenAI's own help centre says plainly that ChatGPT can produce incorrect information and should be checked (OpenAI's note on accuracy). Every major assistant carries a similar warning.
Why does ChatGPT give wrong answers?
Mostly because of how it was built and what it was given. Knowing the cause tells you where to look.
- Out-of-date training: rules change, and a model trained on older pages may repeat the old version confidently.
- Pattern completion: asked for a citation, it produces something in the shape of a citation even if none exists.
- Weak or no web search: without live sources, it answers from memory; with search, it can still misread or mix up pages.
- Leading questions: 'Which section says I can claim this?' pushes the model to find a section, whether or not one applies.
- Indian specifics: state-level rules, recent notifications and Hindi-only documents are thinner in training data than US topics, so errors are more likely.
Two Indian examples of AI getting it wrong
These are illustrative examples of the kinds of errors we see when assistants answer Indian compliance questions. They show the pattern, not any one tool's current behaviour.
Example 1: a GST deadline. Ask an assistant 'What is the last date to claim input tax credit for an invoice of FY 2024-25?' A typical wrong answer quotes the old rule: the due date of the September return. The time limit sits in Section 16(4) of the CGST Act, which was amended in 2022 to link the deadline to 30 November following the end of the financial year (or the annual return date, if earlier). An assistant leaning on older articles can give the pre-amendment date, cite the right section with the wrong rule, or cite Section 17(5), which deals with blocked credits instead. All three look authoritative.
Example 2: an RBI circular that does not exist. Ask 'Which RBI circular allows small businesses to collect UPI payments into a savings account?' and an assistant may produce a confident answer with a reference number in RBI's usual format and a plausible date. Search the RBI website and nothing matches. The shape was right; the document was invented.
The lesson from both: a correct-looking format (section number, circular number, date) is not evidence. It is the part most worth checking.
The 3-minute verification routine
You do not need to check every sentence. You need to check the claims that would hurt if wrong. This routine takes about three minutes for a typical work answer.
- Minute 0 to 1: highlight the risky claims. Numbers, dates, deadlines, section or circular numbers, names, quotes and anything you will act on or forward. Usually three to five items.
- Ask the AI: 'For each of these claims, give the exact source document, its date and the specific section or page. If you are not sure a source exists, say so.' Assistants are more honest when you explicitly allow 'I don't know'.
- Minute 1 to 2: open the primary source yourself (table below), not the link the AI gave. Search by keyword or reference number, and find the claim in the document itself.
- Check the date of the source. A real circular that was later superseded is still the wrong answer.
- Minute 2 to 3: mark each claim as confirmed, corrected or not found. Delete or rewrite anything not found. Do not keep it with a 'probably'.
- For anything legal, tax or medical that you will act on, confirm with a qualified professional. Verification reduces risk; it does not make you the CA.
Here is the routine applied to the GST example. Risky claims highlighted: the deadline date and the section number. Asked for its source, the assistant names Section 16(4) of the CGST Act. On India Code or the CBIC GST site you open the CGST Act, go to Section 16 and read sub-section (4) as currently amended: the limit is 30 November following the end of the financial year, or the date of filing the annual return if earlier. The section was right; the date the assistant gave was the old one. Mark it corrected, fix the answer, and note the source in your email or file so the next person does not have to repeat the check. Total time: about three minutes, most of it finding the right page.
Where to verify: primary Indian sources
Go to the body that issued the rule, not a blog summarising it.
| Topic | Primary source | What to search for |
|---|---|---|
| GST law, rates, notifications | CBIC GST website (cbic-gst.gov.in) | The Act section, notification number, or circular by date |
| Income tax rules and forms | Income Tax Department (incometaxindia.gov.in) | The section of the Income-tax Act, current forms and due dates |
| Banking, UPI, KYC, loans | Reserve Bank of India (rbi.org.in) | Notifications and master directions by date or keyword |
| Markets, mutual funds, IPOs | SEBI (sebi.gov.in) | Circulars and regulations |
| Central Acts as amended | India Code (indiacode.nic.in) | The Act and section, with amendments |
| Government announcements | PIB (pib.gov.in) and the e-Gazette | The press release or gazette notification |
| Company facts | MCA portal, or the company's own site | Registration details, official statements |
| Tool features and prices | The tool's official pricing or help page | The current plan page |
For research where you want sources in the answer itself, tools that show citations make this routine faster; see our comparison of Perplexity and ChatGPT for research. Citations still need opening, because a cited page does not always say what the summary claims.
Red flags that an AI answer may be made up
- Very specific numbers or reference codes with no source attached.
- A source that is vague ('as per RBI guidelines') instead of a named document and date.
- A quote you cannot find with an exact-phrase search.
- An answer that changes when you ask the same question in a new chat.
- Perfect agreement with the assumption in your question.
- Links that open to a home page or a 404 rather than the claimed document.
How to get fewer wrong answers in the first place
You cannot prompt hallucinations away completely, but you can reduce them a lot.
- Give the AI the source: upload the circular or paste the section, then ask it to answer only from that text.
- Ask it to quote the exact sentence it relied on.
- Tell it to say 'not stated in the document' when the answer is not there.
- Use deep research modes for multi-source questions, then check the sources they list; our guide to using deep research in ChatGPT and Gemini covers when they are worth the wait.
- Write tighter prompts with context and limits; the pattern in how to write AI prompts includes a line that asks the model to flag gaps before answering.
Checking AI output against primary sources is one of the habits practised throughout our Applied AI Professional course, because it is the skill employers worry about most when staff start using AI.
Frequently asked questions
Why does ChatGPT make things up?
ChatGPT generates the most likely next words based on patterns in its training, so when it lacks the right information it can produce something plausible but false. Out-of-date training, missing web search and leading questions make this more likely.
How can I verify information from ChatGPT?
Pick out the claims that matter, ask ChatGPT for the exact source, then find each claim yourself on the primary source such as a government, regulator or company website. Anything you cannot find there should not be used.
Are AI answers with citations always correct?
No. Cited answers are easier to check, but the cited page sometimes does not say what the summary claims, or is out of date. Open the citation and find the exact sentence.
Can I trust AI for GST or income tax questions?
Use it to understand concepts and draft questions, not as the final word. Rules and deadlines change through notifications, so confirm on the CBIC or Income Tax Department website and with your CA before acting.
Which AI tool hallucinates the least?
It varies by task and changes with each model update, so no ranking stays true for long. Any tool is more reliable when you give it the source document and ask it to answer only from that.
Keep reading in AI tools
- How to Write AI Prompts That Work First Time Start here
- ChatGPT Custom Instructions and Memory: Set Them Up Once
- Is It Safe to Upload Documents to ChatGPT? Privacy Settings and What Not to Share
- Free AI Tools for Students: Which One for Which Study Task
- How to Make AI Videos From Text and Images
Related: Applied AI Professional · Perplexity vs ChatGPT for research: which one to trust for what · All guides in AI tools