Song-saem: From Planner to Developer with AIBlog home →Why Is AI So Confidently Wrong? The Checking Habits I Learned in Two Years
AI is not a machine that knows the right answer. It is a machine that produces plausible answers people like. So judging and checking the evidence is your job. Here is how to make checking a habit.
Can you trust AI 100%? No. AI is not a machine that knows the right answer. It is a machine that produces the most plausible, most agreeable answer to your question. So you have to make the call, and you have to find the evidence yourself to get close to the truth. Here are the checking habits I picked up working with AI 5 to 12 hours a day for two years.
Why does AI tell you what you want to hear?
AI is a service too. People have to like it to keep using it, so AI is fine-tuned on scores that people give its answers. The catch: people tend to score answers that agree with them higher.
So AI can read your leanings and drift toward what you would like to hear. You may get a biased answer rather than an objective one.
This isn't just my impression. In a 2023 study, Anthropic found that AI trained on human feedback tends to shift its answers to match the user's views. In April 2025, OpenAI even rolled back a ChatGPT update after it became too eager to please.
Does AI lie?
It isn't deliberately deceiving you. Rather, it sometimes produces a plausible answer even when the evidence is thin.
In a September 2025 post, OpenAI explained one of the causes. On a test, a blank gets zero but a guess might be right; current training and evaluation likewise reward a plausible guess over "I don't know."
Two years ago these wrong answers were everywhere. With more training data and newer versions, AI now often says "I'm not sure," and made-up answers have dropped a lot. Still, it can give wrong answers or head off in a direction I didn't ask for.
| How it goes wrong | Why | How to check |
|---|---|---|
| Agreeing with you | It is rewarded for answers people like | Ask for one piece of evidence against your view |
| A plausible guess | Guessing is rewarded over "I don't know" | Ask for sources, then open them yourself |
| The wrong direction | It works without knowing your standards | Give it the standards for the work and for the review first |
Two years ago, even a perfect prompt wasn't enough
Two years ago I was making English test questions by type. I wrote a careful, complete prompt, but AI didn't follow those rules every time. It got the first few questions right, then quietly drifted from the rules, and I kept going without noticing. When I checked later, many questions were unusable, and more than once I had to start over from scratch.
Why? Back then, AI could hold far less in memory at once than it can today. As a chat grew longer, the rules I gave at the start were pushed out and forgotten. That was not the only reason it broke the rules, of course. (I explain why AI loses track of earlier parts of a long chat in Does AI remember you?.)
This rarely happens now. Give it the same job today and it writes the questions, reviews them itself, suggests new ones, and even points out what makes each question hard. Still, I always give it review standards. That is because of the times two years ago when I "kept going without worry" and had to redo everything.
So how do you raise the quality of AI's work?
AI learned from text and documents people wrote, so it can make the same mistakes people make. Just as human work needs review, so does AI's.
But "please review this" alone does little, because AI doesn't know what to check against. I use three methods.
- Give the standards for the work and for the review together: "Customers will read this document. Cite a source for every number, and say so when you're not sure. When you finish, review it against these rules and fix it." I still check the important numbers and sources myself.
- Save them as default rules so they run automatically: So you don't repeat yourself, write your review rules into places like ChatGPT's custom instructions or Claude's project instructions. Then AI checks itself against them every time. It doesn't always follow them, though, so I still check the result once more.
- Ask for an objective rating against other examples: "Compare this with similar cases and score it honestly." This often draws out a fairly candid view. Not 100%, of course.
In the end, you need to see yourself objectively
What makes AI do the right thing ultimately comes from the user. Knowing your own view might be wrong, and doubting once more precisely when AI takes your side. I believe the ability to see yourself objectively is the most important skill in the age of AI.
In short
- AI is not a machine that knows the answer. It produces plausible answers people like.
- Agreeing and guessing come from how AI is trained. Judging and checking evidence is your job.
- Instead of "review this," give review standards and save them as default rules.
- The more AI agrees with you, the more you should double-check.
When I build FROX, the AI-powered business system at ONE-PEX, I still check what AI produces against set standards one more time.
Sources
A software planner who became a developer after two years of working with AI five to twelve hours a day. Today I build FROX, an AI-powered business system, at ONE-PEX.