Song-saem: From Planner to Developer with AIBlog home →
What Can You Build with AI?5 min read한국어로 읽기

Can You Trust AI Stock Predictions? The Safeguards I Built So I Wouldn’t (Building a Stock App, Part 2)

AI’s stock predictions sound convincing even when they’re wrong. So in my app, instead of trusting AI’s explanations, I record its predictions and compare them with real results: predictions locked when made, directions hidden until a bar is passed, five strategies competing with fake money, and past conditions that left 9 of 255 stocks.

Can you trust AI’s stock predictions? I decided to make them checkable instead of trusting them. Each AI prediction is locked the moment it’s made so it can’t be changed, then scored against the actual closing price five trading days later. Until it clears the bar I set, the predicted direction isn’t shown. When I had five strategies compete with fake money, none cleared the bar over the past five years of prices. These safeguards don’t keep AI from being wrong, but they make wrong results hard to hide.

Instead of trusting AI, I put it to the test
Instead of trusting AI, I put it to the test

In my last post, I wrote that I built an app called “Stock Investing for Lazy People” so my older brother could easily check his holdings. Since I’ve never invested in stocks myself, I tried to be extra careful while building it.

Ask AI, “Will this stock go up next week?” and you get an answer. The reasons sound convincing, too. The problem is that they sound just as convincing when they’re wrong.

So in the app, instead of trusting AI’s explanations, I decided to record its predictions and compare them with the real results.

First, predictions are locked the moment they’re made

For each stock, AI predicts the direction five trading days ahead as one of “up, down or flat.” The record can’t be changed later. After five trading days, it’s scored against the actual closing price.

This keeps anyone from changing the explanation after seeing the result: “that’s what I meant all along.” Rather than how persuasive the words are, I wanted to see whether the prediction left in advance was right.

Second, directions stay hidden until the bar is cleared

Just because AI made a prediction doesn’t mean users see it right away. The internal bar I set for showing it has three conditions.

  1. Be scored at least 30 times
  2. Have a hit rate of 80% or more
  3. Beat a simple method that always picks the most common outcome by at least 10 percentage points

The third condition is the comparison. If AI explains things in a complex way but does worse than a simple method, I thought its explanations would be hard to rely on.

The scoring count isn’t taken at face value, either. If you predict five trading days ahead every day, the evaluation periods overlap, and a single market move can be counted in the scores of several predictions.

So only predictions with non-overlapping periods are counted. Filling 30 that way takes quite a while. Until then, users see the accumulated report card instead of the predicted direction.

Still, this bar is a threshold I set for the app. Reaching 30 scorings with a hit rate above 80% doesn’t mean the results will continue or that it will make money.

Lock predictions, and set the pass conditions first
Lock predictions, and set the pass conditions first

Third, they compete with fake money first

Separately, I also compare paper-trading strategies. In the app, they’re set up like five players: statistics, near-the-low, price-swing, trend and AI.

They trade by set rules without real money, with fees and taxes included in the calculations. So that the rules aren’t tweaked in their favor after seeing results, the test rules were fixed first.

Tested on the past five years of prices, zero players cleared the paper-trading bar I set.

It meant something that good results didn’t come easily. But the fact that no one passed doesn’t prove the test is well designed, either. The comparison method and the bar need ongoing review too.

Simulated results on past data differ from real-world performance. Passing a backtest well doesn’t guarantee future results (Investor.gov).

Compete with fake money first
Compete with fake money first

So I changed the question

Seeing how much harder than expected it is to guess “what will go up?”, I changed the question:

“What did stocks you could hold for a fairly long time, without checking daily, look like in the past?”

Instead of picking stocks that will rise, I decided to look at past records against conditions.

ConditionStandard
Company sizeMarket cap of at least 1 trillion won
SwingsOne-year volatility of 40% or less
Long-term trendUp at least 30% over five years
Worst dropMaximum drawdown within 40%
EarningsProfitable three years in a row

Maximum drawdown measures how far the price fell from a peak to a later low during the period. Even if the final return looks fine, a big drop along the way could have made it hard to keep holding.

9 of 255 were left

Applying the conditions to the 255 stocks in this analysis left 9. Seven of them were financial companies.

This doesn’t mean I found nine good stocks. It means their past records met the conditions I set. If one industry dominates, I also need to look at how that industry’s traits relate to my conditions.

And volatility or drawdown being within the conditions doesn’t mean they’ll be safe from now on. I don’t intend to use the list that passed as a buy list.

9 of 255, not a recommendation list
9 of 255, not a recommendation list

Rather than asking for trust, I wanted it to be checkable

The more convincing AI’s explanation, the more I wanted to see the record behind it: what it predicted and when, how it actually turned out, and whether it beat a simple method.

That’s why I built the safeguards: locking predictions, setting the bar for showing them, and testing with paper trades first. They don’t keep AI from being wrong. Instead, I wanted to make wrong results hard to hide or cover with a good explanation.

The safeguards I built so I wouldn’t trust AI have, before I knew it, become the part of the app I care about most.

In short

  • AI’s stock predictions sound convincing even when wrong, so I record them and compare them with real results instead of trusting them.
  • Predictions are locked when made and scored against the actual close five trading days later.
  • Directions stay hidden until they clear the bar: 30+ scorings, an 80%+ hit rate, and 10+ points better than a simple method.
  • All five fake-money players failed to clear the bar over the past five years of prices.
  • Past conditions left 9 of 255 stocks, but that’s a past record, not a buy list.

In FROX, the business system I build at ONE-PEX, I also try to keep to this approach: AI-made results aren’t used as is, but checked against set standards before people see them.

*This post is a record of how I built the app, not investment advice.*

Next (Part 3): When my own ideas turned out wrong in front of the numbers.

Related reading: A Stock App That Never Tells You to Buy or Sell (Building a Stock App, Part 1) · Why Is AI So Confidently Wrong?

  • #주식앱만들기
  • #게으른사람의주식투자
  • #주식앱
  • #AI예측
  • #모의투자
  • #AI앱만들기
  • #바이브코딩
Song-saem · Sanghwa Song, CEO of ONE-PEX

A software planner who became a developer after two years of working with AI five to twelve hours a day. Today I build FROX, an AI-powered business system, at ONE-PEX.

More posts on the blog home →