What to check before you trust an AI chess app

The five-minute version

Before you connect an account to any AI chess training app, check four things: what the "AI" is doing (almost always Stockfish plus some text generation on top), whether analysis runs in your browser or on someone's server, what account permissions it asks for, and whether it can delete everything on request.

Then check one more thing that has nothing to do with privacy: does the app ever tell you that you got something wrong? A trainer that agrees with you all afternoon is a dashboard, not a trainer.

Find out what the AI actually is

In practice, "AI chess training" almost always means an engine doing the evaluation and a language model writing the sentences around it. That split matters, because the two parts fail in completely different ways.

The engine part is reliable within its limits. Stockfish at a stated depth will tell you that 23.Rxd5 loses 180 centipawns and it will be right. The text part is where things go wrong. A language model asked to explain why 23.Rxd5 was bad will produce a fluent paragraph whether or not it has understood the position, and it can cheerfully describe a pin that isn't on the board.

So ask: which engine, what depth or node count, and is the prose generated from the engine's actual lines or written loosely around the evaluation? An app that names its engine and its depth is telling you something checkable. One that just says "advanced AI" is telling you nothing.

Opinion, but strongly held: engine numbers with no explanation are more useful than confident explanations that might be invented. If you have to choose, take the numbers.

Where does the analysis run

There are two honest answers, and they have different consequences. Either the engine runs locally in your browser through WebAssembly, or your positions are sent to a server that analyses them and sends results back.

Local analysis means your games never leave your machine for the purposes of evaluation, and it means the depth you get depends on your own CPU. Server-side analysis means somebody else's hardware, usually deeper, and a copy of your game data sitting in their infrastructure. Neither is wrong. But you should know which one you bought.

One practical tell: open the analysis page and turn off your network connection mid-run. If the evaluations keep appearing, the engine is on your side of the wire.

The permissions you should refuse

Your game archive on Chess.com and Lichess is public and available through their APIs. Reading your last 50 games does not require you to log in to anything or hand over a password. If an app asks for your Chess.com password, close the tab. There is no legitimate reason for it.

Lichess uses OAuth with granular scopes. An app that wants to read your games needs almost nothing. An app requesting board:play, challenge:write, or email:read is asking for the ability to move pieces in your live games, send challenges in your name, or collect your address. Read the scope screen instead of clicking through it.

Then the boring questions, which are the ones that actually bite:

• Can you delete your account and have the stored games and analysis go with it, not just the login?
• Is your game data used to train models, and can you opt out separately from using the product?
• Is anything shared with third parties beyond standard hosting and error logging?
• Does the app store your results under your chess username, which is often linkable to your real identity?
• If it emails you, is that transactional only, or is a marketing list the actual business model?

What the analysis has to show you to be worth anything

Most apps will hand you a single number: average centipawn loss. It is a bad summary, and here is the evidence from our own users' games. Across 267,842 engine-classified moves from 8,421 games analysed on Mated, the average move gives away 92 centipawns against the engine's best. The median is far lower, because a small number of very bad moves carry most of the damage.

That gap is the whole point. Your ACPL is not describing your typical move. It is describing three catastrophes averaged across forty decent ones. An app that shows you the average and stops has smoothed away the only thing you could have fixed.

The same data set gives the shape of those errors: 3.7% of moves are outright blunders and another 4.1% are mistakes, so roughly one move in 13 is a real error before you count inaccuracies, which run at 18.3%. And they are not spread evenly. In that sample, 8.9% of middlegame moves are a mistake or worse, against 2.3% in the endgame.

So when you test an app, look for whether it can tell you where in the game your errors live and what the distribution looks like, not just a mean. If all it can produce is one number per game and a green-to-red bar, you can get that from the free analysis on the site you already play on.

A puzzle set that never beats you is broken

Across 2,259 rated puzzle attempts on Mated, 94.2% were solved first time. That is our own number and it is too high. A set of problems pitched at your actual limit should be failing you regularly — that is what makes it diagnostic rather than decorative.

Use that as a test. Do twenty puzzles in whatever app you are considering. If you get nineteen or twenty, the difficulty calibration is flattering you, and you are practising things you can already do. Solving is pleasant. Failing, then seeing why, is the part that changes your results.

This is also the easiest thing for an app to get wrong in a way that feels good, which is why so many get it wrong. High solve rates make for happy reviews.

The ten-minute test

Take one of your own losses that you already understand. Pick a game where you know exactly what happened — you hung a knight on move 19, or you traded into a lost rook endgame. Run it through the app.

Three outcomes. It finds the move you know about and explains it in terms that match what happened: good sign. It finds the move but the explanation is vague or describes a different position: the engine layer works, the prose layer doesn't, and you should ignore the prose. It misses the move entirely or flags four other moves instead: the depth is too shallow to be useful, and no amount of interface will fix that.

Do this before you pay for anything. It takes one game.

What nobody has measured properly

Here is the honest gap. There is no good public evidence on whether drilling positions from your own games beats solving well-chosen puzzles that have nothing to do with you. It is a reasonable theory — your mistakes recur, so train the recurrence — and every app in this category is built on it, including ours. But it has not been tested at scale with a control group, and anyone telling you it is proven is guessing.

There is a second thing no engine can see: the clock. A blunder with 90 seconds left and a blunder with nine minutes left have the same centipawn loss and completely different causes. One is a time-management problem, the other is a chess problem. Almost no app distinguishes them, and if you review your games knowing the difference, you will get more out of the same data than the software will.

So keep some scepticism in reserve for the layer of explanation on top of the numbers. The engine evaluation is solid. Everything built above it, including the training plan, is a hypothesis about how you learn — and you are allowed to test it against your own rating over two months and drop it if nothing moves.