In traditional software development, you know when a feature is done. Someone writes down what it should do, you build it, and you test it against that. The button opens the right page. The form saves the data. If something doesn't work as expected, you fix it and run the tests again, and once they pass, you move on. We still work that way on a lot of Ari. The pages, the chat window, the login, and the progress screens all have to be designed, coded, and tested the usual way.

As Ari grew, we noticed that the most important parts of it don't fit that process. Ari is a tutor, an assessor, and a coach, though those roles didn't arrive together. The tutor and the assessor came first, and the coach came later, when we saw that learners needed help deciding what to study and when. What we need to know is whether it teaches well, whether it judges a learner's progress fairly, and whether its advice about what to study next actually helps. You can't write a test for that and watch it pass. A reply can come back on time, with no errors and perfect formatting, and still be bad teaching. That was an interesting realization for us as developers. Passing tests stopped meaning a feature was done, and we needed a new way to look at assessment.

Part of the reason is that so much of an AI application lives in its system prompts. A prompt is part of the code, but it doesn't behave like code. You change a few sentences, and the model may or may not do what you meant. Sometimes it does something you never thought to check for. Each new role we added to Ari brought more prompts with it, and more places where things could quietly go wrong. Once an AI application gets to that size, developing it without a way to evaluate it is really hard.

So evaluation became part of how we develop. It started as a nightly job that went over the day's activity. Over time it grew into an application where we run a team of AI agents, each with its own role. Every day, agents check the quality of the tutor, the assessor, and the coach. They also look at the whole experience: whether the screens and the conversation fit together, and whether the thing is actually helpful. They flag anything unhelpful, and anything incorrect or harmful a learner might read. We look at the same activity by day, by week, and by month, because each view shows something different, and we try to find how they connect. Our volunteers' feedback goes into the same picture. What comes out of all that goes back into the next round of changes, and the cycle starts again.

One example from early September shows why. We built a detector for learners who game the system by clicking through answers without really trying, so their progress scores wouldn't overstate what they knew. It worked exactly as designed. Before turning it on, we ran it over real past sessions, and it flagged 22.6% of them. That felt wrong, so we read the sessions it had flagged. Most were students writing "I don't know" or "I'm not sure" when Ari asked them to explain something, or giving a short right answer like "51" and then "ok" when Ari moved on. That's what normal tutoring looks like. A student who says "I don't know" isn't gaming anything. They're struggling, which is the situation Ari is built to help with. There was nothing wrong with the code. The mistake was in how we had defined gaming in the first place, so we rebuilt the detector around the difference between a learner who won't engage and a learner who can't.

An earlier one, from late August, came from a daily review. In long sessions, Ari could only see the most recent part of the conversation, so it lost the beginning, where the student had explained what they were stuck on. We fixed that by keeping a short summary of the middle of the conversation. The fix passed its tests and went live. About two weeks later, the daily review pointed out that our summary only remembered what Ari had done. The student's side was gone, including the questions Ari had already asked, the answers the student got right, and the mistakes they made earlier in the session. That meant Ari could ask the same question twice, re-teach something the student already understood, or treat an old misconception like a new one. Nothing errored, and every reply looked fine on its own. The problem only showed when you read the whole conversation, the way a learner experiences it.

The evaluation itself needs checking too. One report told us the average session lasted 1.0 minute, for sessions that averaged more than 12 messages, which isn't possible. Following that number led us to a timing bug that recorded the first exchange of every session as taking zero seconds. The bug mattered beyond the report, because Ari also used that timing to sense, in the moment, whether a student was doing well, struggling, or drifting away. None of our tests had caught it, and we only found it because the number in the report made no sense.

If you're building anything with AI, you need a new way to look at assessment, and it has to be built into development itself rather than added at the end. The old way still matters. You still test that the buttons work and the site stays up, and page views and clicks still tell you whether people showed up. None of that tells you whether the AI did its job well for the person on the other end, or told them something wrong. To know that, you have to read what your AI actually said, keep checking it after it ships, and check the checks too. It took us a good while of building Ari to see that.