A safety net for
AI that writes code.
TDI watches an AI model while it writes code, catches the slips that matter, and has the model fix them quietly — before the answer reaches you. This page is the full story: what it does, what it has achieved, how good it really is, and where it falls short.
AI writes fluent code — and still makes small, confident mistakes.
It forgets a rule you gave it earlier. It leaves out one piece of what you asked for. It calls a function that sounds real and isn't. It says it added something it didn't. Each answer reads perfectly well, which is why people only find out later, when it has already cost them time.
A bigger model doesn't remove this. TDI is our answer: a separate layer that watches the model and catches these slips as they happen, without changing the model itself.
It watches for six kinds of slip.
Each one is checked against the code itself, not just by asking another AI whether the answer looks right.
Rules you gave
“Use requests, not httpx.” “Standard library only.” “Must run on Python 3.8.”
A few messages later the AI quietly breaks the rule. TDI catches it and reminds the model of your exact words.
Parts of your request left out
You ask for a function called load_config, a --verbose option, and exit code 2 on failure.
The answer looks complete but silently misses one. TDI checks the code for each named piece.
Functions that don't exist
A method on a library that sounds real and isn't.
It reads fine and fails later. TDI checks the names against the real library before you ever run it.
Code that doesn't run
A stray bracket, a misplaced default, a keyword used as a name.
TDI catches code that cannot even load, and the model fixes it first.
Claims the code doesn't back up
“I also added tests for the parser” — and there are none.
TDI compares what the AI says it did with what the code actually contains.
Rules in plain words
In progress“Scores above 90 get an A.”
Checking behaviour rules is the weakest part and still in progress. We list it as such below.
How the fix reaches you. When TDI spots a slip it sends the model a short note you never see, quoting your own words. The model redoes its answer. You get the corrected version, with nothing to click through. Before any note is sent, a final double-check asks whether it is really needed, because a checker that cries wolf is worse than none.
How we got here, including the parts that failed.
- Late September
The idea
AI coding assistants write fluent code and still slip: they forget a rule you gave, drop part of a request, call functions that don't exist. We asked whether a small, separate layer could watch the model and catch those slips as they happen — without changing the model.
- Late September – early October
Many false starts
We built and tested dozens of small approaches. Several failed outright, and we kept the failures. The lesson: one general “is this answer okay?” checker is not reliable. Checking for each specific kind of slip, directly against the code, is.
- 6–7 October
The first checks pass
Broken instructions and broken code reached our bar on data we had held back. A few days later the check for made-up functions did too, at 98.4% — it had been stuck at 92%.
- 8 October
The first full-system test
All checks running together on fresh, never-seen test data: 98.4% of mistakes caught across nearly 10,000 cases, with wrong alarms around 1 in 500. Longer multi-message conversations were the weak spot and stayed on the to-do list.
- 12–14 October
Bigger, harder tests
More conversations, new subject areas, and a stricter standard. We found the system was sometimes “correcting” answers that were fine, so we added a final double-check before any reminder is sent. It cut wrong reminders from 3.3% to 0.4% in our test. The latest full test is below, including the lines we missed.
Strong where we've measured it.
Headline results from our first full-system test (8 October): every check running together on test data nobody had looked at, with each disputed result checked by hand. These are our own tests, not independent ones.
9,616 of 9,774 test cases — our first full-system test, new data
2 in 950 answers — TDI rarely “fixes” what was fine
222 of 228 — against 36% when the model is simply asked again
no correct answer was damaged by a reminder in the sample we checked
Result by kind of slip.
“Real mistakes” are errors the AI actually made. “Planted” means we took a correct answer and deliberately introduced the slip, so we know it is there.
Overall across these six rows: 9,616 of 9,774 caught (98.4%; we are 95% sure the true figure is between 98.1% and 98.6%). After a reminder, 97.4% of the AI's own mistakes were fixed, against 35.5% when the model was simply asked to try again.
14 October: the whole system, on new data — including what we missed.
98 new conversations across 36 subject areas the system had never seen, plus 46 behaviour tests. We set the pass lines before running it and did not change them afterwards.
Five of seven lines met their target; two did not. We publish both.
So how good is TDI?
- Strong: it catches almost all of the mistakes it is built for — about 96–99% depending on the kind — and fixes nearly all of what it catches, with very few wrong alarms.
- Weaker: rules written in plain words, and the wrong-reminder rate, which sits just outside our own limit.
- Not measured at all: other languages, other models, real users, and independent verification.
Wherever AI writes code that people depend on.
TDI is proven on Python code from an AI coding assistant. These are the settings where that matters most. We say plainly which ones we have tested and which are still targets.
AI coding assistants
Products where a model writes code for users and one invented function or dropped requirement costs real time. TDI is built for exactly this.
Teams with strict stack rules
Fintech, health software, government and enterprise teams that set rules like “standard library only”, “no pandas” or “must run on Python 3.8” and need the AI to keep them across a long session.
Unattended coding agents
Agents that write and change code for many steps without a person watching, where an early slip compounds. A quiet watcher that corrects as it goes is the point.
Data and automation scripts
Analysts and operations teams generating Python scripts they may not read closely: reports, file processing, scheduled jobs.
Teaching and training
Coding tutors and exercise platforms where the AI must follow the exercise's rules and not hand over code that doesn't run.
What you should know before relying on any of this.
- Python code only. Other languages are not covered yet.
- Tested on one open-source language model. We expect it to carry over to others, but we have not shown it.
- The test conversations were generated by us and every disputed result was checked by hand by us. Nobody independent has verified them yet.
- No real users yet. Everything is measured on test data, never on live traffic.
- It costs more to run. TDI adds about 10 extra model calls per answer, so it is slower and dearer than the bare model. We have not solved speed.
- Rules in plain words (such as grade boundaries) are below our own bar at 88%.
- Wrong reminders are low (1.0%) but our latest test could not rule out up to 2.6%, above our 2% limit.
What's next: tighten the wrong-reminder rate, lift the plain-word rules past 90%, test on other models and languages, make it cheaper to run, and get the results checked by someone outside 26io. TDI is not a product and isn't available to use yet. We'll publish more as results earn it, and say so plainly when they don't.
Building with AI that writes code?
If a slip costs you time or money, we'd like to hear how. Write to us directly.