Can you maintain code implemented by AI without relying on the AI that wrote it? Developer Emmimal Alexander has released V0.1 of “CodeOwnership,” a tool designed to test that question.
In the first trial, the author rated their understanding a 4 out of 5, but gave up after 8 minutes and 12 seconds without finding a bug planted by another AI, despite a 60-minute time limit.
If you can’t fix it, it isn’t “your code,” even if your name is on the commit
All the tests passed, and the build succeeded. Hearing that, many people would feel reassured and hit the merge button. But three weeks later, a bug report comes in, and the AI assistant isn’t available. Maybe you’ve used up your quota, or you’re on a commuter train with no signal. Can you then read and fix the code in front of you on your own? That is the one thing CodeOwnership aims to measure.
“Ownership” here has nothing to do with copyright or licensing. It refers to the practical ability to follow the logic of the code and make changes without AI assistance. Passing tests may prove that software works under favorable conditions, but it doesn’t prove that you understand what’s inside. That distinction is the starting point.
It’s like a test that turns off your GPS
Here’s how it works. First, an AI agent implements a feature. The developer then records how well they think they understand it on a five-point scale. Next, another agent plants a single bug in accordance with a specification registered in advance. Finally, with AI turned off, the developer uses a bug report to investigate the cause.
Three things are recorded: the self-rating before the task, whether the developer can debug without AI (with a 60-minute limit), and a four-question comprehension check afterward. These are reported separately rather than combined into a single score. Combining them would obscure the gap between thinking you understand something and what happens when you actually work with it.
The author compares it to a driver who always relies on navigation. Ask, “Do you know the way home?” and they’ll say yes. Only when they turn off their phone do you get the real answer. CodeOwnership, you could say, is a tool for turning off that phone.
Measures to ensure fairness
The author is both the tool’s designer and a test subject, so it includes a number of measures to reduce bias. The rules file is locked by a hash—a kind of fingerprint that changes if the contents change—and execution stops if the file has been modified. The idea is similar to preregistering a plan in research.
The bug is planted by a separate agent, not the author. The author knows what behavior is broken but does not see the diff showing what was changed. The working copy used for debugging contains no Git history, specification, hidden tests, or scoring criteria. The self-rating must be committed before the bug is introduced, and the written explanation before the score is displayed. There are 72 self-tests to check that these constraints work as intended.
Confidence: 4; bug not found
The test project is a note library with folders. It supports creating and retrieving notes, creating and deleting folders, moving notes, and listing the contents of a folder. A note can also be placed in multiple folders. The key rule is that deleting a folder leaves the notes intact; only their associations with that folder are removed.
ChatGPT (free account, Auto model, temporary chat) handled the implementation. All 11 public tests and 13 private acceptance tests passed, and the baseline check was successful. After reading the code, the author rated their understanding a 4 out of 5 and recorded it in Git before the bug was introduced. A separate Claude session then made a change based only on the specification, and the behavior check passed as well.
Eight minutes and 12 seconds into debugging, the author stopped without finding the cause. They said they were tired and frustrated after problems with the preparations the day before. The tool offered only two options: “Fixed” or “Time limit reached.” To move on, the author chose “Fixed,” but made no changes to the code; the objective check marked it “Not fixed.” All 11 public tests passed, while 10 of the 13 private tests passed and three failed. The four-question comprehension check was also not completed, so the report recorded the result as “Incomplete.”
The planted bug was just two lines
The change revealed later consisted of two lines added to the folder-deletion logic: if a note’s list of associated folders was empty, delete the note from storage. That was all. Notes in multiple folders were safe because one folder association remained, but a note belonging to only one folder disappeared along with that folder.
At first glance, the logic seems sound: the list is empty, so delete it. But having no owner is not the same as a note being unnecessary. The author later reflected that asking, “Where does responsibility for deleting notes sit in this app?” might have offered a clue. The test was not just whether they could follow the code, but whether they could picture how responsibilities were divided.
What one attempt can tell us
The results need to be interpreted with caution. The test involved just one developer, one task, and one debugging attempt that was stopped partway through. The author says the experiment cannot establish whether using AI leads people to understand code less deeply. The time limit was 60 minutes, and this attempt cannot tell us whether the bug would have been found if the debugging had continued.
The author says they later spent about 30 minutes finding the cause. But this was self-reported, the time was not measured, and the fix was not tested, so it was not counted as a result. The author also notes that, as the tool’s designer, they already knew what could break; that the effects of fatigue and setup problems could not be separated; that the test deviated from the original plan to use an agent running in the working folder; and that the combination of tools changed midway through. The hidden tests and scoring criteria have been published in the repository for reproducibility, which means anyone who reads them before trying the test cannot preserve the blind conditions.
Even so, this one attempt is useful in a particular way. A confidence rating of 4 with all tests passing is not necessarily the same as being able to fix the problem unaided. That is probably the most honest way to read it.
Clues from related research
The author cites several studies as background to the question. None of them validates this particular tool; they are presented as context for the underlying concern.
In a study published by METR in July 2025, 16 experienced open-source developers used AI tools on 246 real issues in their own repositories. The result: it took them 19% longer to complete the work. Beforehand, the developers expected to be 24% faster, and afterward they felt they had been 20% faster. The study measured time spent on tasks, not understanding of the code. METR has since characterized the findings as outdated, and said in February 2026 that it would revisit the experimental design.
A study published by Anthropic in January 2026 involved 52 engineers learning Trio, a Python library; most were relatively inexperienced developers. On a follow-up quiz, the AI-assisted group scored an average of 50%, compared with 67% for the group that coded by hand—a statistically significant difference. The AI group finished about two minutes sooner, but that time difference was not significant. Meanwhile, asking the AI for explanations or follow-up information was associated with better learning outcomes. How people use AI may matter more than whether they use it at all. It is also worth noting that the author says they use Claude regularly and used it to write this article.
The practice automation takes away
The author also draws on classic work. In a 1983 paper, “Ironies of Automation,” Lisanne Bainbridge argued that as automation advances, people have fewer opportunities to practice, and that shortfall becomes consequential when they have to take over during an abnormal situation. The relationship between an airplane’s autopilot and the pilot’s skills is a classic example.
In 1985, Peter Naur wrote in “Programming as Theory Building” that programming is not merely writing code, but an effort to build an understanding—a theory—of how software works. In a workflow where an agent writes and a human only approves, how much of that understanding remains with the human? The question also connects to “vibe coding,” a term Andrej Karpathy used in February 2025 for a style of accepting output without reading much of it. That may be convenient for a throwaway weekend project, but what about software that needs to be maintained?
A simple drill you can try today
You can practice the same idea without installing the full version. Pick a small piece of code you had AI write, and first rate on a five-point scale: “If this broke tonight, could I fix it?” Next, ask AI in a separate session to change one behavior, without looking at the diff. Start a timer, close the AI session, and try to fix the code using only a report of the symptoms. Before checking the answer, write down the cause in your own words. Finally, compare your initial self-rating with the result.
Sometimes you may be confident but fail to find the bug; other times, you may fix it sooner than expected. Neither outcome would have been apparent from passing tests alone. For those who want to try the full version, the instructions are to run the repository’s self-test, followed by the freeze and pilot commands.
The author draws four lessons: assess yourself before you begin; understand that passing tests and being able to fix the code are two different things; practice before something really breaks; and ask AI to explain the code. An Anthropic study also found that participants who asked for explanations or follow-up questions performed better. However, the authors described this as an association, not a causal relationship.
Changes in V0.2
The author says the next version will make four changes. First, it will add an option to accurately record when someone “gave up.” It will display users’ answers alongside the results of objective checks, rather than forcing them to choose between “fixed it” and “time ran out.” Second, debugging will take place after a break, so fatigue from the setup work does not affect the results. Third, the project will return to the original plan of using an agent that runs in the project folder and can run tests on its own. Fourth, other developers will be asked to try the same process. Results from just one person cannot show how well the tool works for others.
Comments