Researchers at Microsoft studied the use of command-line AI coding tools and their impact on productivity among tens of thousands of the company’s software engineers. The tools were Anthropic’s “Claude Code” and GitHub’s “Copilot CLI,” and the study covered about four months, from January 5 to April 29, 2026. Engineers who used a tool during its first week of rollout submitted an average of 24% more pull requests that were merged than those who did not, the researchers found.
Simply providing the tools won’t make them catch on; the first step starts with “the next desk over”
Many executives and managers may imagine that if a company makes an AI tool available to everyone, they’ll all start using it the next day. But this study suggests that workplace relationships—not feature descriptions or announcements—are what determine whether someone gives it a first try.
People were more likely to start using a tool if colleagues who regularly shared code with them, or their direct manager, were already using it. The figures are specific: when more than a quarter of colleagues who shared the same manager as their manager had used a tool in the past two weeks, the odds of trying it rose by as much as 216%. The odds rose by 82% when a direct manager was using it, and by 54% when more than a quarter of the people they regularly exchanged code reviews with were using it.
An increase in odds is not the same as an “increase in probability”
One thing to keep in mind here is the word “odds.” Odds express the likelihood of an event occurring relative to the likelihood of it not occurring. A 216% increase may sound like the number of users more than tripled, but it does not mean that the share of people using the tool rose by 216%. When usage is initially low, the difference can look large, and the actual increase in users will be smaller than this figure suggests.
Even so, the general pattern is clear: managers and colleagues can encourage people to give a tool a try. Rather than reading a manual and getting started, people often think, “If that person is using it, I will too.” This is probably familiar in many workplaces, not just software development. A manager takes the lead in trying a tool and demonstrates how to use it during a review. Such low-key actions may be more effective than training materials.
Whether people keep using a tool depends on their everyday workflow
Whether people who try a tool continue using it is another matter. The study found that continued use was associated with how much coding people did in their regular work. Engineers who wrote a lot of code and submitted pull requests frequently were more likely to see the benefits of AI and make the tool a regular part of their work, the researchers said.
A pull request is a request to have rewritten code incorporated into the main codebase. People who submit them frequently may have more tasks they can delegate to AI. The more hands-on work someone does, the easier it is to feel the tool’s effects.
A familiar environment can make adopting a new tool harder
The study also found that people who had already been using existing AI tools in a different development interface were willing to try the new tools but less likely to stick with them. They may simply find their familiar environment more comfortable.
It’s a bit like switching smartphones. Even if a new phone has better features, the app layout and controls may be so ingrained that you want to go back to your old device. Those introducing a new AI tool may need to consider not just its performance, but also how it fits into the workflows users have already established.
How to interpret the 24% increase in pull requests
The impact on productivity was measured by the number of merged pull requests, a proxy for completed work. Engineers who used a tool during its first week of rollout submitted an average of 24% more merged pull requests than those who did not. The researchers reported that this effect remained stable throughout the four-month study, rather than being a temporary result of initial excitement.
However, productivity here is measured by just one metric: the number of pull requests. This figure does not show that code quality, time spent per request, or the team’s workload improved by the same proportion. It is worth distinguishing the fact that the number went up from the feeling that work as a whole became easier.
Claude Code: 11.4%; Copilot CLI: 24.9%
By tool, the increase in pull requests compared with weeks when no tool was used was 11.4% for Claude Code users and 24.9% for Copilot CLI users. The difference is roughly twofold, but it would be premature to read these figures as a direct comparison of the tools’ merits. Copilot CLI is provided by GitHub, a Microsoft subsidiary, and the tools’ usage patterns and user populations within the company may not be the same.
Even so, few studies have tracked both the spread of adoption and its outcomes over four months in real-world development settings involving tens of thousands of people. While discussions of AI coding tend to focus on “which model is smarter,” these findings offer a look at the realities of adoption: who starts using the tools and who keeps using them. We may be entering an era in which the ability to shape how tools spread in the workplace matters just as much as choosing the tools themselves in determining outcomes.
Comments