A recent study conducted by the Bank for International Settlements (BIS) explored the impact of artificial intelligence on programmer productivity within Ant Group. This research highlighted a significant increase in the volume of code produced by developers who interacted with a Large Language Model (LLM).
The study aimed to empirically assess how AI can enhance productivity in tasks that require cognitive skills. Following the launch of CodeFuse, an LLM tailored to assist programming teams, the field experiment divided programmers into two groups: one with access to CodeFuse (the treatment group) and another without (the control group).
The results were striking, revealing a 55% increase in productivity for the group utilizing the LLM. Notably, about one-third of this productivity surge was directly linked to the code generated by the model. The remaining gains were attributed to enhanced efficiency in other coding tasks, allowing programmers to allocate more time to their work.
Interestingly, these productivity enhancements were predominantly substantial among junior programmers, with senior team members experiencing a less pronounced impact. This disparity appears to arise from the lower engagement levels of senior programmers with the LLM, rather than a lack of utility in the tool itself. The acceptance rate of the LLM’s suggestions did not differ significantly across levels of experience, indicating that senior programmers simply used the tool less often.
The BIS research team concluded, "While senior programmers who utilized generative AI found the suggestions valuable and engaging, the overall usage of the tool was lower among them. This indicates that generative AI can indeed boost productivity, but its adoption varies by experience level, underscoring the need for tailored strategies to optimize its application across different tiers of expertise."
For those interested in exploring the evolving challenges and opportunities that artificial intelligence presents to the banking sector, consider joining Finextra’s inaugural NextGenAI conference on November 26, 2024. Register your interest here.