Generative AI for SDLC: Top Areas for a CTO/VPE to Pay Attention To

by Alex KhizhniakFebruary 8, 2024

While analysts suggest artificial intelligence can speed up coding by 10–20–50%, real-life productivity gains vary among scenarios/expertise/etc., studies reveal.

(Featured image credit)

Table of Contents

We still lack enough data on AI

Since the launch of ChatGPT and code generation tools like GitHub Copilot, a lot of predictions have been made regarding productivity improvements in software engineering. Some analysts and vendors expect that the AI revolution will cut time on development by 2x, while others believe that engineers might be replaced completely—sooner or later.

Anyway, most reports give readers only a fragmented view on the use of generative AI throughout the stages of the software development life cycle (SDLC). From a CTO/VPE perspective, there are still many uncertainties about where to automate and how.

With all the potential that the generative AI (GenAI) brings in, what are the real productivity gains across different types of engineering activities? How much, indeed, engineers save with code generation assistants? What areas of the software delivery process are already being gradually automated by AI? What would engineers delegate to GenAI and what are the concerns for QA, DevOps, cybersecurity, etc.?

Let’s figure it out.

The promise of AI for coding

In July 2023, a group of experts from IBM introduced a study quantifying possible productivity gains for engineers of different seniority (from juniors to architects). In brief, the research forecasts that, by the 5th year, total team’s productivity can increase by 58%, with all members surpassing 50% gains.

(source: IBM)

Improved productivity will eventually have an impact on the final outcomes—including products, processes, and timelines.

Jerry Cuomo

“We estimate that generative AI can lead to a 15–20% increase in the number of new products or features developed, a 10–15% reduction in the number of bugs found in production, a 5–10% increase in customer retention and loyalty, and a 10–15% reduction in time-to-market for new products or features.”
—Jerry Cuomo, IBM Fellow, VP/CTO of Technology

(source: IBM)

Chris Gardner

Similarly, Forrester estimates that developers/coders can gain 20–50% productivity by adopting GenAI, while testers can become more efficient by 15–20% (October 2023). According to the prediction, “all members of product teams will gain above 10% efficiency” by using assistive AI tools for planning and delivery.

“Some have seen productivity rise to 200% and more when experienced engineers use genAI to work with languages or libraries outside their normal zone of development.“ —Chris Gardner, VP and Research Director, Forrester

Other industry reports provided estimates similar to the forecasts described in this section. However, after having AI tools deployed and adopted, real figures may differ, more or less. During 2022–2023, some reports confirmed predictions, while other evaluations were a bit more modest.

Wait, but what are the real productivity gains?

A study among 40 McKinsey developers published in June 2023 revealed tremendous productivity gains when using AI tools. Still, the exact figures varied across types of tasks, as it was observed by other similar research papers. For instance, automation of delivering documentation was done by 2x faster, with 20–45% improvements tracked for code generation and refactoring.

(source: McKinsey)

Experts at McKinsey also noted that “in some cases, tasks took junior developers 7–10% longer with the tools than without them.” Nevertheless, developers with AI tools were 25–30% more likely to complete those tasks, rather than engineers without assistance.

In some scenarios, developers used more than a single AI tool. In this case, they gained additional time improvements of 1.5–2.5x.

(source: McKinsey)

It should be noted that, during McKinsey’s study, generative AI tools occasionally provided incorrect recommendations and introduced code errors. The authors of the report also admitted the existing AI tools are not able to know the specific context, which may influence integrability or security requirements to the code. “Combining multiple frameworks with disparate code logic” may be an issue.

In its turn, GitHub—which is behind one of the top AI assistance tools for engineers—regularly conducts its own evaluations of developers’ productivity. In October 2023, GitHub measured that code reviews with its Copilot Chat were completed by engineers 15% faster. (The tool provides answers about the code and its quality right from an IDE.)

At the same time, developers do not accept anything AI suggests. Another study by GitHub held in March/June 2023 found that an average acceptance rate was ~30%. The rate increased as an expert became more familiar with Copilot.

(source: GitHub)

The study by GitHub also revealed that junior developers were more likely to accept AI-based recommendations, while the rate among senior engineers was lower.

Previous experiments by GitHub discovered that engineers using Copilot had a higher rate of task completion (78% vs. 70%, September 2022). Besides, a survey held between May and June 2022 discovered a 55.8% reduction in task completion time, on average.

(source: GitHub)

In July 2022, Google also revealed that implementing ML-driven assistance for the company’s engineers led to reducing coding iteration time by 6% (i.e., shrinking “time between builds and tests”).

CI/CD vendor Harness evaluated that its customer’s developers made 10.6% more pull requests when using GitHub Copilot, as well as spent 3.5 hours less on a task, iterating more often (September 2023).

David Walker

The Westpac bank operating in Australia achieved a 46% productivity gain when exploring generative AI tools from Microsoft, Amazon, and OpenAI (June 2023).

“When we looked at the time that it took the hand-coding team to actually complete their tasks, on average it was 3.5x longer than it took the other teams to do their tasks.” —David Walker, CTO, Westpac

In May–June 2023, a mobile team of a software company located in Romania observed a 35% improvement in technical onboarding task time. The engineers also spent more than 40% less time on moderate and complex tasks when using AI tools, such as ChatGPT 4.0 and GitHub Copilot.

(source: Technical University of Cluj-Napoca, Romania)

Generative AI across SDLC

Now that we know how much productivity AI is already helping to achieve, let’s have a closer look at some of the specific stages of SDLC where this happens.

AI in DevOps and security

In a recent study, “The State of AI in 2024,” Dynatrace discovered that technology leaders adopting artificial intelligence expect it to become critical for the core DevOps activities, such as:

threat detection, investigation, and response (82%)
automating complex operations (63%)
eliminating false alerts and the manual effort of validating code deployments (58%)

While increasing investments in AI-driven issue prevention/remediation, 87% of IT operations/DevOps leaders believe this effort is “critical to managing multicloud complexity.”

Besides, 95% of the participants are concerned that generative AI could lead to data leakage and improper/illegal use of intellectual property. (See the press release for details.) The opinions among developers are similar. More than a half (59%) have security concerns in this regard, according to a report by JetBrains issued in November 2023.

(source: JetBrains)

At the same time, according to a recent Gartner’s poll held in October 2023, 78% of executives “believe that the benefits of generative AI outweigh its risks.”

A developer’s view on AI

In JetBrains’s report, the developers also shared their thoughts on the types of activities they’d delegate to an AI assistant. Comments/documentation and testing are the top things that most developers would entrust to a robot, while writing code is the core responsibility that engineers would prefer doing by themselves.

(source: JetBrains)

Still, 28% of developers in JetBrains’s survey noted that their companies’ policies limit the use of cloud-based AI tools.

GenAI in QA and testing

A survey by LambdaTest (held in July–October 2023) revealed that the top three GenAI use cases in QA are as follows:

test data generation (50.5%)
test case creation (45.9%)
analytics and reporting (35.6%)

At the same time, 25.6% of QA engineers believe that AI will “bridge the gap between manual and automated testing.” Besides, 60.3% of respondents see the reliability of AI platforms as the most significant concern for QA.

(source: LambdaTest)

Budgets vs. ongoing learning

IDC expects (October 2023) that G2000 companies will allocate more than 40% of their IT budgets to AI-related initiatives by 2025. At the same time, through 2026, 65% of enterprises may fail in unleashing the full potential of AI due to “IT skills mismatch” caused by underinvested training/education.

“As the industry moves forward with this fundamental transition to AI embedded into every business and technology function in the enterprise, IDC believes that every CEO will need to have an AI strategy—and generative AI is the trigger…It is best to get started quickly.” —Phil Carter, IDC (source)

Marci Maddox

In addition to establishing an environment where engineers can learn and grow their skills, things like security and risk control should also matter.

“As industries transition to use GenAI to automate or augment every process, project, or interaction, the measure of success will be in improving productivity and reducing the level of effort to complete the task in a secure, compliant, and trustworthy manner.” —Marci Maddox, IDC (source)

McKinsey recommends fostering ongoing learning and practical knowledge sharing among engineers. The company’s report noted that this approach helped to improve the quality of prompts among participants of the company’s study.

At the same time, according to GitHub, more than a half of developers (57%) believe that AI tools can help them acquire new coding skills (March/June 2023). So, a company adopting AI needs to scale this by providing proper assistance, environment, culture, etc.

Gartner predicts that, by 2025, 30% of enterprises will have AI-assisted development and testing implemented. By 2026, the layouts/interfaces/etc. may be designed by generative AI for 60% of new websites and mobile apps. By 2027, ~15% of new software will be built solely by AI, the analysts expect.

It is clear that the time for AI adoption for software engineering is now.