Article

GitHub Copilot: A 55% Speed Boost in Development – Myth or Reality?

GitHub Copilot has become a topic of significant interest in the developer community due to its potential to accelerate development processes. This article presents a GitHub Copilot review and assesses its impact on enhancing development speed in real-life scenarios.

Since its launch, Copilot has sparked various discussions, ranging from enthusiasm and optimism to scepticism and caution about its potential impact on programming and the future role of developers. Its influence on different aspects of software development - from speed to code quality and learning - continues to be a subject of analysis and discussion.

GitHub Copilot users demonstrated an acceleration in task completion, achieving a 55% faster rate than developers who did not use the tool.

GitHub

Intrigued by the bold claims regarding the speed boost attributed to Copilot, we embarked on a journey to verify its effectiveness, particularly in the realm of AI in software development. In our pursuit of truth, we conducted our own testing of Copilot's use on real projects. To ensure optimal results, we took the following approach:

Varied project selection: We deliberately chose several projects with different tech stacks and architectural approaches, aiming to cover a wide range of use cases.
Diverse developer expertise: We enlisted developers with different levels of experience and competence to test the tool.

The key objective for the team was to conduct a GitHub Copilot review and assess its impact on coding productivity, identify its key influences, and find the most effective ways of using it. The testing period lasted three months to mitigate the potential bias influenced by a learning curve. Let’s dive into the outcomes.

GitHub Copilot review: How it impacts development speed

ELEKS team has recently conducted an in-depth investigation into GitHub Copilot, aiming to assess its impact on developers' tasks, completion duration, and the quality standards of the recommendations it provides. The findings of this investigation can be reviewed here: ELEKS’ GitHub Copilot Investigation – Exploring the Potential of AI in Software Development.

This study's key focus was to explore how the use of Copilot affects different types of projects. We tested and analysed the effectiveness of Copilot in monolithic applications and microservices in both backend and frontend applications to understand where this tool is most effective.

In broad terms, we can assert that the impact of Copilot on development speed is highly variable and depends on many factors. The following are key dependencies that emerge regarding the effective utilisation of Copilot:

1. Size of the existing codebase

Depending on the type of project and code structure, Copilot's impact on development speed varies: in frontend monolithic applications, we got approximately 20-25% development speed improvement; in backend monolithic applications - about 10-15% improvement, and in backend microservices - an average of 5-7%.

Ihor Mysak

Tech Lead at ELEKS

The verdict? Copilot thrives in projects with a large codebase, where it can support developers with existing templates and solutions. However, its prowess dwindles in the microservice realm, characterised by a small codebase. It indicates Copilot's ineffectiveness in projects that are just starting and do not yet contain enough developed solutions.

2. Technological stack

Testing Copilot on projects with different tech stacks showed a significant dependence on the quality of Copilot's suggestions based on the popularity of the technology.

React applications reveal a significant productivity surge, overshadowing the now outdated and less popular Zend framework.
.Net projects find themselves in the middle ground; performance was observed to be intermediate, better than with Zend but not as high as with React, suggesting a correlation with the relative popularity and volume of .Net materials available.

We believe this is because Copilot was trained on public GitHub repositories and had more training material for technologies that were more popular among developers.

3. Quality of code in the existing codebase

Copilot tends to generate higher-quality suggestions with proper and logical naming of variables and methods. Leading us to believe that quality naming helps Copilot better understand the context of the code, providing more accurate and useful suggestions.

Meanwhile, when the naming of variables and methods is unclear or ambiguous, Copilot has less information to analyse, which decreases the quality of its contribution to the development process. Thus, high-quality naming in code not only simplifies the work of programmers but also enhances the effectiveness of artificial intelligence tools.

4. Type of tasks performed by the developer

Despite its effectiveness in certain aspects of development, we also found that Copilot has limitations, especially when generating code that implements new business logic.

Copilot writes only the code according to the prompt, not complete solutions. Copilot is most effectively used for clear and template tasks. The time spent on a detailed description of business logic can outweigh the time needed to implement this business logic without using Copilot.

Ihor Mysak

Tech Lead at ELEKS

While effective in templated tasks, it struggles with the intricacies of new ideas or creative programming. The message is clear: Copilot is a developer's trusted companion in routine, but the realm of innovation demands the touch of human creativity.

Key tips regarding the effective use of Copilot:

Precision in the key: The more precise and detailed the prompt, the higher the likelihood of receiving a quality proposal from Copilot.
Context is everything: Avoid confusion by closing unrelated projects while using Copilot. If multiple projects are open in the IDE, Copilot can confuse contexts and generate suggestions for Project A based on the code of Project B.
Comments matter: Adding comments before creating a class or method enhances autocompletion quality.
File focus: Copilot is sensitive to the open tabs with project files, so one can artificially create a more targeted context for it.

Unleashing GitHub Copilot's potential: adaptability and indirect impact

Adaptability to project-specific environments

One of the most interesting characteristics of Copilot is its ability to adapt to the specifics of a particular project. Over time, Copilot "learns" the coding style and specific features of the project, leading to an improvement in the quality of its suggestions.

Initially, Copilot may provide generic or less precise solutions. However, as the tool accumulates more exposure and interaction within the project, the accuracy and relevance of its suggestions significantly improve. This is especially noticeable in projects with an established coding style and a large amount of existing code for Copilot to "train" on. This adaptability makes Copilot not only a tool for increasing efficiency but also a powerful aid in maintaining code consistency within a project.

Developers have also highlighted Copilot's positive influence on code complexity, noting a shift towards more readable and maintainable solutions, especially among those accustomed to crafting convoluted and intricate code structures.

Elevating automated testing

Copilot doesn't stop at coding; it has also mastered the art of automated testing. The tool offers templates and recommendations for potential test scenarios, allowing developers to save time and resources.

Copilot's ability to generate unique test cases that may not be obvious to developers is particularly valuable. It expands the testing coverage, improving the software product's examination depth.

Interestingly, the quality of tests created with Copilot is directly related to the quality and structure of the tested code. Our developers noted that the clarity of variable names, methods, and the overall structure of the code significantly affect the quality and accuracy of Copilot's test generation. Therefore, the effectiveness of utilising Copilot for writing unit tests depends on the tool itself and the quality of the tested code.

Overall, Copilot has proven to be a useful tool in the process of writing Unit Tests, enhancing not only the speed but also the quality of the final product.

Indirect impact

GitHub Copilot increases the coding speed and improves the overall nature of a developer's work. According to developers' feedback, Copilot allows them to shift focus from routine, time-consuming work to more creative and challenging tasks.

Additionally, Copilot can be an effective alternative to searching the Internet or documentation, reducing the time spent switching between different windows and allowing developers to concentrate on their current tasks. This feature is handy when needing to quickly find answers to questions without being distracted from the main work.

Copilot positively impacts the comfort and satisfaction of a developer. It streamlines getting answers to different questions and helps when there is no opportunity to turn to senior colleagues or find a solution on the Internet.

Olena Hladych

QA Lead at ELEKS

Interestingly, we found a correlation between the soft skills of developers and their satisfaction with using Copilot: developers with less developed communication skills often are less satisfied with its performance, possibly due to difficulties in precisely formulating prompts.

Conclusion

GitHub Copilot is a powerful tool that substantially enhances development productivity in specific scenarios, particularly during unit test composition and when navigating extensive codebases built on popular technologies. However, its efficacy faces constraints in tasks demanding innovative approaches or the creation of novel concepts.

Contrary to the claim suggesting a 55% boost in productivity, the actual outcome fell short. On average, teams experienced a moderate 10-15% improvement in productivity related to generating new code. However, it's essential to highlight various advantages attributed to Copilot utilisation. Overall, developers appraise Copilot as an invaluable tool that contributes significantly to development speed and fostering satisfaction among developers.

We recommend that teams and developers consider Copilot and approach it with an understanding of its potential limitations. The key to effectively using Copilot lies in understanding that it is an auxiliary tool, not a replacement for human intellect and creativity. It can enhance productivity and job satisfaction, reduce the time spent on routine aspects of development, and allow developers to focus on more complex and creative tasks.

Deliver your software project with top development experts at ELEKS.

Contact an expert

Application development

We’ll help you bring your most complex software vision to life with our leading full-cycle custom application development service. So you can focus on delivering an incredible user experience that sets you apart from the competition.

View service

Data science

Deep-dive into your data and boost business performance by understanding what your users really want.

View expertise

FAQs

What are the disadvantages of GitHub Copilot?

GitHub Copilot has limitations in generating innovative ideas and creatively approaching programming tasks. Its effectiveness diminishes when dealing with smaller codebases, which highlights Copilot's inefficiency in projects at their initial stages, lacking a sufficient number of established solutions.

Additionally, the tool may not consistently deliver accurate results, particularly when employed with less commonly used programming languages.

Can GitHub Copilot review code?

Is ChatGPT better than GitHub Copilot?

Will GitHub Copilot replace coders?