Since its launch, Copilot has sparked various discussions, ranging from enthusiasm and optimism to scepticism and caution about its potential impact on programming and the future role of developers. Its influence on different aspects of software development - from speed to code quality and learning - continues to be a subject of analysis and discussion.
GitHub Copilot users demonstrated an acceleration in task completion, achieving a 55% faster rate than developers who did not use the tool.
Intrigued by the bold claims regarding the speed boost attributed to Copilot, we embarked on a journey to verify its effectiveness, particularly in the realm of AI in software development. In our pursuit of truth, we conducted our own testing of Copilot's use on real projects. To ensure optimal results, we took the following approach:
The key objective for the team was to conduct a GitHub Copilot review and assess its impact on coding productivity, identify its key influences, and find the most effective ways of using it. The testing period lasted three months to mitigate the potential bias influenced by a learning curve. Let’s dive into the outcomes.
ELEKS team has recently conducted an in-depth investigation into GitHub Copilot, aiming to assess its impact on developers' tasks, completion duration, and the quality standards of the recommendations it provides. The findings of this investigation can be reviewed here: ELEKS’ GitHub Copilot Investigation – Exploring the Potential of AI in Software Development.
This study's key focus was to explore how the use of Copilot affects different types of projects. We tested and analysed the effectiveness of Copilot in monolithic applications and microservices in both backend and frontend applications to understand where this tool is most effective.
In broad terms, we can assert that the impact of Copilot on development speed is highly variable and depends on many factors. The following are key dependencies that emerge regarding the effective utilisation of Copilot:
Depending on the type of project and code structure, Copilot's impact on development speed varies: in frontend monolithic applications, we got approximately 20-25% development speed improvement; in backend monolithic applications - about 10-15% improvement, and in backend microservices - an average of 5-7%.
The verdict? Copilot thrives in projects with a large codebase, where it can support developers with existing templates and solutions. However, its prowess dwindles in the microservice realm, characterised by a small codebase. It indicates Copilot's ineffectiveness in projects that are just starting and do not yet contain enough developed solutions.
Testing Copilot on projects with different tech stacks showed a significant dependence on the quality of Copilot's suggestions based on the popularity of the technology.
We believe this is because Copilot was trained on public GitHub repositories and had more training material for technologies that were more popular among developers.
Copilot tends to generate higher-quality suggestions with proper and logical naming of variables and methods. Leading us to believe that quality naming helps Copilot better understand the context of the code, providing more accurate and useful suggestions.
Meanwhile, when the naming of variables and methods is unclear or ambiguous, Copilot has less information to analyse, which decreases the quality of its contribution to the development process. Thus, high-quality naming in code not only simplifies the work of programmers but also enhances the effectiveness of artificial intelligence tools.
Despite its effectiveness in certain aspects of development, we also found that Copilot has limitations, especially when generating code that implements new business logic.
Copilot writes only the code according to the prompt, not complete solutions. Copilot is most effectively used for clear and template tasks. The time spent on a detailed description of business logic can outweigh the time needed to implement this business logic without using Copilot.
While effective in templated tasks, it struggles with the intricacies of new ideas or creative programming. The message is clear: Copilot is a developer's trusted companion in routine, but the realm of innovation demands the touch of human creativity.
One of the most interesting characteristics of Copilot is its ability to adapt to the specifics of a particular project. Over time, Copilot "learns" the coding style and specific features of the project, leading to an improvement in the quality of its suggestions.
Initially, Copilot may provide generic or less precise solutions. However, as the tool accumulates more exposure and interaction within the project, the accuracy and relevance of its suggestions significantly improve. This is especially noticeable in projects with an established coding style and a large amount of existing code for Copilot to "train" on. This adaptability makes Copilot not only a tool for increasing efficiency but also a powerful aid in maintaining code consistency within a project.
Developers have also highlighted Copilot's positive influence on code complexity, noting a shift towards more readable and maintainable solutions, especially among those accustomed to crafting convoluted and intricate code structures.
Copilot doesn't stop at coding; it has also mastered the art of automated testing. The tool offers templates and recommendations for potential test scenarios, allowing developers to save time and resources.
Copilot's ability to generate unique test cases that may not be obvious to developers is particularly valuable. It expands the testing coverage, improving the software product's examination depth.
Interestingly, the quality of tests created with Copilot is directly related to the quality and structure of the tested code. Our developers noted that the clarity of variable names, methods, and the overall structure of the code significantly affect the quality and accuracy of Copilot's test generation. Therefore, the effectiveness of utilising Copilot for writing unit tests depends on the tool itself and the quality of the tested code.
Overall, Copilot has proven to be a useful tool in the process of writing Unit Tests, enhancing not only the speed but also the quality of the final product.
GitHub Copilot increases the coding speed and improves the overall nature of a developer's work. According to developers' feedback, Copilot allows them to shift focus from routine, time-consuming work to more creative and challenging tasks.
Additionally, Copilot can be an effective alternative to searching the Internet or documentation, reducing the time spent switching between different windows and allowing developers to concentrate on their current tasks. This feature is handy when needing to quickly find answers to questions without being distracted from the main work.
Copilot positively impacts the comfort and satisfaction of a developer. It streamlines getting answers to different questions and helps when there is no opportunity to turn to senior colleagues or find a solution on the Internet.
Interestingly, we found a correlation between the soft skills of developers and their satisfaction with using Copilot: developers with less developed communication skills often are less satisfied with its performance, possibly due to difficulties in precisely formulating prompts.
GitHub Copilot is a powerful tool that substantially enhances development productivity in specific scenarios, particularly during unit test composition and when navigating extensive codebases built on popular technologies. However, its efficacy faces constraints in tasks demanding innovative approaches or the creation of novel concepts.
Contrary to the claim suggesting a 55% boost in productivity, the actual outcome fell short. On average, teams experienced a moderate 10-15% improvement in productivity related to generating new code. However, it's essential to highlight various advantages attributed to Copilot utilisation. Overall, developers appraise Copilot as an invaluable tool that contributes significantly to development speed and fostering satisfaction among developers.
We recommend that teams and developers consider Copilot and approach it with an understanding of its potential limitations. The key to effectively using Copilot lies in understanding that it is an auxiliary tool, not a replacement for human intellect and creativity. It can enhance productivity and job satisfaction, reduce the time spent on routine aspects of development, and allow developers to focus on more complex and creative tasks.
The breadth of knowledge and understanding that ELEKS has within its walls allows us to leverage that expertise to make superior deliverables for our customers. When you work with ELEKS, you are working with the top 1% of the aptitude and engineering excellence of the whole country.
Right from the start, we really liked ELEKS’ commitment and engagement. They came to us with their best people to try to understand our context, our business idea, and developed the first prototype with us. They were very professional and very customer oriented. I think, without ELEKS it probably would not have been possible to have such a successful product in such a short period of time.
ELEKS has been involved in the development of a number of our consumer-facing websites and mobile applications that allow our customers to easily track their shipments, get the information they need as well as stay in touch with us. We’ve appreciated the level of ELEKS’ expertise, responsiveness and attention to details.