ELEKS’ GitHub Copilot Investigation – Exploring the Potential of AI in Software Development

Although AI hasn't reached a stage where it can independently create, test, and launch software products without human input, compared to the past, it has greatly speeded up the overall turnaround time.

Many AI-driven tools are available that can assist in developing custom applications by generating and autocompleting code, as well as detecting and fixing bugs. The ELEKS Research & Development team thoroughly assessed one such tool - GitHub Copilot - to evaluate its influence on developers' tasks, completion duration, and quality standards for recommendations provided. Let's take a closer look.

What is GitHub Copilot?

GitHub Copilot is a tool for code autocompletion and generation. It is built based on OpenAI Codex and trained on public GitHub repositories. As a result, it suggests code for any programming language. However, the quality of proposals depends on the number of public repositories that are based on a specific programming language and framework.

GitHub Copilot works with program code using Integrated Development Environment (IDE) plugins. Only the following IDEs are currently supported:

• Visual Studio Code
• Visual Studio
• Neovim
• all IDEs from JetBrains

Overview of ELEKS' GitHub Copilot investigation

This investigation aimed to explore the impact of using Copilot on both development speed and quality.

To conduct this study, a small pet project was initiated as a testing ground for various assumptions. With a team comprising four skilled middle developers armed with expertise in React, Redux, TypeScript, Jest, Vite, PHP, Symfony, and Codeception, we started the investigation and defined our goals and approaches as follows.

Key findings from ELEKS' GitHub Copilot investigation

ELEKS' investigation of GitHub Copilot yielded several key findings that shed light on the tool's capabilities and benefits. Here is a more detailed description:

Please note that the investigation outcomes are based on the team's subjective feedback, observations of their work, and code reviews of their solutions.

The GitHub Copilot has two modes of operation:

1. Code auto-completion mode – almost always positively affects performance.
2. Code generation mode based on a comment with a prompt – works well only for typical code structures. It can be counterproductive when generating business logic or non-typical solutions because analysing the proposed code variants takes a lot of time.

The quality of GitHub Copilot code suggestions depends on the following:

• Popularity of the selected tech stack in public repositories. The more popular the technology or language, the higher the quality of GitHub Copilot suggestions.
• Code structure and application architecture. The more typical your solution is, the more effective suggestions GitHub Copilot provides. This is especially noticeable at the start of the project development.

Other advantages of GitHub Copilot:

• Over time, the quality of code suggestions increases. GitHub Copilot learns and adapts to the project structure, code style, and typical approaches to technical solutions on the project.
• Positively influences developer job satisfaction. GitHub Copilot takes on a portion of routine, typical, and boring tasks. The ratio between developing interesting and routine tasks changes, which increases developers’ comfort.

General team findings:

• The tool showed no impact on the code quality—neither positive, nor negative. We think that for now, it cannot replace the code review process, manual code testing done by the developer, running autotests, security testing, and so on.
• The effectiveness depends on the developer's competence in the given tech stack. The better the developer's skills, the more effective this tool proves to be.
• The value of GitHub Copilot for learning new technologies is very doubtful.
• The team formed an impression that code suggestions become better if all code-related files are open in the IDE.
• Like any other tool, GitHub Copilot requires certain skills. The more the team practiced, the more positive feedback was received.
• The feedback regarding increase in performance varies depending on the tech stack, specialist’s competency level, and subjective reasons.

On a side note, during the project, the team encountered several bugs in GitHub Copilot. For example, Copilot proposed code suggestions based on the code from another project open in another IDE window. It offered syntactically incorrect code. Copilot conflicts with the features of some plugins (built-in or added by third-party) that also have a code auto-completion functionality.

Conclusions

GitHub Copilot is a powerful tool that positively impacts development speed and developer job satisfaction. Like any other development tool, it requires time to master it.

At this point, an individual monthly license costs only $10, while the business plan is priced at $19 per month. GitHub Copilot will undoubtedly be financially beneficial for use on the projects, especially for the specialists of middle competency level and higher.

The team predicts that with the growth of experience using GitHub Copilot, the development speed can increase by 5–25%, depending on the technology's popularity and the developer's competency level. GitHub Copilot is guaranteed not to slow down development because developers will gradually gain the expertise and understanding of when it is beneficial to use the tool and when not.

In conclusion, the impact of AI in software development, exemplified by tools like GitHub Copilot, is undeniable. Overall, such tools as GitHub Copilot cannot replace a developer but serve as valuable tool to enhance developers' comfort and productivity, contributing to the evolution of software development in an increasingly AI-driven world.