ELEKS’ GitHub Copilot Investigation – Exploring the Potential of AI in Software Development

Key findings from ELEKS' GitHub Copilot investigation

ELEKS' investigation of GitHub Copilot yielded several key findings that shed light on the tool's capabilities and benefits. Here is a more detailed description:

Please note that the investigation outcomes are based on the team's subjective feedback, observations of their work, and code reviews of their solutions.

The GitHub Copilot has two modes of operation:

1. Code auto-completion mode – almost always positively affects performance.
2. Code generation mode based on a comment with a prompt – works well only for typical code structures. It can be counterproductive when generating business logic or non-typical solutions because analysing the proposed code variants takes a lot of time.

The quality of GitHub Copilot code suggestions depends on the following:

• Popularity of the selected tech stack in public repositories. The more popular the technology or language, the higher the quality of GitHub Copilot suggestions.
• Code structure and application architecture. The more typical your solution is, the more effective suggestions GitHub Copilot provides. This is especially noticeable at the start of the project development.

Other advantages of GitHub Copilot:

• Over time, the quality of code suggestions increases. GitHub Copilot learns and adapts to the project structure, code style, and typical approaches to technical solutions on the project.
• Positively influences developer job satisfaction. GitHub Copilot takes on a portion of routine, typical, and boring tasks. The ratio between developing interesting and routine tasks changes, which increases developers’ comfort.

General team findings:

• The tool showed no impact on the code quality—neither positive, nor negative. We think that for now, it cannot replace the code review process, manual code testing done by the developer, running autotests, security testing, and so on.
• The effectiveness depends on the developer's competence in the given tech stack. The better the developer's skills, the more effective this tool proves to be.
• The value of GitHub Copilot for learning new technologies is very doubtful.
• The team formed an impression that code suggestions become better if all code-related files are open in the IDE.
• Like any other tool, GitHub Copilot requires certain skills. The more the team practiced, the more positive feedback was received.
• The feedback regarding increase in performance varies depending on the tech stack, specialist’s competency level, and subjective reasons.