GitHub Copilot: Pilot test and initial results

Copilot is one of Microsoft’s big bets in the field of generative AI business tools.

To test its real results in companies, at Plain Concepts we have decided to carry out a pilot test on a small group of employees, but it is starting to present some interesting first conclusions that we will corroborate in the following phases of the analysis. Here are the first steps!

Introduction to Copilot

GitHub Copilot acts as an intelligent assistant designed to help developers create code. It works through auto-completion, meaning that when a user starts writing code, GitHub Copilot makes suggestions on how that completed code would look.

It works using the LLM language developed by OpenAI, combining the main functions of ChatGPT and options related to searching for information on the web. This also allows it to evolve and update the answers.

One of the great promises of this assistant is to provide intelligent assistance in real-time, allowing users to improve their creativity, productivity, and capabilities. Let’s see how it is working!

Testing GitHub Copilot

In testing the impact of GitHub Copilot on the Plain Concepts experimental group, we have created 4 distinct groups:

• Unusual technology: Projects where we work with technologies that are not the usual ones in Plain Concepts and that are a challenge for us due to our lack of experience in them.
• A long-term project with a stable team and known technologies: The key in these projects is that we have objective metrics to compare the before and after of using GitHub Copilot. We will go into more detail later on what kind of metrics to use.
• People with little experience: People with little work experience who need regular support from more experienced profiles. The profiles have been supervised by more experienced peers during the experiment to help us validate the result.
• People whose judgment we trust: People who have been with us for a long time and have demonstrated good judgment on successive occasions in various scenarios.

Setting up the experiment

For this first phase of the experiment, 12 people were divided into the four groups mentioned above. In order to do this, representation from multiple units of the company was sought, as well as representation from the main work areas (Web and Mobile Enterprise Application Development, Data & Analytics, and Artificial Intelligence).

The metrics measured were as follows:

• More objective:
  • Velocity
  • Lead Time
  • Cycle Time
  • Code Coverage
  • Error Rates
  • Versatility
  • Time required from more experienced profiles.
  • Defect Density
• More subjective:
  • To be able to successfully execute a project would have been a challenge for us because of working with technologies where we have no experience.
  • Time required from more experienced profiles
  • Time required to start up projects with the minimum technical quality in the usual technologies.
  • Technical quality in the newest areas, such as IA

The participants of the first phase of the GitHub Copilot experiment have received training, intending to make them understand the use cases, how to use it, and so on. Therefore, at the end of this session, GitHub Copilot should not be something new to the participants, and they should be able to take full advantage of it.

The idea is to extend the experiment to bring out new metrics and use cases, if any, to achieve a gradual and constant implementation.

First overall conclusions

After analyzing the results of this first pilot test, the conclusions we have been able to draw are that the areas where GitHub Copilot has been shown to have a positive impact:

• Projects with unknown technologies (different from our usual technology stack).
• More mid-levels of experience, including projects with our usual stack (C#, TypeScript).
  • Improved versatility, autonomy, and quality
• Artificial Intelligence and Data and Analytics areas
• In general, improvement in the metrics of “Error Rates” and “Defect Density”

Main benefits highlighted in using GitHub Copilot

Referring to an Arxiv article, the main conclusion of the paper is that 55.8% of the programmers participating in the study completed tasks faster than the control group. Thus, the observed heterogeneous effects are promising for developers using AI tools to help people transition into software development careers.

In terms of our test, we have detected the following benefits:

• Increased development speed: GitHub Copilot is recognized as helping, in several cases, to speed up the software development process.
• Reduced bugs: Several participants mentioned that GitHub Copilot helps to reduce bugs and defects in the generated code.
• Increased developer autonomy: It is highlighted that GitHub Copilot allows developers to work more independently and with less need for external help.
• Improved code quality: The tool facilitates the writing of cleaner and more readable code, which contributes to higher software quality.
• Optimization of documentation and comments: Some highlight that GitHub Copilot helps in the generation of documentation and comments in the code, speeding up this process.
• Increased developer versatility and autonomy: It is mentioned that GitHub Copilot allows developers to tackle a wider variety of tasks and work in different areas without problems.

Programming languages have been useful for

• Python
  • FastAPI
• Android
  • Jetpack Compose
  • Custom integrations (React Native context)
• iOS
  • Objetive-C (React Native context)
  • Swift (React Native context)
• PowerShell
• JavaScript
• TypeScript
  • React Native
  • React
• HTML
• CSS
• SQL
• Bash
• C#
• Go
• Scala

Areas and projects where we have been able to improve the quality of delivery thanks to GitHub Copilot

• Artificial Intelligence (AI):
  • The quality of the code has been improved. We can compare the quality of the code before and after the use of Copilot, and we can see that the data is improved. The naming of variables, functions, classes, etc., has been improved. Automated testing has been increased and coverage has been improved. The documentation has been extended with examples and comments that fit the context.
  • The experiment follows the best practices and standards proposed by Plain Concepts, and using GitHub Copilot has been of great help in following the proposed best practices and standards.
• Data & Analytics:
  • Optimization of processes that read and write data.
  • Creation of unit tests in Data and Analytics, an area where quality has now been improved.

Next Steps

Over the next few months, we would like to extend the experiment to a larger group of employees to draw clearer conclusions about where GitHub Copilot is most useful and where it can significantly improve results.

Also, if you need a partner to help you discover the full potential of GitHub Copilot, Plain Concepts makes it easy for you.

• We are the first partner in Spain accredited by GitHub.
• We have been working for more than 17 years in the Agile culture, a benchmark in the DevOps community.
• We have a team of more than 350 senior engineers specialized in App Innovation and DevOps.
• AMMP accredited.
• DevSecOps with MVPs.

In addition, we don’t stop at certifications and we offer you an exclusive GitHub Adoption Framework so that you can find the service that best suits your needs, from the best experts. Contact us to find out more!