GitHub Copilot: Exploring AI pair programming tools
As a digital services company at the forefront of government transformation, we’re always exploring how emerging technologies like AI can enhance our development capabilities.
This year we’ve been trialling GitHub Copilot, an AI pair programmer that suggests code as you type. In this blog, I’ll share our experience of using Copilot and how tools like these can accelerate software delivery.
What is GitHub Copilot and why use AI pair programming?
Copilot is a cloud-based AI trained on public GitHub code repositories. This training allows it to suggest code in all open source languages including Python, JavaScript, Java and Go, and also works well with markup formats used by Terraform, among others.
It’s similar to ChatGPT in using a large language model, but Copilot is specialised for code, while ChatGPT is used in a broader context.
As you write code, Copilot provides relevant suggestions to accelerate development. For example, Copilot can generate entire code blocks when given plain English descriptions of what the code should accomplish.
Zaizi started exploring large language models at the start of the year.
Our Design Authority evaluated different AI pair programming tools and selected GitHub Copilot for initial testing given its trusted reputation. GitHub’s use of open source means there should be no copyright issues with its code suggestions.
Importantly, Copilot operates securely— it doesn’t store code snippets or prompts from your editor. It’s essentially an experimental sandbox. As a company that works with secure public sector organisations, this was an important consideration.
We formed a group to conduct a three month Copilot trial across our internal projects and on some of our customer projects where we had obtained permission.
The key benefits
It’s fair to say that our engineers became fans of Copilot pretty quickly. Having an AI whispering code suggestions as you type is, not surprisingly, very useful!
Here are some of the key benefits we saw:
- By handling most rote coding tasks easily, it allowed us to spend less time on repetitive code patterns and more time solving higher-level problems.
- Copilot saved time by generating boilerplate code faster. Setting up standard config files and other repetitive tasks became easy.
- Copilot went beyond just helping you with code completion: with enough context it created a solid starting block for us to tweak and refine the code.
- We saw sizeable jumps in productivity across projects and it was helpful for many job roles – QA, backend, front end, cloud – and seniority levels.
The longer we used Copilot, the better it got. We learned how to give clear instructions to get useful code. By the end of our trial, the team gave feedback that they definitely wanted a tool like this as a permanent part of their toolkit.
Download ebook: How to overcome the inevitable challenges in government agile delivery
The pitfalls of Github Copilot
Copilot isn’t infallible, so oversight remains critical. Soon after starting the trial, we realised the need to provide some initial training on how to provide specific instructions to get maximum productivity gains.
We created guidance and terms of use stating that ultimately developers are accountable for the code, which must go through the standard review cycles and quality checks. For example:
- Copilot sometimes suggests code that seems solid but actually won’t work. We made sure to carefully review each recommendation before using it.
- Developers need to have a clear picture of what the output should look like in order to guide Copilot concisely. Left to its own devices, the solution to larger problems can be messy and lack many of the hallmarks of clear, well-structured code that developers would expect to see.
- If Copilot struggles then it can often be quicker for a developer to fix the problem rather than trying to provide more and more context to Copilot.
- Clear, specific prompting is crucial but takes practice. Without it, Copilot can produce code that doesn’t meet your needs.
Testimonials
Below are firsthand experiences from our developers who tested Copilot:
- “I have found that using co-pilot is beneficial when it’s provided with a clear statement, especially when it has code context to work with. However, if a statement is vague, then naturally, the code generation is hit and miss.”
- “It significantly accelerates my data collection process, allowing me to retrieve the necessary information in seconds, as opposed to sifting through AWS CLI documentation, which, in my personal opinion, is not always the quickest option. While it may not address complex issues, it represents a highly efficient use of my time.”
- “I wouldn’t say there are too many cons with Copilot. Sometimes a property doesn’t come up or an IAM service or action is hard to find, but as long as the engineer has a good understanding of what they want and understands the syntax well, their coding speed will significantly improve.
- “Copilot is a tool that can help with coding, but it is not perfect and should be used with caution. It can be a valuable addition to projects, but it is important to review its suggestions carefully.”
- “Copilot can help complete code quickly, but it’s not perfect. It can suggest inaccurate or buggy code, and it’s best used by experienced developers who can evaluate its suggestions. For beginners, Copilot may cause more problems than it solves.”
- “I liked to write my code as normal and then, if I saw a prompt that matched my intention, I’d just hit TAB and save myself quite a bit of time. As time went on, I learned to write slightly more documentation preceding my code than I might otherwise have, which got things moving along at a good pace.”
What’s the future?
The trial has convinced us that AI-powered coding is incredibly valuable. The productivity boost was significant while the cost was minimal.
We know this technology is evolving rapidly — Copilot is only a year old! To stay on the cutting edge, we’ll broaden our trials to other emerging tools.
In the long term, our goal is to integrate the best AI assistant into our workflows. As these AI pair programmers continue improving, they’ll become integral to efficient, high-quality software development.
And we can bring these benefits directly to our public sector clients. Beyond significant efficiency gains for our consultants, we can share our hands-on knowledge to help government teams take full advantage of AI’s potential.
Along with many other things, AI will transform the future of programming. Our team is excited to be leaders in leveraging these tools to accelerate digital change in the public sector.
If you’re exploring how emerging technologies like AI can transform service delivery, our team has the knowledge to guide you. Get in touch to discuss how we can help accelerate your digital transformation with AI pair programming.
-
My first Regional Scrum Gathering in Stockholm – key takeaways
-
Transformation Day – How do you fit a square peg in a round hole?
-
How product management improves public sector digital services
-
How a business analyst brings value to agile delivery in the public sector
-
Making AI simple: How it can quickly add value to border security
-
Service assessments: A welcome update – for government, and for suppliers
