Secure Self-Hosted Runners for GitHub Actions Leveraging Amazon ECS

While there are plenty of options to choose from when it comes to executing deployment pipelines, GitHub Actions is incredibly popular due to its ease of use and convenient integration with GitHub repositories and features like pull requests and issues. In this blog, I will discuss the security concerns with GitHub repositories that make use of GitHub Actions and demonstrate how you can securely and cost-effectively manage your own self-hosted runners on Amazon Web Services (AWS). I will also provide you with an open-source repository you can use as is or as a starting point for your own needs.

In today’s world, supply chain security is of paramount importance for any organization developing any piece of software. All too often, the process is compromised in some way to allow attackers to inject malicious code, and this can happen at any step, from dependencies, to build, packaging, artifact storage, or deployment. Therefore, it is incumbent upon us as developers to ensure the integrity and security of that process from beginning to end, and this is why it is so critical to secure the infrastructure on which those processes run.

With GitHub-hosted runners, the infrastructure is largely handled for us, but we are still responsible for the secure configuration of the repository and the GitHub Actions workflow itself. When using self-hosted runners, we the developers, and our organization, take on the responsibility of securing those runners and their related infrastructure. We must ensure that attackers cannot gain access to the runner host, the network in which it runs, or the artifact storage that it uses. And in the case of self-hosted runners deployed within AWS, we must ensure that attackers cannot access the account in which they are deployed.

Of particular concern are public repositories using GitHub Actions to access or manage AWS resources. The potential risk here is that because the repository is public, anyone in the world can fork the repository, make changes, and submit a pull request. Without the proper settings configured on the repository, it would be possible for an attacker to use this mechanism to execute arbitrary code on the runner that compromises your application, or your AWS environment.

GitHub-Hosted or Self-Hosted Runners?

Like with anything else there are tradeoffs to be made here and pros and cons to be considered, especially in regards to cost, performance, and security. 

GitHub-Hosted Runners May Be Convenient

GitHub-hosted runners are convenient in that they offload the operational burdens of the infrastructure entirely, but that of course comes at a cost, measured in minutes per month, that for some organizations could be hard to justify. There are also performance considerations: some code bases may be slow to test, compile, and package on GitHub-provided runners, while other code bases may need access to specialized hardware or data that is only accessible within private networks.

Self-Hosted Runners Can Solve Some Challenges

Self-hosted runners can solve many of these challenges but at the cost of maintaining the infrastructure yourself. Smaller organizations may not have the cloud infrastructure experience to do so efficiently, securely, or cost-effectively. Fortunately, Amazon Elastic Container Service (Amazon ECS) using the AWS Fargate launch type can help here by providing container orchestration without the need to manage underlying servers. The underlying compute resources for these containers can be scaled both vertically and horizontally, providing a range of CPU and memory for test and build processes of varying size and complexity. Additionally, we can scale the number of runners, from one to hundreds, which can improve the efficient flow of pipeline executions your code base may need.

Access To AWS Resources

When building cloud-native applications that are being deployed to AWS, the runner itself often needs some level of AWS access. This of course means it needs AWS credentials, and those credentials need to be associated with a policy that grants least privilege, allowing the pipeline to perform its duties while also preventing unauthorized access or escalation.

With GitHub-hosted runners, the way to do this would be to use an OpenID Connect (OIDC) provider in AWS Identity and Access Management (IAM) so that GitHub Actions can securely retrieve temporary credentials. If you are currently using long-term credentials (access key and secret key) stored as GitHub secrets, I would strongly encourage you to refactor that design to use the more modern and secure approach of leveraging an OIDC identity provider.

With self-hosted runners, the way to grant access to AWS resources is to leverage IAM roles for Amazon ECS tasks. This provides the runners with temporary credentials that, again, should be scoped to a policy that provides least privilege access or provides a way for the role to assume other roles for performing other tasks, such as accessing data sets, uploading assets to S3, pushing docker images to an Elastic Container Registry (ECR) repository, or deploying artifacts to compute platforms.

Self-Hosted Runners Made Easy

While there are a number of pieces of infrastructure to manage, Amazon ECS/AWS Fargate can make it both easy to deploy and maintain over time. Let’s take a look at the following diagram and talk through each component:

1. Runners are deployed as a service into an Amazon ECS cluster and configured to use the AWS Fargate launch type inside a private subnet. This offloads the operational burdens of managing EC2 instances, and the private subnet prevents the runners from being accessible directly from the internet
2. Tasks in the service are associated with a task role which should grant least privilege for the operations a pipeline is expected to perform, or alternatively provide the ability for the pipeline to assume other roles to perform various operations
3. Containers pull their image from a private Amazon ECR repository in which a custom image is stored
4. Containers are configured to include a GitHub personal access token (PAT) from an AWS Secrets Manager secret. The PAT will be used to retrieve a runner registration token that the runner script will use to register itself with GitHub Actions
5. Once registered, the runners will then begin long-polling GitHub Actions API waiting for jobs to be picked up and executed

Conclusion

GitHub Actions has become a mainstay for development teams worldwide. For teams or organizations that require secure, self-hosted runners, the pattern described here aims to lower complexity, leverage security-best practices, and gives your developers a cloud-native, scalable, and easily maintainable solution. To that end, Aquia is releasing ready-to-deploy infrastructure as code for self-hosted runners here: https://GitHub.com/aquia-inc/GitHub-actions-iac. Please feel free to engage with us on our project’s issues section!

If you have any questions, or would like to discuss this topic in more detail, feel free to contact us and we would be happy to schedule some time to chat about how Aquia can help you and your organization.