In April 2023, Rezonate research team explored prevalent misconfigurations of GitHub integration with cloud native vendors. GitHub OIDC-based trust relations have been found with the critical misconfigurations that leave connected AWS/GCP accounts vulnerable to potential takeover attacks. Although this issue was discovered and reported in the past, we have found that dozens of GitHub Public Repositories, and potentially many private ones, have demonstrated this critical issue, leaving dozens of companies vulnerable. The team notified recognizable affected organizations, but there may be more private repositories and organizations at risk.
In this article, we will introduce the misconfiguration research process, explain the OIDC implementation process that GitHub uses to authenticate to the cloud, present the misconfigurations that we have identified across various organizations, provide a step-by-step guide for discovering and fixing the problem, and propose how to avoid the issue completely.
Key Points
- Rezonate research team has explored an attack vector that exposed AWS & GCP accounts to unauthorized access through misconfigured OpenID (OIDC) GitHub Actions role.
- Based on our scans, these known misconfigurations exist in dozens of the GitHub public repositories that use Github OIDC provider to AWS or GCP.
- During the last few weeks, the Rezonate team has reached out to a few dozens of vulnerable organizations and personal accounts, providing them with information in order to remediate this misconfiguration.
- To check whether your GCP or AWS accounts have been affected by this misconfiguration, please refer to the “Remediation Guidelines†section.
Background
Cloud and SaaS technologies have made identity and access controls the connective tissue between operational tools and cloud-native environments.
To ensure business operations flow as seamlessly and as fast as possible, IAM and Devops teams are required to share administrative access and create trust relations between 3rd-party tools and their core cloud environments. The key challenge is to limit privileges and conditions on these cases, and to protect and monitor this highly-privileged administrative access.
One of the most common examples of trust relations can be found in a software CI/CD pipeline, where strong privileges and access to an organization’s Cloud infrastructure are provided. We’ve seen dozens of cases where this access path was exploited due to misconfigurations, lost credentials and access keys, and compromised supply chains such as the CircleCI incident, the NPM incident, and many others.
Due to the highly-sensitive nature of these access paths, CI/CD vendors have added support for the OpenID Connect (OIDC) protocol to supply short-term credentials, which are considered more secure. The trend toward OIDC support is reflected through the announcements of GitHub, GitLab, and CircleCI within the last 2 years.
OIDC is a modern, secure authentication method for creating trust relationships, but when not configured correctly, the method can become the root cause of a hidden, critical access risk.
Even though these misconfigurations were reported in the past, in books, as well as in security blogs, the Rezonate research team has identified many organizations that remain vulnerable.
GitHub OpenID Provider Integration
The GitHub OIDC deployment guide highlights many advantages to using OIDC over access keys. OIDC allows you to adopt good security practices, such as:
- No cloud secrets: Eliminating the need to store cloud credentials as long-lived secrets. Instead, workflows will request and use a short-lived access token from the cloud provider through OIDC.
- Authentication and Authorization management: Having granular control over how workflows can use credentials, using the cloud provider’s authentication and authorization tools to control access to cloud resources.
- Rotating credentials: With OIDC, the cloud provider issues a short-lived access token that is only valid for a single job, and then automatically expires.
The following diagram provides an overview of how GitHub’s OIDC provider integrates with GitHub Actions and the cloud provider:
In general, integration involves the following steps:
- Start by creating an Identity Provider in the target cloud environment.
- GitHub Actions will generate an OIDC temporary token containing execution context (such as the repository, organization, or branch).
- GitHub will send a request to the cloud provider, containing the GitHub OIDC token.
- The cloud provider will validate the token and the context, and exchange it with short-lived credentials that allow access to the cloud environment.
GitHub OIDC Integration with GCP & AWS
Integrating GitHub Actions through OIDC includes setting up trust between the cloud infrastructure and GitHub, which workflows can then use to access roles and service accounts.
Configure Trust
The process of configuring trust between the cloud provider and GitHub is similar between different cloud providers, and includes two main steps:
- Create an OIDC Identity Provider
- Create a Role or Service account and establish trust with the recently created Identity Provider.
Let’s examine the setup for GCP and AWS.
Configure Trust with GCP
Based on Google Cloud documentation, the following process creates a Workload Identity Federation, which provides the issuer ID and basic attribute mapping.
Optionally, after configuring the attribute mapping according to the Google Cloud documentation, add Attribute Conditions such as a GitHub organization, repository, or branch.
Now that we have the Identity Provider configured, we can bind it to service accounts and allow GitHub to use them as part of its workflow. In the picture below we can see the github-actions-integration service account connected to the Identity Provider recently created.
Configure Trust with AWS
Similar to GCP, AWS starts with creating a new Identity Provider (OpenID Connect) with the GitHub Provider URL.
Next, we can create an IAM Role and reference the recently created Identity Provider, allowing users federated by the Identity Provider to assume roles in the account.
Next, we name the role, add a description, and modify the access conditions of the role. By default the trust policy condition only includes audience filters.
As mentioned in the GitHub integration documentation, it’s highly recommended that you limit access to specific repositories by adding filters against the token subject. Note that by default the AWS process does not enforce adding additional conditions. To add conditions, click the “Edit Policy†button, which reveals the following warning:
Configure GitHub Workflow
Now that everything is configured from the cloud infrastructure side, the next step is to use the relevant GitHub actions as part of the workflow.
- First, add permissions to the workflow, allowing it to read an OIDC token. Add the following permissions to the beginning of the workflow YAML file:
- Next, add the matching action to the authentication process, based on the cloud provider:
For AWS, use the configure-aws-credentials:
- For GCP, use the google-github-action/auth:
After performing the steps above, the integration process is completed and the workflow can perform operations in the cloud environment.
Potential Misconfiguration
Although the integration between the cloud and GitHub is relatively simple, potential misconfigurations could expose access to unauthorized parties.
GitHub articles refer to Conditional access as part of the integration process, however, the cloud infrastructure is not flagging or alerting when conditional access is not being used.
The lack of notification from the cloud provider during the setup increases the chances that an organization will have misconfigured trust settings. Those misconfigurations may result in roles and service accounts that can be abused by threat actors/attackers, leading to unauthorized access.
As a result of our research, we have discovered two specific types of misconfigurations that can be exploited by attackers to gain unauthorized access to a trusted cloud account.
Misconfiguration – Lack of Subject Condition
This misconfiguration occurs when the user who integrated the role did not add conditional limitations to cloud access. This misconfiguration allows any GitHub organization to access the cloud account.
Misconfiguration – Poorly-Defined Condition Pattern
This misconfiguration occurs when the conditions defined limit the subject, but are not restrictive enough and thus can be bypassed.
For example, the subject condition may include a wildcard to allow all the repositories in the organization to use the same role or service account as part of the GitHub Actions workflow. If the wildcard is mistakenly placed in the organization name (and not in the repository name), the condition allows unauthorized access from any GitHub organization with a name that fits into the pattern.
Identifying Vulnerable Organizations
To Identify vulnerable organizations and understand how prevalent misconfigurations are, we used GitHub search, looking for Actions workflows that use OIDC with AWS or GCP.
Using GitHub Code Search, we executed the following queries:
We split the query into subqueries to avoid reaching GitHub’s result limit, and eventually produced a list of roles and accounts that were potentially vulnerable.
Having the list, the next step was to identify what was misconfigured.
GitHub actions query returned thousands of results, and checking them through GitHub actions would be time-consuming. As a workaround, the team developed a different approach to identify vulnerable organizations using self-hosted runners.
Self-Hosted Runners
Github Self-Hosted Runners is a feature that allows organizations to execute parts of the GitHub Actions workflows in their own environment. The team discovered that by controlling the machine that authenticates against the cloud, we can extract the OIDC token, and perform batch tests on AWS roles and GCP service accounts outside of GitHub.
The batch tests included the following steps:
- Setup a self-hosted runner:
Download and install the GitHub self-hosted runner:
- Configure the runner to proxy all of the traffic through our local proxy:
Use the http-proxy parameter within the GitHub action.
- Intercept a workflow request and extract the Web Identity Token:
After everything was ready, we triggered the workflow by performing a commit to the repository. The commit started a GitHub workflow which performed a network request to assume the target role with the Web Identity token. Then, we extracted the token from the request sent by the runner to AWS.
Decoding the JWT Token reveals our sub, along with other identifiers that are being sent and logged by the cloud provider as part of the authentication process.
- Copy the token to our multithreaded script, which will check the potentially vulnerable roles.
Using a pre-developed script and leveraging Boto3 and Gcloud, we scanned a list of potentially vulnerable roles while having the JWT at hand.
Results
After reducing duplicate roles and service accounts, we scanned approximately 1500 roles and service accounts across GCP and AWS.
As for the first misconfiguration, lack of subject condition, we determined that dozens of the roles were vulnerable, allowing anyone to use them to access the accounts.
The second misconfiguration, poorly-defined condition, was harder to test and required setting up a dedicated GitHub organization per target. Thus, we performed a random test against 20 organizations, finding one of them to be vulnerable to this attack.
The vulnerable accounts included various organizations, including private companies, non-profit organizations, software development studios, and many personal accounts.
During the last few weeks we have reached out to the relevant organizations and people, sending them information regarding the vulnerable roles and sharing remediation guidance.
Remediation Guidelines
Could My Environment Be Affected?
Based on our research, this type of misconfiguration is a risk for AWS and GCP users who use GitHub Actions with modern authentication (OIDC).
Identify Potential Misconfigurations: Lack of Subject Conditions
To identify this misconfiguration In AWS, check for roles that have no subject limitations in its trust policy.
To identify this misconfiguration in GCP, check for service accounts connected to the Identity Provider that has access to the Entire Pool.
Identify Potential Misconfigurations: Poorly-Defined Conditions
To identify this misconfiguration in AWS, locate the StringLike conditions for roles that are connected to the Identity Provider, and check for wildcards in the organization name.
For example, in this case, we have a role that was originally intended to be used by different repositories within the Rezonate organization. Since the StringLike conditions included a wildcard in the organization name, every organization that starts with “rezonate†can assume this role. This means that an attacker can create a new GitHub organization with the name “rezonateX†and get access to the role.
This misconfiguration option does not apply to GCP, as GCP does not support wildcards.
Identity Risks in the Rezonate UI
Rezonate customers should look for the security risk “GitHub Actions Role vulnerable to hijacking†in the Rezonate UI and follow the guided remediation steps attached to it.
Use a Script to Perform a Quick Scan
For the general public, we have released a script to our GitHub repository (link here), which performs a quick scan against the AWS account or GCP project and reveals possible vulnerable roles and service accounts.
