Rezonate is now available in AWS Marketplace

Go back

From GitHub to Account Takeover: Misconfigured Actions Place GCP & AWS Accounts at Risk

Rezonate
May 16, 2023
GitHub Account Takeover of AWS and GCP Accounts

Contents

In April 2023, Rezonate research team explored prevalent misconfigurations of GitHub integration with cloud native vendors. GitHub OIDC-based trust relations have been found with the critical misconfigurations that leave connected AWS/GCP accounts vulnerable to potential takeover attacks. Although this issue was discovered and reported in the past, we have found that dozens of GitHub Public Repositories, and potentially many private ones, have demonstrated this critical issue, leaving dozens of companies vulnerable. The team notified recognizable affected organizations, but there may be more private repositories and organizations at risk.

In this article, we will introduce the misconfiguration research process,  explain the OIDC implementation process that GitHub uses to authenticate to the cloud, present the misconfigurations that we have identified across various organizations, provide a step-by-step guide for discovering and fixing the problem, and propose how to avoid the issue completely.

Key Points

  • Rezonate research team has explored an attack vector that exposed AWS & GCP accounts to unauthorized access through misconfigured OpenID (OIDC) GitHub Actions role.
  • Based on our scans, these known misconfigurations exist in dozens of the GitHub public repositories that use Github OIDC provider to AWS or GCP.
  • During the last few weeks, the Rezonate team has reached out to a few dozens of vulnerable organizations and personal accounts, providing them with information in order to remediate this misconfiguration.
  • To check whether your GCP or AWS accounts have been affected by this misconfiguration, please refer to the “Remediation Guidelines” section.

Background

Cloud and SaaS technologies have made identity and access controls the connective tissue between operational tools and cloud-native environments.

To ensure business operations flow as seamlessly and as fast as possible, IAM and Devops teams are required to share administrative access and create trust relations between 3rd-party tools and their core cloud environments. The key challenge is to limit privileges and conditions on these cases, and to protect and monitor this highly-privileged administrative access.

One of the most common examples of trust relations can be found in a software CI/CD pipeline, where strong privileges and access to an organization’s Cloud infrastructure are provided. We’ve seen dozens of cases where this access path was exploited due to misconfigurations, lost credentials and access keys, and compromised supply chains such as the CircleCI incident, the NPM incident, and many others.

Figure 1: Attacker scans for misconfigured trust relations in GitHub, and uses them to impersonate a CI/CD app, gaining full access to AWS or GCP accounts.

Due to the highly-sensitive nature of these access paths, CI/CD vendors have added support for the OpenID Connect (OIDC) protocol to supply short-term credentials, which are considered more secure. The trend toward OIDC support is reflected through the announcements of GitHub, GitLab, and CircleCI within the last 2 years.

OIDC is a modern, secure authentication method for creating trust relationships, but when not configured correctly, the method can become the root cause of a hidden, critical access risk.

Even though these misconfigurations were reported in the past, in books, as well as in security blogs, the Rezonate research team has identified many organizations that remain vulnerable. 

GitHub OpenID Provider Integration

The GitHub OIDC deployment guide highlights many advantages to using OIDC over access keys. OIDC allows you to adopt good security practices, such as:

  • No cloud secrets: Eliminating the need to store cloud credentials as long-lived secrets. Instead, workflows will request and use a short-lived access token from the cloud provider through OIDC.
  • Authentication and Authorization management: Having granular control over how workflows can use credentials, using the cloud provider’s authentication and authorization tools to control access to cloud resources.
  • Rotating credentials: With OIDC, the cloud provider issues a short-lived access token that is only valid for a single job, and then automatically expires.

The following diagram provides an overview of how GitHub’s OIDC provider integrates with GitHub Actions and the cloud provider:

Figure 2: Overview of GitHub OIDC provider integration

In general, integration involves the following steps:

  • Start by creating an Identity Provider in the target cloud environment.
  • GitHub Actions will generate an OIDC temporary token containing execution context (such as the repository, organization, or branch).
  • GitHub will send a request to the cloud provider, containing the GitHub OIDC token.
  • The cloud provider will validate the token and the context, and exchange it with short-lived credentials that allow access to the cloud environment.

GitHub OIDC Integration with GCP & AWS

Integrating GitHub Actions through OIDC includes setting up trust between the cloud infrastructure and GitHub, which workflows can then use to access roles and service accounts.

Configure Trust

The process of configuring trust between the cloud provider and GitHub is similar between different cloud providers, and includes two main steps:

Let’s examine the setup for GCP and AWS.

Configure Trust with GCP

Based on Google Cloud documentation, the following process creates a Workload Identity Federation, which provides the issuer ID and basic attribute mapping.

Figure 3: Creating a Workload Identity Federation

Optionally, after configuring the attribute mapping according to the Google Cloud documentation, add Attribute Conditions such as a GitHub organization, repository, or branch.

Figure 5: Adding Attribute Conditions

Now that we have the Identity Provider configured, we can bind it to service accounts and allow GitHub to use them as part of its workflow. In the picture below we can see the github-actions-integration service account connected to the Identity Provider recently created. 

Figure 5: Bind service accounts to Identity Provider

Configure Trust with AWS

Similar to GCP, AWS starts with creating a new Identity Provider (OpenID Connect) with the GitHub Provider URL.

Figure 6: Create an Identity Provider in AWS

Next, we can create an IAM Role and reference the recently created Identity Provider, allowing users federated by the Identity Provider to assume roles in the account.

Figure 7: Create an IAM Role

Next, we name the role, add a description, and modify the access conditions of the role. By default the trust policy condition only includes audience filters.

Figure 8: Modify access conditions using filters

As mentioned in the GitHub integration documentation, it’s highly recommended that you limit access to specific repositories by adding filters against the token subject. Note that by default the AWS process does not enforce adding additional conditions. To add conditions, click the “Edit Policy” button, which reveals the following warning:

Figure 9: Missing additional conditions warning in AWS

Configure GitHub Workflow

Now that everything is configured from the cloud infrastructure side, the next step is to use the relevant GitHub actions as part of the workflow.

  • First, add permissions to the workflow, allowing it to read an OIDC token. Add the following permissions to the beginning of the workflow YAML file:
Figure 10: Add permissions to GitHub workflow
  • Next, add the matching action to the authentication process, based on the cloud provider:
    For AWS, use the configure-aws-credentials:
Figure 11: Add matching action to authentication process – AWS
Figure 12: Add matching action to authentication process – GCP

After performing the steps above, the integration process is completed and the workflow can perform operations in the cloud environment.

Potential Misconfiguration

Although the integration between the cloud and GitHub is relatively simple, potential misconfigurations could expose access to unauthorized parties.

GitHub articles refer to Conditional access as part of the integration process, however, the cloud infrastructure is not flagging or alerting when conditional access is not being used.

The lack of notification from the cloud provider during the setup increases the chances that an organization will have misconfigured trust settings. Those misconfigurations may result in roles and service accounts that can be abused by threat actors/attackers, leading to unauthorized access.

As a result of our research, we have discovered two specific types of misconfigurations that can be exploited by attackers to gain unauthorized access to a trusted cloud account.

Misconfiguration – Lack of Subject Condition

This misconfiguration occurs when the user who integrated the role did not add conditional limitations to cloud access. This misconfiguration allows any GitHub organization to access the cloud account.

Misconfiguration – Poorly-Defined Condition Pattern

This misconfiguration occurs when the conditions defined limit the subject, but are not restrictive enough and thus can be bypassed.

For example, the subject condition may include a wildcard to allow all the repositories in the organization to use the same role or service account as part of the GitHub Actions workflow. If the wildcard is mistakenly placed in the organization name (and not in the repository name), the condition allows unauthorized access from any GitHub organization with a name that fits into the pattern.

Identifying Vulnerable Organizations

To Identify vulnerable organizations and understand how prevalent misconfigurations are, we used GitHub search, looking for Actions workflows that use OIDC with AWS or GCP.

Using GitHub Code Search, we executed the following queries:

Figure 13: GitHub Code Query – AWS OIDC Workflows
Figure 14: GitHub Code Query – GCP OIDC Workflows

We split the query into subqueries to avoid reaching GitHub’s result limit, and eventually produced a list of roles and accounts that were potentially vulnerable.

Having the list, the next step was to identify what was misconfigured.

GitHub actions query returned thousands of results, and checking them through GitHub actions would be time-consuming. As a workaround, the team developed a different approach to identify vulnerable organizations using self-hosted runners.

Self-Hosted Runners

Github Self-Hosted Runners is a feature that allows organizations to execute parts of the GitHub Actions workflows in their own environment. The team discovered that by controlling the machine that authenticates against the cloud, we can extract the OIDC token, and perform batch tests on AWS roles and GCP service accounts outside of GitHub.

The batch tests included the following steps:

  • Setup a self-hosted runner:
    Download and install the GitHub self-hosted runner:
Figure 15: Our self-hosted runner
  • Configure the runner to proxy all of the traffic through our local proxy:
    Use the http-proxy parameter within the GitHub action.
Figure 16: The GitHub workflow configured to trigger token generation.
  • Intercept a workflow request and extract the Web Identity Token:
    After everything was ready, we triggered the workflow by performing a commit to the repository. The commit started a GitHub workflow which performed a network request to assume the target role with the Web Identity token. Then, we extracted the token from the request sent by the runner to AWS.
Figure 17: Extract token from runner request – AWS

Decoding the JWT Token reveals our sub, along with other identifiers that are being sent and logged by the cloud provider as part of the authentication process.

Figure 18: Decoding JWT token
  • Copy the token to our multithreaded script, which will check the potentially vulnerable roles.

Using a pre-developed script and leveraging Boto3 and Gcloud, we scanned a list of potentially vulnerable roles while having the JWT at hand.

Results

After reducing duplicate roles and service accounts, we scanned approximately 1500 roles and service accounts across GCP and AWS.

As for the first misconfiguration, lack of subject condition, we determined that dozens of the roles were vulnerable, allowing anyone to use them to access the accounts.

The second misconfiguration, poorly-defined condition, was harder to test and required setting up a dedicated GitHub organization per target. Thus, we performed a random test against 20 organizations, finding one of them to be vulnerable to this attack.

The vulnerable accounts included various organizations, including private companies, non-profit organizations, software development studios, and many personal accounts.

During the last few weeks we have reached out to the relevant organizations and people, sending them information regarding the vulnerable roles and sharing remediation guidance.

Remediation Guidelines

Could My Environment Be Affected? 

Based on our research, this type of misconfiguration is a risk for AWS and GCP users who use GitHub Actions with modern authentication (OIDC).

Identify Potential Misconfigurations: Lack of Subject Conditions

To identify this misconfiguration In AWS, check for roles that have no subject limitations in its trust policy.

Figure 19: Misconfigured Role with no Subject – Example – AWS

To identify this misconfiguration in GCP, check for service accounts connected to the Identity Provider that has access to the Entire Pool.

Figure 20: Misconfigured Role with no Subject – Example – GCP

Identify Potential Misconfigurations: Poorly-Defined Conditions

To identify this misconfiguration in AWS, locate the StringLike conditions for roles that are connected to the Identity Provider, and check for wildcards in the organization name.

Figure 21: Misconfigured Role with Poorly-Defined Condition Pattern – AWS

For example, in this case, we have a role that was originally intended to be used by different repositories within the Rezonate organization. Since the StringLike conditions included a wildcard in the organization name,  every organization that starts with “rezonate” can assume this role. This means that an attacker can create a new GitHub organization with the name “rezonateX” and get access to the role.

This misconfiguration option does not apply to GCP, as GCP does not support wildcards.

Identity Risks in the Rezonate UI

Rezonate customers should look for the security risk “GitHub Actions Role vulnerable to hijacking” in the Rezonate UI and follow the guided remediation steps attached to it.

Use a Script to Perform a Quick Scan

For the general public, we have released a script to our GitHub repository (link here), which performs a quick scan against the AWS account or GCP project and reveals possible vulnerable roles and service accounts.

Tags:

Continue Reading

More Articles
Rezonate
February 27, 2023

TX Group: Eliminating cloud identity risk with Rezonate

Success for Switzerland’s largest international private media company means always staying ahead of the digital curve – and security is no exception. Rezonate makes this possible. “With Rezonate our DevOps and security teams are now enabled to work hand-in-hand and understand the complete identity story - across our IdP and cloud infrastructure. We reduce manual workload, increase productivity and eventually reduce the time to remediate critical risks.” Andreas Schneider, former Group CISO and Olivier Martinet, current Group CISO for TX Group The Challenge: Finding and Fixing Identity ‘Blind Spots’ – Fast Speed is of the essence in the media industry: news happens fast, and it’s imperative to deliver – and secure – it rapidly, as well.  Detecting identity issues and compromises in this complex environment, Schneider says, was like finding the proverbial “needle in a haystack.” He used several different tools to try to uncover every vulnerability, but he knew that he wasn’t seeing the complete exposure map. But finding and closing the identity and access management gaps seemed nearly impossible. AWS’s own insight tools proved difficult even for the engineers to use. So Schneider sought help – and found it in Rezonate. “We had blind spots. There were things we didn’t really think about. We check configuration, for example, but do we check privileges? If a vendor says they need access to something, it is a real challenge to continuously validate need and actual usage.”  The Solution: A team approach that really works Schneider chose Rezonate to handle TX Group’s  identity management for a number of reasons:  Real problem solving.  Rezonate sees the extent to which identities use their access privileges so TX Group can revoke  access to unused resources and applications – the “least privilege” approach.  “I don’t know of any other technology that does this. Rezonate alone could give us real-time visibility into our cloud accounts as well as guidance for quick response. We now know exactly what’s going on and where, every moment.” Rapid response. TX Group can now spot risky accounts and mitigate them with ease using Rezonate, and its security and DevOps teams can work together to resolve the identity and access issues that are so common in the cloud — without slowing or stopping operations. Rezonate accomplishes this feat via its Identity Storyline™, the brains behind the Rezonate platform. Identity Storyline simplifies complex identity and access problems and provides clear guidance on how to resolve them.Now, using Rezonate, TX Group can quickly see, in context, each identity’s behaviors in the cloud – past as well as present – and know which might increase its risk of breach, as well as how to best remediate.Identity Storyline goes beyond static dashboards to answer the dynamic questions that need always-current answers such as Where are our blind spots? Where have identities changed or deviated from patterns of behavior? Where are our active threats? “Without Rezonate, we would not be able to see these kinds of suspicious activities on all our identity providers and cloud accounts. Before, we were seeing just minor parts of our  identity and access risk. We now have the complete picture, and can make decisions with confidence.” User-readiness. The Rezonate platform software is up and running and ready to use in minutes. “Rezonate takes zero trust to the next level. Rezonate is, for me, the one-stop shop security tool for protecting our identities in the correct way – for identifying and remediating threats.” The Outcomes: A full and complete view of identities, access, and privileges via Rezonate’s Identity Storyline™ – leveling up “zero trust” security for the cloud Faster time from risk discovery to risk remediation – from days or weeks to minutes Reduced workload for DevOps and security teams as automation handles detection and remediation before risks become threats Greater productivity as DevOps works hand-in-hand with security  to safely design, create, and deploy Optimized access permissions, ensuring a “least privileges” approach Proactive, prioritized responses to risk and threats
Read More
Ori Amiga
February 27, 2023

CircleCI Breach: Detect and Mitigate to Assure Readiness

On January 4, 2023, CircleCI, a continuous integration (CI/CD) and delivery service, reported a data breach. The company urged its customers to take immediate action while a complete investigation is ongoing. First critical actions recommended by CircleCI were to ‘rotate any and all secrets stored in CircleCI’ and ‘review related activities for any unauthorized access starting from December 21, 2022 to January 4, 2023’. Why is it such a big deal Malicious use of access keys in conjunction with privileged access can have a significant impact on an organization’s source code, deployment targets, and sensitive data across its infrastructure.  CI/CD pipelines operation requires exactly that - high-privileged access which in most cases is administrative and direct access to source code repositories essential for smooth operation - and as such, considered a critical component of the software development life cycle (SDLC).  Start investigating for malicious activity in your cloud environment Data breaches are unfortunately common and should no longer be a surprise. Every third-party service or application has the potential to act as a supply chain vector by an attacker. When that occurs, excessive access that was previously benign can become a critical exposure, allowing the threat actor to exploit the system freely. Here are immediate next steps security and DevOps teams should take to eliminate any possible supply chain risk - those recommended by CircleCI and beyond: Discover possible entry points - Critical first step involves mapping, linking and reviewing the access of all secrets given to the compromised third-party service to fully understand all initial access attempts and possible lateral movement across all supply chain vectors.Specific to CircleCI data breach, Rezonate observed that multiple accounts had a few AWS programmatic access keys with administrative privileges in the CircleCI configuration, allowing for creation and modification of any resource within the account. Threat containment (& traps) - Once you identify any and all keys, the first option is to deactivate or delete them and create new ones (avoid rotating an unused key). However, while you prevent any future use of these keys, you also limit any potential traces of benign or malicious activity. Why? In the case of AWS, Cloudtrail has limited authentication logging for invalid or disabled keys.A second more preferred option is to remove all privileges from users while keeping the keys and users active. This enables further monitoring of activity using ‘canary keys,’ where every access attempt triggers an alert and extracts threat intelligence artifacts (IOCs such as IP address). Activity review & behavioral profiling - Once you capture all suspected keys, you can begin analyzing their activity within the defined range reported. In our case, we used AWS Cloudtrail as the main data source and investigated the access behavioral patterns. The goal is to create a ‘clean’ baseline of activities that occurred prior to the breach. To help define a profile, understand the scope, and identify any potential areas of concern, consider asking the following questions: Reduce the overwhelming number of insignificant incident alerts and the time spent addressing them Increase operational visibility into cloud identity and access security across platforms Discover and monitor third party cross-cloud access Limit permissions and restrict access to the minimum users required without any impact to operations. Once we have a good understanding of normal operation, we can apply the same approach to inspect activities from the date of breach until the present. In this case, the context of workflows, resources, and overall architecture is paramount, so it is critical to collaborate with the dev/infra team to quickly assess, validate, and prioritize findings. Activity review & threat models - Based on the results of previous steps, further questions may indicate a potentially malicious exploitation, such as attempts to elevate privileges, gain persistence, or exfiltrate data. To help pinpoint the most relevant findings, consider the following options: Activities performed outside of our regular regionsAlerting for anomaly of regular access in an attempt to locate compromised resourcesIdentity-creation activities(ATT&CK TA0003)Activities such as CreateUser and CreateAccessKey attempting to gain persistencyResource-creation activitiesDiscover attempts to perform resource exhaustion for crypto mining and othersActivities performed outside of the regular CircleCI IP rangesIdentify any access attempts from external IPs that may relate to known bad intelErrors occurredDetect “pushing the limits” attempts to exploit user privileges resulting in error (e.g. AccessDenied)Spike in enumeration activities(ATT&CK T1580)Detect increased recon and mapping actions (e.g. user and role listing)Defense evasion techniques(ATT&CK TA0005)Detect tampering attempts to limit defense controls (e.,g. DeleteTrail or modify GuardDuty settings)Secret access attemptsDetect bruteforce actions against mapped secrets to elevate account foothold It’s important to consider all suggested actions as part of the overall context, as some may be legitimate while others may be malicious. By correlating them all together, you can reduce noise and false positives.  How Rezonate can help It’s important to note that while this guidance specifically addresses key actions related to the CircleCI data breach, it can also serve as best practice for addressing any risks for any breach. Rezonate automates the actions described above to streamline the compromise assessment process and reduce the time and effort required for manual analysis. Rezonate simplifies discovery, detection, and investigation of the compromise. Work with a system that can automatically correlate and summarize all activities of all identities to save critical time. Working directly with CloudTrail can be challenging, lacking aggregation, data-correlation  and privileged tagging eventually slowing you down.  We have been collaborating with our clients and partners to utilize the Rezonate platform to thoroughly investigate the security incident and assess its potential impact on all activities mentioned here. If you require assistance, please do not hesitate to contact us. Providing support to our clients and the community is a key purpose of Rezonate's founding.
Read More
Rezonate
February 27, 2023

PayMe: Protecting cross cloud identity and access with Rezonate

Empowering micro businesses to achieve more with a full suite of fintech products means building new tools and functionality fast – without cloud identity and access security slowing teams down. Rezonate makes this possible.  “Partnering with Rezonate to protect identity and access allowed both our security and DevOps teams to feel more secure and confident in how fast we’re moving – despite increasing challenges.” Alexander Sorochan, Head of DevSecOps, PayMe The Challenge: Striking the Right Balance Between Speed, Agility, and Security Financial technology (fintech) companies are innovating with unmatched speed and agility to meet new demands in a digital-first world. But securing this fast-growing industry in sync is proving to be more difficult.  For fintech startup PayMe, one of the biggest security challenges has been cloud identity and access management.  “Swiftly detecting and responding to risks across cloud environments is critical,” says Sorochan, “but it’s next to impossible when security teams are managing access for multiple identities in multiple cloud accounts on different platforms.” Piecing together data across identity sources takes time – something most fintech companies simply do not have to spare.  PayMe knew they needed to act quickly to protect their cloud environment with a security tool that could: Increase operational visibility into cloud identity and access security across platforms Reduce the overwhelming number of insignificant incident alerts and the time spent addressing them Discover and monitor third party cross-cloud access Limit permissions and restrict access to the minimum users required without any impact to operations. With Rezonate, PayMe was able to achieve all of the above, and more.  “Rezonate provides unparalleled visibility into one of the core problems facing fintech today: cloud identity and access. Now, we can prioritize exposures and identify threats as they emerge, without sacrificing speed or agility.”  The Solution: A Single Platform for a United Path Forward PayMe chose Rezonate to protect its cloud identities and access for a variety of reasons: Rapid deployment. Rezonate quickly connected with all of PayMe’s identity and cloud providers, enabling self-launch in no time. Within minutes of deployment, PayMe could see, profile and analyze all of their cloud identities across all of their cloud providers; within hours, PayMe was identifying, prioritizing, and mitigating their most critical risks. The result? A complete view of critical findings for immediate prioritization, instant optimization for access and entitlements, and real-time validation for fixes – all in a single platform.  “Within hours of deployment, we understood the complete picture of our cross-cloud identity and access risks. Our DevOps team uses Rezonate daily to understand context and prioritize critical risks. We are now 10X faster and more effective in remediating security gaps.” Reduced complexity. At Rezonate, simplicity is key to quality security. The brains behind the Rezonate platform, Identity Storyline replaces complex graphs with easy-to-understand storylines that trace every identity risk, exposure and threat from root to impact for a panoramic view at every point in time. Now, PayMe’s security team can spot cloud identity and access weaknesses as they are created, and conclusively determine: What they are and their possible impact Who created them in its original intent Where they exist and how abnormal they are Why they have access and how is that usedHow they might impact security and business operations With complete visibility into its cloud environments from Rezonate, PayMe can now optimize remediation and minimize operational impact using a simple, fast, and unified approach.  A united path forward. Rezonate’s platform brings PayMe’s security and DevOps teams together so they can work as one, quickly identifying and remediating risks across the cloud environment in tandem. With Rezonate, PayMe can holistically connect risk, threat, and operational visibility across teams and across the board. Now, PayMe’s security team can respond with confidence – immediately stopping attacks and wiping out risks from within – freeing their developers to work without security slowing them down.  The Outcomes:  Compliance-ready cloud identity and access security in minutes A proactive security stance with complete coverage for cloud-wide environments Context and automation to prioritize and remediate risks  Active threat detection for prevention before progression Minimized excessive access and administrative permissions Better ability to pinpoint risky exposures, reducing identity exposure debt Visibility into AWS and Okta environments in a single platform
Read More
See Rezonate in Action

Eliminate Attacker’s Opportunity To Breach Your Cloud today

Organizations worldwide use Rezonate to protect their most precious assets. Contact us now, and join them.