CircleCI Breach: Detect and Mitigate to Assure Readiness
Contents
On January 4, 2023, CircleCI, a continuous integration (CI/CD) and delivery service, reported a data breach. The company urged its customers to take immediate action while a complete investigation is ongoing. First critical actions recommended by CircleCI were to ‘rotate any and all secrets stored in CircleCI’ and ‘review related activities for any unauthorized access starting from December 21, 2022 to January 4, 2023’.
Why is it such a big deal
Malicious use of access keys in conjunction with privileged access can have a significant impact on an organization’s source code, deployment targets, and sensitive data across its infrastructure.
CI/CD pipelines operation requires exactly that – high-privileged access which in most cases is administrative and direct access to source code repositories essential for smooth operation – and as such, considered a critical component of the software development life cycle (SDLC).
Start investigating for malicious activity in your cloud environment
Data breaches are unfortunately common and should no longer be a surprise. Every third-party service or application has the potential to act as a supply chain vector by an attacker. When that occurs, excessive access that was previously benign can become a critical exposure, allowing the threat actor to exploit the system freely.
Here are immediate next steps security and DevOps teams should take to eliminate any possible supply chain
risk – those recommended by CircleCI and beyond:
- Discover possible entry points – Critical first step involves mapping, linking and reviewing the access of all secrets given to the compromised third-party service to fully understand all initial access attempts and possible lateral movement across all supply chain vectors.
Specific to CircleCI data breach, Rezonate observed that multiple accounts had a few AWS programmatic access keys with administrative privileges in the CircleCI configuration, allowing for creation and modification of any resource within the account.
- Threat containment (& traps) – Once you identify any and all keys, the first option is to deactivate or delete them and create new ones (avoid rotating an unused key). However, while you prevent any future use of these keys, you also limit any potential traces of benign or malicious activity. Why? In the case of AWS, Cloudtrail has limited authentication logging for invalid or disabled keys.
A second more preferred option is to remove all privileges from users while keeping the keys and users active. This enables further monitoring of activity using ‘canary keys,’ where every access attempt triggers an alert and extracts threat intelligence artifacts (IOCs such as IP address).
- Activity review & behavioral profiling – Once you capture all suspected keys, you can begin analyzing their activity within the defined range reported. In our case, we used AWS Cloudtrail as the main data source and investigated the access behavioral patterns. The goal is to create a ‘clean’ baseline of activities that occurred prior to the breach. To help define a profile, understand the scope, and identify any potential areas of concern, consider asking the following questions:
- Reduce the overwhelming number of insignificant incident alerts and the time spent addressing them
- Increase operational visibility into cloud identity and access security across platforms
- Discover and monitor third party cross-cloud access
- Limit permissions and restrict access to the minimum users required without any impact to operations.
Once we have a good understanding of normal operation, we can apply the same approach to inspect activities from the date of breach until the present. In this case, the context of workflows, resources, and overall architecture is paramount, so it is critical to collaborate with the dev/infra team to quickly assess, validate, and prioritize findings.
- Activity review & threat models – Based on the results of previous steps, further questions may indicate a potentially malicious exploitation, such as attempts to elevate privileges, gain persistence, or exfiltrate data. To help pinpoint the most relevant findings, consider the following options:
| Activities performed outside of our regular regions | Alerting for anomaly of regular access in an attempt to locate compromised resources |
| Identity-creation activities (ATT&CK TA0003) | Activities such as CreateUser and CreateAccessKey attempting to gain persistency |
| Resource-creation activities | Discover attempts to perform resource exhaustion for crypto mining and others |
| Activities performed outside of the regular CircleCI IP ranges | Identify any access attempts from external IPs that may relate to known bad intel |
| Errors occurred | Detect “pushing the limits” attempts to exploit user privileges resulting in error (e.g. AccessDenied) |
| Spike in enumeration activities (ATT&CK T1580) | Detect increased recon and mapping actions (e.g. user and role listing) |
| Defense evasion techniques (ATT&CK TA0005) | Detect tampering attempts to limit defense controls (e.,g. DeleteTrail or modify GuardDuty settings) |
| Secret access attempts | Detect bruteforce actions against mapped secrets to elevate account foothold |
It’s important to consider all suggested actions as part of the overall context, as some may be legitimate while others may be malicious. By correlating them all together, you can reduce noise and false positives.
How Rezonate can help
It’s important to note that while this guidance specifically addresses key actions related to the CircleCI data breach, it can also serve as best practice for addressing any risks for any breach.
Rezonate automates the actions described above to streamline the compromise assessment process and reduce the time and effort required for manual analysis. Rezonate simplifies discovery, detection, and investigation of the compromise.
Work with a system that can automatically correlate and summarize all activities of all identities to save critical time. Working directly with CloudTrail can be challenging, lacking aggregation, data-correlation and privileged tagging eventually slowing you down.
We have been collaborating with our clients and partners to utilize the Rezonate platform to thoroughly investigate the security incident and assess its potential impact on all activities mentioned here. If you require assistance, please do not hesitate to contact us. Providing support to our clients and the community is a key purpose of Rezonate’s founding.
Continue Reading
More Articles
PayMe: Protecting cross cloud identity and access with Rezonate
Empowering micro businesses to achieve more with a full suite of fintech products means building new tools and functionality fast – without cloud identity and access security slowing teams down. Rezonate makes this possible.
“Partnering with Rezonate to protect identity and access allowed both our security and DevOps teams to feel more secure and confident in how fast we’re moving – despite increasing challenges.”
Alexander Sorochan, Head of DevSecOps, PayMe
The Challenge: Striking the Right Balance Between Speed, Agility, and Security
Financial technology (fintech) companies are innovating with unmatched speed and agility to meet new demands in a digital-first world. But securing this fast-growing industry in sync is proving to be more difficult.
For fintech startup PayMe, one of the biggest security challenges has been cloud identity and access management.
“Swiftly detecting and responding to risks across cloud environments is critical,” says Sorochan, “but it’s next to impossible when security teams are managing access for multiple identities in multiple cloud accounts on different platforms.”
Piecing together data across identity sources takes time – something most fintech companies simply do not have to spare.
PayMe knew they needed to act quickly to protect their cloud environment with a security tool that could:
Increase operational visibility into cloud identity and access security across platforms
Reduce the overwhelming number of insignificant incident alerts and the time spent addressing them
Discover and monitor third party cross-cloud access
Limit permissions and restrict access to the minimum users required without any impact to operations.
With Rezonate, PayMe was able to achieve all of the above, and more.
“Rezonate provides unparalleled visibility into one of the core problems facing fintech today: cloud identity and access. Now, we can prioritize exposures and identify threats as they emerge, without sacrificing speed or agility.”
The Solution: A Single Platform for a United Path Forward
PayMe chose Rezonate to protect its cloud identities and access for a variety of reasons:
Rapid deployment. Rezonate quickly connected with all of PayMe’s identity and cloud providers, enabling self-launch in no time. Within minutes of deployment, PayMe could see, profile and analyze all of their cloud identities across all of their cloud providers; within hours, PayMe was identifying, prioritizing, and mitigating their most critical risks. The result? A complete view of critical findings for immediate prioritization, instant optimization for access and entitlements, and real-time validation for fixes – all in a single platform.
“Within hours of deployment, we understood the complete picture of our cross-cloud identity and access risks. Our DevOps team uses Rezonate daily to understand context and prioritize critical risks. We are now 10X faster and more effective in remediating security gaps.”
Reduced complexity. At Rezonate, simplicity is key to quality security. The brains behind the Rezonate platform, Identity Storyline replaces complex graphs with easy-to-understand storylines that trace every identity risk, exposure and threat from root to impact for a panoramic view at every point in time. Now, PayMe’s security team can spot cloud identity and access weaknesses as they are created, and conclusively determine:
What they are and their possible impact
Who created them in its original intent
Where they exist and how abnormal they are
Why they have access and how is that usedHow they might impact security and business operations With complete visibility into its cloud environments from Rezonate, PayMe can now optimize remediation and minimize operational impact using a simple, fast, and unified approach.
A united path forward. Rezonate’s platform brings PayMe’s security and DevOps teams together so they can work as one, quickly identifying and remediating risks across the cloud environment in tandem. With Rezonate, PayMe can holistically connect risk, threat, and operational visibility across teams and across the board. Now, PayMe’s security team can respond with confidence – immediately stopping attacks and wiping out risks from within – freeing their developers to work without security slowing them down.
The Outcomes:
Compliance-ready cloud identity and access security in minutes
A proactive security stance with complete coverage for cloud-wide environments
Context and automation to prioritize and remediate risks
Active threat detection for prevention before progression
Minimized excessive access and administrative permissions
Better ability to pinpoint risky exposures, reducing identity exposure debt
Visibility into AWS and Okta environments in a single platform
Read More
How Rezonate Maintains Audit-Ready State Using Rezonate
We all understand the importance of maintaining strong security protocols and controls. That’s why Rezonate decided to invest in the SOC 2 Type 2 compliance early on, and after only one month since our out of stealth announcement, we successfully achieved attestation.
What exactly is SOC 2 Type 2 certification, and why is it important to you?
SOC 2, or System and Organization Controls (SOC) 2 type 2 is a widely recognized set of standards that ensure a company’s controls have been independently examined and tested. The “Type 2” designation refers to the fact that the audit covers a period of time, meaning that a company has not only implemented proper controls, but also demonstrated their continuous effective operation over a period of time.
Which is the key point I want to highlight here: a point-in-time validation vs. continuous readiness.
Rezonate protects Rezonate
Following any compliance requirements can be quite challenging. For starters, you need to fully understand the specific framework by analyzing and interpreting the right categories and controls. Then, using different assessment tools and manual efforts, you compile a list of all requirements, identifying what has been completed and what needs to be done, ensuring that the process is properly documented, logged, and monitored.
So, how can you take steps to remove manual time-consuming actions, excel at all delicate tasks, ensure an error-prone process and achieve zero exception compliance?
At Rezonate, we, the Security & DevOps team, use the Rezonate Cloud Identity Protection Platform (CIPP) on a daily basis for several use cases. As part of our ongoing protection of - our own human and compute resources’ IdP-IaaS identities and every access attempt to and from our cloud-native stack - we ensure continuous compliance readiness across key identity-first trust principles defined by the SOC 2 audit:
Security - Enforce the protection of data and systems, against unauthorized access, enforce MFA, and strengthen access controls. Strict inbound and outbound rules.
Availability - Maintain availability SLAs at all times. Building inherently fault-tolerant systems which do not crumble under high load. Invest in network monitoring systems and DR plans in place.
Confidentiality - Restrict and monitor access to organization’s confidential data and adhere to the principle of least privilege.
We do that with the goal of continuously improving our controls and processes, ensuring that we are always meeting the highest standards in the industry. In a real-world and active environment, drifts may happen, however the process we’ve built around it course-correct itself.
Protect identities, access, systems, and data
We operate in a faced paced environment and therefore our infrastructure changes fast. Yet, we still allow our team the flexibility required to build fast - without compromising security. Using the Rezonate platform, our customers understand the identity security posture with complete visibility of their identities, policies, and access requests to meet all IAM aspects required for the security, availability, and confidentiality principles.
Centralized identity inventory - Up to date inventory of all identities: employees, 3rd party vendors, machine resources, roles, groups, applications, and all required context across your multi-IdP / multi-cloud infrastructure.
Access events - Discover and understand every access performed on or from a monitored identity, since its creation time to its last active session and activity performed.
Privileges analysis - Evaluate entitlements provided to actual usage and true need for access and business operation.
Behavior baseline & drift - Analyze every access request to critical data and application and realize possible risk across our IdPs and cloud infra.
Risky exposures - Detect and better understand critical exposures, new access requests, and policy distribution to our engineering and overall staff. While we evaluate each request and relevant context to uncover potential hidden interdependencies, risk and implications.
Threat detection - Detect any malicious impersonating, access rights, and excessive privileges, while evaluating possible impact, and taking action before damage occurred.
Remediate - Proactively enforce a real-world least privileged access where Rezonate’s DevOps can ‘flex’ policy for unnecessary and risky privileges and ‘relax’ entitlements and access privileges for confirmed benign ones for increased productivity and agility.
We have built this mechanism, all while abiding compliance mandates, to comply and stay audit-ready despite complex architectures to protect our most trusted asset - our customers’ data. Be able to provide required proof for observation period instantaneously without the manual effort involved.
If you want to speak with our team on how we are leveraging the Rezonate platform to protect Rezonate and by doing that, maintain SOC 2 Type 2 audit readiness for everything related to your identity and access, sign up for a demo or simply let us know info@rezonate.io.
Thank you to our partners, EY and Scytale, for their partnership on this and future milestones.
Read More
TX Group: Eliminating cloud identity risk with Rezonate
Success for Switzerland’s largest international private media company means always staying ahead of the digital curve – and security is no exception. Rezonate makes this possible.
“With Rezonate our DevOps and security teams are now enabled to work hand-in-hand and understand the complete identity story - across our IdP and cloud infrastructure. We reduce manual workload, increase productivity and eventually reduce the time to remediate critical risks.”
Andreas Schneider, former Group CISO and Olivier Martinet, current Group CISO for TX Group
The Challenge: Finding and Fixing Identity ‘Blind Spots’ – Fast
Speed is of the essence in the media industry: news happens fast, and it’s imperative to deliver – and secure – it rapidly, as well.
Detecting identity issues and compromises in this complex environment, Schneider says, was like finding the proverbial “needle in a haystack.” He used several different tools to try to uncover every vulnerability, but he knew that he wasn’t seeing the complete exposure map.
But finding and closing the identity and access management gaps seemed nearly impossible. AWS’s own insight tools proved difficult even for the engineers to use. So Schneider sought help – and found it in Rezonate.
“We had blind spots. There were things we didn’t really think about. We check configuration, for example, but do we check privileges? If a vendor says they need access to something, it is a real challenge to continuously validate need and actual usage.”
The Solution: A team approach that really works
Schneider chose Rezonate to handle TX Group’s identity management for a number of reasons:
Real problem solving. Rezonate sees the extent to which identities use their access privileges so TX Group can revoke access to unused resources and applications – the “least privilege” approach.
“I don’t know of any other technology that does this. Rezonate alone could give us real-time visibility into our cloud accounts as well as guidance for quick response. We now know exactly what’s going on and where, every moment.”
Rapid response. TX Group can now spot risky accounts and mitigate them with ease using Rezonate, and its security and DevOps teams can work together to resolve the identity and access issues that are so common in the cloud — without slowing or stopping operations. Rezonate accomplishes this feat via its Identity Storyline™, the brains behind the Rezonate platform. Identity Storyline simplifies complex identity and access problems and provides clear guidance on how to resolve them.Now, using Rezonate, TX Group can quickly see, in context, each identity’s behaviors in the cloud – past as well as present – and know which might increase its risk of breach, as well as how to best remediate.Identity Storyline goes beyond static dashboards to answer the dynamic questions that need always-current answers such as
Where are our blind spots?
Where have identities changed or deviated from patterns of behavior?
Where are our active threats?
“Without Rezonate, we would not be able to see these kinds of suspicious activities on all our identity providers and cloud accounts. Before, we were seeing just minor parts of our identity and access risk. We now have the complete picture, and can make decisions with confidence.”
User-readiness. The Rezonate platform software is up and running and ready to use in minutes.
“Rezonate takes zero trust to the next level. Rezonate is, for me, the one-stop shop security tool for protecting our identities in the correct way – for identifying and remediating threats.”
The Outcomes:
A full and complete view of identities, access, and privileges via Rezonate’s Identity Storyline™ – leveling up “zero trust” security for the cloud
Faster time from risk discovery to risk remediation – from days or weeks to minutes
Reduced workload for DevOps and security teams as automation handles detection and remediation before risks become threats
Greater productivity as DevOps works hand-in-hand with security to safely design, create, and deploy
Optimized access permissions, ensuring a “least privileges” approach
Proactive, prioritized responses to risk and threats
Read More