Automated build pipelines for medical device manufacturers and industrial leaders like Dräger and Bosch were caught in the wide-reaching net of the Trivy-to-LiteLLM attack chain. This sophisticated operation, uncovered by security researchers, exposed the inherent vulnerabilities in the modern software supply chain where a single compromised utility can compromise thousands of downstream entities. The scale of the breach is staggering, with a massive 152.5 GiB archive of stolen data being recovered, containing over 415,000 individual capture files. This dataset is not merely a collection of static files but a real-time snapshot of sensitive credentials harvested directly from active memory during automated build processes. The threat group responsible, known as TeamPCP, utilized a specialized piece of malware called SANDCLOCK to systematically drain environment variables, private keys, and authentication tokens from systems that developers and engineers thought were secure and isolated.
The breach represents a significant escalation in supply-chain warfare, moving beyond simple repository poisoning to active, multi-stage credential harvesting. By targeting the intersection of security tools and automation, the attackers exploited the trust placed in widely used GitHub Actions. As organizations increasingly rely on automated workflows to manage code deployments, the security of the underlying runners becomes the ultimate prize for state-sponsored and criminal actors alike. This incident serves as a stark reminder that the tools meant to protect us, such as vulnerability scanners and compliance checkers, can be turned into Trojan horses if the integrity of their update mechanisms is undermined. The repercussions of this breach continue to unfold as major technology firms and government cloud tenants work to assess the full extent of the exposure and rotate thousands of compromised secrets.
1. Executive Summary: The Scope of the Compromise
The investigation into the LiteLLM supply-chain attack has revealed an extensive archive labeled supplychain_victims.tar, which serves as a centralized repository for the loot gathered by TeamPCP. This 152.5 GiB collection consists of hundreds of thousands of individual text files, each meticulously named with timestamps and a local loopback identifier. These files provide a granular look at the data exfiltrated between March 19 and March 24, 2026. Unlike traditional data breaches that focus on user databases, this attack prioritized the backbone of the tech industry: the credentials that allow automated systems to communicate, deploy code, and manage cloud infrastructure. The sheer volume of the data suggests an automated and highly efficient harvesting process that remained undetected for the critical early days of the campaign.
The SANDCLOCK malware utilized in this campaign was specifically engineered to operate within the constraints of CI/CD environments, where speed and automation often come at the expense of deep forensic monitoring. By embedding itself into the Trivy GitHub Action, the malware gained immediate access to high-trust environments across 898 different GitHub owners. This widespread reach allowed the attackers to infiltrate the development pipelines of some of the most prominent companies in the world, including NVIDIA, IBM, and Microsoft. The data within the archive confirms that the attackers were not just interested in LiteLLM itself, but used it as a stepping stone to reach larger, more lucrative targets. This strategy of lateral movement through the software supply chain demonstrates a high level of planning and execution.
2. Primary Observations: Patterns in Stolen Data
A deep dive into the harvested files reveals a startling consistency in the types of data targeted by the SANDCLOCK malware. Approximately 98% of the sampled files contained active GitHub tokens, specifically the GITHUB_TOKEN automatically generated for Actions jobs. While these tokens are often short-lived, the sheer scale of the harvest allowed the attackers to maintain a persistent window of opportunity to manipulate repositories and exfiltrate source code. The presence of these tokens across thousands of runs suggests that the malware was effectively able to bypass standard security boundaries within the GitHub Actions runner environment. This systematic collection indicates that the attackers understood the value of ephemeral credentials when harvested at a massive scale.
Furthermore, the method of data acquisition was particularly invasive. Rather than simply reading environment variables from disk, the malware performed on-host secret harvesting by reading directly from process memory. The use of the 127.0.0.1 tag in the file naming convention indicates that the secrets were dumped in-process on the victim host rather than being intercepted over a network. This technique allowed the attackers to capture “live” secrets that may only exist for the duration of a specific build or deployment step. By escalating privileges to root on the compromised runners, the SANDCLOCK stealer could sweep everything from SSH keys and cloud credentials to Kubernetes tokens and .env files, leaving behind a comprehensive record of the victim’s internal configuration and security posture.
3. Incident Timeline: The Trivy to LiteLLM Infection Path
The architecture of this attack followed a logical and devastating progression, starting with the compromise of a trusted security utility. On March 19, 2026, TeamPCP successfully backdoored the aquasecurity/trivy-action on GitHub, a tool used by thousands of developers to scan their containers for vulnerabilities. Because many CI/CD pipelines are configured to automatically fetch the latest version of their security tools, the poisoned action was immediately pulled into active development environments. This initial move provided the attackers with a massive distribution network, effectively turning a security tool into a primary infection vector. The automation that modern DevOps relies on became the very mechanism that facilitated the rapid spread of the malware.
Once the backdoored Trivy action was running within LiteLLM’s CI/CD pipeline, it targeted the specific credentials needed to further the attack. The malware identified and exfiltrated LiteLLM’s PyPI publishing tokens, which are used to upload new versions of the software to the Python Package Index. With these tokens in hand, the attackers published malicious versions of LiteLLM, specifically versions 1.82.7 and 1.82.8, on March 24, 2026. This second-order compromise allowed the attackers to move from a single GitHub Action to a popular software library, potentially infecting every developer and system that downloaded the latest LiteLLM updates. The transition from a tool compromise to a library compromise represents a sophisticated method of self-propagation within the software ecosystem.
4. Dataset Overview: Analysis of the 152.5 GiB Archive
The structural analysis of the recovered archive provides insight into the operational efficiency of TeamPCP. The directory contains over 415,000 timestamped files, with a size distribution that ranges from minimal beacons to massive dumps of secret material exceeding 30 MB. The temporal distribution of the captures shows two distinct waves of activity. The first wave peaked between March 19 and March 20, coinciding with the initial backdoor of the Trivy action. After a brief lull, a second surge occurred on March 23 and March 24, which aligns with the period when the malicious LiteLLM packages were published and began infecting downstream users. This two-phase approach suggests the attackers were actively managing their campaign based on the success of each preceding step.
The data within these files is formatted as a JSON-style secret store, making it easy for the attackers to programmatically process and utilize the stolen credentials. Each entry includes the name of the secret, its value, and a boolean flag indicating its status as a secret. This level of organization within the exfiltrated data allowed the threat actors to quickly categorize and prioritize the most valuable assets, such as cloud administrative keys or private repository access. The archive also includes runner identifiers that point to a mix of GitHub-hosted runners and self-hosted environments using the actions-runner-controller. This diversity in the victim landscape highlights that no specific hosting model was immune to the SANDCLOCK stealer’s capabilities.
5. Types of Stolen Information: Beyond Simple Keys
The breadth of information captured by the SANDCLOCK malware extends far beyond basic passwords or API keys. In addition to the ubiquitous GitHub tokens, the archive contains a vast array of cloud access credentials for major providers such as AWS, Google Cloud, and Firebase. These keys often grant extensive permissions within an organization’s cloud environment, allowing for everything from data exfiltration to the spinning up of unauthorized resources for crypto-mining or further attacks. The exposure of AWS role-to-assume configurations is particularly concerning, as it suggests the attackers could potentially pivot through different accounts and services within a compromised organization’s cloud infrastructure.
Moreover, the breach hit critical software development infrastructure by capturing tokens for private container registries and package managers like JFrog and ECR. This type of access is invaluable for a supply-chain attacker, as it allows them to inject malicious code into internal software builds that are never exposed to the public internet. The theft of SSH keys and code-signing passwords further compounds the risk, as these assets can be used to forge trusted identities and sign malicious software, making it appear legitimate to automated security checks. By gathering such a diverse set of authentication materials, TeamPCP ensured they had multiple avenues for long-term persistence and further exploitation within their targets’ networks.
6. Redacted Evidence: Visualizing the Exposure
Evidence recovered from the 152.5 GiB archive provides a chilling look at how organizations’ most sensitive data was displayed to the attackers. The capture files follow a consistent schema where environment variables and memory-resident secrets are laid out in clear, structured formats. In the redacted samples, researchers have seen variables like “github_token” and “AWS_ROLE_TO_ASSUME” alongside the metadata that identifies the specific build environment and project. While the actual values of these secrets have been masked in public reports for security reasons, the metadata alone is enough to confirm the high level of access the attackers obtained. The transparency of this data within the attackers’ archive illustrates the total lack of encryption for secrets once they are loaded into the memory of a running CI/CD process.
The “isSecret” tag found within the JSON records is a particularly notable detail, as it shows that the malware was intelligent enough to distinguish between ordinary environment variables and those specifically marked as sensitive by the GitHub Actions platform. This capability demonstrates that the attackers were not just performing a blind dump of information but were actively seeking out the keys to the kingdom. Seeing these records laid out in such a clean, readable format underscores the vulnerability of modern automation. When a developer marks a variable as a “secret” in their CI/CD settings, they expect it to be protected, yet this breach shows that once the environment is compromised at the root level, those protections are essentially bypassed by on-host harvesting techniques.
7. Impacted Entities: A Global Corporate Map
The list of organizations affected by the LiteLLM and Trivy breach reads like a directory of the global Fortune 500. The victim manifest includes nearly 900 GitHub owners, many of whom are pillars of the technology, finance, and industrial sectors. Major names such as Microsoft, NVIDIA, and IBM appear alongside financial giants like PayPal and S&P Global. For these companies, the breach represents more than just a leak of credentials; it is a fundamental challenge to the security of their development processes. In many cases, large organizations had dozens of different repositories affected, suggesting that the malware spread horizontally across different teams and projects that all relied on the same automated security tooling.
The impact was not limited to the private sector, as the data also indicates the compromise of a US GovCloud AWS tenant and other sensitive government-linked infrastructure. The presence of global entities like Deloitte, FedEx, and John Deere highlights the universal nature of the software supply chain risk. Industrial leaders like Bosch and Kärcher, who are increasingly integrating digital services into their physical products, face unique risks from the exposure of their firmware and software build pipelines. Each of these organizations must now deal with the fallout of the breach, which includes not only the rotation of thousands of secrets but also the long-term task of verifying the integrity of every piece of code that passed through their pipelines during the window of infection.
8. Threat Assessment: The Cascading Risk Factor
The security community has categorized this breach as having a high-to-critical severity level due to the potential for cascading failures. When an attacker gains write access to a repository or the ability to publish packages, they can effectively use that platform to launch subsequent attacks against any user of that software. This is exactly how the campaign moved from the Trivy action to the LiteLLM library. The risk of second-order supply-chain propagation means that the initial 898 compromised organizations are just the first layer of a much larger web of potential victims. Every customer, partner, or end-user who consumes the software produced by these companies is now at risk of being the next target in the chain.
Beyond the threat of software poisoning, the harvested cloud credentials present a massive risk for account takeover and infrastructure manipulation. With administrative-level access to AWS or GCP environments, attackers can delete backups, steal customer data, or establish backdoors that persist for years. The ability to forge trusted software signatures using stolen code-signing passwords is perhaps the most dangerous outcome, as it undermines the entire foundation of software trust. If a malicious update is signed with a valid corporate key, most security systems will allow it to run without question. This combination of cloud access and software signing capabilities gives the threat actors an unprecedented level of control over the digital ecosystem.
9. Attacker Strategies: Understanding the TeamPCP Playbook
The success of TeamPCP’s campaign can be attributed to their sophisticated understanding of modern development workflows and the MITRE ATT&CK framework. Their primary method of initial access was a classic software dependency compromise, but they elevated the execution by targeting a security tool rather than a generic utility. This choice was strategic, as security tools often require higher levels of permission and are integrated deeply into the CI/CD process. Once inside the runner, the SANDCLOCK malware performed rapid privilege escalation to gain root access. This allowed it to move beyond the restricted environment of the user-level process and scan the entire host for sensitive files and memory-resident secrets, a tactic known as credential harvesting from files and memory.
The group’s exfiltration strategy was equally efficient. By saving the stolen secrets to local text files before sending them to their central archive, they minimized the risk of network-based detection during the harvesting phase. The uniform structure of the exfiltrated data suggests that they had built a robust backend system to handle the massive influx of information from thousands of concurrent build runs. This level of preparation indicates that TeamPCP is a well-resourced threat actor with a clear focus on industrial espionage and large-scale disruption. Their ability to manage a campaign of this magnitude over several days without being immediately shut down highlights the current imbalance between attacker speed and defender visibility in the CI/CD space.
10. Identification and Detection: Indicators of Compromise
For organizations looking to determine if they were part of this massive breach, there are several specific indicators of compromise to monitor. The most obvious sign is the presence of the timestamped files following the
Another key detection sign is the specific JSON format used in the exfiltrated files, particularly the inclusion of the “isSecret”:true flag. Any log or file containing this string on a build machine is a definitive sign of a SANDCLOCK infection. Furthermore, organizations should audit their cloud provider logs for any unauthorized use of service accounts or the assumption of roles by entities outside of the normal build process. Because the malware escalated to root, any unusual system-level activity or changes to environment files like /etc/environment or .env should be treated with high suspicion. Reviewing historical runner logs for the execution of the aquasecurity/trivy-action version that was active during the breach period is also a critical step in the identification process.
11. Immediate Response: Securing the Perimeter
In the aftermath of the discovery, the most urgent task for any organization involved was the immediate rotation of all credentials identified in the logs. This included every API key, GitHub token, cloud access key, and SSH key that could have been present in the CI/CD environment during the infection period. Because the malware was capable of reading memory, simply changing passwords was not enough; every token had to be revoked and replaced to ensure that the attackers could not use previously harvested data to re-enter the system. This massive undertaking required coordination across multiple departments, from DevOps and Security to Finance and Legal, as the scope of the exposure touched nearly every part of the modern enterprise.
Beyond simple credential rotation, impacted entities had to perform a thorough forensic audit of all code changes and package releases made during the attack period. This was necessary to ensure that the attackers had not used their access to inject malicious code into production systems. Verification of the integrity of every software artifact became a priority, with many organizations reverting to known good states from before the March 19 window. Alerting impacted users and regulatory bodies was another critical step, particularly for those involving government cloud exposure or sensitive customer data. Restricting the reach of service accounts within cloud environments by implementing stricter identity and access management policies helped to limit the potential damage from any missed credentials.
The response to the LiteLLM and Trivy breach highlighted the critical need for a more resilient approach to managing software dependencies. In the weeks following the incident, organizations moved away from using static, long-lived API keys in favor of short-lived, identity-based credentials like OpenID Connect (OIDC). This shift significantly reduced the window of opportunity for attackers to utilize stolen tokens. Security teams also implemented stricter dependency locking, ensuring that all external actions and libraries were pinned to specific, verified versions rather than automatically pulling the latest updates. These structural changes were complemented by the deployment of automated scanning tools designed to detect secrets being written to disk or transmitted in cleartext. By monitoring underground forums for leaked company data and adopting a zero-trust posture within the CI/CD pipeline, the industry worked to build a more robust defense against future supply-chain campaigns. These proactive measures were essential for restoring trust in the automated systems that drive modern technological progress.
