A fault with an update issued by cybersecurity company CrowdStrike led to a cascade effect among global IT systems Friday, with industries ranging from banking to airlines facing outages.
Banks and healthcare providers saw their services disrupted, and TV broadcasters went offline as businesses worldwide grappled with the ongoing outage. Air travel has also been hit hard, with planes grounded and services delayed.
At the heart of the issue is Texas-based cybersecurity vendor CrowdStrike. On July 19th, the cybersecurity firm experienced a major disruption following an issue with a software update.
In the aftermath of this global software crisis, two critical questions have emerged: What happened? And how do we prevent something like this from happening again?
The Blue Screen of Death
Friday, the 19th of July 2024, gave us a glimpse of what Y2K could’ve been — a total network meltdown affecting 8.5 million users around the world. Banks, airlines, healthcare institutions, and multinational corporations ground to a halt. According to insurers, Fortune 500 companies lost approximately $5.4 billion on the day.
CrowdStrike, a manufacturer of cybersecurity software, released an update on their Falcon sensor with incorrectly programmed memory access. This caused Windows computers to crash repeatedly, preventing millions from starting their devices. It quickly became known as “the blue screen of death.”
The update was rolled out to millions of computers simultaneously, preventing them from doing business for hours. The consequences were widespread and devastating. We’ve compiled a list of the best tips from experts on how software companies can avoid massive shutdowns when rolling out updates.
Here are ten things software companies can learn from the CrowdStrike incident:
Pre-Deployment Testing
Pre-deployment testing is essential to identify and mitigate potential vulnerabilities before releasing software into production. Laveena Ramchandani, software testing lead at EasyJet, advocates for dedicated testers rather than allowing developers to test their own code. “It’s just not their role,” she explains. “We think of not just happy paths. Our first thought is: ‘What can go wrong?’ So we start with the negative parts”.The logic error in the Falcon sensor update could have been identified and rectified with rigorous testing. Rigorous testing protocols can simulate various scenarios, including edge cases and stress conditions, ensuring the software’s robustness under different circumstances.
Prioritize Incident Response Training
Incident response training is crucial in cybersecurity as it prepares organizations to handle and mitigate the impact of security incidents effectively. This training enables personnel to promptly respond to cyber threats.
The CrowdStrike Falcon team was able to quickly identify and remediate the logic error to reduce the extent to which the system was down and impacted. This is a clear example of the importance of well-prepared incident response teams. Proper incident response training involves developing a comprehensive incident response plan and drills and staying updated with the latest threat intelligence.
International Cybersecurity Cooperation
Software providers like CrowdStrike operate in a global context, which was only highlighted by the international crisis. The global reach of the outage affected systems worldwide. International cooperation and information sharing between organizations are vital to addressing widespread issues swiftly and efficiently.
International cooperation also facilitates the development of global cybersecurity standards and frameworks, promoting consistency and interoperability in security practices. Joint efforts in research and development can lead to innovative solutions to emerging cyber threats, benefiting all participating nations.
Conduct Regular Audits and Testing
Regular audits and testing are critical components of a robust cybersecurity strategy. Audits involve systematically reviewing and assessing an organization’s security policies, procedures, and controls to identify weaknesses and ensure compliance with industry standards and regulations.
Testing, on the other hand, includes activities such as vulnerability assessments, penetration testing, and security scans to detect and address potential vulnerabilities before they can be exploited.
The CrowdStrike outage demonstrated the importance of regular audits and testing. The faulty update that caused system crashes could have been detected through more frequent and thorough testing protocols. By conducting regular audits and tests, organizations can identify and rectify security gaps, ensure the integrity of their systems, and maintain a high level of security.
Cybersecurity Expertise and Funding
As cyber threats become increasingly sophisticated, the importance of cybersecurity expertise and funding cannot be overstated. Skilled cybersecurity professionals are essential for developing, implementing, and managing effective security measures.
Adequate funding is crucial to support these efforts, allowing organizations to invest in advanced security technologies, conduct regular training, and stay updated with the latest threat intelligence.
The CrowdStrike outage highlighted the need for high expertise and resources to quickly identify and remediate the issue. The complexity of cybersecurity threats and the sophistication required to manage and mitigate them, along with increased investment in cybersecurity expertise and funding, are essential to developing robust systems and preventing similar occurrences. With the growing frequency and complexity of cyberattacks, organizations must prioritize building and maintaining a strong cybersecurity workforce.
This includes not only hiring skilled professionals but also investing in their continuous education and training. Adequate funding ensures that these professionals have access to the necessary tools and technologies to protect the organization’s assets effectively. Additionally, a well-funded cybersecurity program enables organizations to implement comprehensive security measures, conduct regular audits and testing, and develop robust incident response plans.
Balance Efficiency with Security
Balancing efficiency and security is crucial in today’s fast-paced digital environment. While operational efficiency is important for business success, it should not come at the expense of security. While rapid update deployment is important, the CrowdStrike outage demonstrated that prioritizing speed over thorough security checks can lead to severe consequences.
Ensuring that security measures are not bypassed or overlooked in the pursuit of efficiency is essential to prevent vulnerabilities that could be exploited by cyber attackers. This involves implementing security protocols and controls that are integrated seamlessly into the organization’s processes, allowing for both efficiency and robust protection.
Maintain Transparent Communication During Incidents
Effective and quick communication is vital for tech companies, especially during a cybersecurity incident. Timely communication ensures that all stakeholders, including customers, employees, and partners, are informed about the situation and the steps to resolve it.
The CrowdStrike outage highlighted the importance of quick and transparent communication. Timely updates and clear communication with customers helped mitigate the impact and guide them through remediation steps. Prompt communication can prevent the spread of misinformation, reduce panic, and maintain trust. It also enables coordinated efforts to mitigate the impact of the incident, as everyone is aware of their roles and responsibilities.
Clear communication protocols and channels are required to ensure information is disseminated quickly and accurately. By prioritizing quick communication, tech companies can enhance their incident response capabilities, minimize the impact of security incidents, and protect their reputation.
Implement Phased Rollouts for Updates
Phased update rollouts are an effective strategy for managing the deployment of new software or system changes. By rolling out new versions in stages, organizations can monitor the impact of the changes on a smaller scale before a full-scale deployment. This approach allows for the early detection and resolution of issues, minimizing the risk of widespread disruption.
The CrowdStrike outage highlighted the potential benefits of phased rollouts. If the update had been deployed in phases, the logic error might have been identified and corrected before it impacted many systems.
Phased rollouts also enable organizations to gather feedback from a smaller group of users, allowing for further refinement and optimization of the update. This method not only reduces the risk of major issues but also enhances the overall quality and reliability of the software.
Ensure Business Continuity with Backup Servers and Alternative Data Centres
Backup servers and alternative data centers are critical components of a comprehensive IT strategy, especially for businesses that rely heavily on digital operations. They serve as a safeguard against data loss and system failures, ensuring business continuity and minimizing downtime. The CrowdStrike incident highlighted the need for robust disaster recovery plans to quickly restore affected services and reduce operational impact.
Backup servers are dedicated servers used to store copies of critical data and system configurations. Their primary function is to provide a recovery option if the primary system encounters a failure or data corruption.
Alternative data centers provide an additional layer of protection by hosting copies of the primary data and applications in geographically separate locations. In a disaster, operations can switch to the alternative data center, ensuring that services remain operational and data remains intact.
Automate Routine IT Processes to Minimize Human Error
Automation of routine IT tasks, such as backups, updates, and system monitoring, is essential for efficiency and reliability. Automation can help minimize human errors, such as those that might have contributed to the logic flaw in the CrowdStrike update. By automating routine IT processes, organizations can ensure more consistent and reliable system management.
Automated systems reduce the likelihood of human error, ensure process consistency, and free up IT staff to focus on more strategic tasks. For instance, automated backup solutions can schedule and perform regular backups without manual intervention, ensuring that backups are timely and comprehensive. Similarly, automation tools can manage updates and patch installations, keeping systems secure and up-to-date without constant oversight.
Conclusion
The CrowdStrike outage was the biggest software crash in recent memory. With millions of users and industries affected and billions lost in revenue, the outage shone a spotlight on the importance of cybersecurity in software updates. Several interventions should have and could have been deployed to prevent and mitigate the risks of an outage. The recurring theme across experts highlights that shortcuts cannot be taken when rolling out new code on a mass scale.
Both Crowdstrike and Microsoft worked tirelessly to resolve the issue, and the lessons learned will only propel the industry forward. Strengthening software development with strong cybersecurity protocols is how IT professionals can avoid further disruptions and provide seamless software updates.
