We’ve learned a lot in the last ten days about the software update from Crowdstrike that crashed approximately 8.5 million Windows-based PCs. In retrospect, 8.5 million is a fairly small portion of the global Windows environment, 7 or 8% (according to MSFT less than 1% of Windows machines were impacted), but millions of business computers going down at the same time had a huge impact on many enterprise networks and the delivery of services around the globe.
With the Crowdstrike incident causing an estimated $5B in damages and prompting multiple lawsuits, some have argued we’re now paying the price for things like “software monoculture”, poor “security by design” choices, or the absence of liability for software providers. Much of the blame will likely be pinned on Crowdstrike and faulty patch management, which can be a notoriously difficult operational process to execute well, both for individual organizations and especially at Internet scale. However, it is important to understand the role of public policy in contributing to this incident.
In 2009, Microsoft voluntarily agreed to open up its about-to-be-globally-launched Windows Vista operating system making changes to its security (among other things). This decision, taken in consultation with EU regulators, followed years of inquiry by the European Commission and other regulators into anti-competitive practices and ultimately the 2004 Order that required Microsoft to make certain products interoperable and un-tied. In light of this Microsoft decided (consistent with the 2004 remedies) to gain the good graces of the Commission, circumventing its own OS kernel security product (Patch Guard), by building kernel-level application programming interfaces (APIs) so that third-party security software vendors could legitimately gain access to parts of Vista they needed to provide security products and services. Thus, a thriving market for producers of “endpoint security” was fostered, into which existing (Microsoft, Symantec, McAfee, Trend Micro, Kaspersky, ESET, etc.) and new competitors (e.g., Crowdstrike in 2011) entered. Today that global market is estimated to be worth nearly $15B with numerous competing firms, including the well-respected Crowdstrike which has about 20% of the market providing services to numerous corporations and government agencies. Another perspective would be that Microsoft’s 2009 decision created a vulnerability and increased risk by facilitating access to its desktop OS kernel, something that other (much smaller at the time) OSes didn’t allow as easily. However, this ignores the fact that implementing cybersecurity has always been about making difficult tradeoffs.
Fast forward 15 years. The Internet and the devices connected to it are wildly different. The dominant mobile platforms (Apple iOS, Google Android) have eaten into the market share of Microsoft’s desktop OS. Since 2009, mobile and desktop environments have flipped positions in the market. These mobile platforms now face the same regulatory pressures Microsoft encountered, with European policymakers using the Digital Markets Act in an attempt to increase competition by mandating the openness of designated “gatekeepers”. Apple and Google, along with all the other platforms that have been legally classified as gatekeepers, emphasize the risks to security that such openness may raise. Platforms will no longer be able to maintain accurate contact data about their users. Many of the users will have no credentials in the platform they are visiting. Any time one competitor gains a new customer, it becomes a customer of all the other platforms. The inability to combine user identification data across different platform services may inhibit the detection and mitigation of threats. Opening up a closed platform to the sideloading of unvetted apps bypasses the security checks competing platforms now conduct, making app security fall to the lowest common denominator. Nonetheless, the platforms have begrudgingly complied with the law. Once again, we need to understand that we are making a tradeoff, between security and ease of market entry. We will have to wait and see if new, unexpected vulnerabilities will be created by mandated interoperability.
Background about the Crowdstrike incident and its impacts:
- European Commission’s 2004 order forcing interoperability and un-tying of Microsoft products: https://ec.europa.eu/commission/presscorner/detail/en/IP_04_382
- Microsoft’s voluntary changes to Vista, opening up its security to updates from third-party vendors: https://www.seattletimes.com/business/microsoft-makes-vista-changes/
- Microsoft on why do security solutions leverage kernel drivers? https://www.microsoft.com/en-us/security/blog/2024/07/27/windows-security-best-practices-for-integrating-and-managing-security-tools/#why-do-security-solutions-leverage-kernel-drivers
- Market share in endpoint security: https://www.crowdstrike.com/resources/reports/idc-worldwide-modern-endpoint-security-market-share-report/
- Crowdstrike External Technical Root Cause Analysis — Channel File 291: https://www.crowdstrike.com/wp-content/uploads/2024/08/Channel-File-291-Incident-Root-Cause-Analysis-08.06.2024.pdf
- Tech Analysis: Addressing Claims About Falcon Sensor Vulnerability: https://www.crowdstrike.com/blog/tech-analysis-addressing-claims-about-falcon-sensor-vulnerability/
- Current European Commission legislation requiring interoperability of gatekeepers https://digital-markets-act.ec.europa.eu/about-dma_en
- Apple iOS kernel security: https://support.apple.com/guide/security/operating-system-integrity-sec8b776536b/1/web/1#sec41bf3cd61
- Android kernel security: https://source.android.com/docs/security/overview/kernel-security
- OS kernel verification: https://cseweb.ucsd.edu/~dstefan/cse227-spring20/papers/sel4
- Germany’s BSI calls meeting hopes MSFT will commit to restricting access to kernel: https://www.wsj.com/articles/german-cyber-agency-wants-changes-in-microsoft-crowdstrike-products-after-tech-outage-05b82c19
Seriously? – “However, it is important to understand the role of public policy in contributing to this incident.”
Hi, Brenden; this is a new perspective on the vulnerabilities that allowed the CRWD software update. My question is, how does the interoperability mandate create the same vulnerability on iOS machines? Thanks
It’s a good question. Probably not the same one given how Apple prevents modifications of kernel and driver code. Nonetheless, according to DMA gatekeepers must allow effective interoperability with an OS, hardware or software applications (Art. 6(7)), and allow the installation and use of third-party apps or app stores that do not endanger the integrity of the device or OS (Art. 6(4)). The problem is a platform can’t know in advance what vulnerabilities might emerge with mandated, effective interoperability.