The cause of the CrowdStrike mess has been revealed by the company, together with the steps it has taken to ensure nothing similar can happen again.
The company is facing a deluge of lawsuits over the estimated $5B worth of financial losses incurred by its clients, but the small print in its contract may protect it …
A quick recap on the outage
A huge mistake by cybersecurity company CrowdStrike last month caused a global IT outage on a massive scale, with airlines, banks, health services, and more affected – including some 911 centers.
Airlines were forced to ground flights, broadcasters were taken off-air, retailers were unable to accept payments, hospitals couldn’t book appointments, and much more.
- A faulty update was issued by the company
- Because CrowdStrike has kernel access to Windows, that crashed machines
- Machines couldn’t be rebooted without a manually-applied update
- The scale of the disaster was because most major corporations use CrowdStrike
CrowdStrike cause
What wasn’t known until yesterday was the exact nature of the faulty update, and how it was issued globally without the problem being spotted in testing. CrowdStrike has now explained both.
In brief, the company wanted to make it easier to issue new threat updates to client PCs. To do this, it used a new approach which allowed dynamic configuration of the threat-detection. Protected PCs watched for updates based on a template comprising 21 pieces of data.
The crash occurred when CrowdStrike issued a template instance which contained only 20 pieces of data, one less than expected.
Therefore, the attempt to access the 21st value produced an out-of-bounds memory read beyond the end of the input data array and resulted in a system crash.
That then brings us to the second question: How was this not picked up in testing? In short, because the testing wasn’t done with real data.
The selection of data in the channel file was done manually and included a regex wildcard matching criterion in the 21st field for all Template Instances, meaning that execution of these tests during development and release builds did not expose the latent out-of-bounds read in the Content Interpreter when provided with 20 rather than 21 inputs.
CrowdStrike says it has made changes to ensure the instances match the templates, and has added runtime bounds to ensure that even if there’s a mismatch, it won’t cause a crash. Finally, it will in future do a staged rollout, so any problem which does make it through will only affect a limited number of PCs.
Small print may protect it from lawsuits
CrowdStrike is also facing a slew of lawsuits from large corporations, small businesses, and even its own shareholders. Wired reports:
On July 29, Delta informed CrowdStrike and Microsoft of its intent to sue over the $500 million it claims to have lost as a result of the outage. A class action lawsuit has been filed by law firm Labaton Keller Sucharow on behalf of CrowdStrike shareholders, claiming they were misled over the company’s software testing practices. Another law firm, Gibbs Law Group, has announced it is looking into bringing a class action on behalf of small businesses affected by the outage.
However, CrowdStrike terms and conditions impose strict limits on liability, and Jonathan Cardi – a law professor specializing in civil liability cases – says that negating this may prove challenging.
Those hoping to recover financial losses will need to find creative ways to frame their cases against CrowdStrike, which is insulated to a great extent by clauses typical of software contracts that limit its liability, Cardi says. Though it may seem intuitive that CrowdStrike be on the hook for its mistake, the company is likely to be “pretty well-guarded” by the fine print, he adds.
Photo by Ivan Vranić on Unsplash
FTC: We use income earning auto affiliate links. More.
