Bonus: The Global Story - Crowdstrike 26:27

Hello, this is the Global News Podcast from the BBC World Service, with reports and analysis from across the world. The latest news seven days a week. BBC World Service podcasts are supported by advertising.

Well, we got a minute. I'm gonna buy that truck I've been wanting. Wait, don't you need, like, weeks to shop for a car? I don't. Carvana makes it super convenient to find exactly what I want. Hold up. You're buying a car on your phone? Isn't that more of a laptop thing? You can shop wherever you want.

I like to do my research, read reviews, compare models. Plus, Carvana has thousands of options. How'd you decide on that truck? Because I like it. Oh, that is a great reason. Go to Carvana.com to sell your car the convenient way. It's that time of the year. Your vacation is coming up. You can already hear the beach waves, feel the warm breeze, relax, and think about...

Work. You really, really want it all to work out while you're away. Monday.com gives you and the team that peace of mind. When all work is on one platform and everyone's in sync, things just flow, wherever you are. Tap the banner to go to Monday.com. Hello, this is Valerie Sanderson from the Global News Podcast. I'm here to introduce you to another pod from BBC World Service. It's the Global Story. They take a deep dive into one big news story every weekday.

Let me hand you over to Lucy Hockings for a flavour of what to expect when you subscribe. This episode looks at the fallout from the biggest IT outage in history, CrowdStrike.

A glitch at a major cybersecurity firm that brought banks, hospitals, airports and thousands of businesses to a standstill. Chaos and growing passenger outrage as Delta Airlines... Routine appointments, prescription requests and test results were all affected... People had never even heard of CrowdStrike before today. Now everybody's wondering, how did this happen? We live in an ever more digital world and the power to derail it, even by mistake, is concentrated with just a few corporations that keep the whole system running.

So what do we know about these cyber giants? And how vulnerable are they to accidents or attacks that could shut down wide sections of society? With me today is the BBC's cyber correspondent, Joe Tidy. Hi, Lucy. Joe, you're the person we always turn to when we need to understand what's going on in the digital world.

But I wanted to take you back to the day or maybe not even the day, the moment where we found out about the CrowdStrike outage and what your first gut visceral reaction was. Well, I remember well, because I was up at about just after 6am and I looked up a quick look at Twitter before I got on with my kind of morning routine stuff.

and I thought, oh, this is interesting, what's happening in Australia. Sounds a bit weird. Sounds like it's quite serious for the Australians. I wonder if that'll be a story.

And what, of course, transpired was that it all started in Australia because that's where the people were awake when the update hit. It was at about three o'clock when suddenly all of our computer screens flashed blue, the dreaded so-called screen of death. Throughout the morning, it got more and more serious as obviously more countries came online and realised that their computers were also badly damaged.

I am flying from Delhi to Polkapel and we are absolutely affected by... And in a real state because of this cascading wave of chaos around the world.

So, Joe, what actually went wrong? CrowdStrike is this cybersecurity company, one of the giants of cybersecurity, a US tech firm, about a quarter of the market, loads of FTSE 100 and index companies, some of the biggest in the world. They use CrowdStrike.

And what they were doing was putting out a really basic standard update. It was a very small file because they'd got some new research on some latest cyber technique that hackers were using. So they thought, right, we'll just push this out like we do four or five times a month, something like that. But something was terribly, terribly wrong with the code. And

And it caused what we call a logic error or a logic bug right at the heart of a Windows computer. This was right to what we call the kernel of a Windows system. So if you think about a tree, for example, you think about going down right into the roots, into the heart of that tree, the foundations that hold it all together. That's where CrowdStrike and other products like it are kind of sitting in a system.

So when they put out an update that goes wrong, like this one, it really can bring everything crashing down. And this was, they said, due to a problem with their testing. They normally put these things through tests, obviously. They would have their own dummy computers that they put their code out to first.

in a quarantined environment and then they put it out in waves normally in waves to their customers around the world they didn't do that here and now they've said and they've promised they are going to start doing that now they're going to improve their testing and Windows is saying also that they're going to try and restart the conversation about whether or not products like CrowdStrike should have such deep access into Windows machines.

It was the sheer scale of it, Jo, though, that became the story. The impact, not just on people who needed to do their daily work, but if you needed to go to the doctor, if you had a flight to catch, if you were trying to use online banking. I mean, give us an idea of just how massive this was.

millions of computers we now know 8.5 million computers received the update automatically downloaded it and then in what we call it it bricked the systems it caused this blue screen of death

which means that your computer is in a kind of reboot cycle. You can't get out of it unless you get fingers on keyboards and actually turn off the computer and get an engineer to look at it. It is a huge number of computers, but the important thing to remember with this outage is these were really important computers. These were computers running airlines, hospitals...

banks, you name it. These were the biggest companies in the world because CrowdStrike is a quite an expensive product. We're talking about a really complex and sophisticated system that's right at the heart of your company's computers.

that's doing a normally a very good job, very trustworthy job of keeping you safe. So what happened here was the biggest companies in the world were all brought down. And that meant, as you say, we had thousands of flights cancelled. I think I saw a figure this morning about 40,000 worldwide, something like that. We saw lots of point of sale terminals in everything from cafes to supermarkets go down, lots of offices having nothing to do. Don't forget the broadcasters, not the BBC, but...

One of our main competitors went off air? Oh, of course. Yeah. I mean, this is a problem. I forget how many organizations were affected because it was so massive. But of course, Sky News, which I used to work for, they were off air for a few hours. And in fact, when they managed to get back on air, the presenters were using pen and paper and talking to each other on mobile phones. And I think

Was it ABC in Australia was affected and they went either off air or massively disrupted. So it's really hard to kind of downplay how big this outage was. And I know that in cyber, the world is always ending and it's always, you know, things are always terrible. But this genuinely was the biggest outage I've ever reported on. And I think the biggest one since I've been a cyber reporter.

You might be pleased to know I'm sitting here with a pen and paper. Oh, old school. I'm old school. There's no tech around me today. Joe, my first response when I started to see what was happening in Australia was that it was a cyber attack. Is that common? Do people jump to that conclusion all the time? Every single time. Without fail. I love that you're laughing. Well, as soon as it's a bit of a joke. I love that it's like me. Well, it's a bit of a joke in cybersecurity because people jump to cyber attack

I don't know what it is. Maybe that's the kind of most dramatic and interesting reason for outages. But normally, always, it's something really mundane and really boring that causes these giant problems. And of course, as soon as people start tweeting about it being a cyber attack, it becomes that becomes the narrative. And then you have to work really hard to try to convince people actually know someone's forgotten to renew a certificate or someone's pressed the wrong button.

And had it in a way you'd been waiting for this moment? Was there a sort of air of inevitability about it? No, I mean, we do have outages quite often in cyber, especially with our kind of alt

ultra-connected world that we live in. And there are many outages that by the time me and the tech team get our heads around what's happened, they've already been fixed. Let's say, for example, that Twitter is out or Instagram is down and you think, oh, that's affecting a lot of people. But almost as soon as you've written it and published it, they've solved it.

So when you have something like this, which over the course of a few hours, it didn't look like it was going away. In fact, it looked like it was getting worse. You think, OK, yeah, this is a pretty serious situation and not the kind of thing that you expect and sort of have run of the mill that happens every week or every month.

Given that CrowdStrike control almost a quarter of the entire cybersecurity market, I'm also a bit embarrassed to say I hadn't really heard of them. Yeah, and that's completely understandable because, as I say, this isn't your consumer product. This is kind of really rich companies. That's the kind of product they use. There aren't many companies like that. I can probably count, I don't know, four or five of them. But CrowdStrike is definitely one of the absolute top, top companies.

With CrowdStrike, you get AI-powered threat protection, industry-leading intelligence, and a team of experts to detect threats and prevent breaches before they happen. It was founded in 2011 by Dmitry Oparovitch, who is a cybersecurity expert. He's now moved on and he writes lots of books about geopolitics and does podcasts and things. It's a very, very trusted brand for these giant multinational companies. We have had cybersecurity products put out, dud, uprooted,

updates before. But when it's CrowdStrike and when it's this bad, that's when you really feel it. And it also showed, I think, how dependent we are on companies like CrowdStrike. I mean, are we putting all of our digital eggs into this one basket, Joan? Is that a problem?

Yeah, we've had this debate a couple of times in the last few years because there have been big outages, not as big as this, of course, but there have been. For example, when in 2021, Meta, which owns Facebook and Instagram and WhatsApp, had a six hour outage.

and we look back now and we think, oh, well, that's no big deal. That's six hours of, you know, maybe you can't post pictures of your lunch to your friends or whatever. But actually, we know, of course, that around the world, in the developing world particularly, WhatsApp in particular, that is a huge source of business. It is the key piece of software, whether you're a taxi driver or you're a small business owner in Bangladesh or Nigeria or India. These products are relied upon by hundreds of millions of people, and when we have outages like this...

Someone said, I wrote a piece back then about it, and someone said this really brilliant phrase, which has always stuck with me. It's, when stuff like this happens, you have hundreds of millions of people sometimes just sitting and waiting for a small team of engineers in California to fix the issue.

So Joe, I've heard of Meta, but I hadn't heard of CrowdStrike. How many other companies are we talking about that could disrupt our daily lives in this way if they went down, if there was some kind of outage? Well, there's a few. So there's obviously, I don't know if you've heard of Cloudflare, but that is a giant content management system company. They run huge parts of the internet backbone. And there's Fastly as well, which does a similar job. Akamai,

And we have had situations where they've been affected and they've brought down not just a few websites. We're talking hundreds or thousands of websites. And I'm looking at a list now from a story that I did a couple of years ago. And in July 2021, for example, there were 48 services, including Airbnb, Expedia, Home Depot, Salesforce, that went down for around an hour when there was a problem with a domain name system, a DNS, at Akamai.

So there you go. We're talking about those kind of problems are happening. I reckon probably we'll have one big one, nowhere near as big as CrowdStrike, but one big one probably every year for the last kind of three or four years. And the issue we've got, of course, is as these companies become bigger, they become better, which means they become more reliable. They become possibly cheaper. More and more other companies go all in with them and rely on them so much.

And you have this issue of giant, not monopolies, but giant companies that just own a huge part of the market and

And part of the issue with a company like CrowdStrike is that when you employ an endpoint detection system like CrowdStrike's Falcon system, you have to embed that into your computer network. And it takes quite a lot of effort and time and investment to take that out and put a new one in. So I think what we see is these giant problems, these giant outages from companies that not only we rely on without knowing it, but the companies that we use rely

rely on them. And they actually are kind of, in a sense, they're a bit stuck because yes, everything works normally 99% of the time. But when they don't, it's very hard to get out of those contracts and get out of those products.

So, Jo, we know that the matter and CrowdStrike outages were accidental. And because I do like a bit of drama, as mentioned, I'm one of those people who did leap to the notion that it was a cyber attack. Can I ask you about that? Because surely they're so big, they've got so much influence that they are targets for cyber attacks.

Are they well protected? Have they got all the right measures in place, do we assume, to protect them from cyber attack?

But I was listening to some analysis recently that said that, wow, there must be what we call APTs, advanced persistent threats, which are hacker groups that are government run. They must be looking at this thinking, hang on a minute, why are we trying to hack hundreds or thousands of different companies when we can just go for one and we can get 8.5 million computers? So, yeah, I think you'd look at it and you think, wow, that's sort of inspirational information.

In a sense, with CrowdStrike's outage, it performed what was the worst downtime that you could ever imagine. And it wasn't malicious. It was accidental. So, yeah, you'd think, well, hang on, how can we do that in a malicious way? We have seen malicious attacks in the past. In 2017, there were two. There was the WannaCry attack and there was the NotPetra attack.

which were two malicious cyber attacks which spread uncontrollable crypto worms around the world, which locked up lots and lots of systems. It looked at first like an attack just on hospitals in the UK, but it's now becoming clear that this malicious software has run riot around the world. Russia, the United States and many points in between have been hit by what's now a common form of cybercrime. But...

Again, we're not talking about the scale of CrowdStrike. The WannaCry one, for example, was about 300,000 computers in 150 different countries, thought to have come from North Korea. The NotPetya one was a similar number, thought to have come from Russia, that attack. So we have seen malicious attacks, nothing like this. And I'm certain, as you say, that there will be malicious hackers thinking, well, hang on, this is inspirational.

So we've looked at how reliant our businesses and institutions are on just a few large digital companies and what can happen when there's a problem. Next, I'd like to find out how resilient the digital world can be and how worried we should be about a worst-case scenario.

Whoa, easy there. Yeah.

It's that time of the year. Your vacation is coming up. You can already hear the beach waves, feel the warm breeze, relax, and think about...

This is The Global Story. We bring you one big international story in detail five days a week. Follow or subscribe wherever you listen. I'm speaking to the BBC cyber correspondent, Jo Tidy.

So, Jo, we've been talking about the disruption caused when just one of these big cybersecurity companies or online platforms suffer an outage. I don't want to sound alarmist, but tell us about the worst case scenario if we did see several of these service providers or even one of them go down. Is that something that people should be worried about, especially as more and more of our world is digitized?

I don't think so. I think it was and is an alarming situation when one company like this can cause such havoc. But I do have faith that these cybersecurity companies are well protected, especially after something like this. You know, we've been through this, we've kind of got through it, and there are so many lessons being learned, not only about reliance on too few companies, but also about testing and protecting systems better. I don't think we're in a position where we're going to have simultaneous security

malicious attacks like we saw with CrowdStrike on lots of different companies.

We have had cybersecurity companies affected in the past. There was the SolarWinds attack. Hackers launched a broad and indiscriminate effort to compromise the network management software used by both government and the private sector. Which was a malicious attack by thought to be Russia against a cybersecurity company in the US to get inside lots of US departments to kind of do spying, kind of traditional espionage. As a

As of today, nine federal agencies and about 100 private sector companies were compromised, including networks of companies whose products could be used to launch additional intrusions.

You could argue that it's not been done to the scale of the SolarWinds attack, but no doubt UK, US allies all doing the same to other countries, to adversaries. But I don't think there's any need for anyone to be really scared about the CrowdStrike thing being times by five or something and happening simultaneously. You don't need to down tools and move to the woods just yet.

That's good to hear. Can you reassure us then around things like our power, the electricity grid, the water systems, even our national defence systems? Are they well protected too from outages? That's a trickier one. The truth is that there are lots of countries around the world which are kind of putting themselves on a war footing now.

And we know, for example, from researchers that China is and has been gaining access to critical infrastructure networks. A big warning on Capitol Hill yesterday from FBI Director Christopher Wray about possible hacking attacks by the Chinese government. China's multi-pronged assault on our national and economic security make it the defining threat of our generation.

PRC hackers are targeting our critical infrastructure, our water treatment plants, our electrical grid, our oil and natural gas pipelines, our transportation systems, and the risk that poses to every American requires our attention now.

The kind of issue we have here, of course, is that we don't know how deep that access is. We don't know whether or not that's persistent or whether or not they've been kicked out. We only hear from one side as well, because, of course, all the information that we get about hacking comes from Western countries alone.

So, for example, there's this Five Eyes Alliance, which is the UK, the US, Australia, Canada and New Zealand. And they're always putting out warnings to people who run critical infrastructure organisations saying,

To be aware, because about six months ago, the warning went out that China is embedding itself inside networks ready for if there is some sort of war that they could, I don't know, cause disruption or switch off supplies, that kind of thing. Absolutely, all countries spy. Our countries spy. All governments have a need to be covertly informed. All countries seek strategic advantage. But the behavior we're talking about here goes well beyond traditional espionage.

And then about two weeks ago, in fact, last week, we had warnings from the same alliance that North Korea is doing the same. But we only ever hear from that side. So there's no doubt, of course, that the Five Eyes alliance is probably doing the same to China and Russia and North Korea. So there is a sort of imbalance of knowledge here that I've never really been comfortable with as a reporter. But it's one that we have to acknowledge is there. Otherwise, of course, we're not telling people what's happening here.

We don't know whether or not this could lead to major outages because we haven't been at that kind of World War III level of conflict, and hopefully we will never get there. What we do know is that Russia has on multiple occasions switched off the lights in Ukraine through cyber attacks. It happened twice in, I think, 2016, and it was confirmed to have happened this year as well.

Russia against Ukraine has always been seen as a bit of a kind of test bed for what can go wrong. Russia has been attacking Ukraine with cyber for many, many years now. And I think what's interesting there is that what we've learned by watching and observing the Ukraine conflict is that although cyber is kind of insidious and you can do lots of damage and you can cause disruption, nothing will really beat a missile because all the disruption we're seeing in Ukraine

None of the cyber stuff comes anywhere near to what we are seeing from kinetic attacks against critical infrastructure networks and factories and facilities. Joe, is there an argument that we should be changing the whole way that this works, that we're talking about a small number of very big companies having this major influence over the way the world runs and that we've got to come up with a different kind of arrangement? Absolutely.

I suppose that's what monopoly legislation is all about, breaking up companies to make them smaller and improve competition. I think that is the conversation that's being had right now. Obviously, it's not for me to say whether or not something like CrowdStrike should be broken up or whether or not people should...

Because what are the consequences now for CrowdStrike? Oh.

Oh, well, there's obviously a massive insurance bill to be paid. Whether or not CrowdStrike will foot that bill, we don't know yet. It's going to take a long time to work out who pays the money here. We're talking billions of dollars. If you take, for example, the most visual representation of any IT outage is always for some reason,

airlines and airports because people are there unhappily with their luggage waiting for their flights they're looking at boards you know saying cancelled or delayed this is our fourth cancellation i just wish i we'd given up and stayed home we're flying to key west but now we cancelled everything because you can't even rent cars we barely got a hotel room last night we were lucky to even got one and every one of those people we always put out on broadcasting

They're entitled to compensation, obviously. So who pays the money? I look back at the NotPetya attack in 2017, and there was a shipping company called Maersk, which had a huge amount of disruption. They were back to pen and paper, couldn't ship things around the world. And it cost them hundreds of millions. And it took years and years to decide who would pay that bill.

So, Joe, there are surely calls for more oversight of the industry after what's happened. Oh, definitely. And the CrowdStrike CEO has been called to a Senate hearing and they're going to be sat there like we've seen lots of times recently. Mr. Zuckerberg, let me start with you. Did I hear you say in your opening statement that there's no link between mental health and social media use? With the tech bros sat there kind of being shouted at by senators. That's not a question. That's not a question. Those are facts, Mr. Zuckerberg. That's not a question. Those aren't.

There's going to be a lot of discussion now over the next couple of years at least just to find out how can we prevent this kind of thing happening again.

Do companies like CrowdStrike need to have legislated better testing, that kind of thing? And as mentioned, we are talking now about whether or not there's just too much access to our computers that these cybersecurity companies have. Because when you go down to the kernel of a Windows system, you're kind of like you've got God mode access to a computer, which is partly why.

The CrowdStrike product, Falcon, is good. Like others, it's very good at what it does. But it also means that when things go wrong, it is absolutely catastrophic. So a big picture philosophical question to end on then, Joe. If we look to the future, do you think we as a society will become more resilient to

as technology improves? Or do you think actually we'll get more vulnerable because more and more of our lives are taking place online? Well, I think there's a really interesting kind of weird issue and situation that we're in where

The CrowdStrike thing was absolutely massive. The outage caused huge amounts of disruption for people around the world. But I'm actually quite surprised at how resilient we all were. Life went on. Yes, it was really horrible. And I'm not I wasn't sat in those airports looking at the departures boards with my entire family screaming at me. So I'm sat here with in a sort of privileged position. I wasn't kind of at the heart of it. And also, I wasn't one of the IT engineers.

working over the weekend to try and get their systems back up and running. And I know it was horrendous for them, but we did bounce back pretty quickly. And I think within about three days, things were pretty much back to normal through lots of pain. Yes. But the average person, every, every living everyday life, they kind of were impacted, but they got through it. So I do feel a little bit positive that if something so bad as the crowd strike outage meant that life went on, that does sort of fill me with a bit of hope. And,

And the other interesting thing, I think, which might help us explain why there was quite good resilience is because we have had years now, about four or five years of this thing called ransomware, which is a form of cyber attack that brings companies to their knees and encrypts their systems and means they can't run their computers anymore.

And lots and lots of companies have been hit. They're being hit all the time. But more importantly, it's become something that every CEO in every company knows about and every organization knows about and worries about. So I like to think that the resiliences are being built into systems in case ransomware happens. And in this sense, the CrowdStrike outage was a bit like a ransomware attack.

So I think we are in a position where we're learning each time and we've been through years of pain with ransomware that I think if the CrowdStrike outage would have happened maybe four or five years ago, it would have been way worse. Hope, resilience, learning. It's a good way to end. And Joe, do you know what this noise is? That is the sound of my pen being put down on the piece of paper. You can't hack a pen. Can't hack a pen. Thanks so much, Joe. Good to speak to you. Thanks, Lucy. Cheers. Bye.

If you want to get in touch, you can email us at theglobalstoryatbbc.com or send us a message or a voice note. Our WhatsApp is 44 330 123 9480. Thanks for taking the time to listen. Remember, The Global Story won't always appear in our feed, so if you'd like to hear more episodes, just search for The Global Story wherever you get your BBC podcasts and follow or subscribe. Bye for now.

Did you know that it's 50 years this week since Richard Nixon became the first US President in history to resign from office?

To mark this monumental moment, Witness History brings you five programs about influential events in US presidential history. And with all the amazing twists and turns in the current race for the White House, what a time to bring you them. You'll hear about the closest US election in history and from the man who was in the Situation Room during the raid on Osama Bin Laden. That's Witness History from the BBC World Service. Listen and subscribe wherever you get your BBC podcasts.