Chapters
Shownotes Transcript
WNYC Studios is supported by Zuckerman Spader. Through nearly five decades of taking on high-stakes legal matters, Zuckerman Spader is recognized nationally as a premier litigation and investigations firm. Their lawyers routinely represent individuals, organizations, and law firms in business disputes, government, and internal investigations and at trial. When the lawyer you choose matters most. Online at Zuckerman.com.
Radiolab is supported by Progressive. Most of you aren't just listening right now, you're driving, exercising, cleaning.
What if you could also be saving money by switching to Progressive? Drivers who save by switching save nearly $750 on average, and auto customers qualify for an average of seven discounts. Multitask right now. Quote today at Progressive.com, Progressive Casualty Insurance Company and affiliates. National average 12-month savings of $744 by new customers surveyed who saved with Progressive between June 2022 and May 2023.
Potential savings will vary. Discounts not available in all states and situations.
Yeah, it's another day, but what kind of day is it really? A day of doom scrolling? Laundry day? Nah, it's a White Claw day. Light and refreshing tasting. Uniquely cold-weight filtered. There's an iconic flavor for everyone. So grab a pack, grab your friends. Make the most of each day with those who matter most. When you're together with White Claw, every day is a once-in-a-lifetime experience. You just gotta grab it. White Claw. Grab life by the claw. Please drink responsibly. Parts also with flavors. White Claw Salsa Works, Chicago, Illinois.
Prime One Day Delivery is fast. How fast we talking? We talking Bunchy and Cornhole sets you order two minutes to go for your backyard party tomorrow. Delivered fast. Peaky Torches and Bug Zappers delivered fast. Can I come to your party? I'm a great conversationalist. I make friends fast. Fast One Day Delivery on 20 million items. It's on Prime. Listener supported. WNYC Studios.
Hey, everyone. We have a brand new story for you. Cannot wait to tell it and for you to hear it, but I'm actually going to make us both wait for just a second while I say two things. First, thank you. Thank you for listening, especially sustaining members of the lab. We are listener supported. It's how and why we do what we do.
That was thing the first. Okay. Thing the second is if you want to join the lab, now is a really good time to do it. Too sweet in the pot. If you join this month, you get a free Radiolab t-shirt. It'll make you look great, but also approachable. Anyway, to join and get that shirt, go to Radiolab.org slash join. That's Radiolab.org slash join. On to the show. Wait, you're listening? Okay.
Okay. Okay. You're listening to Radiolab. Radiolab. From WNYC. Okay, so I am, this is unusual because I usually prep so much and I have not really prepped that much at all. All right, you're flying fast and loose. I'm flying very loose here. With secret intel about the government and poison? No. Something. Is this about poison? Maybe. Okay.
Okay, let's just do the thing. All right. Hey, I'm Lulu Miller. I'm Latif Nasser. This is Radiolab. Okay. Basically, I was just like nosing around and I found this article in a journal called Nature Machine Intelligence. Nature Machine Intelligence. Okay. And this is a few months ago. It was in March. I don't even know what I was looking for when I was... And I just found this paper that
And it had this weird kind of boring title that I didn't understand, but I like started reading it. And the tone of it, there was something about the tone of it that was sort of breathless. Like, oh my God, we just discovered this thing and it's kind of scary. And we're not the only ones who are able to find this thing. We're not the only ones who are actually looking for this thing. And it felt like calamitous. ♪
And by the time I finished it, I was breathless. Like, I was like, oh my God, like, is this what I think it is? Because if it is...
This thing is terrifying. Okay, what, what, and what's the thing? Okay, so let me just start from the beginning. I was just trying to manufacture out of nothing a kind of open tease. No, I mean, I'm in. Okay, so here's what happened. So our scene begins with... La, la, la, la, la, la, la, la. You're muted. Oh, I'm muted. This guy named Sean. Hey! I'm muted. Duh. Okay, cool. All right. Okay. Um...
So, Sean, I just, like, I got to say, when I read your paper, basically my jaw dropped and I wanted to hear you tell the story. Yeah, I'm happy to tell you how it came about. It took quite a long time to get out. Well, before we get there, if you don't mind my interrupting, let's actually rewind a little bit. Like, who are you? What is your company? What do you do? So, my name is Sean Eakins. I'm the CEO of Collaborations Pharmaceuticals.
And this is a company based in Raleigh, North Carolina. I founded it 2015. And we work on using computational approaches for doing drug discovery.
Which basically means what they do is they use AI to discover new medicines. You're a medicine hunter and you do it through computers. Exactly. Yes. So essentially what they do is they've built like using a lot of like open source technology, a lot of like open source like databases of drugs. They basically created this computer program that's kind of like a search engine. And so they call it MegaSyn.
Or Megasyn? How do you say it? Yeah, Megasyn. So it was a really quick name for megasynthesis. Oh, and sort of synthesizing drugs that exist with receptors in brains? Kind of. It's a little complicated. It's one of those strange things where, like, I don't use it. I have one of my employees that basically codes it and puts it all together. So that employee... Sorry, let me take my mask off here. His name's Fabio. Oh, oh, oh.
Okay. Fabio Urbina. Okay. Sean and Fabio. Sean and Fabio. I love it. So what they do is they typically work with rare diseases. Which aren't considered profitable. Big Pharma has ignored them. There's no drugs for them. So what they'll do is they'll take one of these diseases. That usually only a few hundred people have. And they'll be like, okay, we need a drug that will do a very specific thing in the body to stop this disease. To stop the person from getting sick. Right. So they'll tell Megasyn.
we need a drug that can do this very specific thing. And then they'll hit search and Megasyn will comb through all the available drugs that have been discovered, all the drugs that have been even evaluated, like this giant network of basically every drug that's ever been created. And if from that, Sean and Fabio can't find a good match... Well, we're kind of out of luck. That's the end of that. Except it's not. Because Megasyn can do this other thing. It can put together a drug, basically.
So how this works is drugs are basically just made up of molecules. And the thing about molecules that work as drugs is they have a certain molecular weight range. They have certain properties. They're distinct. You can look at them and say, OK, that's a drug or, oh, that looks nothing drug like. And so using all these public databases and just inputting all this information into Megasynth about chemistry, molecular engineering. In a sense, we've we've tried to train it to make things that a chemist would make.
And so with that knowledge, what they do is they take the rare disease and they say, okay, Megasyn, in the infinity of molecules that could be drugs that don't even exist yet, can you make something... That might be active against our disease of interest. That could work here. So Fabio will enter all this stuff into Megasyn, hit run, and within minutes... It'll spit out...
These brand new, never before seen molecules. Molecules that look like drugs. And then Sean and Fabio can go through these molecules and say, okay, this is the one we want to do this thing we need it to do. Huh. Huh. To disrupt this disease that humankind doesn't have a cure for. Oh my gosh. The speed of that just needs a moment. Yeah, yeah. It's kind of amazing. It's incredible. It's incredible. Yeah. Yeah. Um...
Okay, so this is, I mean, this is where things start to get dark. So it's 2021. And these guys, they are just doing their thing and they get invited to this conference called the SPIES or SPIES Convergence Conference. Kind of international security conference. And the goal of it really is to understand how technologies can be misapplied. Part of the theme is like this idea of like dual use.
So, okay, for example, dual use is like a nuclear bomb and nuclear power plants come out of the same technology, like the so-called like double-edged sword. And so we got this invite. And of course, we're thinking, oh, well, that's interesting. Why, why, why us? How can we misuse wonderful drug discovery tools? It's never even occurred to them.
So the location was good. It was in Switzerland. They're kind of excited. I was like, oh, the pictures look really good about going on a free trip to Switzerland, as anybody would be. Jumped at the chance. In the end, it was a Zoom conference, which was a bit, you know, not as exciting. But they were like, OK, well, let's think of something like we could think of a way that we could, you know, we got this invite. Like, surely there's something we could do.
So they're sort of like brainstorming, like, okay, if we were like really evil, what would we do? How would we misuse what we know? They called it like their Dr. Evil plan. What would Dr. Evil do here? So yeah, it was very weird feeling. So they're like imagining and thinking it up. Well, we were running out of time when Sean has this idea. I hadn't given it a lot of thought. It was pretty quick.
So one of the things about Megasyn is if we are trying to generate a new drug, we want to make sure it's not toxic. Fabio basically programmed this filter in Megasyn so that if the side effects of the medicine are going to be worse than the medicine itself, like, or than the disease itself, like, not interested, not interested. Don't bother. Because, of course, it doesn't matter if your drug cures all cancer. If it stops your heart from beating, it's
going to not be a good drug. It'll save my life, but it'll also kill me. So I'll pass. Yeah, exactly. Got it. And so Sean, well, he thought, what if instead of going right, we went left?
What if we flipped the filter? What if we did the exact opposite, like photo negative of that filter? Just spit out the deadliest? Yeah, exactly. And so... I was using a 2015 Mac. That night. Just did a couple of copy and paste changes. Typed a one where there was a zero and a zero where there was a one. It was that simple. It was literally that simple. He hit run on Megasync. And then went home for the evening.
Next day. I did some work in the morning and then I think it was around noon, I just sort of opened up this file. And they check what they have. Fabio said, all right, we've got a list here. It's like overwhelming. Tens of thousands of molecules. And so they skimmed like the top crop. These top toxic molecules. And I looked at them. And so what they did is they would pick out a Megasyn molecule. Put it into a public database. Sort of like a database search to see if these molecules already existed. So they're going through this database.
To see if Megasyn had created anything terrible. Looking, looking. When all of a sudden they come across a match. With a super hideous molecule. Called...
VX. So what exactly is VX? It's the thing that... World news today, a toxic substance was indeed the murder weapon. Do you remember that Kim Jong-un poisoned his half-brother in an airport? Two women now in custody... That's VX. ...wiped his face with the toxic substance... It's a nerve agent. ...developed in the United Kingdom in the 1950s. Banned by the United Nations and classified as a weapon of mass destruction, it is considered one of the most lethal chemical substances...
Ever made. Ever created. I mean, one way to think about it is if you think about salt, a few of those crystals of salt, if it was VX, would be enough to kill you. And if you did get exposed, your muscles would start to twitch, your pupils would dilate, you'd start sweating, then you'd start to vomit. After that, your muscles would go completely slack. You'd be paralyzed, including your diaphragm, which would stop working. So you'd start to suffocate and
And within a few minutes of being exposed, you would die. Oh.
That's horrific. Yeah, I mean, it's awful. It's very awful. And Megasyn basically independently created it with the push of a button. What did you think was going to happen? I mean, realistically... Maybe it was just what did happen. Yeah, well, it was... I think what did happen was just the ease of it. I thought we would maybe get a few things that looked like VX. You did? Yeah. And...
we found a few, you know, in the literature, in publications. A few horrible things humans had already created, and they figured that'd be it. But what we got was...
Thousands of different molecules that look like VX. Thousands of brand new, never-before-seen molecules that were actually predicted to be more potent than VX. Massively more potent. Like, orders of magnitude. This is just bad, right? Yeah, it's bad. It was like the alarm bells started ringing at that point. Because, according to Sean, if any chemist got their hands on this and wanted to make some of these molecules into weapons, if they did...
because no one knows they exist, these weapons would be untraceable, undiagnosable, incurable. - Wow, Latif, this is so scary. - It's really, really scary. - I didn't sleep. I didn't sleep. I did not sleep. There was that gnawing away, you know, we shouldn't have done this. And then, oh, we should just stop now. Like, just stop. But in a minute,
We're going to keep going. Stay with us.
I'm Maria Konnikova. And I'm Nate Silver. And our new podcast, Risky Business, is a show about making better decisions. We're both journalists who moonlight as poker players, and that's the lens we're going to use to approach this entire show. We're going to be discussing everything from high-stakes poker to personal questions. Like whether I should call a plumber or fix my shower myself. And of course, we'll be talking about the election, too. Listen to Risky Business wherever you get your podcasts.
Lulu, Latif, Radiolab. So Sean and Fabio have opened Pandora's box of chemical weapons. Yeah, now we have this sort of file in our computer and all of a sudden holds all these warfare agents. Was there part of you that was just like, delete, delete, delete, and just pretend this never happened? Just unthink the thought experiment. That definitely crossed our mind. Yeah, we definitely had that sort of reaction to this. Like, I don't want to know. I don't want to know anymore.
But then they figured, wait a second. Other people could do this. You do not need a PhD to do this. You just need some basic coding knowledge, a basic laptop, and then all the data is available online for free. This could be something people are already doing or have already done. You know, these tools are in the hands of people that there is no control of.
You know, anyone could do this anywhere in the world. And so the two of them in this moment were faced with this dilemma of now that you know what you know, what do you do?
Do you tell people? Well, we have to make people aware of these potential uses of the technology and show people that, yes, these technologies can be misused. So maybe people could prepare for it. Try to prevent it. Exactly. But at the same time, if you tell people... We could inspire instead of prevent. Then maybe people that would want to do this see how far they could push it. What do you do? What do you do? What do you do? It's exactly the double-edged sword.
So what happened? So they make this decision together where they're like, we got to go to this international security conference and we got to say this out loud. But we're not going to show anybody the specifics and we're going to tell them just enough to
that they know that this is a really serious problem. Yeah. And we got to flag it and hopefully someone smarter than us can figure out a solution to this. And so they go to the conference, they present at the conference, they then publish a few months later a comment in this journal, which is basically effectively what they say at the conference. And it
It blows up. Well, maybe that's the wrong word to use, but like it, it goes everywhere. Wired, Scientific American. There was a thing on it in the Washington Post. There was a thing on it in The Economist. There was a thing on it in this thing. And a lot of people like me stumble upon this thing. There's a lot of like active
discussion about it from not just chemical weapons people, but also AI people and people in the pharmaceutical industry and philosophers and, you know, weapons people and like all kinds of different people are like weighing in on it and thinking about this and like, what do you do here? So it's kind of, I think they got what they want, but they also like every night they're going to bed thinking like, tomorrow I could wake up and some horrible thing could happen and it could have been because of me. Lulu? Yeah.
Lulu, are you there? Hello? Hello, Lulu? Oh my gosh. Sorry, sorry, sorry. I think my mic just conked out. I might need to just use my computer mic right now. Can you hear this? Yeah, yeah, yeah. Let's just, if you can hear me, let's just keep going with this. No problem. That's fine. Just in one sentence, can you just introduce yourself for me? Okay.
Okay. So in the midst of reporting this story, I ended up talking to this expert. So my name is Sonia Benoit Graham Gormley. She teaches at George Mason University. The biodefense program. And I study weapons of mass destruction, particularly biological weapons. And I found Sonia because, you know, I just, I needed somebody else to talk to about this. I needed some kind of outside perspective. Because when I read... Freaked out.
Sean and Fabio's paper. I legitimately thought this was very, very terrifying. Yeah, that's the impression it gives. And my point is that the thought experiment is just a thought experiment. It just shows that it is possible to identify new molecules, but there's a long way between the idea and
and the production of an actual drug or an actual weapon. Like, if you're a chemist who's going to make something that just exists on paper, like, that takes a long time, a lot of investment, a lot of work, a lot of thought. And it's already hard enough to do it with chemicals that already exist.
She said there's plenty of examples of scientists... Who try to transform them. Who try to tweak them just a little bit. To make them more harmful. And very often they failed. Because it's just super, super difficult. That's the point. And I told her it was still kind of hard to wrap my head around it because...
The way that I was thinking about it, I was actually thinking of like the anarchist cookbook. The molecules is almost like recipes or something, right? It's like this thing spat out 40,000 recipes or whatever, right? And then you just go to the store and get some fertilizer and no one make it. And it's the exact proportion. Exactly. So I was worried about that. Right. I'm picturing that too. Is that the right analogy or would you use like a different analogy? And she was like, no, actually, that's the same.
metaphor I use when I'm teaching my students. But think about it this way. If we take the analogy of a cake, making a cake. These molecules are like really fancy cakes. And based on what I read, right, based on the article, what they have is a list of different ingredients. And to your mind, it would take a five-star Michelin chef with a whole kitchen full of, you know, sous chefs
to kind of figure that out and to ultimately make any of those cakes. Exactly. Exactly. It's like a level of craft. It's a level of craft and expertise. It's like, yeah, you could make the David in theory if you had marble and a chisel, but like... Right. But it's like, I mean, to take a step back, I feel like your takeaway about this paper is that you...
You have not lost any sleep over this paper. No, no, I think I found it a little bit too alarmist. Huh. So Sonia is soothing us. Yeah. Are you going to set this whole section to like a nice lullaby bedtime? Like, it's okay. It's okay.
No, because there's still a problem, which is that, I mean, Michelin star chefs do exist. And, you know, I don't know if this could go on air just yet, but I don't think it's private or anything, but we were just contacted by the White House this morning. Oh, my God. Really? Yeah. We weren't expecting that. So what did they say? They want us to brief them, um,
Holy crap. Whoa. So we ended up doing a Zoom call with folks from the Office of Science and Technology Policy and the National Security Council, which was very surreal. So this was just in March 2022. Yeah.
Sean, Fabio, and some of their team got on this Zoom call. With these folks in their office in the White House with the White House crest in the background. So they do this presentation for the White House folks. I think they were worried that we were, you know, kind of crazy people and we were just going to let all of this information out there. But at the end of their presentation, there's like a Q&A and
the White House folks start asking them questions. Is this information, you know, sort of locked away somewhere? And it is. So, yeah, you know, one of the first things to do is put in a file and encrypt it really heavily. And it's on a computer that is not connected to the internet. It's air-gapped. Fabio has it on his machine, locked away, encrypted. But as the White House staff kept asking questions, Sean was like, oh. I was just waiting for the question. He's like anticipating it. It was just, when are they going to ask the question, the elephant in the room, right? And then finally they asked the question. Uh,
Can we have the data? And it's like another one of these moments where Sean has to decide, you know, maybe the government should know about this so they can anticipate it, try to regulate it. But on the other hand, now you're actually handing over the list to one of the most powerful governments in the world. Like if anybody has access to Michelin star chefs, here they are. And Sean, it's his call, right? Because he's the CEO of the company.
And Sean says, you know, no, no. And to me, that was like, I was like, oh my God, how would you even? And how did they respond? Well, the reaction was basically, I think it was like, okay, fine.
I mean, it was pretty, all right, well, you just have to accept that. And what's your rationale for saying, why not share it with them? I just didn't feel that I wanted to hand it over to them. We had other scientists reach out to us as well, asking exactly the same question. And I told the White House that I didn't feel like, you know, we should do that. We should give it over to them. And I just didn't feel like it was right. I mean- So you were basically like,
no to you, White House, and no to everybody. We're not sharing this with anybody. And you've told everybody about the existence of the list, but you're not sharing the list. What's the impetus there? I mean, it's bad enough to put the article out there and tell people how to do it and watch it sort of spread out across the globe. But if I gave them the list of molecules, it's almost like a red flag to a bull, right? The chemists out there are just going to try to figure out, all right, which ones are the easiest ones to make and then just make them. And I don't want to be the person that has to
say, yeah, I'm responsible for that. And so for now, the list just sits on Fabio's computer, kind of waiting to be used in the case of an emergency in case someone was to recreate it somewhere else and actually start making some of these things.
reporting this story, I talked to a lot of people, AI experts, weapons experts, that kind of thing. And also like from my own expertise, studying the history of science, this is a thing that just happens where something gets created and
And someone finds a way to use it for the opposite purpose, right? That's the story of the nuclear bomb and nuclear power. It's the story, and this is one we've actually done on the show before, of the nitrogen fixing process that can give you fertilizer to feed billions or, you know, gunpowder and explosives. This kind of thing just happens over and over and over in history.
But I think the thing that struck me about what Sean and Fabio were doing is how when it came to change this machine from making drugs that would help people, it's just how easy it was. I just did a couple of copy and paste changes. Flip it. Typed a one where there was a zero and a zero where there was a one. It was that simple. It was literally that simple. Like the line between this thing doing good and this thing doing bad felt similar.
So thin. And it was in thinking about this that I reached out to an old friend of mine, a guy named Yen Liu. He studies ancient Chinese medicine. He now works at the University of Buffalo. And I called him because he always would tell me this thing.
that when it comes to drugs, that an individual drug can be good or bad for you. It could go either way. That it's all about the dosage or also about the intent of the person giving it to you or about the body of the person who's taking it. It's more about the dosage
The context. Than it is about the drug itself. Exactly. And I showed him Sean and Fabio's paper, and he was like, oh, this feels like a toxicology paper. This separation between pharmacology and toxicology, we know we have a long tradition, especially in the West. It's a sense of separation between what is good and what is bad. But he's like, pharmacologists and toxicologists are studying the same molecules. But it's like, we just want them to be separate. Exactly. But every substance is...
has the potential to either heal or to kill. And that's what I keep thinking about, about the story, is that there's no real line here. Everything is capable of everything, and it's just a matter of how we choose to use it. Yeah. It's like the idea that there is a line or that you could ever stay on one side of the line is...
I don't know what the myth, the blind spot. Yeah, it's a bitter pill to swallow. But it is also apparently a sweet pill. Right, a spoonful of sugar to help the medicine go down. No, like the opposite side of the bitter is sweet. Like it's a bitter pill, but it's a sweet pill.
at the right dosage or something. But maybe like the kind of fact that most of us are not Michelangelo's or great chefs, like that that alone, it's the sadness in terms of like curing diseases, but maybe it's the comfort. Yeah. I don't know if that's my silver lining takeaway. It's like, oh, we're too dumb to make chemical weapons. That's kind of my, I mean, that's actually just sifting through all of this is like, I think I will be able to
sleep a little better at night by thinking about the fact that most of us are not Michelangelo. The Michelangelos of chemical weapons, yeah. Yeah. But even sculpting, like there are a lot of people who are like woodworkers or carpenters or crafts people. Yeah, but there's like no one who's Michelangelo. That's what I like. But then you just need a 3D printer to just, you know. I mean, I think it's like why I'm settling on Michelangelo is like, because I kind of still don't think a 3D printer could do it. Could print to David? Yeah. Yeah.
Can't it, though? Aren't there, like, if you go to any museum gift shop, like, you'll find... Yeah, but they're not. Yeah, but they're not. They're not. How different are they, though? Very. The keychain Davids? Yeah, like, they are different. I don't think they're that different, though. I think it's really different. I don't think it's that different. Well, you don't get the, like, how the lights...
But the average person will be a lucky person. It's like that in that whole artistry. Humans want it.
So mark your calendars, I guess.
This episode was reported by me, Latif Nasser. It was produced by the ever non-toxic Matthew Kilty with production help from actual pastry chef Rachel Cusick. Original music and sound design contributed by Matthew Kilty with mixing help from Arianne Wack. And given Lulu's mic situation, I'm going to sign off for both of us. I'm Lulu Miller. I'm Latif Nasser. Thank you for listening.
Radio Lab was created by Jad Abumrad and is edited by Soren Wheeler. Lulu Miller and Latif Nasser are our co-hosts. Susie Lechtenberg is our executive producer. Dylan Keefe is our director of sound design. Our staff includes Simon Adler, Jeremy Bloom, Becca Bressler, Rachel Cusick, Akedi Foster-Keys, W. Harry Fortuna, David Gable, Maria Paz Gutierrez,
Our fact-checkers are Diane Kelly, Emily Krieger, and Natalie Middleton.
Hi, this is Susanna calling from Washington, D.C. Leadership support for Radiolab's science programming is provided by the Gordon and Betty Moore Foundation, Science Sandbox, a Simons Foundation initiative, and the John Templeton Foundation. Foundational support for Radiolab was provided by the Alfred P. Sloan Foundation. ♪
