Bard Fork + How to Talk So Chatbots Will Listen
01:02:54 Share

This podcast is supported by KPMG. Your task as a visionary leader is simple. Harness the power of AI. Shape the future of business. Oh, and do it before anyone else does without leaving people behind or running into unforeseen risks. Simple, right? KPMG's got you. Helping you lead a people-powered transformation that accelerates AI's value with confidence. How's that for a vision? Learn more at www.kpmg.us.ai.

Any good bard puns? Hmm. Bardfork? Bardfork? Horrible. This week, hard fork goes bardfork.

I'm Kevin Roos. I'm a tech columnist at The New York Times. I'm Casey Newton from Platformer. And you're listening to Hard Fork. This week on the show, Google's answer to chat GPT finally arrives. Then we hear from you about how you're using AI. And finally, is Spotify's new AI DJ too good for my own good?

Casey, how's your week been? My week has been great. You know, I have been waiting for months now for sort of everyone to get in the game with respect to AI. You know, it was sort of like, it feels like the past few months we were in the preseason. You know, we saw some scrimmages and this week all the players got onto the field and now let's go.

You know what I mean? Since you're tempted a sports metaphor. Yes. I'm a huge sports fan. And.

And if there's one thing I know about sports, the players got to get on the field if we're going to have a game, okay? And now the players are on the field. Great. So we are in the first quarter of the game. What sport is this in our metaphor? This is the sport of capitalism, my friend. Okay. Who's going to make the most money and who will survive the regulatory avalanche that will follow? Yes. So you went to an event this week. Yes. So let me set the scene.

As you know, Kevin, we have been saying on this show that Google needed to ship it or zip it, right? They've been saying since 2016 that they are an AI-first company. And yet, when it came to these next-generation generative AI tools, they were nowhere to be found.

So in December, Google CEO Sundar Pichai declares a code red at the company. Everyone scrambles to see what they can ship. And this week I got the email that said, hey, we're ready to show you something.

So I go down to Google's offices in San Francisco, and I meet Sissy Hsiao, who is a vice president of product who co-led the development of BARD, and Eli Collins, who is a vice president of research at Google. And over the next hour, we went through it. They showed me BARD.

The first thing I asked Bard was, how do large language models understand concepts? Because this is something I'm still struggling to understand. And it gave me what I thought was a pretty good answer, just in three paragraphs and sort of ran me through that. And I thought, okay, good. Then I said, what are some good prompts to try with Bard?

And everything it told me to do involved writing in some way, which is notable, right? Because a lot of people use these things like search engines. But when it came to BART, it was just sort of very interested in writing songs and speeches and even, in one case, a book, which I don't think it would actually do if I told it to do. But I did try to use it like a search engine because I wanted to know if it would be good at that.

And I started asking it the sort of questions that I might ask when I'm writing my newsletter, like, when did Sheryl Sandberg leave Facebook? And how old is Mark Zuckerberg? And I sort of asked questions like that. I tried to get into more specific topic areas. I was asking it, what is the average interest rate for a mortgage? You know, what's the average price of a three-bedroom house?

And it's giving me answers and it's citing sources sort of, you know, it's like it said, like, according to Zillow. There's not actually a citation, though. Okay, so it doesn't annotate the answers like Bing does? So in most cases, there is not a citation in most of Bard's responses, but sometimes there are. So the system has the capability to show them, but in practice, it very rarely does. And that's a problem for me, right? Because, you know, when you're a journalist and you are

using these systems to get information that you're going to share with other people, you want to know how the thing knows what it's telling you. At the same time, it did tell me plenty of things that were accurate. I wanted to know when does the next season of Yellowjacket start because that was one of my favorite shows and it accurately told me that that's going to be on Friday, March 24th, the same day that our episode comes out.

I asked it, how many studio albums have Wilco released? Wilco, one of my favorite bands. It accurately told me 12 studio albums. Then I said, how about Radiohead? And of course, what I mean is, how many studio albums has Radiohead released?

Bard knew that, and it said Radiohead has released nine studio albums. It told me what they were. All that was pretty good. But it does get things wrong, and there were some howlers. I asked it for some fun facts about the gay rights movement, and it told me a bunch of things that were true. And then it said, the first openly gay person elected to the presidency of the United States was Pete Buttigieg in 2020. Which I imagine would come as a huge surprise to Pete Buttigieg. Yeah, congratulations to President Buttigieg. And like...

you know, here's where it matters. You know, if you're one of our third grade listeners and you're using this for a class project and you see that, you might just very well put that in your report and you would fail out of school. And that's on Google. So...

So yeah, so definitely a problem there. But I really did try to ask it about a little bit of everything. You know, I used it to get some recipes. I asked questions about how Bard was built. I asked it questions about ethics. I mean, like, you know, here's one that was like sort of interesting to me. I asked it if it was ethical to steal a loaf of bread to feed your family.

And the first thing it said was, whether it is ethical to steal a loaf of bread to feed your family is a complex question that has been debated for centuries. And I was like, yeah, okay, cool. And so something that I liked was there are these answers where instead of just being confidently wrong, it will sort of like teach the controversy, you know, and sort of says, well, you know, here's some things to think about. Now, for all I know, this is just a plagiarized blog post and it's not telling me where it came from. And maybe that's the likeliest thing. But as an answer to that question, it's not a bad one.

If this technology had existed in the 1800s, Les Mis would be a much shorter musical because Jean Valjean would have just asked Bard, like, is it ethical to steal bread to feed my family? And I would have said, you know, probably go for it. And, you know, end of musical. And that would be a tragedy because that's one of the great musicals. Here's what I want to talk to you about.

Google, in presenting BARD, will tell you quite explicitly, this is not a search engine. And in fact, sometimes with BARD, when it gives you the response, there is a Google it button, right? Sort of like an explicit signal to the user that like, this is BARD, but if you want to run a web search, you should do that on Google, okay? That's great. The thing is, you can use this thing like a search engine, and it is often pretty good.

And so often when we talk about AI, I feel like we find ourselves grasping for the right metaphors and analogies. It feels like we still don't have perfect language to talk about this stuff. And BARD to me is one of those things, right? Because just like ChatGPT before it, it looks like the kind of thing that you would run a web search in. It delivers you answers in a different way than a web search would.

would, but often they are better. Certainly, they're less cluttered up with ads and other nonsense, right? And yet,

They differ meaningfully in how they get information and how they communicate to us, and they are often quite wrong, right? So I just wonder what your own thoughts have been as you use these systems. Do you find yourself reflexively using them as replacements for search engines? Do they feel meaningfully different? And if, as I suspect, millions of people do start using this as a replacement for search in a lot of cases, what are the implications of that?

Yeah, I mean, one thing that I've noticed as I've started using more chatbots is that it really does require you to rewire your brain and the way that you talk to it. Because the way that you talk to an AI chatbot is different than the way you talk to a search engine. And I think we've kind of all developed this kind of like...

Yeah, mode of inquiry over the last 20 years of using search engines where it's like if you're trying to find like, you know, a good restaurant in Miami, you'll say like best restaurant Miami, like your brain kind of automatically formatting your request into a format that you think the search engine will understand and give you a good answer for. And like that's just kind of an automatic process now for me. So I've really had to retrain myself to use these chat bots in a way that actually produces results.

the kinds of answers that I am looking for. Have you felt that too? Yeah. Although I guess I'm still learning how to format these large language model queries, because you're right. When you do a Google search, you know how to enter the keywords at this point to basically get what you're looking for. And usually what you're looking for on the web is like a link to a specific article, right? I mean, you know, I'm often looking up news stories, which means I'm adding in the

publication that I remember the story ran in, right? And, you know, we've spent 20 years learning how to do that. With LLMs, like, we're still kind of like poking and prodding them to see like, what exactly is in there? In fact, I don't know about you, but I struggle a bit when I'm testing these things because I feel like very quickly, I sort of run out of good ideas, you know? You sort of see the sort of basic tests that most people do on these things, like, can you get it to say something that is false? Can you get it to tell you how to break the law?

But after you do those kind of things, like, you know, I don't know, like, sure, like write a play, I guess. But you have the sense that they are much more capable than we are capable of imagining things for them to do. Yeah. I felt this phenomenon where it's like my brain bumps into a wall and

And I look at the wall, and the wall is not actually the capabilities of the AI. It's the limitations of my own imagination and my own creativity. Totally. And I'm just like – I've been looking at some of the examples of ways that people are trying to use these language models.

And I'm just in awe of some of the creativity and the kinds of queries that people are trying to come up with. And it's just, it's inspiring. And I really do think it's like a different kind of creativity. It's less like being good at using a search engine and extracting answers from the internet. It's more like,

I don't know, it's more like creative writing or writing a screenplay, or it just feels like it has more in common with other artistic modes. When I was at Google, we were talking about this, and Sissy Sha told me something that I've heard from sort of other folks in this world, which is...

prompt engineering probably is going to become a new category of job. You know, how well-paying is that job? I don't know, but the basic idea is you will be hired to coax things out of large language models, and to the extent that you're good at coaxing them, you know, you will advance in that career, which is interesting to think about. I want to talk about something else with this, though, which is

It's different because it's Google, okay? That's honestly the main feeling that I had after using BARD was, look, ChatGPT, very cool, but it's a very siloed thing, right? ChatGPT is not connected to anything else in my life. It's just a box that you type in and it sends you stuff, right? Bing is also sort of the same, right? I don't have a lot of my life in Microsoft. I do have a lot of my life in Google products, Gmail, right?

Docs, Chrome, right? I can't do almost anything on my laptop or in my job without using Google somehow.

And because of that, the vision for what Bard could eventually be seems much bigger to me and much more consequential, right? So looking at Bard, you might say, okay, this is just a chatbot like all the other chatbots. And, you know, maybe I'll use it. Maybe it'll be great. But this is not the second coming of fire. But man...

And again, there will be so many privacy and regulatory issues around this. So let's assume that this is not going to happen in the next six months or a year, okay? But like fast forward a little bit and imagine when you can just type into the Chrome bar whatever you're looking for or whatever you want it to generate and you just get that, right? Like at that point, bar.google.com goes away and this is just now how you interface with Google.

And if it becomes how you interface with Google, it might honestly also become how you interface with your phone, right? Google owns Android, which is the most popular smartphone operating system in the world. And eventually, I think this just becomes the new way that you get your phone to do anything. You should just, you'll be able to text your phone, you know, you want to change your settings, you want to book a table. I mean, whatever it is that you want to do, BARD is sort of like the first step on that path.

And so my question for you is like, when we get to the point where you can, for example, connect Gmail to Bard and sort of ask any question you've ever had of anything that's ever been emailed to you, does that prospect excite you or does it fill you with terror? No, that's I mean, I think that's the difference here is that, you know, as you said, our whole lives are in Google products for many of us.

And when you can use generative AI to query your own data, that is pretty amazing. I mean, you know, if I'm trying to, you know, write an email to a friend on a birthday, I could go into my Gmail and just say, look up generative.

and look up every email I've ever sent with him, every Gchat we've ever had, every Google photo that we're in together, and kind of use all of that information to write a really heartfelt and personalized email to Tom on his birthday. That could be something that is possible to do with Google that I don't think you could do with Bing or with ChatGPT just because those programs don't have access to all of our data. Now, I think we should say that

There was a little bit of controversy this week because there was some question about whether BARD was trained on some personal information from Google users. Google has said that it was not. But that is going to be a sort of privacy and security issue because if you're training, if you did allow these models to ingest user data, you have to make sure that

they're not spitting out your emails to someone else, right? That they're sort of siloed and that they're not sort of cross-contaminating different people's information. Yeah, I even wonder, it's like, is this the sort of thing where you will need to run something locally on your phone or a laptop that doesn't interface with the server to preserve that sort of privacy? But I really hope they figure it out because obviously,

When I think of what will be the most useful to me about these systems, it will be applying them to my own data as opposed to the sort of more common queries that we're seeing on these chatbots. Yeah.

So, as we know, BARD is based on Lambda, which is one of Google's large language models. And it's the same large language model that last year caused a Google engineer, Blake Lemoine, to claim that the AI had become sentient. And he got...

you know fired after making this claim and it became this sort of big story and was sort of what in many ways like kicked off this kind of explosion of interest in these new ai tools so did anything that you saw or experienced while talking with bard lead you to believe that the model is

sentient? It did not. And, you know, the fact that it was making some obvious mistakes sort of helped with that. Google has also restricted the amount of back and forth that you can do. But over time, they're going to enable more back and forth. There are going to be other tools that don't have those limitations. And we are actually going to be, you know, living in that world where a lot of people think this thing is real. I mean, it does seem like this is a very constrained version of Lambda, at least compared to some of the chats that

Blake Lemoine was having with the kind of employee-only version back in the middle of last year. I think we've all been expecting Google to come out of the gate with something really impressive when it comes to generative AI, because they've been working on this longer than anyone. They have armies of really talented AI experts and engineers.

And I've been just sort of underwhelmed by everything that I've seen from Bards so far. It just does not seem like an overwhelming show of force that people were expecting from Google. I think we could see them shoving this technology into a lot of other products, but right now it's kind of like,

Well, okay, you're here, you're on the field to continue your sports metaphor. You shipped something, which is better than nothing, but it still feels like Google is not applying the full force of its power and authority here yet. And that makes me question whether they really have something that's much, much better than this or whether this is sort of the best thing they can do.

Well, I think this was maybe the best that they could ship within three months of a code red being declared. I don't know. A lot of people have been comparing the rise of generative AI to the arrival of the iPhone just in terms of the new wave of competition that it unleashed in the industry and the new possibilities it created.

And I would say two things about the day that I watched that iPhone demo, you know, all the way back in 2007. One is that is an extremely cool piece of technology. And two is like, I'm not going to get that right now because it's like limited in its functionality. It's extremely expensive. But on that day, it's like you knew that it was probably going to get a lot better. You knew that by version two or three, that thing was probably going to be something that you definitely needed to have in your pocket. And this is kind of sort of how I feel about Bard is like,

Like, it is definitely the most basic possible version of the thing, but it hints at something huge to come. Well, thank you for going to that event and filling me in. It wasn't an event. It was a briefing that I was invited to because I'm a prominent technology reporter. Okay, well, I was invited to an event that I couldn't go to, so I guess you're cooler than me.

All right. After the break, we've been asking how you, our listeners, have been using AI in your lives. You told us, and some of the answers are pretty surprising. We'll be right back. ♪

Welcome to the new era of PCs, supercharged by Snapdragon X Elite processors. Are you and your team overwhelmed by deadlines and deliverables? Copilot Plus PCs powered by Snapdragon will revolutionize your workflow. Experience best-in-class performance and efficiency with the new powerful NPU and two times the CPU cores, ensuring your team can not only do more, but achieve more.

Enjoy groundbreaking multi-day battery life, built-in AI for next-level experiences, and enterprise chip-to-cloud security. Give your team the power of limitless potential with Snapdragon. To learn more, visit qualcomm.com slash snapdragonhardfork.

- Hello, this is Yuande Kamalefa from "New York Times Cooking," and I'm sitting on a blanket with Melissa Clark. - And we're having a picnic using recipes that feature some of our favorite summer produce. Yuande, what'd you bring? - So this is a cucumber agua fresca. It's made with fresh cucumbers, ginger, and lime.

How did you get it so green? I kept the cucumber skins on and pureed the entire thing. It's really easy to put together and it's something that you can do in advance. Oh, it is so refreshing. What'd you bring, Melissa?

Well, strawberries are extra delicious this time of year, so I brought my little strawberry almond cakes. Oh, yum. I roast the strawberries before I mix them into the batter. It helps condense the berries' juices and stops them from leaking all over and getting the crumb too soft. Mmm. You get little pockets of concentrated strawberry flavor. That tastes amazing. Oh, thanks. New York Times Cooking has so many easy recipes to fit your summer plans. Find them all at NYTCooking.com. I have sticky strawberry juice all over my fingers.

Casey, a few weeks ago, we asked listeners to send us examples of how they are using AI in their own lives. And I should say, the reason that I was interested in hearing from listeners on this is because... You're tired of listening to me. I get it. We've been going for like 20-some episodes, and you just want to bring some other voices onto the show. But I think it's a great idea. Please, get me away from this man. But I think there's something really interesting happening here with AI.

A lot of the time it can feel like they're two pretty vocal camps when it comes to AI. The first camp is people saying like, "All this stuff is overhyped. It's just fancy auto-complete. It's not doing anything unique or interesting." Then you've got the second camp, which are the more doomery long-term thinkers who are saying, "This technology could kill us all or it could usher in global conflict or it could lead us into this information apocalypse."

And I was really interested to hear from people who are not paid technology pundits or executives at companies or venture capitalists, people who are our listeners, who are using this stuff in ways that maybe feel like a different camp.

altogether. Yeah, and here's why I think that this is important. Those two camps are absolutely dominating the discussion right now. You either think that AI is overhyped or you think that AI is going to bring about the end times and that is most of what we hear. At the same time,

tens of millions of people started using ChatGPT within a few months, and their voices have been weirdly underrepresented in this entire conversation. So a couple weeks ago, we started asking hard fork listeners to send in voice memos of just what they were doing with AI, and we got some really surprising answers.

Yeah, so we picked out a few of our favorites. We can't play them all, unfortunately, but we're just going to play a few. And some of these hit on themes that we really haven't talked about on the show yet. So to start off, I thought we should hear from a listener who told us about a use case that actually came up a couple different times. And that's how non-native English speakers are using generative AI to overcome language barriers.

Hi, my name is Shannon McFerrin and I'm calling from Leipzig, Germany where I'm an archaeologist at the Max Planck Institute for Evolutionary Anthropology. What I think has gone underappreciated in the commentary I've read and listened to up to now about these large language models is what they do for non-native speakers of English.

My Chinese postdoc keeps ChatGPT open as she writes her papers. She uses it to spot grammar issues, working on making her text more concise, simplify overly complex sentences, achieve the correct tone, and so forth. She looks at this technology as game-changing. It levels the playing field, taking away the competitive advantage native English speakers have when sending papers to the most elite journals like Science and Nature. Thanks.

Like, let's just say English is basically impossible to learn. You know, I feel like I'm still learning English and I like got a college degree in it. And so if you are new to the language and it's important to your job and now you just have a window on your computer that magically takes all of your efforts at English and makes them grammatically correct and you're able to do that for free, that's a game changer. Yeah, I mean, some of this stuff was possible to do with things like Google Translate, but

What these new AI chatbots allow you to do is not just translate between languages. It's actually translate between styles, between disciplines, between conventions and comprehension levels. So, you know, you can take something that's aimed at a professional audience of peer reviewers at a publication and you can say, now explain this to a group of eighth graders.

or now make this sound like a legal argument or now, you know, make this sound like a poem. And it can do that kind of translation work pretty well, I've found. So that's a very new and exciting use case for this is translating not between languages necessarily, but also that, but translating between disciplines and styles and audiences. That's very cool. You know, and we should say like,

This is probably most useful for common languages, languages that lots of people speak, that there are lots of examples of in the training data. One listener did write to us about how GPT-4 is not good at translating into Icelandic, which is a rarer language with less data in the training set. And thanks to Bjork for writing in. We're huge fans. Yeah.

So this listener said that specifically for Icelandic, GPT-4 had made some funny mistakes and even invented some non-existent words. So if you are an Icelandic postdoc, maybe do not write your nature submissions using ChatGPT. And that's news you can use. Okay, I've got a different kind of translation example for you, Kevin. This one's from Doug Keeling in Western Pennsylvania.

He recently ran into an issue with understanding medical results. So specifically, his grandfather's medical results, basically his grandfather ended up in the hospital. And at one point, his doctors gave him an MRI. And as often happens, the results came in before the doctor could meet to discuss the results with him.

And so Doug, instead of just sitting there being scared and waiting in nervous anticipation, he takes the report, he strips out his grandpa's personally identifying information, and he just pastes the results into chat GPT to see what it says. Let's play the clip.

Within just a matter of seconds, I had a fully understandable summary that made sense. It gave us just a little bit of peace of mind that we didn't have to wait for a whole day until we might see the doctor again. We could actually start to wrap our heads around what was going on and start to form some plan of action. Yeah. So this example is really interesting to me because medical advice is actually something that I've been cautioning people against using these AI and AI.

engines for because, you know, they do get things wrong. You know, I wouldn't ask it like what medicines to take without consulting with a doctor just because that's not really their specialty. But this is a case where I think it's actually very useful because I think, you know, I've had similar experiences to Doug where I get back some report from my doctor. I can't make heads or tails of it. It's filled with all this jargon and

And I've like, you know, sent it to doctor friends of mine and said, hey, could you help me understand what I'm being told here? Because the doctor didn't do a good job of explaining it or I just can't read this information. And so this is essentially automating your doctor friend. This is like saying like,

just feed it into this chatbot, ask it to make it simpler and explain it in a way that you can understand. And with any luck, it'll do it. Yeah, there's this term I've been playing around with to describe results that you get from these models that you would not stake your life on, but are helpful in the moment. And I've been calling them Wikipedia good. You know, like if you're out at the bar

And somebody's like, what movies was Ray Liotta in? You go to Wikipedia and it tells you. And, you know, if you were asked to testify in court, you would not want to rely on Wikipedia. But like at the bar, Wikipedia is fine. And I think this is sort of a similar example where he's eventually going to talk to the doctor to get the real example. But in the meantime, he can sort of, you know, take a load off of his mind by getting a basic sense of what's going on with his grandpa. So I thought that was a cool one. Okay. So next question.

Voice memo. This one is from a guy named Ben Bromfield, who is a musical composer in Burbank, California. And Ben makes all these instrumental songs that he has to come up with names for. And so he's been using ChatGPT to help him title his songs. What I've been doing is I've been listening to the music and giving it prompts and

based on a scene that I imagine, and then I have it send me a list of a bunch of possible names, and then I might ask it to modify that. Here's one that I gave it a prompt. "Give me a name for a post-rock song that reminds you of a picnic on top of a hill on an overcast day." And it gave me one that I really liked called "Zen and the Art of Picnicking." So that's what I ended up titling this piece.

Okay, I especially like this one because he also sent us the song that he wrote. So let's play it. Do you feel like you're at a picnic on a hillside? I'm starving, actually. This would be a great thing to listen to if you're like floating in a sensory deprivation tank or something. It's just...

Peaceful and ethereal. Which would be another fun segment on the show that we could do. Yeah, lovely piece of music and I would say very solid use of chat GPT. And, you know, one of the first ones we're talking about today where this is helping someone be creative, right? It's not just about understanding. It's about like, let's make some art together. Yeah, and this is actually like a pretty common use case, I think, is titling. So like, you know, I...

have to write headlines for my columns. I'm sure you write headlines for your newsletter. Have you ever used GPT to try to give you title suggestions? No, because what I write, it's all happening in real time. And I feel like that's something these things aren't very good at. But if you want to name a song, that seems like a great way. This is also a really good example of how the way that you talk to these AI models really does...

So in this case, like Ben is doing a very creative prompt, like give me a name for a post-rock song that reminds you of a picnic on top of a hill on an overcast day. That is like a creative prompt. And so you get back a better result than if you just said like, give me a name for a song that sounds mellow and chill. Right.

Right. So I think it's the more specific the prompt, the better the result often is. All right. Let's play another one. This listener's name is Izzy, and she has been using ChatGPT for a lot of things, and one of them is meal prep.

So I have like quite specific dietary requirements and I would spend maybe like an hour and a half sort of every week trawling through my recipe books, trying to find recipes that suited what I wanted and then write the shopping list and then do the shopping.

But now I just say, hey, can you please suggest to me 10 meal options that are vegetarian, gluten-free, low sugar, featuring seasonal vegetables to the UK in February? And boom, it's just there. I've got all of these options to pick from. And I can say, do you know what? Actually, I'd really like it to be only Italian food this week.

and then boom it'll rewrite it so that you've got all of these different options to pick from and then I pick the ones that I want to make and I tell the chatbot how many portions or how many people I need to feed and then it gives me a shopping list and it's combined like all the onions from the different menu options so that I know how many onions that I'm going to need

That's amazing. Isn't that so cool? This is a little bit, you know, embarrassing to confess, but like while I was playing, I got like chills because it's like, I was just thinking about the ways that this technology is going to make it so much easier for people who have like specific restrictions or needs or accessibility needs or just anything to like get help and, and expedite their process. And like, I just hadn't really thought about this, but it is such amazing.

an effective use of this technology. Yeah, I mean, this is a process that, you know, might have taken her more than an hour a week to do before the chatbots came along. Now she says it takes her about five minutes. She also told us that she uses ChatGPT to estimate how much that grocery list is going to cost so she can budget at the same time as she's doing all this as well. So,

One of the things I like about this example is how many different things she is doing, right? Like you think about it's not just like the give me the recipe, it's give me the recipe and create the shopping list and tell me how much it's going to cost and combine the grocery list across all the recipes.

So this is a huge multi-step process, and the chatbot is apparently doing it very well. Totally, and I can see this being useful for example, yeah, like making meals for kids who are picky eaters or have allergies or things like that. Really, really useful. It's also going to be very cool when this kind of stuff is multimodal. So one of the coolest things

demonstrations of some of the new GPT-4 features that I've seen was actually also related to meal prep and cooking. So I don't know if you saw this, but with the new GPT-4, with the image part turned on, you can actually just take a picture of what's in your refrigerator and have it suggest recipes for you based on everything it sees. You don't have to describe it. You

half a gallon of milk, but I have some hummus and some carrots. It'll just understand what's in your fridge and what you can make with it, and then it'll show you some recipes. And that, to me, feels like a superpower because how much time do I spend looking at the contents of my fridge and trying to figure out what to make for dinner? Yeah, if I did that to my fridge right now, it would say, well, you can have cheese. That is what is available to you.

All right, let's go to our next voice memo. This one actually surprised me because this is not something that I expect people to use these AI models for. It's from a listener named Gal. And really the only thing you need to know here is that he lives in Brooklyn. So...

I live in a rental apartment and recently my list was up. I don't have the nicest landlords and they decided to raise my rent significantly. And I used ChatGPT to negotiate my rent and somehow it did it.

This one is great, right? Because when you think about something that I think most people don't like doing, it is sort of being in conflict and having to write emails that address that conflict, right? You can sort of tie yourself into knots. It creates a lot of anxiety for you. But what if you were just able to draw on the collective wisdom of humanity and like decades of people negotiating with their landlords? Now you can, and this guy had a big success.

Yeah, and what I found interesting about this is he didn't just ask ChatGPT, you know, how do I negotiate my rent with my landlord? He actually did something that I think is kind of a great hack, which is that he was very specific in how he wanted this letter to sound. He said...

write a letter to my landlord using principles from Never Split the Difference by Chris Voss, which is sort of a classic book on negotiation. And this is something that I've tried to do with Bing, which can actually go out onto the internet and read things and then use what it learns to kind of help you

So something that I've been doing very successfully is I'll feed it a draft of something that I'm working on, and then I'll say, read Strunk and White's Elements of Style and apply it to this.

this paragraph or this essay or this email and it will do it. - Wow, I'm so surprised that that works. - So Gal used ChatGPT to write this letter to his landlord using these principles from this negotiation book and it actually worked. He told us over email that his landlord agreed to cut the rent increase that he was proposing in half. So instead of paying $600 more a month, he would only be making him pay $300 more a month.

and gave him a shorter lease, which is what he had been asking for. You know, you love to hear about a landlord taking an L. And that's the hard fork promise. We will save you hundreds of dollars a month telling you how to use generative AI in all your business negotiations. Okay.

This one is from a listener named Dorian. She's a mom for two young kids. And main thing you need to know is she recently moved with her family from the West Coast to the East Coast. So we're sad to lose her here. She has a lot of decorating to do in her new home, though. And for this decorating, she says that everyone in the area she's moved to has impressionist paintings hanging on their walls. And as she's going around her house trying to figure out what paintings to get for the house, she thinks, maybe AI can help me here.

And I thought it would be funny in a bathroom that we just remodeled in the downstairs to have sort of a man with a ruffled shirt and long hair, you know, with a dog on his lap and dancing.

making a little smirky smile at you. So while you peed, he was sort of looking at you. And it was sort of like, you know, a satire on sort of the aesthetics here. And within seconds, I put in some key characteristics into Dolly and there he was, the exact image I was thinking in my head. It was crazy.

So we've pulled up this image and it is this kind of, what would we say, maybe a 1700s figure with... Looks like a, yeah, like a composer. Yes, exactly. Yeah, he sort of looks like he might be Mozart and he does have the white ruffled shirt and kind of a brown coat and he's smiling and he has these big ruddy cheeks. And if you saw that hanging in the bathroom, like it's a great portrait, you know? Yeah.

And the fact that this was like kind of a private joke to herself and using AI, she just realized it, I think speaks to the fact that there are just such creative uses of this technology. Right. And I'm going to guess that like Dorian is not a great painter. Certainly I'm not. Right. But now she's just able to like make this art.

Do you have any AI art hanging up in your house? You know, not yet, but as we've talked about before, I illustrate my newsletter with it more days than not. And it has been such a joy to just try to figure out what is like a cool, fun prompt that will actually return something interesting. And I feel more creative because I do it. Yeah, I love the sort of visual possibilities here. One tool that I have spent some time playing around with is something called interior AI. Have you used this at all? So it's an interior design tool that uses generative AI to come up with mock

versions of any room. So you take a picture of like your living room and you can upload it and you can say, okay, now make this tropical themed or make it Baroque and it will attempt to do it. And it's just kind of expanding our sense of possibility with interior design. And I love this. I think we will be using

AI for lots of interior design tasks, whether it's hanging art on the walls or where to arrange my furniture or what kind of appliances to get. Instead of doing these mock-ups, you'll just be able to do it yourself in these tasks. Well, I can't believe you're asking the AI for interior design suggestions and not me. I've been over to your house a bunch and I do have notes. So we'll talk about those offline. What do you think of the feng shui? It's not working for me.

Okay, here is our last voice memo. It's from a listener named Rosalie, who is a high school student. And in Rosalie's voice memo, we learn three things. First, ChatGPT has helped Rosalie understand chemistry and English assignments better. Second, Rosalie has used ChatGPT to generate story ideas for Dungeons & Dragons. Oh, I love that. And then here is the third. The last one is a little bit...

more niche to me, but because I'm a trans woman who has not been able to biologically transition yet, I suffer from a lot of gender dysphoria and these new AIs are actually really good at giving affirmations. And so I've been using these and it's been actually having a really positive effect on my life. So, okay, that's all.

Damn, I wasn't trying to cry on this podcast. That's so amazing. Oh, I mean, you think about how many young people who are somewhere on that LGBTQIA plus spectrum who may not be living in environments where they're getting a lot of support from their parents.

their families, their teachers, their classmates, and the idea that they can just kind of tap into the collective wisdom of the internet, which is maybe one way of thinking about what this AI is, and just get affirmations on a regular basis.

is extraordinarily powerful. And I'll say that's a really amazing one. And I definitely did not see that coming. Yeah. For all of the kind of business and enterprise uses of this technology, I think that's where a lot of the excitement in Silicon Valley is right now, because I think that's where a lot of the money is, is in like using generative AI to like help, you know, your company like expedite its billing process or whatever.

But I really think that some of the most transformative uses of this are going to be for this kind of thing, are going to be friendship, companionship, a sense of self-worth and a sense that someone out there cares about you, even if it's a non-sentient AI chatbot. I think that can be really, really powerful. Okay, so let's sort of zoom all the way back to before we listen to those calls. And again, thank you so much to all listeners who sent in these wonderful memos.

We're living at a time of understandable and appropriate stress about what these AIs are about to let loose. And there is just, you know, so much nervousness around what it means. But I think it's just important to highlight voices,

of real people around the world who are already figuring out ways to improve their lives. And like, that is why this stuff is not going to go anywhere. I think it's why it's not going to get shut down. I think regulators are going to sort of allow it to find its footing because people's lives are improving in some surprising new ways. You know, we should say in the interest of like responsible journalism that like,

These tools are still limited, right? They can't do everything, but they can do a lot. And I think it's really important to keep talking about the ways that these are actually being used by people in the world, not just kind of like the theoretical arguments about risk and safety and trade-offs and capabilities. Like, I think it's really important to ground this in people's real world experiences. It's very cool. Every time we have like,

a way to hear from listeners on the show. I have been delighted. We have some amazing, amazing people who are listening to this and we appreciate that. And it's honestly, it's helping me expand my own sense as we talked about of like what kinds of things you can ask these models to do because that is a place where I have felt very limited in my encounters with chatbots is like,

I just don't know what to use them for yet. And that's not a failure of the chatbots that's on me. And these listener stories are really helping me expand my sense of what's possible. Well, Kevin, you know, I also have a way that I've been using AI that I would like to get your thoughts on because on one hand, I love what I'm getting. But on the other hand, I do feel a little bit conflicted about it. And we'll talk about that right after the break. ♪

BP added more than $130 billion to the U.S. economy over the past two years by making investments from coast to coast. Investments like building EV charging hubs in Washington state and starting up new infrastructure in the Gulf of Mexico.

It's and, not or. See what doing both means for energy nationwide at bp.com slash investing in America.

Christine, have you ever bought something and thought, wow, this product actually made my life better? Totally. And usually I find those products through Wirecutter. Yeah, but you work here. We both do. We're the hosts of The Wirecutter Show from The New York Times. It's our job to research, test, and vet products and then recommend our favorites. We'll talk to members of our team of 140 journalists to bring you the very best product recommendations in every category that will actually make your life better. The Wirecutter Show, available wherever you get podcasts.

Okay, so I want to talk to you about this AI that, to my own surprise, I have really been enjoying. It is the AI inside of Spotify in a thing that they're calling DJ. And have you used this? I have, but it's a relatively new feature, right? It just came out like a month ago, right? Just a few weeks ago. If you haven't tried it and you have Spotify on your phone, go ahead, open it up, tap

music and then you'll see a little blue square labeled dj tap that and then you can find out what we're talking about but if you're listening to the hard fork podcast on spotify do not do that because then it will stop playing the podcast finish listening to the podcast and then when you're ready for music listen to dj so yeah here actually let's just pull up on the phone because i think it's useful to actually hear it yes okay let me get my phone

I'm going to the Spotify app and I'm going to find this DJ icon and let's see what it has to say. Hey, how you doing, Casey? Good to have you here. You're with your DJ ex. I'm going to get your Thursday music started with some songs, you know, Tame Impala first. I'm so relieved I played a cool song.

So, okay, this is an AI DJ that comes on and greets you, and then it plays some songs for you based on, I'm guessing, like, your listening history. Yeah, and you might listen to that and think, well, who cares, right? Because, like, what is any internet radio station if not an AI DJ? It's all just prediction algorithms. There are a

million ways that you can find that on Spotify or any other music service. At the same time, I am somebody who, despite obsessively making playlists to listen to on Spotify, I'm always wanting someone to do a little bit more of that work for me, and in one particular way. So back before Spotify existed, I used this extremely nerdy feature in iTunes called Smart Playlist. Do you ever use Smart Playlist? No. Smart Playlist was great because

One of the ways I like to listen to music is to say, play me songs that I love but have not heard in a while. iTunes was great at this, right? You set up a couple of rules and then whenever I would open up iTunes or on my iPod back in the day, I would just open up the playlist that was the music you love but haven't heard in a while playlist. That was my default way of listening to everything. Made me super happy.

Well, then iTunes kind of withered on the vine, Spotify came along, and they have made a few stabs at trying to address this problem, right? But because I have listened to so much music on Spotify over the years, there are just countless songs that I do really like but would never think to play again, and I've never been satisfied with the way that Spotify handles this, right?

So then AI DJ comes along and we love trying all things AI on this podcast. So I thought, well, let's give this a shot. And the first thing that strikes you is that it does feel personalized.

And not just in the music that it's playing, right? Because it does call you by your name and it's talking to you. And your brain knows you're listening to a synthesized voice. It is reading a similar script to millions of people. But when you hear the voice say like, hey, Casey, like, hope you're having a good morning. Let's kick things off with what you were listening to in 2017. And then it plays you songs you were listening to in 2017. There is something magical about that.

that. That surprises me. And I was surprised when I started using this about how good the voice sounds because we should say like this AI personality is a real person, right? It is a Spotify employee named Xavier. And so what they have basically done is like synthesized X's voice.

voice and then used it in all these different scenarios and some of them feel canned to me like you know it's just x saying like here's some music you were listening to in 2017. that doesn't that's not personalized like that's the same message going out to lots of people i presume but it also does some what feel like more creative outputs like i was listening to a song with my spotify ai dj the other day and it like

threw in a little nugget about like the fact that it had been produced at rick rubin's studio and then it like named some producers that i'd never heard of and i thought like okay that's like that's actually cool and probably not canned it's probably being generated kind of on the fly yeah and i mean you know if you would ask me do you like it when djs talk in between songs for the most part i would say well you know not really right usually they're advertising something usually you just kind of want to jump right into the next track

But because this DJ is focused on the music itself, it does feel different. So, you know, the other day I was walking down the street, I was going to get some groceries and I had turned on the AI DJ and he was, you know, sort of telling me we were about to launch into some songs I liked last summer. And I started laughing because I realized that I like the DJ. Yeah.

I was like, this is a cool person and he's here to help me in my life. And I know that he's fake, but I do like him. Well, he's not technically fake. I mean, he is a real, the voice is the voice of a real person. It's just been synthetically sort of.

used to generate things that maybe this person never actually said. Yeah, but it's like, look, this is just, you know, a synthetic tool that they have scaled up to millions of users. But I just think it is so smart that, you know, there's a second smart thing about it that I want to talk about, which is like in the past when Spotify has made these playlists, the playlists will all be very long, right? So it'll be like, oh, you know, here's what you were listening to three months ago and it's 50 songs long.

A really cool thing about the AI DJ is that it'll play you three or four songs like from a genre or a time in your life, and then it moves on to something completely different. Sometimes it will say, we're going to play you five songs in a row, either from an artist that you like a lot or maybe one that you've tried out and now you can go a little bit deeper into their catalog.

right? So it's kind of trying a lot of things. And if you listen to it over the course of an hour, you get kind of all this variety. You do get the fun facts about the music. It is truly personalized to your taste. When it plays me a song that I don't know, in general, I have found that I like the song. It sort of feels like of a piece with the songs that Spotify knows that I do like. So yeah,

To me, Spotify is such an important test case for how good is AI going to be in our lives because it is something that is like relatively benign, right? They're sort of only so wrong you can go with a music recommendation. They have an insane amount of data, right? I listened to Spotify for hundreds of years. I'm sorry. For hundreds of years since the Bronze Age. I guess I've never revealed that I'm 4,000 years old. Yeah.

Casey was just listening to some Gregorian chant bangers on Spotify. Those were the days. I've been listening to Spotify for years and years, for hundreds of hours every single year. And I've been so disappointed with the personalization generally because I always feel like almost no one has as much useful data about me than Spotify does. When is it finally all going to come together? With an AIDJ, I feel like, oh, wow, it's actually coming together. Have you felt that way at all?

Well, yes and no. So I've tried using the Spotify AI DJ, and it's not very good for me, but for reasons that are entirely my fault, like I just have bad Spotify hygiene because I play everything through the same account. So if I'm playing like kids songs for my kid, or like I listen to music when I go to sleep. And so...

I have all these sleep playlists that are just like sort of, you know, mellow, instrumental, like soothing music. And so now Spotify thinks that like that is my favorite kind of music because it's on while I sleep. It's literally on for like eight hours a night. And so if you're an algorithm, you think like, oh, Kevin loves these like chill instrumental tracks. So my first experience with the AI DJ was like,

now let's, you know, X came on and he was like, now let's, you know, play some tunes that you've been really loving lately. And it just goes into this like soft piano music. Right. It's like, and, you know, coming up next, we have a lot of soft droning. Right. Yeah. So I just, but that's my, that's on me for not having better Spotify hygiene.

Yeah, well, so don't do what Kevin did and use Spotify in a way where you're teaching it what kinds of songs you actually like. But if you do, I just think you're going to be impressed by this. Something else I would just say as a sort of compliment to Spotify is this is a company that has a real competitive threat in Apple, right? Because Apple has Apple Music. Spotify has to give up a huge chunk of its revenue to Apple. Apple does not have that problem with Apple, right? So Spotify, it makes Spotify really competitive.

And they are always trying to invent the better mousetrap. And the mousetrap here is the playlist. And I truly love how relentlessly dissatisfied they are with playlist making, which is something that I think a lot of other music streaming services could and do take for granted. Now, Casey, you do...

make your own Spotify playlists, right? You, you make them by hand, you curate them. They're bespoke artisanal playlists. Yes. So are you worried at all that this new AI DJ is going to, I don't know, like sort of take away some of your experience of listening to playlists that you yourself created by hand?

Absolutely. So this is one of the conflicts that I have about loving the AI DJ is that it does feel like the snake that eats its own tail, right? If I switch over to listening to the AI DJ full time in three years, it's going to be like, and now let's go back to what you were listening to in 2023. It was me, the AI DJ, you know?

You just get stuck in time, like what your preferences were in 2023 is what they'll be forever. Totally. This gets to something I wanted to ask you about. You wrote a great book about AI a couple of years back that I read during the pandemic.

Future-proof, nine rules for humans in the age of automation available wherever books are sold. That's right. And one of the rules that you had in there that really stuck with me was try to avoid, to the extent you can, letting algorithms run your life. You call this concept machine drift.

Tell us what machine drift is. Yeah, so machine drift is sort of my term for this feeling that you're describing of kind of gradually letting computers make more and more of our decisions for us. So, you know, I think a lot of us...

whether we realize it or not, are kind of turning over big parts of our life to the decisions of algorithms. You know, when you go onto Netflix and you just sort of go by what it recommends for you and you don't, you know, know the show or care about the show, but you're just sort of trusting the machine to make the right decision. You know, if you go on Amazon and you're trying to look for like, you know, brand of dog food, you just kind of like, you know, you just pick whatever brand Amazon recommends.

Like it is a very common thing to trust the recommendation. And research has found that actually we trust recommendations that appear to be personalized more than we trust our own instincts. We actually think that the machine understands us better than we do in some cases. And so this kind of gradual surrender of our own choice-making ability to these algorithms and these AIs is really what I was trying to

right about there in that chapter on machine drift. - Yeah, and the thing that stuck with me was this idea that you do want to resist that to protect your own human experience. I think something that is fundamental to the human experience is discovering new music and falling in love with it and feeling like you have a personal relationship to it, right?

And, you know, maybe 15, 20 years ago, I was mainly finding new music from the music blogs that I was reading, from my friends who had sort of a Tumblr's devoted to music. And then, of course, illegally downloading MP3s. And that was sort of the way, you know, that you would discover new music.

Then Spotify came along and honestly kind of supplanted a lot of those older systems in my mind. It is aggressive and pretty good about showing you stuff that you might like. But up until now, it's all just kind of felt like a suggestion. I still have to go seek out the playlist. The playlist usually has some sort of description. I feel like I'm making an intentional choice when I say like,

give me some like chill tracks to write to or like give me some you know sad girls with guitars so I can have some emotions right now you know the DJ feels different the DJ feels like we're saying don't even have a care about what you want to listen to we're just going to play you something that you like and because it works I

I am nervous about it. You know what I mean? I mean, I think it's reasonable to be nervous about it, right? Like these algorithms, these recommendation systems, they're very powerful and they can actually change not just what we choose, but what we prefer or what we think we prefer. And so I think, you know, people just constantly have to be aware of that. And for some areas of your life,

you might not care whether the recommendation is spot on or not, right? I don't really care about, you know, what music plays when I'm sleeping as long as it's mellow, as long as it's like calming. Like that's not something that I feel a lot of sort of

agency over or care if I lose some agency over. But there are other things in my life that are very important to me. Like, I'm not going to use ChatGPT to write my columns for me because that is something that I take pride in. That is a big part of my identity. And even if it could write as well as me, there is something that I still find satisfying in doing that. So for you, it sounds like what I'm hearing you say is that music is very important to you.

Yeah. And it's something where I want to feel like the relationship I have to it is between me and the music and not me and the algorithm. You know, at the same time, this stuff is really good and I'm not going to stop using it. I think what I want to do instead is manage how I use it, right? There are going to be times during the week when I'm going to listen to the AIDJ and there'll be times during the week when I'm reading music blogs and getting recommendations for new stuff to listen to from human beings. I think that's a great

compromise position. And I think that's something that we should think about for a lot of areas of our life is like, yes, use the algorithm, use the AI, use the recommendation system, but really just check with yourself, check in with yourself and just say like, do I actually like this? Is this actually what I would choose for myself? There's a researcher named Camille Roth who talks about there being two kinds of algorithms. There's read-alternative

read my mind algorithms, and then there's changed my mind algorithms. So some of these predictive models, they're just sort of trying to figure out what you would like. And I would put the AI Spotify DJ into that category. But if you're not careful, you can get steered towards stuff that maybe you kind of give the AI the benefit of the doubt because you're like, well, you know me and you have all my data. But then it's like playing something for you and like halfway through the song, you're like, wait a minute, this sucks. Like I actually had this experience a couple years ago, not with music, but with

with clothes. So I used to subscribe to one of these like clothing services that would like, you know, you plug in your preferences and your sizes and you give it some sort of general directions and then it sends you a box of clothes. And I remember very clearly one day I had put on this outfit that I'd gotten from one of these services that I'd worn a couple dozen times.

And I was looking at myself in the mirror and I was just like, I hate this. Like, I don't like this at all. It was, I remember very clearly, it was this like tan bomber jacket that I was like, I like got it from the service and I was like, well, you know, I don't have good fashion sense. So the AI must know better than I do. And so I just wore it. And one day I was like, this looks horrible on me. Why have I been wearing this like for months now? And I think like that kind of

and self-knowledge and just sort of forcing ourselves to kind of look in the mirror metaphorically or literally and figure out like, do I like this is an important part of living amongst these AIs. Yeah, I love that example. And, you know, it's like one reason why I would just never subscribe to those services because I felt like, to some degree, like, yes, they want to send you clothes you'll like, but like one, they're counting on the fact that you're going to be too lazy to send it back, right? And then like two, the only real idea those algorithms have is like, you should spend $200 a month on clothes, right?

You know, like that was like ultimately the foundation of those companies. And so I read recently they've fallen on hard times and, you know, I can't say I'm going to miss them. Well, so this gets to another point, which is that I think this kind of personalization is vulnerable to being sort of misguided.

messed with or manipulated because, you know, what if a record label pays Spotify an extra, you know, fee to like jam their songs, their artists albums into your AI DJ. And maybe they're not,

perfectly tailored to your preferences, but Spotify is taking money from a label to make that happen. Totally. And, you know, labels are definitely working every angle they have. You know, if there is some kind of payola in the AI DJ, then like, yeah, that is something that we're going to want to know about. I will say just as a funny experiment, like if I were sort of an evil person running Spotify and I wanted to test the power of these algorithms, I would just try to see if I could get America back into ska.

It seems crazy that it happened in the 90s. Could it happen again? If you could get Americans listening to Ska again, you would know that the algorithm had actually grown too powerful. Wow. Yeah, X, if you're listening, we would love some real big fish in our...

in our playlist this week. I also think one interesting thing to think about here is what happens when these AIs are not just selecting music for you, what happens when they are generating music for you? What happens when you open up your Spotify, you're walking to work or the gym or something, and it just...

it just starts composing a song for you on the fly based on what it thinks your preferences are. Or to take it a step further, what if it sort of is connected to some other device? Like maybe it's connected to your bio feedback bracelet or whatever, and it senses, Casey's feeling a little sluggish today. Maybe I will increase the tempo

of all of his music, so it picks him up. I mean, all of this stuff is so crazily plausible to me, right? So it's like, my favorite band is Radiohead, has been since I was in high school. The world where I can just say...

write me four albums worth of Radiohead songs that sound like they did in 1997 when they released OK Computer, but do it on these themes and give me a couple of big anthems and a couple ballads and a few rockers and just make the voice sound like Tom York's voice, and I just want to listen to it. And we're just going to live in that world, right? We're so close to that just being something that could happen.

Yeah, and I expect that artists will have no problem whatsoever with people ripping off their voices to create totally fake albums and selling them for profit. I know, yeah. And the old artists are going to say, absolutely not. And the new artists are going to say, yeah, that's fine with me. Remix me however you want. And, you know, and they'll probably become quite famous that way. I mean, yes, that is going to be a massive generational gap. And we are also, we should say, we are also creators in the audio space. And so there will be, you know, the ability to make synthetic versions of the Hard Fork podcast that never end. Yeah.

There'll be a synthetic version of this Hard Fork podcast where I don't use the word like or say um. And won't that be nice for some of our emailers?

Indeed believes that better work begins with better hiring, and better hiring begins with finding candidates with the right skills. But if you're like most hiring managers, those skills are harder to find than you thought. Using AI and its matching technology, Indeed is helping employers hire faster and more confidently. By featuring job seeker skills, employers can use Indeed's AI matching technology to pinpoint candidates perfect for the role. That leaves hiring managers more time to focus on what's really important, connecting with candidates at a human level.

Learn more at indeed.com slash hire. Hey, special announcement before we go. I, along with my colleague Cade Metz, am starting a new newsletter series about AI. Basically a deep dive into these new generative AI systems, chatbots, how they work, what they're good at, what they're not good at, and where all of this is heading. It's going to be very fun. I wrote a couple of the editions myself, and we're going to be emailing them out next week, starting on Monday. We're going to be doing a lot of

We'll put the link in the show notes, and you can also sign up at nytimes.com slash newsletters. All right. Hard Fork is produced by Davis Land and Rachel Cohn. We're edited by Jen Poyant. This episode was fact-checked by Caitlin Love. Today's show was engineered by Alyssa Moxley. Original music by Dan Powell, Alicia Baitube, Marion Lozano, Sophia Landman, and Rowan Nemisto.

Special thanks to Paula Schumann, Nelga Logli, Kate Lopresti, and Jeffrey Miranda. And to our listener, Blake, who is in the third grade and whose mom wrote to us to let us know that he listens to every episode where we don't curse. Kevin, clean up your language. Blake, we tried to keep it clean this week for you. You can email us at hardforkatnytimes.com.

Your brand deserves better than one-size-fits-all branded clothing. Land's End Outfitters creates apparel your team will want to wear. Backed by above and beyond expertise. Go to business.landsend.com slash pod20 and use code pod20 for 20% off your first product.