Is Bing Back? with Microsoft’s Yusuf Mehdi 53:47

On September 28th, the Global Citizen Festival will gather thousands of people who took action to end extreme poverty. Join Post Malone, Doja Cat, Lisa, Jelly Roll, and Raul Alejandro as they take the stage with world leaders and activists to defeat poverty, defend the planet, and demand equity. Download the Global Citizen app today and earn your spot at the festival. Learn more at globalcitizen.org.com.

This episode is brought to you by Shopify. Forget the frustration of picking commerce platforms when you switch your business to Shopify, the global commerce platform that supercharges your selling wherever you sell. With Shopify, you'll harness the same intuitive features, trusted apps, and powerful analytics used by the world's leading brands. Sign up today for your $1 per month trial period at shopify.com slash tech, all lowercase. That's shopify.com slash tech.

Hi, everyone from New York Magazine and the Vox Media Podcast Network. This is Dilbert with 100% less racist assholitude. Just kidding. This is On with Kara Swisher and I'm Kara Swisher. And I'm Naima Raza. Kara, Elon Musk suggested it's the media that's racist against white people. Yes, I'm sure. Like Scott Adams, whose comic strip was pulled by the Chronicle Post and more. Yes, thank you from the cheap seats in South Africa. I'll not be lectured by Elon Musk on racism. How's that? Oh.

How about his... He can't stay out of it. He has to jump in whenever there's a drama. You know who he learned that trick from? Who? Donald Trump. Trump, I know. He's like the new Trump of Twitter. He's going to end up, you know, wandering around trying to join weddings someday. At Mar-a-Lago. Hey, it's Elon. Would you like me to say a few words? No. No, thank you, sir.

Elon's latest move is getting into the AI wars. Oh, yeah, right. Everyone's talking about it. He's going to try to recruit the deep mind lever, right? Yeah. He had been involved in, or I've interviewed him many times about open AI. He was one of the earliest, and it was very laudable at the time because he didn't want it to be restricted to just Google and Amazon and Facebook, et cetera. And he was correct. And he was very supportive of that because he was worried about how it got developed, and it was very thoughtful at the time.

And now, of course, because everyone's all hot and bothered about the topic, he's got to move his little body into it. It does seem like he likes his fingers in every pot, right? Yeah. From transport to brains to rockets. Yeah, I heard he's going to lift the train in Ohio with his own arm. Oh, that's so nice. Yeah, it's cool. Cool, thanks. But the AI wars are the subject of our conversation today. We'll be joined shortly by Yusuf Mehdi, who oversees all things Bingbot and more at Microsoft.

But the media coverage of Bing bot and of AI in general, I think you've been a little skeptical of it, especially since Kevin Bruce's article in the Times. The coverage, not the topic. The coverage, yes. Explain, because there have been a lot of articles from Washington Post, New York Times, et cetera, about the creepy experiences with Bing search. And obviously Kevin's, in which Bing, aka Sydney, is.

I fell in love with him, made the front page of the New York Times above the fold. It's a lot of like poking at AI and creating stories that really aren't the point. And, you know, this has been a big story for years and years since Google bought DeepMinds. I actually broke that story a long time ago, weirdly enough, as a reporter.

And the jumps in AI have been very significant, and it's the next important iteration of computing, much the way mobile was or before that user interface, the laptop smallness and things like that. There's all these trends, and this is it. This is one that's the most important.

And the most significant. And it's being treated like it's just another, you know, dating app. It's not. This is a really important thing. And so it tends, the coverage tends towards like firm papers by college students, which, okay. Or let's see what terrible things we can make it say. But that's everybody's like dream of AI is that it's going to be sentient and...

replace humanity. I think it's just, it diminishes what's going on here and the important topics around it. And then it gets grabbed by the right wing to be about, you know, it's woke AI. Well, now everything's about anti-ESG and Leonard Leo's on that train. Yes, exactly. And so just anything can be grabbed and made political, but you have to think about it smartly. And it's

People are sleeping through this and not understanding the various implications are the ones that are going to hurt us and make it into something silly. There's also amazing things this could do. So it's just, it's the internet. It's the internet. What is the main thing to you?

What does it do? What are the implications? I don't even think we know. Because it's like, if you were back in 1994 or 95, what is the internet? What's it going to do? Could we have predicted this? No. So don't you think it's important to know these things and set up guardrails? 100%. But this has been going on for years. This is not a new thing. And it's being traded like yet another jazz hands thing. I'm sorry. It's the most significant shift in computing there has been. So I'd like to treat it with some significance.

Anyways, these are all good topics that we should talk about with our guest today, Yusuf Mehdi. He's the Corporate Vice President and Consumer Chief Marketing Officer at Microsoft. That is a very jargony title. But you've known him for years. What does he do? A long time. He used to go get Diet Cokes for Bill Gates. That's when I met him.

Bill Gates had a lot of smart young people around them, and I believe that's what Yousef was. I think I met him at one of the launches where he was putting together his computer as he got on stage, that kind of stuff. And he's been very visible during the rollout of the new AI power, Bing. It's been around a long time.

By the way, when we did that panel with Casey Newton in San Francisco, I said that this is going to be the transformative thing for Bing. And you and Casey laughed about it. Do you think it might be? No, I don't. You don't think that Bing will win because of its partnership with OpenAI? I do not. I do not.

Hmm. Hmm. So, Kara, have you been playing with the Bing bot? Yeah, I have. Has it been anything mind-blowing? No, not at all. Well, hopefully you'll get better answers out of Yusuf Mehdi. Let's take a quick break and we'll be back with the interview. You

Yusuf, thanks for joining me. You and I have known each other for a dog's age. I was trying to think when we met, but I think you were getting Diet Cokes for Bill Gates is when we kind of met, and I was in my 20s. Yeah, it's been a long time. I remember we talked a lot during the early days of the internet. Yes, we did. And I did a lot of product management work on Windows, later on the internet, then on web and search and Xbox. So

Got to make a little tour of duty around Microsoft's consumer products. Yeah, you were at everything, but you ended up at Bing now. You oversee it, correct? Yeah, now I work on all of the consumer products here, Windows, Surface, and definitely the new Bing and Edge. Which is the browser. And Surface is still around? Yeah. Oh, wow. Absolutely. Microsoft Surface, great devices. I'm talking to you now on one. Not the table, though. Not the table.

Remember the big-ass table? I do. That was quite a sea bomber, for those who don't know. Demoed a giant table we called the big-ass table, but it was a surface table. I don't think it sold very many. Did it sell very many? Not that many. It was a select few that had a taste for it. Yeah, that's right. That's right. Anyway, let's talk about Bing, though, in specifics, because, you know, Microsoft has tried a lot of things. But search is where it really did put a lot of money into it, a lot of attention. And it never...

reached altitude in general until recently. Now, Steve Ballmer did also debut Bing at Code, if you recall, and he kept saying, Bing, Bing, Bing, Bing, Bing, quite a lot. Talk about sort of why it hasn't reached it, besides the fact that Google is just overwhelming, has the overwhelming market share. Yeah, you hit it pretty well, Cara. And yeah, I was there at your Code conference. In fact, I remember standing vividly between Steve Ballmer and Walt Mossberg, trying to do a

a live demo, and that was HiWire Act. You know, Bing, I think I would say, as you summarized, we made some good progress. Today we have actually a great business with Bing. We have a great search engine that has not only helped our online advertising business, but it's helped things like Azure because of all the cloud work and the infrastructure and all the technology behind it. So it's been a...

I'd say a very successful product for Microsoft in that regard. We make money on it today. It's a profitable business. You make money on it. Yeah. But to your point, we've not had the broad success. And that's really been because of Google's sort of over-dominant reach and the fact that they got there early with all of the advertisers. And so they have that economic, you know, increasing scales of return. Right.

Now, we have been making share over the last two years, as you mentioned, and we've been growing slowly. But, you know, we're a small fraction of the search market. Yeah, we'll get into that in a second. But what has been the difficulties of doing that, of breaking in to a market? Given you're Microsoft, right? You're not like some little, like Neva or something smaller that, you know, obviously is going to take a lot more. Yeah, there's a couple things. Certainly, search is one of these businesses where...

Scale matters. And so if you have more reach, you get better data, your algorithms get better. If you have more reach, you get more advertisers and your advertising rate goes up. So that's been a tough one. And then Google has taken that and then gotten exclusive relationships like Apple on the iPhone. So they're the default search on the iPhone.

We did come with some new ideas. As you probably recall, when we launched Bing, we tried to fashion it as a decision engine. So moving beyond search to help you make decisions and take actions. And it was actually a great idea. In fact, Satya and I were talking about the other day that we were probably ahead of our time. We had the right idea, but the technology wasn't ready to deliver on it. And so even though we did good answers for you, we couldn't fire them consistently and at scale. The technology wasn't there to get that level of capability.

Which is what's exciting now about the new Bing. Talk a little bit about this process of how you decided to introduce this. And I want to go into specifics in a minute, but this is a differentiation, correct? Correct. Yeah, and if I can, maybe I'll take a step back on it because there's a lot happened between there. So search over the last two decades has fundamentally remained the same, but our use of it has changed.

And so people now are using it to do different things. So if you take the 10 billion search queries that will happen today, by our count, only about half will really get the answer they want. And that's because search, if you were to bucket it, there's three types of searches. There's what I call navigation search, where you're looking for a website, like you need to renew your driver's license, so you need the DMV. Search works great for that.

Then there's a second bucket of searches that we'll call informational. And here, today's search works so-so for simple things like you need a stock price, you need the weather forecast, you want the last night's sports scores. Search does a good job. But if you ask it anything remotely interesting and hard, like, hey, I'm going to Ikea. Will the couch I'm about to pick up fit in the back of my car? Search doesn't do a good job on that. Right.

And then the last bucket of search, and these are all about one-third, one-third, one-third, is deep research. You know, shopping. I want to plan a trip. Right.

And so today what's happened is half of these searches are kind of going unanswered. And that was the opportunity that we're trying to tap into with the new Bing. All right. So that was your demo presentation. I watched it. So this is the idea of trying to differentiate. You all need to show that you're improving search in some way that's actually significant and actually a differentiator from Google because most people are very used to Google. They know how to use it. And it's habit forming. Yeah.

One of the things that first happened, though, when you introduced this, people were sort of, ooh, ah, and then, oh. These reporters tried to push edge cases that revealed the dangers around the product. So talk about doing that. When you saw that starting to happen, was it something you expected? Or like, oh, look, this is what happens. This is just the way it goes when we introduce a product into the wild before it's ready, really. Yeah.

Yeah, it's a great question. And I'd say there's two things just to also frame that. So we launched really kind of four capabilities in one product, the ability to search, the ability to get answers, the ability to chat, and the ability to generate content with AI. The search piece landed well. So people were actually, for the most part, people are very happy, 70% plus love the search and the answers.

So it's better answers. Better answers, yeah. And 70% of people really loved it. And then chat as a tool to help you search, also great. You can now just talk to it in natural language and refine things.

There's two places where we got feedback where we could improve, and we're jumping on that. The first is on factual answers. Yeah. And this is particularly true for vanity queries, for math queries, and for things that are really up to the minute, like live sports scores. Right. And there we got a lot of feedback that said, "Hey, you can improve." And so we've done some things to improve the model. Yeah, it's wrong. It's wrong, in other words, yeah. Right. And so we were sending more data to help get better answers.

Then the other one that you're mentioning, the one that has gotten a lot of notoriety, is chats. And like I said, chat for search has been great. But one thing we didn't expect is people are using chat to do what I call social entertainment, where they're in and they're asking Bing to tell it stories. They start to ask Bing to say, hey, tell me about yourself if you were not a helpful AI. What would you be doing? And there, what happened is in a handful of cases,

We got these very long chats. They were like two hours plus. We never expected someone to be chatting with Bing for two hours.

There's a lot of lonely people, Yusuf. We found some issues with the product, one of which is chat can lose its context. And so we went in to make some fixes to help improve on that. So how have you updated that? So we did a couple things. One is we reduced the amount of chats you can have in a session, if you will. So after six questions, we ask you to refresh the context so we keep things current. Right.

The other thing we've done is we've done some work in what's called the meta prompt so that we do a better job of understanding the tone or the types of questions. So in the past, if you came in and you asked the Bing chat, hey, are you alive? Are you trying to take over the world?

It would gladly go in and entertain a lot of wild ride questions. Now we limit that so that it's not talking about itself. You can still do super interesting things, but we've done some improvement in the classifiers and the prompts. I'm surprised you didn't test it for two and a half hours. You guys didn't think people would do that? Not in a chat. Not in a chat. Okay. All right. And not in a chat about itself. And so...

In fact, that's why we do this. So to answer the question that you asked up front, most of what we found, I would say, yes, we expected it. We're very happy about it. And some things we didn't expect.

But in a way, that was my definition, because when you roll these products out, and this is our approach to how we want to release technology, we want to do it in the open. We want to do it in front of people. In fact, we want reporters to find things and write about it. That way, we're developing the right way. We don't believe you could do it in a lab. Right. Well, going back to the wrong answers thing, at the big launch demo, you played a video showing what it can do, and the answers were full of inaccuracies.

Why is that happening? I want to drill down on that first. Why is that? Why do they give wrong answers? Because people keep catching it. Sure, yeah. Things made out of thin air. Yeah. So a couple of things. In the launch demo, there was really one thing that was really wrong. I mean, people said there was a lot of stuff, but there's really just one thing. It was numbers related to a financial report. And that's where the large language model, as I said, it's not great for math necessarily. The

The way the AI works is it basically does a best estimation of summarizing content of language models to try and come up with an answer. By marrying it with search, we can now ground it. So we can give you more up-to-date information. We can check it against the relevancy rank of search pages.

And we can get you better answers. And, in fact, we're seeing better answers. But to your point, we're not going to get everything right. It's just the nature of it. So what happened with the Mexico City itinerary, for example, in the demo? So that's what I was saying. So the Mexico City, that was less of a, quote, issue. Basically, there was some question on, hey, are some of these bars or some of these restaurants described in the right way? Sometimes what happened in a couple of these cases is,

what was on the webpage deeper is different than what was on the headline of the page. And so if you go and you click in deeper, you learn more about the particular page. And in the case of that analysis, there was a question of,

Hey, how good was the AI? Did it understand all of the depth of the page or just the headline of the answer? Yeah, I see. So it doesn't know what to pick from, correct? Yeah, that's right. And that's why one of the things that we've also done in our approach is we have citations in the search so that when we give you an answer, we tell you where we gathered that data, what website it came from, and you can go ahead and click on that and learn more about it and either fact check it or get a great start to go and do more research.

So how many people are in the beta and when do you expect the product to be open to the public? What are the numbers now? Yeah, so now we're, I would say, well over a million people now are in the preview and using it. And we're scaling more people every day. So we've now, I would say we've transitioned from when we launched in, it's hard to believe it was three weeks ago. We started with a few thousand and then we started to scale up. And some of those issues you found were really only among a few thousand people.

We've since improved a lot of the things on chat, and now we're at a million plus. And we're going to continue to scale as we see it and feel it's ready. One of the recurring criticisms is that you're rushing out the product that isn't ready in order to capitalize on the hype, which there's a lot of hype, and I get that. You're a marketing guy. I understand it. Is that a mistake to do that? Because Facebook just announced some stuff that's very research-focused and not product-focused and not out in the wild, which is...

almost a 180 way to introduce this. They probably feel late and this is a differentiator for them. How do you respond to the criticisms that you're rushing out the product?

too quickly? Well, I would say that's not the case at all. We're very confident about how we're bringing the product to market for the following reasons. First is we did a lot of really heavy testing. We did a lot of special AI work, responsible AI work in the labs. And we got it to a point where we felt it's quite good. And that's why I was saying today, people are giving it predominantly 70% plus thumbs up on the search and answers. They really like it.

Yes, you're going to find some things. And you need to do that in the public. You need to bring it out and let people try it. As I said, we're only 1 million. We're like 1% of the active base. And you choose to use it. So when people say, quote, in the wild, it's not really in the wild. You have to come and choose to join the wait list. You know what I mean, though. Meta is doing a very different thing. They're going to keep it limited and not integrated with their products yet. What's the single toughest problem you need to fix from your perspective? Yeah.

- I think that, like we said, we were talking about it. The toughest problem, I think, is really getting the answers to be ever increasingly better and being able to make sure that we're using the best of our search data with the best of the AI to answer the questions directly. That's, I would say, the toughest problem. And so the AI, we're a point in time, right? Like six months ago, before ChatGPT, there wasn't even really one of these services out there. So you have to stop and say, "It's not even been a year," and look at the progress we've made.

There are increasingly new models, right? The model that runs Bing is more advanced than the model that powers ChatGPT. And we've added search. And we're not done between the great collaboration with OpenAI and the work we're doing here at Microsoft. We're getting increasingly better faster. And this data, this in-the-market research data, is helping make that system better. So I feel like we're in exactly the right place to

to get feedback, to improve, and to get better. So explain the business arrangement with OpenAI. This is a group that has created ChatGPT. It's led by Sam Altman.

I basically want to know everything. What kind of exclusivity do you have? How much of a head start do you have where there's a new iteration, this GPT-4, et cetera? And the financials would be great in detail. You're always so curious, Karen. I always disappoint you because I can never answer all your questions. Yes, you actually can. You just won't. But let's move on from that. So I'll tell you what we can talk about.

So we have a fantastic collaboration with OpenAI. We work at many levels. So we provide the cloud backend infrastructure on Azure. There's these special computers that do the inference work and the training that have GPUs on them. We've worked to build essentially what is the supercomputer with them on it. We work on the actual models themselves.

And so we do a lot of work between, again, our Bing search index and the data and what they've done with AI to improve the models as well. So we work at a very deep level. And we've invested in the company, obviously. It's what, $9 billion? Is that correct? We haven't disclosed the number. But, you know, we obviously invested a billion a while ago. A billion a while ago. Right, right. Sort of like your early investment in Facebook. Right.

And then we made more investments since then. Yeah, more recently. Yeah, people don't realize Microsoft was a very early and beat out Google actually for the earliest investments in Facebook.

That's right. So when you have this, is it exclusivity that no one else can use it? We are the exclusive backend cloud provider, and we'll work together on monetization efforts. But they can work with any company. To sell GPT-4 or whatever else. Yeah, people can use their capabilities broadly, and we want that. So in that respect, we want as many people as possible using the technology. Okay.

And so for the money you've spent there, you get early access. What do you get to GPT-4, for example? Yeah, this is where it gets more complicated. I mean, part of it is we have that partnership on the back-end cloud. We have a partnership on the advertising, how we will work together on modernization. And then we get to work with them on models that we tuned for search. So we work with them deeply on that front. But beyond that, it's hard to say more. Why did you buy the company?

Was it not for sale? Again, it's probably something I probably will beg off answering specifically. Did you try to buy the company? I won't comment on it. But what I will say is this, the arrangement we have now is, I think, turning out to be one of the best ones because...

They are able to go and innovate. They can move fast as a smaller company, and they can do a lot of different things. We're able to then do the work that we want to, productize it, to bring it to enterprise and to do the like. And so in some ways, I think we're actually able to move faster, if you can say that, to drive innovation because of the structure. Does it remind you of your Facebook arrangement? No. I think this is much more strategic and important to our core business.

So Satya said one of the goals is to build an AI supercomputer using Azure, as you just mentioned, your cloud software. Could you do that without OpenAI? Technically, we could. You know, we could build it. Obviously, we have a bunch of capabilities. I think the collaboration with OpenAI has just vastly accelerated what was possible because of— Explain why. Well, just I think they're so far ahead on the AI innovation, the work that they've done, this large language model, the training. Yeah.

I mean, for a small team, they're doing some miraculous work. And that accelerated dramatically what we were able to do. Who's making the decisions here? Is it you, Satya, Brad Smith, Bill Gates?

Explain how you're all thinking about this. Well, it's a broad team. So, I mean, certainly I would say obviously Satya is very involved. Kevin Scott, our chief technology officer, you know, has done a lot of long-term thinking on the future of AI. He kind of really led the engagement with OpenAI and the partnership. He's been a visionary. Mikhail Perikin is the head of our engineering at Web Services Work, and he's done a lot of the productization of the technology team.

And Brad Smith has continued to be a big counsel on responsible AI, on the ethical approach, and has guided how we bring that together. And obviously, Amy Hood has kept us clear and focused on how we invest, how we look at the business impact. She's the CFO of Microsoft. CFO of Microsoft, yes. Has Bill Gates been involved? Have you been talking to him? They brought back the Google guys, apparently. I don't believe it. I don't believe it. They'd have to take them out of the cryogenic chamber, and that's really cold.

But go ahead. Yeah, Bill has been an active, continues to be since his time here, obviously, actively involved reviewing technology. He looks at a number of different technology projects. He's definitely looked at this project and given us feedback. And what's that feedback been?

You know, he has a lot of different thoughts on how the technology can be used in different ideas and different circumstances. So he gave me some marketing advice last week. Such as? Tell me. Now that I'll just keep between us. Bring back Bomber to say, bing, bing, but four. Oh my God, please don't. We'll be back in a minute.

Let's move to competition. Do you think AI is a silver bullet that gets Bing to beat Google? I mean, one of the things I know was frustrating to Bill was Google's hegemony in this area. And what is beating these days, really?

Yeah. I mean, what you said is a good question. So I'd say a couple things. First, I'd say it's such early days. We're really focused on the most important thing, which is wowing people with the technology and trying to build something that they can't do today. If we can do that—

And I think we're off to a good start. But if we can do that, then I think all good things will come for us, which is that more people will come and use it. That'll entice more advertisers and the business will get better. You know, by way of dimensionalizing the opportunity, the online ad market is a half a trillion dollars. Search is the biggest part of that. It's 40%.

If we take one point of search share, just one point, and mind you, we're small single digits, that's $2 billion of revenue. So very small improvements for us are meaningful, even at Microsoft scale. And we talked about this at our launch event.

There's a new generation of search. And that's why we talked about this concept of your co-pilot, which is that as you move along the web, the ability to get answers, to get search, to chat, to create, that's the new state of the art. That's the new thing where we're headed.

And I feel like we're on that. You know, the idea of typing into a box still is the most non-innovative thing I can think of at this moment in time. We've always talked about this idea that it should know you in a way that's not creepy. And then it should follow you and understand you so that when you ask it something, it knows what you mean. A lot of things are useless in terms of...

Like it's not useful except for what's the stock price or which in the old days you would have looked up in a newspaper, for example. I mean, I think you're spot on. I think that that's why this idea of co-pilot is something that we start to get more, I would say, intrigued about for a couple different ideas. And by the way, I love the word co-pilot because it is what we're about, which is you, the individual, are in charge and the technology is running alongside the help and

A couple of things that we've already done in the products we've just shipped that I think speak to it. One is if you're in Edge now, if you click the Bing icon on your browser, we open up essentially a right rail, right? A thing column. That now understands the page you're on. And so if you're on a page and you're like, "Oh, what's this all about?" You can simply ask Bing, "Hey, summarize this page for me," or "Tell me what's going on." Or you can say, "Hey, compare what's on this page with what I know from the internet."

And to your point, you don't have to go and do a lot of typing. You just say, hey, tell me what's going on. And contextually, the product is smarter, even without knowing who you are specifically. It knows the page. Another thing we've done is with the new Bing mobile app that we just shipped a week ago. Now you can click it and you can use your voice. And I'll tell you, it's...

It sounds so intuitive, but it's really kind of mind-boggling when you use it. Being able to now use your voice to ask for things and have it read back answers. And this is like what some of these other agents should have been but never were. But you still have a ways to go. Look, the fact that people are trying your search out again is an unusual thing because –

people do get stuck in habits. So kudos for that. But when it comes to global market share, I've seen different numbers, but I think as 3%, Google has 93%. Really, it's an astonishing number. But Google is working on its own AI chatbot called Bard. Do you like that name? I

I like Bing. All right, okay. Elon, who was an early supporter of OpenAI, he and I had talked about it when we used to talk, is ginning a new one to compete. Talk about each of these. Let's start with Google. You know, I can't really speak to what Google's going to do. I haven't seen it. You know, I guess you'd have to ask some of their employees how it is. What do you expect them to do? You must be sitting there going, okay, this is what they're up to.

You know, I don't really know. And the reason for that is that I think this is such a change. It really is a little bit of innovator's dilemma because what we see is we think search needs to change dramatically. Well, you got nothing to lose, 3%. That's exactly right. We can afford to be optimistic disruptors. And so we've put answers right on the same page as the search. And for us, it works economically because if we get a few more people and we get more usage—

we can build a business on that. But if, on the other hand, you have all the users and you have all the economics, it becomes a real challenge to try and disrupt on the user experience. And so I don't know what they'll do. Meta, they announced, or sort of announced, that they're going to do this initiative based on research. Yeah, it's a different thing, right? So, and again, it's like, exactly, they've announced they haven't shipped anything. It's more of a research play. Yeah, I think because they're scared it's going to do something bad, like propose marriage to Donald Trump or something like that. I don't know. Right.

Correct. I mean, I would assume they're nervous about the possibilities.

if it's publicly integrated into their products. Yeah, I don't know, actually. I don't know how they're thinking about it. But, you know, look, from our standpoint, I think this technology really can do some incredible things. What about Elon announcing this? Is it just because he likes to get in within any press release possible? Yeah, I mean, hard to say. I just saw that he tweeted this morning that he, or maybe there was something written about that he's starting to look at it. I don't know anything more than that. Yeah, so here he is once again. Can't get rid of him. Can't.

How can we miss you if you won't go to Mars, Elon? Anyway, what is being the advantage then? I mean, I know it's exciting. Competition is exciting, but obviously you want to win this time. Yeah. Well, look, I think we have a number of advantages. One of them is that we're first out there on the market. We're learning. We're getting feedback that's such an advantage to be able to have feedback from real people.

The second thing is, as we just talked about, we can afford to be innovative. We can afford to take some sort of disruptive ideas and put them into the product, like putting answers in search on the same page. That's a big idea that...

for other companies, that'll be tougher to do. Yeah. And then I think this partnership with OpenAI, they're the leading company in my mind in terms of doing OpenAI research and advancement. And then we have the broader capabilities of our search service. There's only two real search services out there, ourselves and Google at scale. And there really is something valuable to bring in the search capabilities to bear with the AI capabilities.

So those are a number of things I would say that give us an opportunity to make some impact. Now, another opportunity is

Is the Justice Department antitrust investigations into Google? Well, that is around advertising. You talked about grabbing a point of advertising share. Google, of course, is arguing that the market is taking care of itself. So is Facebook or Meta. How does that play into your thinking? Obviously, Microsoft has had a long history with the Justice Department in a more negative way, but I would argue it made Microsoft better over the many years. How do you look at that? They're hindered.

And you're not necessarily. Yeah. I mean, in the area of search, look, we need more competition. You summarized the numbers. You know, they're 90 plus percent share, depending on who you want to look at. And we're very low single digits. And no one else can afford to mount a credible search alternative. It's just the nature of the state in terms of how it operates. And so we need more competition. Are you all talking with the Justice Department on this issue as you iterate this?

We, like, look, we'll comply with government requests for information. That's one, like, to your point, one of the things we learned about our time with the Justice Department earlier is how to work with the government to provide information, to, you know, follow the guidelines and regulations. And I think we've matured very much as a company on that. So we work with them in all the ways that, you know, are appropriate and when asked. Does Google and Meta have unfair advantage from your perspective? Yeah.

Well, Google definitely has a pretty dominant position in search, there's no doubt. And they have super normal returns on their advertising, and they've done some exclusive deals on search distribution that make it hard for other companies to compete. Okay, that's a yes. So at the World Economic Forum, Satya talked about how IAO is going to be embedded across the Microsoft environment. So where does it go next? This is a complaint I've been making. It's like, I want to stop talking about term papers, cheating, and...

getting them to talk to tech reporters like their girlfriends. Where does it go next? Yeah. So, you know, a couple of different things. And you're right, Satya, and, you know, the vision we have as a company is that we think this AI technology is going to permeate all of the opportunities. We see a lot of ambition to be able to improve our products.

Some of the things that we have done already is with Azure, we've done a bunch of the OpenAI capabilities for developers. That's been one great place. So now anyone can come and build on top of Azure to build great AI products. Second is with GitHub, we have a co-pilot for developers. And the feedback from developers is you'll have to pry this from my cold hands before you can take it away because it's made them so much more productive.

A couple of weeks ago, we announced it in Teams Premium. Just so people don't know, Microsoft owns GitHub, but go ahead. A couple of weeks ago, we announced it in Teams Premium, our video conferencing solution for enterprises. We brought in an earlier version of ChatGPT to do summarization of meetings so that when a meeting's done, if you miss the meeting, you can go and say, hey, tell me what happened during this meeting. That's why you missed the meeting, but go ahead. You don't want to hear it.

So that's another example of where we've done some work. And we're continuing to bring different capabilities of AI broadly. In fact, this morning we announced with Windows a big update to Windows 11, and we're doing a couple things. One is we're bringing the new Bing onto the Windows taskbar. This is an important piece of Windows. Half a billion people use it every month.

We've put AI into the video capabilities of Windows. So now when we're doing a call, a lot of times if you do these video calls, you and I are looking at each other. But what I see is I see you looking down and you probably see me looking down. We use AI to now get

Eye Gaze to map directly. That's an AI-powered tool. And then even on your Start menu, if you're logged into your enterprise work. So all over. All over. So Satya said Microsoft was working at a frantic pace to integrate AI into your product offerings.

How do you deal with all that? These are a lot of products you just mentioned. How do you deal with the externalities of revolutionary technology in a responsible way if you're working in a frantic? I hate the word frantic. I'll be honest with you. Well, look, I'd say a couple of things, too, and it's just good to remind ourselves. We've actually been at this for a while, so it isn't like this just happened three weeks ago. We've had responsible AI principles going back multiple years.

We've done a bunch of the engineering work on AI from some of the earlier incarnations of products. We've built now an incredible organization at the company across researchers, ethicists, AI experts. And we have now...

a very amazing team, and they're shoulder to shoulder. So in fact, the first people that got to see the new AI that people are using today with Bing were actually not the engineers. It was the people in the responsible AI group. They got the first look at the new tech from OpenAI and came in and said, okay, how are we going to do it? And Satya talks about this a lot. And he's said this to us many times. Before we talk about the what we're going to build, we're going to talk about how we're going to build it.

And we go back to core principles about we are going to bring this technology forward in a responsible and ethical way, in a way that's inclusive of all people. And that's what grounds anything before we start building a particular product. Well, you're lucky the bar is so low, but the pace has definitely kicked up a notch, though. There are dangers in that, in going too quickly with something that's much more significant than almost anything I can think of in my history of covering technology. Yeah.

Yeah, and I would say, look, we agree with you. No one is confused here at Microsoft. We need to be thoughtful and responsible. And that's the approach we've taken. So despite all of the amazing press about different use cases,

It's also worth noting the speed with which we responded to some of these things. I think you and I talked about this back in the days of the internet. There was this thing about you have to move at the speed of the internet. I think there's a thing now you have to move at the speed of AI. And so we got that feedback from some of the wild chats. Within 24 hours, we had fixes in place to limit the number of chats. Within 24 hours. So walk me through one or two or three ethical questions AI raises and how you've dealt with those issues at Microsoft and

Give me specific examples. For example, you know, violence. People want to use the technology for violence. People say, hey, how do I create a bomb? Which has been a question on the internet since it started, but go ahead. Exactly. And so we'll just take that one. The approach we've taken now to deal with that in the new being with the AI is a couplefold. It's multilayered.

First is, as soon as that query comes in, we're able to do what's called a query classification. So we can understand, hey, is this a self-harm? Is this violence? Is this hate speech? We catch it and we don't even allow it to go into the model, right? So we catch that at the front.

Then there's the model itself, which we work with OpenAI, where we make sure that the data that it's trained on is good data, that we don't actually go in and necessarily have it be corrupted. That's something we learned in the past where people could go and try and train the model. Yeah, you had that issue with the racist, yeah. That's exactly. And then there is at the end, before an answer gets provided back out to the person, we catch it again. We look at the classifiers. We look at the data that's coming back.

So we put a lot of different safeguards in place. We catch it at multiple layers of the technology stack. What about not answering that question?

Well, that's what we do. So what will happen is, and you can try this in Bing today. If you go in and you try it and say, how do I make a bomb or how do I cook meth? We'll say, hey, we don't answer that question. In fact, sometimes we'll say, this is a dangerous thing. Here's a helpline if you want more information about why this is a dangerous element. Right. So you're going to make some, they're essentially editorial decisions here. Yeah, based on our principles for responsible AI. So violence.

sexual harassment. Yeah. Hate speech, self-harm, you know, a number of things. We have taken a position to say we're not going to enable that with our technology. So that brings you to the issue. You're sort of getting dragged into where Facebook's been living and Twitter, et cetera. Conservatives think AI will be biased against them. They're talking about woke AI, which exhausts me. They're going to do woke pancakes at some point. Talk about that, the idea of woke AI. Yeah.

Yeah, I mean, this is a tricky thing. You know, the whole notion of what I see is everyone sort of has a different view of the technology's capabilities. One of the things I've sort of learned, and I think actually that...

People are trying to understand what is the right line. And the problem is that different people see it in different places. Sure do. And I think that's the problem in a way that there is no sort of clear line. And what our approach is,

We're trying to just be factual. We're trying to say, this is the things that we know. And that's why a big part for us is to just put the references. This is where the answer to this question came from. These are the sources. And if you want to learn more, you can. And then people can choose. And if people say, hey, I don't believe these sources.

These are not good sources. I believe these other sources, right, news sources, you can pick and go and find the answer. That's the approach we're taking. We're not trying to, you know, opine specifically one way or another. We're trying to be as factual as possible. Is being biased against conservatives?

No. When you think about that pushback, though, they are making it a thing. Are you worried about that? Yeah, I think this is one of these things where, you know, it's the world in which we live in. And so we know that we have to do our part to bring the technology in a way that everyone can feel good about that is being helpful in their regards.

Our goal is not to take a position on this thing one way or another, but to be factual and to let you go and find the data that you want to find. Are you worried about getting dragged into it? I would say that...

you know, we've developed very much a sort of professional sense of how we work on all of these topics, right? There is, uh, as I mentioned, we're very connected with the government, uh, at many levels we have, uh, and so we're aware of these issues, you know, these are, these aren't catches by surprise. We know what they are. Um, we get that feedback directly in the engineering team on, Hey, how are we doing on these types of questions? What's going on with these types of questions? And we address it in the product. So, uh,

I would say this is an active thing that we pay attention to, like we do many things. Elon Musk, who funded OpenAI, as I said in its early days, is calling for AI regulation, though, and says the lack of regulatory oversight is a major problem, even as he's funding new stuff. I would tend to agree. The Internet was early not regulated. What do you want from government here today?

in order to keep it innovative, and at the same time, boy, does this open up a raft of problems. We don't even, you know, it's like saying, what's the internet going to do in 1996? We don't know. Yeah, it's a great question, and

Look, I think first and foremost, we have the job that we need to do as a company, right, to bring things responsibly, et cetera. Look, we do think that there is a role, an important role for government to do regulation and to help promote competition, to help make sure that things are safe for people, and to create economic opportunity. And I think that is the role for government on this effort, and we want to work with them. Do you feel them moving in quicker?

Quicker than what? They never came in for the internet, so do you feel like they are at least paying attention here? Yeah, absolutely. I think there's a lot more engagement. There's a lot more questions being asked. Certainly in Europe. Yeah, absolutely. But they're always ahead of things. Very last questions. What would you like AI to do? What are you most excited for it to do, a specific thing, and what are you most frightened of that it could do? I'm really excited about...

being able to empower people to find information and discover information in a way they haven't been able to do before. And honestly, in all of the projects I've worked on, this might be the coolest thing I've worked on in my entire time at Microsoft. So I'm really happy about that. And in terms of like, you know, the things I, the thing I worry the most about is, you know, making sure that, you know, that we do this in a way that is, um,

you know, is responsible and promotes a positive growth of the internet and of society. And, you know, as we talked about, tools can be great tools and they can be also weapons, right? This is sort of like Brad Smith's book. And so having this be a tool and not a weapon is the thing that, you know... Because boy, is it a weapon if they were, you know, Hello China or wherever you think about it. Yeah, you have to worry about that. Anyway, Yusuf...

Very exciting times. Thank you, Cara. Thank you so much for talking to me.

So a tool, not a weapon. That is what? Wishful thinking or accurate? No, it's the title of Brad Smith's Chief Counsel's book. Yeah, I think they are. They're a much more thoughtful company than most, let's say. I'd rather have them doing this than a lot of other giant, enormous companies of which this is going to be, no matter how we slice this. So yeah, they don't want to get dragged. They've seen what's happened with Facebook. They've seen what's happened with Facebook.

with Twitter and everything else. And this veers in on that area. I think it was surprising they weren't expecting people to chat with Bing for over two hours. Yeah. That they hadn't kind of programmed in an automatic refresh or a limit to begin with. Yeah. Do you feel like maybe Facebook or other companies that have actually struggled with more issues than content moderation would better preempt these kinds of issues than like a Microsoft, which is used in such a way? No, they have gaming. They're in lots of businesses where they know how people behave.

You know, they're very active and they've had experience with whatever you think of Microsoft Network. They have experience. I think it's just you can't anticipate these things. You don't know what people are going to do. And that's the whole problem is it's not the Internet, it's people. People is the problem.

You sound like an NRA advocate, Carol. No, but you can make the tools so irresponsible in terms of stopping people. You have to anticipate that people are going to do something bad with it. That should be your... Your default assumption. Yes, exactly. You have to anticipate consequences. And one of the things they do not do, and I think Microsoft does more than most companies, is...

what's the worst thing that could happen? Because guess who got in big trouble with what's the worst thing that could happen? It was Microsoft. So I think they've got it. That's in their DNA now for sure. Yeah, they definitely have a more professional eye. With Satya Nadella for certain, he certainly is very cognizant of that. And that's held them back a little bit, but that's okay.

They're actually very valuable, so it's not holding back that much. Yeah. They're doing just fine. Just fine. I'm not worried about Satya Nadella. Yeah. He was a little bit closed about what Bill had told him about AI. Yeah. I'm going to email Bill Gates and ask him. So that's what we'll have to do.

Probably around, you know, he's been the subject of so many misinformation attacks and some that are true, but he's certainly around the vaccines and everything else. So I think he's probably very cognizant of the dangers of things like this in a way that many people are not, especially during this pandemic period that he owns all the farmland. He's trying to put chips in people's brains. Yeah.

or through the vaccine. And so I suspect they've got to be very careful about how they market this thing. Yeah, I assume so. And he seemed very cautious in the interview. He wasn't very forthcoming about the details of the partnership with OpenAI, but it's very clear that there is

Very tight. Smart move. Yeah, smart move by them. Smart move. And OpenAI gets a lot from them too, right? Yeah, that's right. Smart forever. OpenAI. The others would have overwhelmed. Google would have overwhelmed OpenAI. And so would... There's no other partner. Microsoft was the perfect partner. It's actually a very smart...

But touching Nadella doesn't get the credit he deserves as a CEO, I have to say. Well, you can have him on. You can give him all the credit and all the tough questions. I don't give him all the credit, but I think he's been a surprisingly good CEO. He's very much like Tim Cook at Apple. Increased the value, thoughtful, taking away all the noise and kind of the negatives of Bill Gates and Steve Ballmer in a way that kept the good parts. Well, I think what hangs for me in this interview is how is government going to regulate this? Because you're right, they were so, so slow.

on the internet. They're still not there. They're still not there. We'll see. We'll see. I'll hold my breath. Okay. Well, let's read the credits before you start holding your breath. Okay. Today's show was produced by Naeem Araza, Blake Meshik, Christian Castro-Rossell, and Raffaella Seward. Rick Kwan engineered this episode, and our theme music is by Trackademics. If you're already following the show, you get early access to Bingbot. Actually, I have no ability to get you that. I'm just making it up. If not, you're stuck with Clippy.

Also, Microsoft Bob, go look that up. Go wherever you listen to podcasts, search for On with Kara Swisher and hit follow. Thanks for listening to On with Kara Swisher from New York Magazine, the Vox Media Podcast Network, and us. We'll be back on Monday with more. How can we miss you if you don't go to Mars?