Is The Eurovision Song Contest Rigged?

Publish Date: 2023/5/18

Chapters

You're a podcast listener, and this is a podcast ad. Reach great listeners like yourself with podcast advertising from Lipson Ads. Choose from hundreds of top podcasts offering host endorsements, or run a reproduced ad like this one across thousands of shows to reach your target audience with Lipson Ads. Go to LipsonAds.com now. That's L-I-B-S-Y-N-Ads.com.

I think that the very fibre, the nature of the Eurovision since its inception was a very political statement anyway. It was meant to bring together all the European countries this side of the Iron Curtain and kind of make a big splash for reunion after the war. So it's by definition a political operation.

Hello and welcome to the FiveThirtyPolitics Podcast. I'm Galen Drew. Today's episode should probably come with a nerd warning. So on Monday's podcast, we joked about polling pollsters about their favorite poll. And well, today we're going to actually take a look at a poll of pollsters.

Pew Research recently released its findings from a survey of pollsters they've been doing for years. Essentially, they ask pollsters about the methods they use to reach Americans. And what they found is that the polling industry of today bears almost no semblance to the polling industry from the year 2000.

At the beginning of the century, there were about 29 national pollsters, all of which almost exclusively relied on real human beings randomly dialing respondents by phone.

Today, there are about 69 national pollsters and only 4% exclusively use random digit dialing. The number of pollsters and methods of polling have grown significantly. So here with me today to talk about what this means for those of us trying to figure out what Americans really think is Vice President of Methods and Innovation at the Pew Research Center, Courtney Kennedy. Welcome to the podcast. Thank you so much.

I should also mention for any Eurovision fans out there, especially anyone perplexed by last weekend's results, that later in the show, I'm going to be joined by a statistics professor from University College London who created a Bayesian model, yes, a Bayesian model to detect bias in the Eurovision voting system. And you thought a poll of pollsters was going to be the dirtiest thing you heard today. Anyway, we will get to that. But Courtney, let's talk about polling.

So first of all, what are pollsters doing instead of having real humans randomly dial Americans by phone? They're doing a huge variety of things. And that's one of the interesting sort of high-level findings. It's not that the field has settled on one methodology, but what we found is a number of different methodologies being used, different flavors of online polls. We've got

online polls done with what are commonly noticed like opt-in panels or river samples. You have other online polls that are actually recruited offline to try to retain elements of probability-based sampling.

And then with this study, we also saw the uptick of, you know, kind of more emergent methodologies like texting. We found that about 10% of national public pollsters are using text. Usually not as their only way of reaching the public, but it's sort of a supplemental way to reach people. Is it clear? Do we have any sense of which is...

the most accurate? I know that's not specifically what this study looked at, but is there any emerging wisdom about what the best methods even are now? Definitely not a consensus. My read of the literature, though, is we have recent elections, and that definitely gets the most attention. If you look at elections the last four or five years,

It seems that there's really not one methodology that's like head and shoulders above the rest. And that null finding is actually a little bit surprising because on paper, some methods look way better, like they should deliver a much more representative random sample. But because of all the vagaries involved with election polling and trying to predict this

future population of voters that we don't know yet for a lot of reasons that just hasn't borne out. So I would say if you look at recent elections, no clear leader in terms of like the most accurate methodology. Now, if you take a step back, there are a bunch of studies that gauge the accuracy of different polling methods, not using elections as the benchmark, but using benchmark data from other things we know from the government, like

what share of Americans have health insurance or what share of Americans have a driver's license or smoke every day, right? There's a lot of stuff that the government measures really well and they provide good estimates, not perfect, but good. And so there's a whole raft of studies that do what are called these benchmarking exercises where you take the federal government benchmark, you try to measure the same thing in different polling methodologies and you see which of those methodologies returned the more accurate results. Now,

That set of research, I think, does point to that polls that do start their process with a random sample of the public, whether that's phone numbers or whether that's addresses, they do tend to return more accurate estimates. But it's not in the context of elections, which, again, I recognize is usually what draws most of the attention.

Right. And do you think that's in part because there are certain biases amongst the public in terms of how liable they are to want to respond to a poll? Like asking about whether or not you have health insurance, which may seem somewhat personal, isn't ultimately the same thing as asking who you're going to vote for.

Yeah, I think there's a couple of factors. One thing, I looked very deeply with a number of colleagues on behalf of APOR, this national polling organization. We did a postmortem on the 2016 pre-election polls.

And, you know, to your earlier question, we looked to see if there was any particular methodology that did better in 2016. And if you had to pick one, it was actually IVR, right, Interactive Voice Response or robopolls. So if you get a call on your cell phone and you pick it up and it's sort of an automated question and answer type thing, that's a robopoll. Those in 2016 were...

did pretty well. But it's not because they're awesome at getting a pure representative sample of the U.S. public. They're not. They skew incredibly old and incredibly white. But in 2016, that turned out to be kind of helpful if you were trying to get an accurate read on the level of Trump support. So I think of it as

In elections, you can get these weird things of different errors canceling, right? And it might not behoove you to do really well with young, diverse segments of the public in an election where they are actually less likely to vote. So my read is that the methodology that's going to give you

you know, the most representative sample of all U.S. adults is not necessarily going to give you the best sample or the most accurate prediction of what's going to happen in an election. Again, because we don't have, you know, universal voting and we see differential voting rates in different parts of the population.

That's a really interesting point. And one of the things that your survey shows is that a lot of pollsters are using more than one method at the same time to try to get to the bottom of whatever question they're asking. Like, how common is that now that pollsters are just kind of throwing spaghetti at the wall or trying to cancel out different biases with different methods?

Yeah, that's my favorite finding from the study is that like almost 40% of national pollsters are doing two or more approaches to try to reach the US public. And by that, I mean, two or more sample sources. So like phone numbers or addresses or two or more modes like online, phone, paper, etc. And sometimes they'll use multiple approaches.

approaches like that in the same poll, like, oh, we did some interviewing online and we did some interviewing with robocalling, or sometimes they'll split it across two different polls and do one poll using one method and one poll using another. There's just a tremendous mix right now. And it gets to something that I found really interesting was that you saw more of this

commitment to change, more churn in methodology after 2020 than we did after 2016. And the share of pollsters that are using three or more methods, which is a lot, right? Three or more methods tripled just in the two years.

after 2020. And for a while, like at Pew, we were one of them. We were doing phone surveys still, kind of winding that down. We had moved online and we found a need to do sort of an annual survey that offers paper response or online response. And we were not alone. There was other polling organizations doing that as well. So

Yeah, it's, it seems that there's no, you know, methodology that's, that's perfect for getting everybody in the public. So you do see a lot of this combination of different approaches to try to get young and old, you know, kind of using different methods.

Yeah, I found that finding in the survey really interesting, that there was more iteration after 2020 than after 2016. And I think amongst the public, at least the emotional response to the polling error in 2016 was probably bigger than the polling error in 2020, even though, importantly, the polling error in 2020 was actually larger than the polling error in 2016. My thinking on this was,

because of course the survey doesn't tell us why that happened, but having, you know, spent a lot of time in the trenches myself, I was thinking, you know, a lot of pollsters came to the conclusion that after 2016, what they needed to do was, you know, wait for education. So for example, if we know that say two thirds of Americans don't have a four-year college degree, but

only 40% of your sample doesn't have a four-year college degree. You've got to count the people who don't have a four-year college degree like one and a half times or that's basically how weighting works.

And then we thought that like that was gonna be the solution. And of course, that's not a method of sampling. That's just a waiting method, which this survey doesn't look into. But after 2020, once all the pollsters were doing that and then there was still error, they're like, oh, maybe we have to actually go out and figure out a different way of trying to reach people in addition to just waiting. Like, of course, we don't have a clear sort of why as to why this was the case, but does that, does my theory sound plausible to you?

I think that's all correct. The other layer that I would add in from talking with people in the industry

is that after 2016, you saw everyone was concerned, right? But as you say, the response to 2016 was fairly muted and it was more technical, it was more in the waiting. Part of it too was like after 2016, we didn't know if that was just a one-off bad year, you know, for the field or if it was actually the start of a longer lasting error that we were going to have to grapple with. And frankly, whether we like it or not, 2020 provided an answer, right?

As you said, the same systematic errors under representation of all kinds of Republican candidates, it repeated, and if anything, was a little bit worse. And so when I talked to colleagues after 2020, there was like a different tone in people's voices. It was like, oh, I am now scared straight. I am going to commit to not just tweaking around the edges with waiting, but actually fundamentally changing how we are –

where we're sampling and how we're reaching people in our polling. So yeah, the tenor of the conversation absolutely changed in 2020. And I think it's because we got an answer that this is not going away easily. How do you address the somewhat of a conundrum that polling was very accurate in 2018 and in 2022 amidst all of this turmoil in the actual presidential election polling? Well,

I mean, this might sound a little simplistic, but the data suggests to me, if you look at 2018 and 2022, that if Donald Trump is not literally on the ballot, that the polls tend to be more accurate. And, you know, if you look at how he was communicating about polling, I mean, it's no secret, like a prominent message of his

was to say, to tell his supporters, don't take polls, to spread this narrative that polling is rigged and so forth. I mean, you know, stuff that was not helpful to the polling community. And I think a lot of his supporters, the data suggests, they listened to him and they did stop participating in polls. So I think you have the message that was being sent out right by the candidate. Yeah, that was a tough thing for the industry to fight.

Yeah, which is, it's important to mention, different from lying to pollsters, which I think is like a common theory that people have out there, which is that some people are telling pollsters they're going to vote for somebody else. It's more an issue of they're just not responding to the pollsters at all. That's right. Yeah. And there have been a couple studies done on that sort of shy Trump phenomenon. We did one at Pew and others have done them. My read of that is like...

That might account the sort of notion of people who voted for Trump but purposefully saying they did not vote for Trump. That might be like one or two points in the error that we see after an election. But if polls are missing by larger margins, if polls are missing by like five, six, seven points, that's non-response. That is like you did not have enough Trump voters in your sample to begin with, as far as I could tell.

So as I mentioned at the top, once upon a time, there was one way to conduct a high quality poll and it was live callers doing random digit dialing. And that was the easy answer. Now though, what is the most common method? So we haven't come to some conclusion about what is the most accurate, but in the meantime, what are people relying on the most?

That would be online surveys that are done with what is known as opt-in samples. I did an analysis using some data from FiveThirtyEight not too long ago, and I looked at, you know, of all the polls that are trying to gauge like presidential approval, what share are different methodologies? And over 80%

public polls by that measure were online opt-in. So if you have to ask me what's the most popular one, it is that flavor. But I would stress that there's a second version of online surveys that, you know, again, is more probability-based, starts with a random sample of the public. And those are gaining some traction. I think we saw in our study that about 30% of national public pollsters are doing that version of online

of online polling. So you really do see both.

And the field went through sort of 10 years, for the last 10 years, a real steep curve of tremendous growth in online opt-in. But it looks like that might have abated a bit. It's by no means going away because of the cost dynamic as such. That's all many people can afford to do. But it's no longer looking to be the case that online opt-in is going to simply take over the field and push out

all other forms of polling. I think that that was an open question five years ago. The growth was so tremendous, but it's looking like those other methods are actually holding their own. Do you think today that

the pursuit of accuracy or lower costs is driving a lot of the change that we're seeing in the industry. Because when you talk about online opt-in polling, for any student of statistics, you're immediately like, okay, well, you're not starting with a random sample.

So how do I even trust this poll? How heavily do you have to wait to get yourself out of the conundrum that you've created in the beginning, which is that you didn't start with a random sample? And so is it really just costs that are at play there as opposed to pursuit of accuracy?

I think it depends on who's doing the poll. And some organizations, we're lucky at Pew and maybe Gallup and some others, we have the luxury of being able to afford either type and being able to pay close attention to accuracy and to measure that.

and to really put our resources into doing the more expensive version. But I recognize that there's a lot of survey researchers out there that just feel that the probability-based version is outside of what they can afford. And so for them, it might be a question of, if I do a survey, it has to be opt-in or I don't do the survey at all, right? And so I try not to be too judgmental because I recognize that that's the equation for some people. And so I guess another way to put it is...

You know, for some people, cost is really the driver. They just don't have the luxury of focusing too much on accuracy, but some organizations do. And you tend to see the major media and some of the higher profile state and college survey centers being able to focus on accuracy as well as cost. What do pollsters have to do when they begin with an opt-in online sample to try to get more accurate results?

You have to wait the heck out of it. But to be honest, we all do. Even probability-based pollsters have to do a tremendous amount of waiting. But when I look at the opt-in space, I think the approach that CBS News takes...

and some others look to buy mine sort of best in class in that they will match to voter file records where they can. They will do very sophisticated modeling to try to reduce as much error as we can. They start from a place of recognizing that the sample they get is not really representative and the burden is on them

to use all the statistical firepower that they have to make it representative through various modeling approaches. What do you think is the biggest challenge facing pollsters today? Is it still just reaching a portion of the Republican electorate that has opted out of polling, or does it go beyond that? I think that's part of it. I think the other is that, as I mentioned, with

Everybody just would love to just do surveys online, but if accuracy is really important, you're going to struggle with that. And so the question is, how do you reach that segment of the public that is not keen on participating in online polls? Because we've seen it in the data. Online panels, any online panel I've ever worked with, tends to underrepresent people who are more religious, people who are more conservative, people

people who have less formal education and people who vote for Donald Trump. So you're going to systematically underrepresent a really crucial part of the population with just online methods. But I said a few minutes ago,

Over 80% of public polls are just online, right? So we have this real conundrum of a wildly popular methodology being used, but a lot of research showing that that methodology is missing part of the population. And so, you know, we're trying to come to grips with that. And the resolution is not obvious. And so we see a lot of organizations taking different approaches to it.

For you, what's at stake in getting this right? You know, why care so much about this question?

Well, I've spent my career in survey research, and I care very deeply that what we do is important, that the information, being able to give the public nonpartisan, unbiased information about what's going on, that that's deeply important. They can't just rely on politicians to assert that the public thinks X or thinks Y. You need somebody you can trust to

to say, you know, here's how issues are playing out in the public. So I believe deeply in sort of what the polling field can do when it's doing it well. And I recognize that, you know, 2016 and 2020, the industry took just tremendous reputational hit. You know, when you talk to normal people, friends and family, you hear it. You hear the skepticism, you hear the loss in trust. And

So as someone that cares about the field, I really want to solve that problem. I talk with my colleagues every day, like, what can we do to fix this? Is it a waiting solution? Do we need to try new modes? What are people across the field doing? It's what keep me up nights for sure.

Was there anything, of course, we mentioned on the podcast earlier this week that the conference for the Association of American Public Opinion Researchers was last week. Was there anything you saw at the conference that gave you hope? Yes, yes. One frustration I've had for a couple of years is that some of the organizations that do some of the more rigorous, more expensive probability-based polls, they tended to not have

be very aggressive in how much modeling and statistical adjustments they were doing because I think they were leaning on, oh, well, we're starting with a random sample. And yeah, you can start with a random sample, but not everybody participates. And so what I saw is that like,

in terms of who's statistically more adept in solving more problems, you saw more creativity and initiative with people using opt-in data or non-probability data. And I feel like some of the probability-based pollsters are finally starting to catch up and recognize. And what I'm really getting at is waiting on things like party and waiting on things like past vote. Because one thing you saw at APOR in a lot of different presentations was

is everybody had, even if you had the right share of Republicans in your poll in 2020, almost everybody had too many Republicans who voted for Biden, right? And so you have to really be aggressive. Even if you're starting with a random sample, you have to be aggressive, I think,

and you're waiting to try to fix these things. And I'm hearing more recognition that, yeah, we have to wait on pass vote or party or something like that. We have to try to address the fact that people who are more interested in politics are way more likely to take surveys. Just more recognition and discussion of some of the limitations that we're having as a field. I think I mentioned on this podcast, in fact, while we were covering it, that the 2022 election to me

felt like the first post-polling election since horse race polling became so ubiquitous. And what I mean by that is that both the general public and even experts themselves were questioning the data. So for example, data would come out showing Democrats doing fairly decently in House districts or in different competitive races around the country.

And experts and the general public alike seem to sort of ignore it and say, OK, well, you know, Republicans are doing well. We expect Republicans to do well, et cetera, et cetera. And ultimately, in the end, a lot of that polling that got second guessed was correct. So I guess my question to you is.

Are we in some ways post-polling? Like, have we lost the public as far as the sort of, oh, we look to polling to give us some sense of what the future is going to look like? Are we in a different era now where it's just viewed differently?

I think in the beltway circles, yes. Yeah. I think that that whole narrative of second guessing polling and correcting for the last cycles, errors in polling, that's not going to go away anytime soon. And frankly, some of that's fair because we did have as an industry two pretty bad elections back to back. But I would not describe it as post-polling because as much as we might struggle as a field,

I just can't get around the fact there's always going to be a demand for good polling. There were years where people were sort of hyping, you know, like, oh, if we just scrape Twitter, right, we can get a more accurate read on the public. Well, that turned out not to be true, right? There's been a lot of people have taken runs at replacing polls and trying to get

get out there where the public's at, but none of it works. I mean, frankly, the poll, it remains our only sort of best tool to try to get a cross section of how the public is, you know, reacting to whether it's Black Lives Matter movement, you know, Dobbs decision, whatever. It remains our best tool. And I just can't, I don't see a future where the demand for polling information goes away. Yeah. And I mean,

When you look at the people or the organizations that are making financial decisions based on whether or not polling is accurate, like they're still using it. Corporations are still using it. Businesses are still using it. Campaigns are still using it. Like as much as it's popular to sort of doubt it, the people who actually are staking their fortunes on these questions are still using it. And of course, you know, for us,

for people who think that it matters in a democracy what people think, it's far better than showing up on a random street in Pittsburgh and just asking the first four people you see. So I am, you know, I'm rooting for you. I'm rooting for the entire industry here.

Thanks. Yeah, I totally agree. I mean, right, marketing is a multi-billion dollar industry and it's largely rooted in doing surveys to figure out, you know, how people are reacting to messages and products and whatnot. And, you know, I think part of it is also

you know, companies and campaigns, they're going to keep doing polling, they're going to have this information. And so the question is whether the public deserves to have the same type of information as leaders of industry and leaders of politics, you know, I think so. I think it's in the public good to have the public have access to the same information as other people do.

All right. Well, thank you so much for joining me today. Courtney Kennedy is the Vice President of Methods and Innovation at the Pew Research Center. Thanks for having me.

Last weekend was the annual Eurovision Song Contest. If you aren't familiar with it, first, what are you doing? Second, go do some YouTubing. And third, stick with me here. So it's an annual competition that has been running since 1956, where each European country and some guest countries enter one song that's generally chosen through competitions in each of the countries.

The most famous winners include ABBA and Celine Dion. And if you're wondering, Celine Dion is Canadian. What was she doing in Eurovision? She was representing Switzerland. I'm not really sure what was up with that. But anyway, don't ask.

So last weekend, Sweden won the contest for the seventh time, a win that was only made possible because of the judges' points. The public preferred act was Finland. And I was watching the contest and tweeting about it stupidly. And we got some responses from listeners asking us to dig into Eurovision voting methods. And so this is that. That's what we're doing today.

Here with me to discuss is Gianluca Baio, Professor of Statistics and Health Economics at the University College London. Welcome to the podcast, Gianluca. Thank you very much. Thank you for having me. And it's very interesting that I often get called to give my expertise on the Eurovision, which kind of annihilates all the serious work that I do in my research and in my day job. But it's also awesome. So I'm very happy about that.

Okay, maybe we'll give you the opportunity at the very end of the podcast to plug any other serious work that you've been doing. However, you know, voting methods, we take voting methods very seriously on the FiveThirtyPolitics podcast. So this is not just some silly cultural exercise as far as we're concerned.

No, no, of course not. And I should say that listeners might be surprised to know just how much social science research has been dedicated to Eurovision voting patterns. When I started Googling, there was an awful lot of research out there. But I was drawn to your analysis in particular because one, you actually created a Bayesian model to investigate bias in the voting. And then two,

I was scrolling through your Twitter feed and saw that you had a somewhat similar reaction to me to the contest last weekend, which is that Moldova should have won. Absolutely, yes. I love their act.

It kind of grew on me as well because the first reaction wasn't like, oh, I love this. But then by the end of the song, I was convinced, yes. Nocturne, nocturne, nocturne con luna

It kind of had everything. It was a catchy tune. The performance was actually compelling. The production quality was good. Like Sweden, come on. It was just like one woman singing solo a song that didn't sound all that different from the song she had already won with however many years ago. That was eight years ago. So yeah, I don't know what was going on. Dreams play.

I think you're absolutely right. When we set out to do our work, which was back in 2012 to 2014 through some kind of collection of the data doing this on the side,

The main motivation for us was that at the time the UK was in the run-up to what had then become Brexit and there was the phase in the political sort of environment where some of the politicians were just trying to plug anything into the kind of context whereby Europe in general is a bad component of social life and the UK were always disadvantaged in many different aspects.

You would think that the Eurovision is just this silly thing, like a contest that goes on every year with silly music that nobody really likes. It's just a good night of fun. But it was used as a platform to actually make much bigger points. And to us, to my co-author Marta,

who's also my wife, I should say, so we had like another vested interest in there. But it seems strange to us that people could just make those claims basically based on no real evidence. Basically, people were just saying, oh, the fact that we don't win at the Eurovision and that we always come last is evidence that Europe hate us and therefore we should Brexit.

And so the point was, well, let's see if that's true. Let's see if there's any signal in the actual data that there's an active discrimination against the UK Act. Well, we're about to dig into that. But I should say, the year after Brexit, didn't the UK receive zero judges votes and like zero public votes? So it would seem that maybe there was some bias. Yeah, the Act was incredibly crappy, though.

No, I mean, there's all sorts of things going on. And if you want, we can dig deeper into that. And I think I have some data-driven and more scientific speculations and some wilder speculations, which I think are actually equally valid in the grander scheme of things. Of course, I would...

I would stick more preferably to the scientific ones, but I can come to that if you like. At 5.38, we start with the science, then we can get weird. But I think first we should explain how exactly the voting works to any uninitiated listeners. Yes.

So in the first instance, when the Eurovision was established back in 1956, there was a jury giving scores to each individual act and then the act with the highest number of votes would win the contest. Then later on, I think in the mid-70s perhaps, I can't remember the exact date, they introduced this kind of hybrid system where parts of the votes come from a jury of experts and the other part comes from experts.

Televoting, essentially. So people were asked to submit their votes initially through phone and then now with their app, the Eurovision app. And then the votes are combined. So there are... The experts can give votes in the scoring system of 12, 10, 9, 8, 7, 6, 4, 3, 2, 1, I think, and 0. So you only score for 12 of the countries and the other acts get 0 points.

Televoting happens in a slightly different system whereby people can just use their app and this year I think they can submit up to 20 votes so you could in theory vote 20 times for the same act or for different acts if you wanted to and they're tallied all together and then re-proportioned to again re-give some kind of 12 to 12, 10, 9 and so on all the way down to zero.

And I should say, I tried to vote for Moldova, but it was going to cost 20 euro to cast 20 votes for Moldova. And so, I mean, what is this like poll tax system that they're using in order to allow Eurovision televoting? And also, this was the first year America could participate. So I was excited to do so, but then I ultimately didn't because I'm anti-poll tax.

I think it was a lot cheaper from Europe to do it, but I'd agree with you that Moldova was good, but not that good to invest 20 dollars or 20 euros for that.

Indeed. Okay. So to what extent... You mentioned the UK's conundrum in the run-up to Brexit. To what extent have there been accusations of bias? Was that the first time or is this sort of part of the game from the beginning? I think it has been part of the game from the beginning because obviously the system lends itself to accusations such as these. You had, especially in the beginning, a set of experts who would vote on the closed doors. So...

who knows what happens and I think that again there might be an element of other things happening over and above the songs because you could argue that when you do proper elections and you vote on the basis of a manifesto or something that is such an important thing like normal voting that maybe people can make decisions based on rationality maybe or maybe not

This is just about a song, so the quality of the offering is absolutely subjective, so it's very, very hard to make an absolute judgment. So the accusations has always been there. And what are those kinds of accusations? Is it to say, like, in the UK's instance, "Oh, the rest of Europe just doesn't like the UK"? Is it more sinister than that? Is it like, you know, "The West doesn't like the East"? Or that there are other kinds of motives at play?

So I think that historically, particularly during the Cold War, and again this started in the 50s and then went on throughout the entire duration of the Cold War, and I think that actually initially the countries in the Soviet bloc were invited in the Eurovision and they had their own version of the Eurovision.

However, there are countries like Lithuania, Estonia, Latvia, who are at the border with the Western world. They would pick up the signal from the Swedish television, for example, and they were actually watching the Eurovision, which was frowned upon by some of the regimes there. And so the original accusations was of kind of block voting, whereby all the countries in the Soviet Union would just tend to vote for each other and not consistently for other countries, irrespective of the quality of the song.

Later on, and I think that's what happens mainly in the UK, there was this kind of accusation that the UK was singled out and countries wouldn't vote for the UK Act.

for other reasons than the lack of quality or perceived lack of quality for their act. So much so that Terry Wogan, the commentator for the BBC, just quit on air at some point and said, this is all rigged and nobody loves us, so I'm not going to do it next year. And politicians also jumped onto that bandwagon and said,

raise the argument that, well, this is evidence that we should get out of the EU because Europe doesn't love us. And that's yet another example of that. Okay, so you built a Bayesian model to investigate how much of this is actually true. And what did you find?

Yes, exactly. So the main research hypothesis, if you want to be a bit pedantic and pompous, was that let's see if there is actually evidence in the data that there is a systematic way in which different countries tend to vote for each other and whether there's a strong signal that country A would always score highly country B and always score poorly country C.

So what we did was to build up this model that I think that the interesting point from the kind of Stasi point of view was that, you know, sometimes you do models to try and do prediction, like when you try and predict what the outcome of the general election would be. In our case, that wasn't necessarily the main output. We didn't want to predict who would win next year. What we wanted to do is to kind of balance out some of the potential outcomes

confounding elements in the signal that you would get so that you can try and isolate whether there is a systematic directionality of the votes from Spain towards Italy all the time or rather from Sweden towards Denmark all the time. That is,

over and above everything else, like a natural flows of the votes, which would kind of explain why a certain country tends to win just because of the favor of other countries. And the reverse of that would be why a country does never win because others kind of don't want to vote deliberately for them.

And the output of the model, what we actually saw, taking into account some contextual factors like the year of the competition and some characteristics of the act. Is it like a group song? Is it a solo song? Is it sung by a female artist or a male artist? Is it sung in the national language or in English? So accounting for all of these different things and then trying to account for some kind of geographic relationship

So, are you more likely to vote more highly for the country next door? Are you more likely to vote more highly and to score highly the countries who share some cultural background, like the Soviet bloc that you have commonalities in terms of your own language, for example?

After leaving that aside, taking that into account essentially, what we were after, the main output of the model was the residual, what's left. And is what's left a strong signal that you still tend to vote over and above everything else more strongly from a country to another? And actually what we saw is that there isn't such a strong signal in that direction. There were some...

Pairs, for example, Denmark tended to highly score Sweden consistently, but not in a way that you would deem this screws up all the results and that's the reason why Sweden wins. And more importantly, we didn't find any evidence of the reverse. So any strong signal of any country not voting ever, irrespective of everything else for any of the other performers.

So in this situation, you're controlling for a lot of factors where people might assume there is some bias, right? Like voting for your neighbor or voting for a song that is sung in English. In fact, I was actually in Stockholm yesterday.

a couple months ago and was at the ABBA museum. And they make the argument that when they won with Waterloo, the idea was that they needed to sing in English in order to like win Eurovision and, you know, become a success. So are there actual, I mean, looking at some of the other social science research, um,

Are there actual biases when it comes to whether it's a solo or group act, whether it's sung in English or the language of the country that is putting the song forward, whether it's a female or male, etc.?

I think that many of the models, certainly ours, found that there are some small effects of all of these different things. So generally speaking you may be more likely to score highly if you have songs sung in English historically and I think that this might not necessarily be the same over time. There may have been some kind of change point whereby up to a certain year this was the kind of trend and if you sung in English you'd have

higher chance of attracting more interest. And then it's kind of reversed and perhaps in the last few editions you'd have more acts singing in their own language than in the previous past. So the signal isn't even stable over time. But anything that you would pick up had a magnitude that wasn't huge. Essentially, there was nothing that you could just say, well, yes, that explains it. That's the reason why. If you have a song that is this, you'd certainly get higher votes by country, blah, blah.

There was nothing that suggests anything like that. So there's not like if someone were to hire you as a consultant, like Moldova next year is like Gianluca, we're hiring you to give us some pointers on how we can win Eurovision. Could you give them pointers in terms of if you really want to win, these are the subtle things you can do to improve your odds?

I don't think that there's a fail-safe measure. As a matter of fact, after our original paper, I had an MSc student doing some kind of analysis and what she did is to try and get data off Spotify to control for the song's characteristics, some of the hard data that you might link to the song, like in terms of

beat per minute or whether you would have like a more rock song or something like that. Yeah. And she was trying to figure out what are the key features that make the Eurovision winner. I think that there are some caveats with that analysis and we never actually pursued it

much further than that and mostly the problem was that not all the songs have this kind of hard data that we could find very easily anyway. And so there was lots of messiness and it wasn't a clear signal anyway. I think you might have some little advantages coming from all of these little features

Like, you know, you have a female act, maybe that increases the chances of getting higher votes from some of the countries. But that isn't what actually gets you to win the Eurovision. There's no single factor that would drive your success. That's a good point. We haven't even talked about the pop versus rock versus techno angle here. But essentially...

There's got to be a magic to it. It's not like a pop song is necessarily more likely to win than a rock. Of course, last year's winners were, congratulations, Italy, Maneskin, which is a rock group. So it's not, you know, it was like the opposite of...

compared with this year's, which was a single female act that was singing a pop song last year. It was a group act singing in Italian, and it was a rock song. And it was a very non-Eurovision song as well, Måneskin. It was like the opposite. ♪

But I think, again, that kind of feeds into the realm of speculation. I don't have any real evidence here and actually I've never even looked at it and I don't even want to. But I think if you start looking at the coverage that Eurovision gets, and again, you might suspect this is a silly thing, nobody should care, but they do and they start caring way before it actually starts. And there's a lot of kind of

propaganda almost coming all the way from February, March, so a couple of months, a few months before the actual contest and they start saying "oh this year Sweden is the hot favorite" and when Italy won we were the strong favorite, you know, the bookmakers were already betting that Italy would do very well and I think that, you know, nowadays you can listen to the songs well in advance of the contest. I think by February you can listen to all of the songs that will actually enter the the contest later in May

But there's probably more to it. There's probably a kind of a conscious act of the various countries who say, well, we're very, very interested in actually hosting the context next year, which isn't a non-trivial thing because, you know, to organize the whole Eurovision is a very costly enterprise. Typically, the national broadcaster will be responsible for the organization under the supervision of the Eurovision machinery.

And, you know, again, this is more in the realm of conspiracy theory, perhaps, than statistics. But you always get a clear explanation as to why a country was likely to win. Like in this case, Sweden have won this year and next year is going to be the 50th anniversary. I think that ABBA won the contest for the first time or participated for the first time.

When Italy won, it was on the back of a very difficult period because of COVID and Monaschim were highly rated before going into the contest. There was a lot going on even without thinking about the quality of the song.

And I think that kind of swings the mood of eventually the people who vote in the televoting, maybe the jurors as well. So there's more to it, I think, which is hard to capture with a model. Yeah, yeah. I heard that theory that Sweden won because ABBA won with Waterloo in 1974. So it'll be a celebration of...

Eurovision, its legacy in terms of creating, I guess, its probably most famous winner of all time and it'll be in Sweden, etc. etc. I don't mean it in a way it's rigged, it's deterministic. Sweden say next year we want to host it and make us win. I don't think it works like that.

But certainly there's an element of if a country is kind of willing and they kind of invest in an act that is geared in a certain way, but also there's the whole context and good reasoning, then maybe kind of the seed is planted in people's head and that helps. Well, and it seemed like the seed was more planted in the judge's head than it was the public's head, because going into the contest this time,

And Finland's song Cha Cha Cha, which is sort of a more like hard techno, almost, I don't know how exactly you would describe it, went viral on TikTok. Everyone was listening to it. And it ended up winning all of the public's votes. So there were kind of two competing energies at play there, which is that, yes, because people got to listen in advance and it made its rounds and went viral on TikTok, it

the public was all in on Finland even before the contest actually took place. But I guess for the sake of Eurovision's legacy and sort of brand promotion or whatever, maybe it made more sense to the judges to support Sweden.

I'm curious in all of this, if there's any other conclusions that you've come to or social science research that you've looked at in voting patterns that, you know, is worth acknowledging before we wrap up our serious conversation about Eurovision voting patterns.

No, I mean, like I said, I think that what was interesting is to try and debunk the idea that you can make claims based on hard evidence, which really wasn't there. And I think that that's what the model found. We didn't find, if you try and do some kind of series analysis and try and take away all the contextual factors and account for them, then what's left, which would indicate some kind of act of deliberate action,

going one way or the other really wasn't there. And if it was there, it was to a magnitude that wouldn't swing the results either way. And I think that's the main conclusion. I think that there may be more. I think that there are things that are very difficult to match up in a modelling, in a statistical modelling. I think that it's true that the fact that they have these two different systems, the jurors and the televoting system,

maybe lends itself to some kind of rebalancing. It was pretty obvious that Fjällräb would get lots of the favours of people in the televoting. It was the classic Eurovision act geared towards the kind of crazy night, oh, these guys are just off the hook, let's vote for them.

So if somebody wanted to rebalance things, they probably would have made sure that the jurors would vote in a certain way. But to be fair, Sweden did get a lot of the popular vote as well. Finland won the popular vote, but it wasn't like a million to zero. Sweden did really good on both. And eventually, I think that's what kind of pushed them off the line. Yeah, for sure. I mean, so you did find, though, ultimately that...

There's some favorable bias towards countries that you're close to or share a cultural affinity for. I mean, would that mean that maybe there is a disadvantage for the UK because it's an island? I mean, obviously, there are countries that are still close to it, but it's an island and has built its own sort of like identity and culture that is separate from Europe. I mean, I think that it's...

The British culture as well in Europe is a dominant one. If you look at every sort of possible music industry that is in Eurovision, the British industry is kind of dominant. Most of the acts who sell like a gazillion records or are top of the hits charts in Spotify or whatever, they come from Britain anyway. So I don't think that there's any active dislike or not wanting to...

reward the UK for a participation. It is fair and I think it is recognised here as well that a lot of the acts that have actually entered when the UK came last a few years ago

Maybe people in Europe weren't happy that the UK had done Brexit, but also the song really wasn't that great. And, you know, it wasn't a huge surprise that you wouldn't get a lot of votes. It's just what it is. So I think that the geographical aspects in our model, we had like a structure of neighbourhood whereby the countries that are closest to you would be considered closer in terms of the geographic and possibly cultural level.

And so there is an element of accounting for that. And we do see, for example, some signal that tends to say that Ireland tends to score the UK Act relatively highly, but the reverse isn't true. So by the same token, you could say that actually Ireland should be agreeable with the UK and the UK doesn't vote highly for Ireland, even though they both speak English and they are arguably close by some metrics.

Well, it's also funny because Ireland has historically done much better than the UK in Eurovision history. Yeah, yes. This is all really interesting stuff. Also, I was mistaken. I said that Manuskin won last year. Manuskin didn't win last year. Ukraine won last year, which is also an important sort of like maybe political signal, which is support for Ukraine amidst a Russian invasion. It was, yes. Undoubtedly.

So yeah, I mean, it seems like there's a lot of politics in this awfully sort of like kooky, fun song contest.

I'd agree, I'd agree, yes. And I think that the very fibre, the nature of the Eurovision since its inception was a very political statement anyway. It was meant to bring together all the European countries this side of the Iron Curtain and kind of make a big splash for reunion after the war. So it's by definition a political operation.

Interesting. All right. Well, I said that at the end of the podcast, I would let you shout out any serious non-Eurovision work that you wanted to mention. I know you're a stats professor and a health economist. What should our listeners know that's not necessarily related to Eurovision?

I think a lot of the normal serious stuff that I do, particularly since becoming the head of my department, is very, very boring. And I don't think your listeners want to know anything about that. So, yeah, probably best to just Google and find the Eurovision stuff and some football prediction stuff as well. Maybe that's interesting. Oh, interesting. OK, well, have you ever thought of creating an actual prediction model for Eurovision?

I did and that's why I kind of was pushing my student to try and find these data on the hard qualities of the song. It was more of a summer kind of let's have some fun with the project and it never actually went to a fully fledged modeling and paper. But I think the difficulty would be in finding the data, a consistent data set that you could match up with the Eurovision data. Makes sense.

All right. Well, we're going to leave it there. Thank you so much for joining me today, Gianluca. Thank you very much. Gianluca Baio is a professor of statistics and health economics at the University College London. My name is Galen Drew. Tony Chow is in the control room and also on video editing. You can get in touch by emailing us at podcasts at 538.com. You can also, of course, tweet at us with any questions or comments. If you're a fan of the show, leave us a rating or a review in the Apple Podcast Store or tell someone about us.

Thanks for listening, and we will see you soon.

Is The Eurovision Song Contest Rigged?

FiveThirtyEight Politics

Chapters