Nate Silver and the Ascendance of Expertise by Bora Zivkovic | November 14, 2012
Total Page:16
File Type:pdf, Size:1020Kb
Permanent Address: http://blogs.scientificamerican.com/a-blog-around-the-clock/2012/11/14/nate-silver-and-the-ascendance-of-expertise/ Nate Silver and the Ascendance of Expertise By Bora Zivkovic | November 14, 2012 Nate Silver is now a meme (also source ADVERTISEMENT of the image on the left). I usually pepper my posts with links, but today I feel lazy, so I listed a bunch of links at the bottom – hours of fascinating reading you can have after you read my post! Who is Nate Silver? Nate Silver likes to play with numbers. He started out with sports, then burritos, then politics. He, using statistics, correctly predicted most (not all, but almost all) presidential and congressional races in 2008, 2010 and 2012. Back in 2010, he came to ScienceOnline and moderated a session (together with Arikia Millikan) on using math to study human behavior online – the Web Science. What does Nate Silver do? Twenty years ago, there were only a few pollsters out there and they did relatively few polls. Today, there are many polling organizations and they, especially in the home stretch of an election, poll incessantly, every day. They do national polls, statewide polls, even local polls. Over the years, they refine their methodology. Some predict outcomes better than others, for a variety of reasons. Nate Silver averages all the polls, weights each poll according to the statistics of past performances, and produces a dailychanging set of numbers predicting outcomes of various electoral races. For the Presidental elections, unlike pundits focusing on national polls, he rightly focuses on state polls, especially in swing states, in order to predict the winner of the Electoral College – the only thing that really counts (we can discuss if that is right or wrong, but that is how the game is played now, so that it what he measures). What did Nate Silver not do? As a couple of bloggers (see links at the bottom) pointed out, Nate Silver did not do Big Data. These are pretty small and limited data sets he has at his disposal. In aggregate, they are powerfully predictive, but that is not Big Data, though the motivations and methodologies are similar. As Silver started in sports statistics, being a part of the Moneyball movement in baseball, people assume that what he is doing now is the same thing. But it is not. It is also not the same as what he did with burritos, though that comes closer. In baseball (and later in basketball, though horse racing and betting industry has been doing this for a century at least), there are hard blogs.scientificamerican.com/a-blog-around-the-clock/2012/11/14/nate-silver-and-the-ascendance-of-expertise/?print=true 1/7 data. Player hit the ball or did not. Caught the ball or did not. The ball ended in a spot X or did not. It was a homerun or it wasn’t. Empirical data. Are two players good buddies or not does not matter that much at that level – they are both professionals and will do their best regardless of interpersonal relationships, body language and other subjective parameters. Thus, stats in sports work well, as they are based on clearly measurable things. From such stats, one can rank players and teams, and predict with quite a high degree of accuraccy which teams will win and which will lose. Or which horses have which odds for winning a race. So again, What did Nate Silver do? People focus on numbers, imagining they are hard data. But remember that the numbers come from polls. Polls are questionnaires. What Nate Silver did was social science. Polls ask questions. People answer them differently. They may have conscious or unconscious biases. They will have different backgrounds and different levels of being informed. Some will lie on purpose, to skew the polls, as part of their activism. Some will lie unconsciously because they are afraid to tell what they really think. People respond differently if they are polled over their landline phones (public) and differently if called on their cellphones (private), and differently in online polls versus being asked in person, face to face (e.g., in exit polls). Some people put a lot of thought into their answers. Others want to do it as fast as possible and go with gut feeling, or even almostrandom responses. Different pollsters will ask similar questions, but with slightly different wording. And we know that wording affects the responses. The order of questions also affects responses. Each pollster can only reach a limited number of people, so the small sample size results in a pretty large error. But when Nate (and others) averages the polls, he increases the sample size, thus reducing the error. When he takes into account the past accuracy of pollsters and weights them accordingly, he further reduces the error. People who lie in opposite directions cancel each other. Pollsters who are biased in opposite directions cancel each other. A wellbalanced, weighted average can take care of all of that, and produce a much more accurate prediction. But importantly, it is still not numbers from physical measurements. It is statistics (and yes, Statistics is a subdiscipline of Mathematics) applied to messy human minds and brains and emotions and moods. It’s people! Why Nate Silver now? blogs.scientificamerican.com/a-blog-around-the-clock/2012/11/14/nate-silver-and-the-ascendance-of-expertise/?print=true 2/7 A lot of it has to do with the current political climate. I wrote my thoughts about it on my Tumblr as I did not think it was appropriate to post it here, but go take a look. In short, it is a backlash to alternative reality, alternative facts, alternative science, alternative math. It is a backlash to the self perpetuating cycles of mutual lying between rightwing media, rightwing politicians, rightwing donors and rightwing voters, each preventing the others from straying one millimeter away from this alternative fantasy world. It is a backlash against antiempiricism, antiscience, antifacts, headinthesand, “we make reality now” mindset. Practical solutions require dealing with the world as it is, not the world one imagines to be or wants it to be. And when postmodernism in public life reaches a point of saturation, and when people have had enough of it, and when there is a backlash, people will go for as extreme opposite as they can find. In this case: math. Numbers. Hard, cold numbers. Unbiased analysis. No “gut feelings”. Which is why they go for Nate. Which is why they tend to ignore that Silver’s numbers are people. Why Nate Silver and not other numbers guys? Because Nate is a blogger. Really. Others put data out there as well (see links at the bottom). Nice graphs and charts and tables. Great numbers, essentially the same as Nate’s. But they don’t tell a story about the data. He does. He’s been doing it for years. He has regular readership. He has a recognizable voice. He has earned trust not just by the strength of his predictions, but also by the strength of his writing, his personality that shines in his blog posts, his transparency about his thinking and about his methodology. People focus on Nate and trust Nate because he is an expert, but more importantly because he is an expert who can tell the story. An expert who can explain stuff in ways that people understand. He narrates his work and his numbers. Why Nate Silver’s blog? A number of people, some unhappy that other numbercrunchers did not reach Nate’s fame (or rightwing wreath), explain his prominence by the fact that his blog is hosted by New York Times (see links at the bottom). Even the NYTimes public editor suggested that his fame is due to the association with the NYTimes brand. This is upside down. And she got instant and strong backlash. It is Nate who is the brand. NYTimes profits more from having Nate on their site (the traffic to his blog just before and during the election day dwarfed all the traffic to everything else on their site) than he does from being associated with them. He is strengthening their brand, by being an expert on site, rather than the other way round. NYTimes reported on Nate’s traffic in a pretty vague way – number of site visits that included visits to Nate Silver’s blog. But we know that very few people go to sites via homepages. Older people and people within the news business may still have that habit. But most people do not. I bet that at least 90% (and more likely 99.99%) of the traffic to Silver’s blog on the election day did not come from the NYTimes homepage, or any other page on the site. It came from direct links, social media, “dark social”, emails, bookmarks, RSS feed readers, searches, etc. Nate Silver and the Ascendance of Expertise What New York Times does smartly, to enhance its brand, is to hire people with real expertise, people like Nate Silver (and Paul Krugman etc.) and give them a prominent spot on the site (and even sometimes in the paper version). Washington Post does the same with Ezra Klein. Many media outlets, including the one you are on right now, have set up blogging networks specifically in order to attract and host writers with real expertise. As I wrote a couple of weeks ago: Landing on the New York Times page after you followed a link tells you something about it, to a certain extent.