Nate Silver is now a meme (also source of the image on the left).
I usually pepper my posts with links, but today I feel lazy, so I listed a bunch of links at the bottom - hours of fascinating reading you can have after you read my post!
Who is Nate Silver?
Nate Silver likes to play with numbers. He started out with sports, then burritos, then politics. He, using statistics, correctly predicted most (not all, but almost all) presidential and congressional races in 2008, 2010 and 2012. Back in 2010, he came to ScienceOnline and moderated a session (together with Arikia Millikan) on using math to study human behavior online - the Web Science.
What does Nate Silver do?
Twenty years ago, there were only a few pollsters out there and they did relatively few polls. Today, there are many polling organizations and they, especially in the home stretch of an election, poll incessantly, every day. They do national polls, state-wide polls, even local polls. Over the years, they refine their methodology. Some predict outcomes better than others, for a variety of reasons.
Nate Silver averages all the polls, weights each poll according to the statistics of past performances, and produces a daily-changing set of numbers predicting outcomes of various electoral races. For the Presidental elections, unlike pundits focusing on national polls, he rightly focuses on state polls, especially in swing states, in order to predict the winner of the Electoral College - the only thing that really counts (we can discuss if that is right or wrong, but that is how the game is played now, so that it what he measures).
What did Nate Silver not do?
As a couple of bloggers (see links at the bottom) pointed out, Nate Silver did not do Big Data. These are pretty small and limited data-sets he has at his disposal. In aggregate, they are powerfully predictive, but that is not Big Data, though the motivations and methodologies are similar.
As Silver started in sports statistics, being a part of the Moneyball movement in baseball, people assume that what he is doing now is the same thing. But it is not. It is also not the same as what he did with burritos, though that comes closer.
In baseball (and later in basketball, though horse racing and betting industry has been doing this for a century at least), there are hard data. Player hit the ball or did not. Caught the ball or did not. The ball ended in a spot X or did not. It was a home-run or it wasn't. Empirical data. Are two players good buddies or not does not matter that much at that level - they are both professionals and will do their best regardless of interpersonal relationships, body language and other subjective parameters. Thus, stats in sports work well, as they are based on clearly measurable things. From such stats, one can rank players and teams, and predict with quite a high degree of accuraccy which teams will win and which will lose. Or which horses have which odds for winning a race.
So again, What did Nate Silver do?
People focus on numbers, imagining they are hard data. But remember that the numbers come from polls. Polls are questionnaires. What Nate Silver did was social science.
Polls ask questions. People answer them differently. They may have conscious or unconscious biases. They will have different backgrounds and different levels of being informed. Some will lie on purpose, to skew the polls, as part of their activism. Some will lie unconsciously because they are afraid to tell what they really think. People respond differently if they are polled over their land-line phones (public) and differently if called on their cell-phones (private), and differently in online polls versus being asked in person, face to face (e.g., in exit polls). Some people put a lot of thought into their answers. Others want to do it as fast as possible and go with gut feeling, or even almost-random responses.
Different pollsters will ask similar questions, but with slightly different wording. And we know that wording affects the responses. The order of questions also affects responses.
Each pollster can only reach a limited number of people, so the small sample size results in a pretty large error.
But when Nate (and others) averages the polls, he increases the sample size, thus reducing the error. When he takes into account the past accuracy of pollsters and weights them accordingly, he further reduces the error. People who lie in opposite directions cancel each other. Pollsters who are biased in opposite directions cancel each other. A well-balanced, weighted average can take care of all of that, and produce a much more accurate prediction.
But importantly, it is still not numbers from physical measurements. It is statistics (and yes, Statistics is a sub-discipline of Mathematics) applied to messy human minds and brains and emotions and moods.
It's people!
Why Nate Silver now?
A lot of it has to do with the current political climate. I wrote my thoughts about it on my Tumblr as I did not think it was appropriate to post it here, but go take a look.
In short, it is a backlash to alternative reality, alternative facts, alternative science, alternative math. It is a backlash to the self-perpetuating cycles of mutual lying between rightwing media, rightwing politicians, rightwing donors and rightwing voters, each preventing the others from straying one millimeter away from this alternative fantasy world. It is a backlash against anti-empiricism, anti-science, anti-facts, head-in-the-sand, "we make reality now" mindset. Practical solutions require dealing with the world as it is, not the world one imagines to be or wants it to be.
And when postmodernism in public life reaches a point of saturation, and when people have had enough of it, and when there is a backlash, people will go for as extreme opposite as they can find. In this case: math. Numbers. Hard, cold numbers. Unbiased analysis. No "gut feelings". Which is why they go for Nate. Which is why they tend to ignore that Silver's numbers are people.
Why Nate Silver and not other numbers guys?
Because Nate is a blogger. Really. Others put data out there as well (see links at the bottom). Nice graphs and charts and tables. Great numbers, essentially the same as Nate's. But they don't tell a story about the data. He does. He's been doing it for years. He has regular readership. He has a recognizable voice. He has earned trust not just by the strength of his predictions, but also by the strength of his writing, his personality that shines in his blog posts, his transparency about his thinking and about his methodology.
People focus on Nate and trust Nate because he is an expert, but more importantly because he is an expert who can tell the story. An expert who can explain stuff in ways that people understand. He narrates his work and his numbers.
Why Nate Silver's blog?
A number of people, some unhappy that other number-crunchers did not reach Nate's fame (or rightwing wreath), explain his prominence by the fact that his blog is hosted by New York Times (see links at the bottom). Even the NYTimes public editor suggested that his fame is due to the association with the NYTimes brand.
This is upside down. And she got instant and strong backlash. It is Nate who is the brand. NYTimes profits more from having Nate on their site (the traffic to his blog just before and during the election day dwarfed all the traffic to everything else on their site) than he does from being associated with them. He is strengthening their brand, by being an expert on site, rather than the other way round.
NYTimes reported on Nate's traffic in a pretty vague way - number of site visits that included visits to Nate Silver's blog. But we know that very few people go to sites via homepages. Older people and people within the news business may still have that habit. But most people do not. I bet that at least 90% (and more likely 99.99%) of the traffic to Silver's blog on the election day did not come from the NYTimes homepage, or any other page on the site. It came from direct links, social media, "dark social", emails, bookmarks, RSS feed readers, searches, etc.
Nate Silver and the Ascendance of Expertise
What New York Times does smartly, to enhance its brand, is to hire people with real expertise, people like Nate Silver (and Paul Krugman etc.) and give them a prominent spot on the site (and even sometimes in the paper version). Washington Post does the same with Ezra Klein. Many media outlets, including the one you are on right now, have set up blogging networks specifically in order to attract and host writers with real expertise.
As I wrote a couple of weeks ago:
© 2012 ScientificAmerican.com. All rights reserved.
Landing on the New York Times page after you followed a link tells you something about it, to a certain extent. You still have to figure out if you trust the article you are about to read. Your expectations are higher than if it was Daily Mail, but you are still on guard. How do you decide in advance? By the name in the byline. If it is Maureen Dowd, you expect entertainment, but not much depth. If it's David Brooks, you expect seductively beautiful writing that is based on pseudo-sociology he picked out of thin air to conform to his ideology. But if it's Paul Krugman, you know you will get a better understanding of some aspect of economics because the guy knows his stuff - he is an expert.And you know exactly what you'll get if you see the byline of Nate Silver. Expertise engenders trust. When I write about biology, my readers trust me as I am an expert. When I write about media, people trust me a little bit less because my expertise in this came later, was not "official" (i.e., no graduate school degrees), and is mostly based on my own impressions and experience, though my track record so far has been pretty good. When I write about politics...why would anyone trust me? Everyone has a political opinion, right? What is important to note is that there is hunger out there for expertise. I started as a political blogger. Back in 2003/2004 there was a bunch of us starting political blogging. We each tried to add a particular angle, or bring in our other expertise (I focused on psychology of ideology, a nascent field now called 'political psychology'), but mainly we pontificated about politics and performed acts of media criticism and of political activism. After the 2004 election, many of us specialized. Ezra Klein focused on health care and became a "Go To" person for it, resulting in his hire by the Washington Post. Many others did something like that and got hired as campaign managers, or writers, or consultants, etc. I focused on science and ended up at Scientific American. In January 2005 I started a science blog, separate from my political blog. And instantly, with my very first post, the new blog reached the same traffic levels as the old blog. There were comments, questions. In science, I was an expert, and people trusted me and were hungry for information. I said and wrote this many times, but long posts that do not shy away from nitty-gritty details (including numbers, formulae, technical terms if explained first, even Latin names for animals - see super-successful Tetrapod Zoology blog right here on the network) do extremely well. They may not get an instand surge of low-quality traffic from Slashdot, Digg, Reddit, Stumbleupon or Fark, but they accrue tons of traffic over time. Such pieces are not seen as entertainment, but as resources - something to be saved, bookmarked and shared with friends. Such pieces keep getting re-discovered and re-shared for years after initial publication. They provide value that a one-hit wonder, entertaining piece does not. They provide value that standard, short, news pieces do not - they provide context and detail and quality of explanation that comes from expertise, something that a 400-word piece cannot possibly contain, as there is not enough space for it. Longform writing works. What is expertise? How does one become an expert? There are two ways. There is the 20th century method (yes, 20th century is an outlier on everything), in which one does hands-on research on a very narrow project while, hopefully, reading a little bit more broadly, resulting in an official badge of expertise - an MS or PhD or MD or some such degree. And then there is the historically traditional method that is making a big come-back now - having a deep interest in the topic and doing it yourself, reading, discussing with others, doing own research, blogging about it, writing and reporting on it for years, establishing oneself as an expert on the topic. This is how the most respected journalists became most respected - by becoming the Go To experts on a particular topic. The generalists and pundits - or, if you want, foxes as opposed to hedgehogs - are the reason why the audience is losing trust in the traditional media. They have seen expertise, and they are not going back. There is something importantly different about l'affaire Silver, though. Most of the cases in the past were impressionistic. We used our own 'gut feelings' to say that a particular blog post by an expert X was better than a traditional news article by a journalist Y. But now we can back up our gut feelings with numbers. This case is empirical. Expert blogger Nate Silver was correct, while pundits and traditional bloviators were not...and here are the numbers. How does expertise fit inside the new media ecosystem? It is easy here at Scientific American. We are an expert publication almost by definition. When news breaks, and there is a science component to it, others come to our site to get the reliable scoop on it. Generalist news organizations link to our articles on the scientific aspects of news stories. All our editors are experts on the topics they write about (and some even have the 20th century badges of expertise, i.e., PhDs and such). And then we have the blog network, where we have about 50 additional experts in other fields. Being on, or regularly reading, Scienceblogs.com over several years, where science bloggers were treated as 'media', taught us a lot. We learned from one another, learned from our own mistakes, and learned by analyzing mistakes of traditional media. We encountered and studied the traditional journalistic ethics and best practices and incorporated the best of it into our blogging. The Pepsigate scandal was a particularly useful teaching moment for all of us. We became better writers, better journalists, and better bloggers. The distinctions between these blurred. But we remained experts in our domains. And we resisted some of the traditional media trappings. Being Web natives, we vehemently resist the alien concept of "word count". No blogger I know ever counts words in their posts (if they do, they are too ashamed to say it publicly). The post is done when it's done, when all the historical, philosophical, social and methodological context is included, all details hashed out, all conclusions finalized. And we know that #longform works best. And we resist detached "objectivity". We know we gain rapport and trust with our readers if we insert ourselves into our stories, explain what is the personal connection, where does our expertise on the topic come from, what are our potential biases on the topic, why are we particularly excited about this topic and decided to write about that and not about something else. As I said yesterday, the traditional and new forms are fusing, learning from each other, getting better as a result, and we are all better off because of it. The line between blogs and columns, and between beats and obsessions is getting fuzzy, and that's a good thing. Many traditional journalists are now also blogging, experimenting with forms and formats, and then transferring those into their more traditional writing. This is why forward-looking media organizations are hiring experts. And why the pundits and bloviators, once their contracts expire or they retire, will gradually disappear from the media ecosystem (this will take many years, especially on TV which is the most resistant to change). This is why journalism schools are training experts. This is why media organizations are hiring bloggers. And then some of those bloggers get desks in the office, salaries equal to staff, benefits, etc. One day, that will be the norm. Let's hope. Links: Nate Silver: the verdict. Under Attack, Nate Silver Picks the Wrong Defense The Times's Washington Bureau Chief, and Legions of Others, in Defense of Nate Silver New York Times wants to hold Nate Silver to newsroom standards Sorry, Margaret, You Need to Get Out More Your Employee Is an Online Celebrity. Now What Do You Do? Nate Silver probability map vs. Actual map Three Lessons From The Nate Silver Controversy Here's What the New York Times' Nate Silver Traffic Boom Looks Like In defense of Nate Silver: Pundits bare their misunderstanding. 'How Can That Be?' More on the 'They Can't Both Be Right' Saga Wrath of the Math: Obama Wins Nerdiest Election Ever Silver Medal In Defense of Nate Silver, Election Pollsters, and Statistical Predictions The Nate Silver backlash Data, uncertainty, and specialization: What journalism can learn from FiveThirtyEight's election coverage Nate Silver gets a big boost from the election Why Math is Like the Honey Badger: Nate Silver Ascendant Nate Silver of 538.com and his critics in the press corps. Get your literacy up. Nate Silver's Braying Idiot Detractors Show That Being Ignorant About Politics Is Like Being Ignorant About Sports In defense of Nate Silver -- and basic math Today's War on Nate SIlver: Quiet Flows the Don Edition The Passion of Nate Silver (Sort Of) Pundits versus probabilities What's FiveThirtyEight Good For?: The Inevitable Nate Silver Backlash How did Nate Silver Get the Election Odds so Wrong? Math and Discipline -- Why Nate Silver's Accuracy Isn't About "Big Data" Nate Silver the Real Winner of Election 2012 How did Nate Silver predict the US election? Among the top election quants, Nate Silver reigns supreme Drew Linzer: The stats man who predicted Obama's win Was Nate Silver the Most Accurate 2012 Election Pundit? Climate science is Nate Silver and U.S. politics is Karl Rove Debunking Two Nate Silver Myths Whatever Nate Silver Does, Isn't Science How a nerd named Nate Silver changed political reporting forever. Nate Silver: Why I Started FiveThirtyEight Pundit Forecasts All Wrong, Silver Perfectly Right. Is Punditry Dead? Can Nate Silver's example save political journalism? Gallup is very upset at Nate Silver Nate Silver on the Election, Pundits, and His Drunk Alter Ego Foxy Nate Silver and why old-media hedgehogs could soon be old news Follow Scientific American on Twitter @SciAm and @SciamBlogs. Visit ScientificAmerican.com for the latest in science, health and technology news.
© 2012 ScientificAmerican.com. All rights reserved.
