Why you shouldn’t use Twitter to predict elections

Yesterday the Social Science Computer Review published a new paper by Harald Schoen, Oliver Posegga, Pascal Jürgens, and me on the futility of trying to predict elections with Twitter. This claim shouldn’t be all that controversial, but apparently it is.

Quite a few studies have identified fundamental problems with Twitter-based predictions. Especially Daniel Gayo-Avello has written multiple times about systematic flaws in these attempts, so has Takis Metaxas, and Mark Huberty. More fundamentally Fernando Diaz and colleagues have identified central challenges in interpreting social media data as proxies for survey results because of irregular fluctuations in the services’ user base and difficulties to consistently map this activity to established sentiment or opinion measures. The predictive potential of other digital services has also been challenged. For example, David Lazer, Gary King, and colleagues illustrated general challenges of working with these data by focusing on Google Flu Trends. So skeptics are in good company. Still, where there is a business case, there also is hope.

Building on earlier work, in our new paper we focused on checking the central claim underlying the arguments of proponents of using Twitter to predict elections, implicitly assuming that mentions of political actors somehow indicate political support. On the face of it, this argument seems highly unlikely but a series of apparently positive findings give the impression that there might be something to it. If we look closely at the data, though, we find that Twitter mentions much more likely indicate public attention towards politics, a concept that sometimes–but far from always–might be correlated with political support. As we state in the article:

“Our evidence raises doubts with regard to the validity of Twitter-based metrics as indicator for political support. In addition, trends in the daily mention counts of political parties showed no systematic link with trends in opinion polls. Instead, the dynamics in the daily mention counts of parties appear to correspond with media events, media coverage of politics, and controversies. It, therefore, appears far more likely that Twitter-based metrics measure public attention toward politics than political support. Sometimes attention might be a covariate of support, but this relationship is far from stable. Our analysis underscores the importance of approaching the use of digital trace data in the measurement and analysis of political and social phenomena cautiously and emphasizes the importance of using established standards of social science methodology in indicator validation to avoid premature conclusions.”

Thus, studies identifying Twitter to be a predictor of electoral results might in fact have identified cases in which public attention, manifested in Twitter messages, was correlated with public support. But, as the many negative findings by us and other researchers indicate, this relationship is far from universal. To use Twitter as indicator of electoral fortunes might thus be simply betting on the stability of the link between attention and support in any given election. A stability which various politicians at the center of controversy and scandal might doubt.

To be sure, this is not to say that Twitter-data do not hold significant potential for public opinion research. As we have stated elsewhere, Twitter messages mirror parts of social and political life mediated through the interests, attention, and motivations of Twitter users. There is a host of topics in which Twitter-data might be productively used:

Likely candidates for political phenomena to create digital traces are political media events, intense media coverage of politics, or public controversies. Accordingly, future research may focus on using Twitter data to analyze which kind of political information attracts Twitter users’ attention and is distributed online. This gives rise to important questions concerning the sources, that is, the media, political elites, or social networks, and conditions successful in getting Twitter users to pay attention to political information. Thus, Twitter has the potential to become a source of insight into conditions and dynamics of attention toward politics.

Unfortunately, much of recent research attention has been focused on trying to make Twitter-data fit other measurements of political and social life. While the hope for finding cheap and readily available proxies for expensive and slow survey-based measures of public opinion is understandable, Twitter might be an unlikely candidate to provide this proxy. So instead of overextending our expectations scholars should probably leave predictions to consultants and instead:

“[…] should acknowledge the conditional nature of findings more freely and be more careful in considering and analyzing the consequences of potentially varying data-generating mechanisms. This might lead Twitter-based research to free itself from inflated early expectations to find proxies of public opinion in Twitter-data and instead focus on the potential of digital trace data in yielding insights into public attention toward political information. Digital trace data may thus provide valuable information for public opinion research, though on different phenomena than those on which prior research focused.”

Andreas Jungherr, Harald Schoen, Oliver Posegga, and Pascal Jürgens. 2016. Digital Trace Data in the Study of Public Opinion: An Indicator of Attention Toward Politics Rather Than Political Support. Social Science Computer Review. (Online First). doi: 10.1177/0894439316631043 [Online Appendix]

This paper’s findings have be covered by The Hill, The Guardian, The Washington Post, Newsweek, Reuters, Ars Technica UK, Fortune, Politico EU, The World Economic Forum, and others.