For the last seven years or so, Pascal Jürgens and I have tried to tease meaning out of data collected on Twitter in our research. As classically trained social scientists, this has not always been a smooth ride. Working with Twitter data has been tremendously interesting and rewarding but as we were both trained neither in software development nor code-based data collection or management, it is fair to say that our learning curve was very steep. As Twitter has become an increasingly popular research object and environment, we decided to share some of our experiences in form of a tutorial and an accompanying starter-kit with code examples for the code-based collection, preparation, and analysis of Twitter data.
A Tutorial for Using Twitter Data in the Social Sciences: Data Collection, Preparation, and Analysis
Abstract: The ever increasing use of digital tools and services has led to the emergence of new data sources for social scientists, data wittingly or unwittingly produced by users while interacting with digital tools. The potential of these digital trace data is well-established. Still, in practice, the process of data collection, preparation and storage, and subsequent analysis can provide challenges. With this tutorial, we provide a guide for social scientists to the collection, preparation, and analysis of digital trace data collected on the microblogging service Twitter. This tutorial comes with a set of scripts providing researchers with a starter kit of code allowing them to search, collect, and prepare Twitter data following their specific research interests. We will start with a general discussion of the research process with Twitter data. Following this, we will introduce a set of scripts for data collection on Twitter. After this, we will introduce various scripts for the preparation of data for analysis. We then present a series of examples for typical analyses that could be run with Twitter data. Here, we focus on counts, time series, and networks. We close this tutorial with a discussion of challenges in establishing digital trace data as a normal data source in the social sciences.
Our imaginary reader was very much a social scientists starting out in the code-based work with data collected on Twitter. This and our own background as social scientists and not as computer scientists might make the tutorial and our code examples seem rather quaint to more formally trained eyes. Still, we hope what the tutorial lacks in this regard it makes up through our perspective in approaching Twitter data as object for research in the social sciences.
While we aim to update the tutorial and our code in the future, please let us know in the meantime if you find mistakes or run into trouble with the code. The easiest option for this is by opening an issue in the GitHub repository with our example code.
You are of course free to use and adapt the scripts provided in our code examples in your research or teaching. If you do so, please cite the package by providing the following information:
If you want to cite this tutorial please provide the following information:
Pascal Jürgens and Andreas Jungherr. 2016. A Tutorial for Using Twitter Data in the Social Sciences: Data Collection, Preparation, and Analysis. Available at SSRN: http://ssrn.com/abstract=2710146