Capturing Data from Social Media

Key Software

Prerequisites for Twitter Data Crawling – Twitter API

What is Twitter API

Twitter API stands for Twitter Application Programming Interface. It is a set of “standardized methods” (Bucher, 2013) that enable interaction with the Twitter platform and programmatically access its features. With Twitter API, developers, researchers and users can now build applications that interact with Twitter in various ways, including tweeting, engaging, and retrieving tweets.

According to Twitter:

“At a high level, APIs are the way computer programs ‘talk’ to each other so that they can request and deliver information. This is done by allowing a software application to call what’s known as an endpoint: an address that corresponds with a specific type of information we provide (endpoints are generally unique like phone numbers).”

Twitter API access levels and versions

Source: https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api

Obtain Twitter API

Create a Developer Account

Obviously, you’ll need a Twitter account to create a developer account before applying for Twitter API access. If you do not already have one, follow the instructions on the Twitter website to sign up for an account.

Apply for Twitter API Access

You’ll then need to fill out an application to gain access. It will require you to provide some basic information, including your academic profile and project description. Depending on the complexity of your project and the workload of the review team, the approval of your application could take days to weeks.

Information you need to provide about your project includes:

What’s your research project’s name?
Does this project receive funding from outside your academic institution?
In English, describe your research project.
In English, describe how Twitter data and/or Twitter APIs will be used in your research project.
Will your research present Twitter data individually or in aggregate?
In English, describe your methodology for analyzing Twitter data, Tweets, and/or Twitter users.
In English, describe how you will share the outcomes of your research (include tools, data, and/or resources).
Will your analysis make Twitter content or derived information available to a government entity?

Generate Twitter API credentials

Once your application gets approved, you can create a project and app to access the Twitter API and then generate your app’s API keys and user access tokens to capture Twitter data using your preferred programming language (such as Python, R, or JavaScript).

Note: Twitter only displays your access token and secret when you first generate it due to security considerations. You may revoke or regenerate them at any time, rendering your existing tokens invalid.

Using R to collect Twitter data

Useful R Package

Reference and things to Read

VOSON Lab http://vosonlab.net/SocialMediaLab
Bucher, Taina. “Objects of intense feeling: The case of the Twitter API.” Computational Culture 3 (2013).
Kim, Annice E., et al. “Methodological considerations in analyzing Twitter data.” Journal of the National Cancer Institute Monographs 2013.47 (2013): 140-146.
Twitter Bots Tutorials https://digitalinspiration.com/docs/twitter-bots
Twitter Bots Tutorials (A Full Guide for Beginners) https://learn.g2.com/how-to-make-a-twitter-bot
Littman, Justin. “Where to get Twitter data for academic research.” (2017). https://gwu-libraries.github.io/sfm-ui/posts/2017-09-14-twitter-data
Trupthi, M., Suresh Pabboju, and G. Narasimha. “Sentiment analysis on twitter using streaming API.” 2017 IEEE 7th International Advance Computing Conference (IACC). IEEE, 2017.
Pfeffer, Jürgen, Katja Mayer, and Fred Morstatter. “Tampering with Twitter’s sample API.” EPJ Data Science 7.1 (2018): 50.
Hino, Airo, and Robert A. Fahey. “Representing the Twittersphere: Archiving a representative sample of Twitter data under resource constraints.” International journal of information management 48 (2019): 175-184.
Aguilar-Gallegos, Norman, et al. “Dataset on dynamics of Coronavirus on Twitter.” Data in Brief (2020): 105684.