Prerequisites for Twitter Data Crawling – Twitter API
What is Twitter API
Twitter API stands for Twitter Application Programming Interface. It is a set of “standardized methods” (Bucher, 2013) that enable interaction with the Twitter platform and programmatically access its features. With Twitter API, developers, researchers and users can now build applications that interact with Twitter in various ways, including tweeting, engaging, and retrieving tweets.
According to Twitter:
“At a high level, APIs are the way computer programs ‘talk’ to each other so that they can request and deliver information. This is done by allowing a software application to call what’s known as an endpoint: an address that corresponds with a specific type of information we provide (endpoints are generally unique like phone numbers).”
Twitter API access levels and versions
Obtain Twitter API
Create a Developer Account
Obviously, you’ll need a Twitter account to create a developer account before applying for Twitter API access. If you do not already have one, follow the instructions on the Twitter website to sign up for an account.
Apply for Twitter API Access
You’ll then need to fill out an application to gain access. It will require you to provide some basic information, including your academic profile and project description. Depending on the complexity of your project and the workload of the review team, the approval of your application could take days to weeks.
Information you need to provide about your project includes:
- What’s your research project’s name?
- Does this project receive funding from outside your academic institution?
- In English, describe your research project.
- In English, describe how Twitter data and/or Twitter APIs will be used in your research project.
- Will your research present Twitter data individually or in aggregate?
- In English, describe your methodology for analyzing Twitter data, Tweets, and/or Twitter users.
- In English, describe how you will share the outcomes of your research (include tools, data, and/or resources).
- Will your analysis make Twitter content or derived information available to a government entity?
Generate Twitter API credentials
Note: Twitter only displays your access token and secret when you first generate it due to security considerations. You may revoke or regenerate them at any time, rendering your existing tokens invalid.
Using R to collect Twitter data
Useful R Package
Reference and things to Read
- VOSON Lab http://vosonlab.net/SocialMediaLab
- Bucher, Taina. “Objects of intense feeling: The case of the Twitter API.” Computational Culture 3 (2013).
- Kim, Annice E., et al. “Methodological considerations in analyzing Twitter data.” Journal of the National Cancer Institute Monographs 2013.47 (2013): 140-146.
- Twitter Bots Tutorials https://digitalinspiration.com/docs/twitter-bots
- Twitter Bots Tutorials (A Full Guide for Beginners) https://learn.g2.com/how-to-make-a-twitter-bot
- Littman, Justin. “Where to get Twitter data for academic research.” (2017). https://gwu-libraries.github.io/sfm-ui/posts/2017-09-14-twitter-data
- Trupthi, M., Suresh Pabboju, and G. Narasimha. “Sentiment analysis on twitter using streaming API.” 2017 IEEE 7th International Advance Computing Conference (IACC). IEEE, 2017.
- Pfeffer, Jürgen, Katja Mayer, and Fred Morstatter. “Tampering with Twitter’s sample API.” EPJ Data Science 7.1 (2018): 50.
- Hino, Airo, and Robert A. Fahey. “Representing the Twittersphere: Archiving a representative sample of Twitter data under resource constraints.” International journal of information management 48 (2019): 175-184.
- Aguilar-Gallegos, Norman, et al. “Dataset on dynamics of Coronavirus on Twitter.” Data in Brief (2020): 105684.