Social Media Mining Toolkit (SMMT)

The set of tools collected and presented here are designed with the purpose of facilitating the acquisition, preprocessing, and initial exploration of social media data (mostly Twitter for now). This centralized repository depends on other widely available libraries that need to be installed. We separated this toolkit in three categories (each one on an individual folder): 1. Data Acquisition Tools: Utilities to gather data from social media sites 2. Data Preprocessing Tools: Utilities to parse social media 'raw' data and to separate by terms 3. Data Annotation and Standardization Tools: Utilities to make automatic NER annotations on preprocessed tweets, plugins to use popular annotation tools and NER systems

Usage

1. Install dependencies (below)
2. Clone repository
3. Make sure you have your Twitter API keys handy if you are gathering any Twitter Data
4. Each tool and their usage is described on the README file on each category of tools folder.

All the libraries used in this toolkit can be installed using the following command.

sh requirements.sh

Note: If you would like to setup headless browsing automation tasks, please install additional dependencies given below.

Dependencies and versions used

1. Python 3+

2. Spacy v2.2

pip install spacy 
python -m spacy download en
python -m spacy download en_core_web_sm

1. Twarc pip install twarc

2. Tweepy v3.8.0 pip install tweepy

3. argparse - v3.2 pip install argparse

4. xtract - v0.1a3 pip install xtract

NOTE: If you are using the scraping utility, install the following dependencies. These dependencies are needed for the headless browsing automation tasks (no need to have a screen open for them). Configuration of these items is very finicky but there is plenty of documentation online.

1. Xvf sudo yum install Xvfb

2. Firefox sudo yum install firefox

3. selenium pip install -U selenium

4. pyvirtualdisplay - v0.25 pip install pyvirtualdisplay

5. GeckoDriver - v0.26.0 sudo yum install jq

and then use the provided utility:

bash SMMT/data_acquisition/geckoDriverInstall.sh

If you still have issues or the Firefox window is popping up through your X11, follow this: https://www.tienle.com/2016/09-20/run-selenium-firefox-browser-centos.html

Twitter Keys

This is a very important step, if you do not have any Twitter API keys, none of the software that uses Twitter API will work without it

How to cite our work:

If you used SMMT and liked it, please cite the following paper:

R Tekumalla and JM Banda. "Social Media Mining Toolkit (SMMT)". Genomics & Informatics, 18, (2), 2020. https://doi.org/10.5808/GI.2020.18.2.e16

Social Media Mining Toolkit (SMMT) Extra Information

Data Acquisition Tools:

1. Twitter hydration tool - This script will hydrate tweet ID’s provided by others.
2. Twitter gathering tool - This script will allow users to specify hashtags and capture from the twitter faucet new tweets with the given hashtag.

Data Preprocessing Tools:

1. Twitter JSON extraction tool - While seemingly trivial, most biomedical researchers do not want to work with JSON objects. This tool will take the fields the researcher wants and output a simple to use CSV file created from the provided data.

Data Annotation and Standardization Tools:

1. Spacy dictionary-based annotation pipeline This is the tool that will require the most work during the hackathon. This pipeline will be available as a service as well, with the user providing their dictionaries and feeding data directly.
2. Dictionary generation tool This tool will transform ontologies or provided dictionary files into spacy compliant dictionaries to use with the previous pipeline.
3. Manual annotation hooks to tools like brat annotation tools

This work was conceptualized for/and (mostly) carried out while at the Biomedical Linked Annotation Hackathon 6 in Tokyo, Japan.

We are very grateful for the support on this work.

Proposed functionality of SMMT V1.0