Skip to content

Latest commit

 

History

History
96 lines (72 loc) · 4.59 KB

README.md

File metadata and controls

96 lines (72 loc) · 4.59 KB

COMP551 Project 2: IMDB Sentiment Classification Competition

This project was completed as part of the W2019 COMP551 course and involved implementing a text sentiment classifier for an in-class Kaggle competition.

This repo contains the software used to produce the first-place submission of the competition and achieves 95.53% and 95.18% accuracy on the public and private leaderboards, respectively.

Due to the rapid nature of Kaggle competitions, the code turned out quite messy. I have decided to leave things as they were, but if you have any questions feel free to let know. Finally, I would like to acknowledge my teammates Justin Whatley and Raed Abdo for their contributions to this project.

Installation Instructions

  • If possible, use a Windows 10 64-bit environment with 32GB+ RAM. Linux is mostly untested, but should work.

  • Install pre-requisites depending on which part of the project you want to run:

    • venv (scikit-learn, PyTorch 1.0.1, etc.) - Python 3.6 
      pip install -r requirements.txt
    • FastAI (only required for ULMFiT) - Python 3.6 
      conda create --name <envname> --file requirements_fastai.yml

  • Downloading and installing the dataset

    • Download the dataset from Kaggle: https://www.kaggle.com/c/comp-551-imbd-sentiment-classification
    • Move the contents of 'train' and 'test' into 'project_src/data/'.
    • Run the 'utils/n1_convert_dataset_to_dataframe.py' utility which should load the dataset and convert it into a pickled dataframe object saved in 'project_src/data/preprocessed_data.

  • Make sure that you are using an IDE (e.g., PyCharm or Visual Studio Code) and that the project root (i.e. the folder that contains project_src) is in your $PATH, otherwise the module imports probably will fail. Some IDE's will do this for you automatically.

Running the deep learning models

Recommended System Pre-requisites

  • Windows 10 64-bit
  • 64 GB RAM
  • NVIDIA GPU w/ 16 GB+ VRAM
  • CUDA 10

We recommend using a cloud-computing service with a Tesla V100 or greater.

ULMFiT

Setup

  • Create and activate the FastAI environment (see above).

Training

  • Navigate to deep_learning/ulmfit.
  • Run train_lm.py to fine-tune a language model.
  • Run train_classifier.py to train a classifier.
  • Run make_predictions.py to create the predicted probabilities and classes.

OpenAI GPT

Setup

  • Make sure 'pytorch_openai_transformer' is in the current shell's PATH or PYTHONPATH so that the modules will import properly.
  • Install the spacy en module (python -m spacy download en)

Training

  • Navigate to deep_learning/pytorch_openai_transformer.
  • Either run train_folds.bat or python train_imdb.py --fold_id 0

Evaluation

  • Run python train_imdb.py --fold_id 0 --make_predictions. The output should consist of the predicted probabilties and classes.

BERT

Training

Fine-tuning a LM (Optional)
  • Install "en" for Spacy (https://spacy.io/usage/models)
  • Create the training corpus using "make_finetune_corpus.py"
  • Run "run_lm_finetune.py", e.g.
python3 run_lm_finetune.py --train_file preprocessed_data/imdb_train_corpus_0.txt --bert_model bert-base-uncased --output_dir finetune_0_uncased_512_1e --max_seq_length 512 --on_memory --do_lower_case --do_train --train_batch_size 12 --num_train_epochs 1
  • Use the resulting trained LM for the next step.
Training a classifier
  • Create the pre-processed data using preprocess_data.py, ensuring that you select the correct max sequence length.
  • Modify run_experiment.py so that BERT_PRETRAINED_PATH points to either bert-base-uncased or wherever you saved the fine-tuned LM (if applicable). Furthermore, if you pretrained a LM, you will need to obtain the vocab and config files to pair with it (i.e., placed in the same directory). They can be found at https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz and https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt.
  • Run the script run_experiment.py 0, where the first argument specifies the fold ID. If
  • Create the submission file using make_predictions.py. The output should consist of the predicted probabilties and classes.

Stacked Ensemble

  • Collect the 15 validation prediction, and 15 test prediction files from the 3 models and place them into deep_learning/stacked_ensemble/inputs and deep_learning/stacked_ensemble/inputs_test.
  • Run deep_learning/stacked_ensemble/train_stacker.py.