Skip to content

Latest commit

 

History

History
117 lines (85 loc) · 5.61 KB

README.md

File metadata and controls

117 lines (85 loc) · 5.61 KB
Raw

Sparkify-Churn-Prediction-PySpark

Machine Learning on a Large 12 GB dataset with Pyspark. Sparkify is an imaginary app just like Spotify. The data consists of event logs. The dataset is provided Udacity and is publically available in the S3 buckets given below:

  • mini-sparkify-event-data (150 MB) - s3n://udacity-dsnd/sparkify/mini_sparkify_event_data.json
  • sparkify-event-data (12 GB) - s3n://udacity-dsnd/sparkify/sparkify_event_data.json

About

Project Overview

Predicting churn rates is a very challenging and common problem that data scientists and analysts face in any customer-facing business. They have to ascertain it before so that steps can be taken to prevent users from churning. The goal of this project was to help Sparkify music service retain their customers.

The main task of the project is to make a prediction, whether the customer is about to churn. Such a prediction can be made for each customer by a binary classifier model. The following tasks should be completed to create the model:

  • Analyze and preprocess the data to extract usable features for each customer
  • Train classifier and then evaluate it

Input Data

Data is the log of all activity on Sparkify (imaginary app like Spotify). Two datasets are used:

  • sparkify_event_data.json – large dataset with 26 million records

  • mini_sparkify_event_data.json – a tiny subset of the full dataset, which is useful for preliminary data analysis.

Both files contain the following data:

# Column Type Description
1 userId string Unique identifier of the user
2 artist string Name of the artist related to the song related to the event
3 auth string authentication of the customer : whether he is logged in or not
4 firstName string First name of the user
5 gender string Gender of the user
6 itemInSession bigint Item in session
7 lastName string Last name of the user
8 length double Length of the song related to the event
9 level string Whether the account is free or paid. User can change the level, so events for the same user can have different levels
10 location string Location of the user
11 method string Get method or Put method
12 page string Type of action which corresponds to the event : this can be clicking "Next Song", "Upvoting" etc
13 registration bigint Registration
14 sessionId bigint/long Session id
15 song string Name of the song related to the event
16 status bigint/long Response status from the page: 200, 404, 307
17 ts bigint/long Timestamp of the event
18 userAgent string The browser used and other relevant information eg: Mozilla/5.0

Files and Folder Description

  • churn-prediction.ipynb : Exploratory Data Analysis, Feature Engineering and Classification using the mini-event-sparkify-data on my local system.
  • reference.ipynb : Feature Engineering and model training on AWS EMR with mini-event-sparkify-data with 1 m5.xlarge node cluster. This is used as a reference for training with large data.
  • training-aws-emr.ipynb : Actual training on the large event-sparkify-data with 5 m4.xlarge node cluster
  • run.py : script to run the flask application
  • errors.py : script to handle errors in user input
  • df_ml.xlsx : dataset engineered from event-sparkify-data which has 1 row of information per user. This is the data that is used for actual model training.
  • requirements.txt : The dependencies for this application
  • models : Saved models after training on large dataset on AWS EMR
  • templates : html templates
  • static : css styles

I recommend going through the files in the above order

Implementation

The original data has 26 million rows that cannot be processes in a single machine. That is why I have used Spark clusters to analyze data and predict customer churn. I use PySpark and SparkML libraries to implement the solution. The implementation of the project consists of two parts:

  • EDA, Feature Engineering and Machine Learning on the dataset
  • Developing a flask based application for actual use case by people

Demo

Repository Structure

- app
| - templates
| |- form_css.html # home-page template
| - static 
| |- main.css # home-page style
|- run.py  # Flask application
|- errors.py # Handle Exception

- demo
|- gif.gif # animation with prediction of churn

- model
|- model-gbt # PySpark Gradient Boosted Trees model
|- model-rf # PySpark Random Forest model
|- model-gbt-one-hot-encoder-estimator # PySpark Gradient Boosted Trees model with OneHotEncoderEstimator instead of OneHotEncoder (This has been used in the application)
|- model-rf-one-hot-encoder-estimator  # PySpark Random Forest model with OneHotEncoderEstimator instead of OneHotEncoder

- README.md
- churn-prediction.ipynb # Python 3 notebook, which contains analysis of small dataset
- reference.ipynb # PySpark kernel notebook on AWS EMR, which contains analysis of small dataset
- training-aws-emr.ipynb # Pyspark kernel notebook on AWS EMR, which contains analysis of large dataset
- df_ml.xlsx # Extracted Dataset for triainig model
- requiremects.txt # Dependencies of this application

Conclusion & Improvement

I tried training with two models Gradient Boosted Trees and Random Forest, and have got an F1 score of 80% which is good for actual use case.

Some improvements that can be made are :

  • Using cross-validation and param-grid-builder to find out the best hyper-parameters
  • Trying other models such as Neural Networks, Support Vector Machine etc
  • Extracting out new features that can predict churn better