RoboReviews_Destill_BERT.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Robo Reviews Project\n",
    "## The new product review aggregator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download Libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/stopwords.zip.\n",
      "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
      "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
      "[nltk_data] Downloading package omw-1.4 to /root/nltk_data...\n"
     ]
    }
   ],
   "source": [
    "# Libraries\n",
    "import pandas as pd  # pandas and numpy for data manipulation\n",
    "import numpy as np\n",
    "import sklearn as skl # scikit-learn for machine learning models (e.g., clustering, classification).\n",
    "import transformers # transformers and torch for sentiment analysis and text generation.\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import re\n",
    "from sklearn.model_selection import train_test_split\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "import nltk  # nltk for text preprocessing (tokenization, stop-word removal, etc.)\n",
    "# Download necessary nltk datasets\n",
    "from nltk.corpus import stopwords\n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt')\n",
    "nltk.download('wordnet')\n",
    "nltk.download('omw-1.4')  # For WordNet lemmatizer\n",
    "from transformers import AutoTokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary of the Workflow:\n",
    "1. Preprocessing: Clean, tokenize, and normalize the text data.\n",
    "2. Sentiment Classification (Positive, Neutral, Negative): Create model to classify customer reviews based on their sentiment.\n",
    "3. Meta-Category Classification: Group reviews into 4-6 meta-categories.\n",
    "4. Blog-style Article Generation: Generate articles based on the product categories, including the top 3 products, complaints, and worst product."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 Preprocessing: Load and Explore the Dataset\n",
    "\n",
    "We will focus on these relevant columns:\n",
    "\n",
    "reviews.text: The review content (text) for sentiment analysis.\n",
    "reviews.rating: The numerical rating that can guide sentiment classification.\n",
    "categories: Product categories, which we can use to group reviews into meta-categories.\n",
    "reviews.date, reviews.numHelpful, and reviews.username: Can be used later for insights or blog generation.\n",
    "\n",
    "Preprocessing Steps\n",
    "Clean the review text: Remove any unwanted characters and normalize the text.\n",
    "Handle missing values: Remove rows where reviews.text or reviews.rating is missing.\n",
    "Sentiment labeling: Use reviews.rating to label reviews as positive, neutral, or negative.\n",
    "Tokenization: Convert the cleaned text into numerical format using TF-IDF or embeddings.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_43/286049223.py:3: DtypeWarning: Columns (1,2,3,7,8,11,14,23) have mixed types. Specify dtype option on import or set low_memory=False.\n",
      "  df = pd.read_csv(file_path)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>dateAdded</th>\n",
       "      <th>dateUpdated</th>\n",
       "      <th>name</th>\n",
       "      <th>asins</th>\n",
       "      <th>brand</th>\n",
       "      <th>categories</th>\n",
       "      <th>primaryCategories</th>\n",
       "      <th>imageURLs</th>\n",
       "      <th>keys</th>\n",
       "      <th>...</th>\n",
       "      <th>reviews.numHelpful</th>\n",
       "      <th>reviews.rating</th>\n",
       "      <th>reviews.sourceURLs</th>\n",
       "      <th>reviews.text</th>\n",
       "      <th>reviews.title</th>\n",
       "      <th>reviews.username</th>\n",
       "      <th>sourceURLs</th>\n",
       "      <th>reviews.dateAdded</th>\n",
       "      <th>reviews.userCity</th>\n",
       "      <th>reviews.userProvince</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AVpgNzjwLJeJML43Kpxn</td>\n",
       "      <td>2015-10-30T08:59:32Z</td>\n",
       "      <td>2019-04-25T09:08:16Z</td>\n",
       "      <td>AmazonBasics AAA Performance Alkaline Batterie...</td>\n",
       "      <td>B00QWO9P0O,B00LH3DMUO</td>\n",
       "      <td>Amazonbasics</td>\n",
       "      <td>AA,AAA,Health,Electronics,Health &amp; Household,C...</td>\n",
       "      <td>Health &amp; Beauty</td>\n",
       "      <td>https://images-na.ssl-images-amazon.com/images...</td>\n",
       "      <td>amazonbasics/hl002619,amazonbasicsaaaperforman...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>https://www.amazon.com/product-reviews/B00QWO9...</td>\n",
       "      <td>I order 3 of them and one of the item is bad q...</td>\n",
       "      <td>... 3 of them and one of the item is bad quali...</td>\n",
       "      <td>Byger yang</td>\n",
       "      <td>https://www.barcodable.com/upc/841710106442,ht...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AVpgNzjwLJeJML43Kpxn</td>\n",
       "      <td>2015-10-30T08:59:32Z</td>\n",
       "      <td>2019-04-25T09:08:16Z</td>\n",
       "      <td>AmazonBasics AAA Performance Alkaline Batterie...</td>\n",
       "      <td>B00QWO9P0O,B00LH3DMUO</td>\n",
       "      <td>Amazonbasics</td>\n",
       "      <td>AA,AAA,Health,Electronics,Health &amp; Household,C...</td>\n",
       "      <td>Health &amp; Beauty</td>\n",
       "      <td>https://images-na.ssl-images-amazon.com/images...</td>\n",
       "      <td>amazonbasics/hl002619,amazonbasicsaaaperforman...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>https://www.amazon.com/product-reviews/B00QWO9...</td>\n",
       "      <td>Bulk is always the less expensive way to go fo...</td>\n",
       "      <td>... always the less expensive way to go for pr...</td>\n",
       "      <td>ByMG</td>\n",
       "      <td>https://www.barcodable.com/upc/841710106442,ht...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AVpgNzjwLJeJML43Kpxn</td>\n",
       "      <td>2015-10-30T08:59:32Z</td>\n",
       "      <td>2019-04-25T09:08:16Z</td>\n",
       "      <td>AmazonBasics AAA Performance Alkaline Batterie...</td>\n",
       "      <td>B00QWO9P0O,B00LH3DMUO</td>\n",
       "      <td>Amazonbasics</td>\n",
       "      <td>AA,AAA,Health,Electronics,Health &amp; Household,C...</td>\n",
       "      <td>Health &amp; Beauty</td>\n",
       "      <td>https://images-na.ssl-images-amazon.com/images...</td>\n",
       "      <td>amazonbasics/hl002619,amazonbasicsaaaperforman...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://www.amazon.com/product-reviews/B00QWO9...</td>\n",
       "      <td>Well they are not Duracell but for the price i...</td>\n",
       "      <td>... are not Duracell but for the price i am ha...</td>\n",
       "      <td>BySharon Lambert</td>\n",
       "      <td>https://www.barcodable.com/upc/841710106442,ht...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AVpgNzjwLJeJML43Kpxn</td>\n",
       "      <td>2015-10-30T08:59:32Z</td>\n",
       "      <td>2019-04-25T09:08:16Z</td>\n",
       "      <td>AmazonBasics AAA Performance Alkaline Batterie...</td>\n",
       "      <td>B00QWO9P0O,B00LH3DMUO</td>\n",
       "      <td>Amazonbasics</td>\n",
       "      <td>AA,AAA,Health,Electronics,Health &amp; Household,C...</td>\n",
       "      <td>Health &amp; Beauty</td>\n",
       "      <td>https://images-na.ssl-images-amazon.com/images...</td>\n",
       "      <td>amazonbasics/hl002619,amazonbasicsaaaperforman...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://www.amazon.com/product-reviews/B00QWO9...</td>\n",
       "      <td>Seem to work as well as name brand batteries a...</td>\n",
       "      <td>... as well as name brand batteries at a much ...</td>\n",
       "      <td>Bymark sexson</td>\n",
       "      <td>https://www.barcodable.com/upc/841710106442,ht...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AVpgNzjwLJeJML43Kpxn</td>\n",
       "      <td>2015-10-30T08:59:32Z</td>\n",
       "      <td>2019-04-25T09:08:16Z</td>\n",
       "      <td>AmazonBasics AAA Performance Alkaline Batterie...</td>\n",
       "      <td>B00QWO9P0O,B00LH3DMUO</td>\n",
       "      <td>Amazonbasics</td>\n",
       "      <td>AA,AAA,Health,Electronics,Health &amp; Household,C...</td>\n",
       "      <td>Health &amp; Beauty</td>\n",
       "      <td>https://images-na.ssl-images-amazon.com/images...</td>\n",
       "      <td>amazonbasics/hl002619,amazonbasicsaaaperforman...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://www.amazon.com/product-reviews/B00QWO9...</td>\n",
       "      <td>These batteries are very long lasting the pric...</td>\n",
       "      <td>... batteries are very long lasting the price ...</td>\n",
       "      <td>Bylinda</td>\n",
       "      <td>https://www.barcodable.com/upc/841710106442,ht...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 27 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     id             dateAdded           dateUpdated  \\\n",
       "0  AVpgNzjwLJeJML43Kpxn  2015-10-30T08:59:32Z  2019-04-25T09:08:16Z   \n",
       "1  AVpgNzjwLJeJML43Kpxn  2015-10-30T08:59:32Z  2019-04-25T09:08:16Z   \n",
       "2  AVpgNzjwLJeJML43Kpxn  2015-10-30T08:59:32Z  2019-04-25T09:08:16Z   \n",
       "3  AVpgNzjwLJeJML43Kpxn  2015-10-30T08:59:32Z  2019-04-25T09:08:16Z   \n",
       "4  AVpgNzjwLJeJML43Kpxn  2015-10-30T08:59:32Z  2019-04-25T09:08:16Z   \n",
       "\n",
       "                                                name                  asins  \\\n",
       "0  AmazonBasics AAA Performance Alkaline Batterie...  B00QWO9P0O,B00LH3DMUO   \n",
       "1  AmazonBasics AAA Performance Alkaline Batterie...  B00QWO9P0O,B00LH3DMUO   \n",
       "2  AmazonBasics AAA Performance Alkaline Batterie...  B00QWO9P0O,B00LH3DMUO   \n",
       "3  AmazonBasics AAA Performance Alkaline Batterie...  B00QWO9P0O,B00LH3DMUO   \n",
       "4  AmazonBasics AAA Performance Alkaline Batterie...  B00QWO9P0O,B00LH3DMUO   \n",
       "\n",
       "          brand                                         categories  \\\n",
       "0  Amazonbasics  AA,AAA,Health,Electronics,Health & Household,C...   \n",
       "1  Amazonbasics  AA,AAA,Health,Electronics,Health & Household,C...   \n",
       "2  Amazonbasics  AA,AAA,Health,Electronics,Health & Household,C...   \n",
       "3  Amazonbasics  AA,AAA,Health,Electronics,Health & Household,C...   \n",
       "4  Amazonbasics  AA,AAA,Health,Electronics,Health & Household,C...   \n",
       "\n",
       "  primaryCategories                                          imageURLs  \\\n",
       "0   Health & Beauty  https://images-na.ssl-images-amazon.com/images...   \n",
       "1   Health & Beauty  https://images-na.ssl-images-amazon.com/images...   \n",
       "2   Health & Beauty  https://images-na.ssl-images-amazon.com/images...   \n",
       "3   Health & Beauty  https://images-na.ssl-images-amazon.com/images...   \n",
       "4   Health & Beauty  https://images-na.ssl-images-amazon.com/images...   \n",
       "\n",
       "                                                keys  ... reviews.numHelpful  \\\n",
       "0  amazonbasics/hl002619,amazonbasicsaaaperforman...  ...                NaN   \n",
       "1  amazonbasics/hl002619,amazonbasicsaaaperforman...  ...                NaN   \n",
       "2  amazonbasics/hl002619,amazonbasicsaaaperforman...  ...                NaN   \n",
       "3  amazonbasics/hl002619,amazonbasicsaaaperforman...  ...                NaN   \n",
       "4  amazonbasics/hl002619,amazonbasicsaaaperforman...  ...                NaN   \n",
       "\n",
       "  reviews.rating                                 reviews.sourceURLs  \\\n",
       "0            3.0  https://www.amazon.com/product-reviews/B00QWO9...   \n",
       "1            4.0  https://www.amazon.com/product-reviews/B00QWO9...   \n",
       "2            5.0  https://www.amazon.com/product-reviews/B00QWO9...   \n",
       "3            5.0  https://www.amazon.com/product-reviews/B00QWO9...   \n",
       "4            5.0  https://www.amazon.com/product-reviews/B00QWO9...   \n",
       "\n",
       "                                        reviews.text  \\\n",
       "0  I order 3 of them and one of the item is bad q...   \n",
       "1  Bulk is always the less expensive way to go fo...   \n",
       "2  Well they are not Duracell but for the price i...   \n",
       "3  Seem to work as well as name brand batteries a...   \n",
       "4  These batteries are very long lasting the pric...   \n",
       "\n",
       "                                       reviews.title  reviews.username  \\\n",
       "0  ... 3 of them and one of the item is bad quali...        Byger yang   \n",
       "1  ... always the less expensive way to go for pr...              ByMG   \n",
       "2  ... are not Duracell but for the price i am ha...  BySharon Lambert   \n",
       "3  ... as well as name brand batteries at a much ...     Bymark sexson   \n",
       "4  ... batteries are very long lasting the price ...           Bylinda   \n",
       "\n",
       "                                          sourceURLs  reviews.dateAdded  \\\n",
       "0  https://www.barcodable.com/upc/841710106442,ht...                NaN   \n",
       "1  https://www.barcodable.com/upc/841710106442,ht...                NaN   \n",
       "2  https://www.barcodable.com/upc/841710106442,ht...                NaN   \n",
       "3  https://www.barcodable.com/upc/841710106442,ht...                NaN   \n",
       "4  https://www.barcodable.com/upc/841710106442,ht...                NaN   \n",
       "\n",
       "   reviews.userCity reviews.userProvince  \n",
       "0               NaN                  NaN  \n",
       "1               NaN                  NaN  \n",
       "2               NaN                  NaN  \n",
       "3               NaN                  NaN  \n",
       "4               NaN                  NaN  \n",
       "\n",
       "[5 rows x 27 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the dataset\n",
    "file_path = 'combined_amazon_reviews.csv'\n",
    "df = pd.read_csv(file_path)\n",
    "\n",
    "# Explore the dataset\n",
    "df.head()  # See the first few rows\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['id', 'dateAdded', 'dateUpdated', 'name', 'asins', 'brand',\n",
      "       'categories', 'primaryCategories', 'imageURLs', 'keys', 'manufacturer',\n",
      "       'manufacturerNumber', 'reviews.date', 'reviews.dateSeen',\n",
      "       'reviews.didPurchase', 'reviews.doRecommend', 'reviews.id',\n",
      "       'reviews.numHelpful', 'reviews.rating', 'reviews.sourceURLs',\n",
      "       'reviews.text', 'reviews.title', 'reviews.username', 'sourceURLs',\n",
      "       'reviews.dateAdded', 'reviews.userCity', 'reviews.userProvince'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "print(df.columns)  # Check the available columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                        reviews.text  \\\n",
      "0  I order 3 of them and one of the item is bad q...   \n",
      "1  Bulk is always the less expensive way to go fo...   \n",
      "2  Well they are not Duracell but for the price i...   \n",
      "3  Seem to work as well as name brand batteries a...   \n",
      "4  These batteries are very long lasting the pric...   \n",
      "\n",
      "                                        cleaned_text  \n",
      "0  order 3 one item bad quality missing backup sp...  \n",
      "1    bulk always less expensive way go products like  \n",
      "2                          well duracell price happy  \n",
      "3  seem work well name brand batteries much bette...  \n",
      "4                 batteries long lasting price great  \n"
     ]
    }
   ],
   "source": [
    "# Drop rows with missing reviews or ratings\n",
    "df = df.dropna(subset=['reviews.text', 'reviews.rating'])\n",
    "\n",
    "# Initialize stopwords \n",
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "# Function to clean and preprocess text\n",
    "def clean_text(text):\n",
    "    # Remove non-alphanumeric characters\n",
    "    text = re.sub(r'[^\\w\\s]', '', text)\n",
    "    # Convert to lowercase\n",
    "    text = text.lower()\n",
    "    # Remove extra whitespace\n",
    "    text = re.sub(r'\\s+', ' ', text).strip()  # This ensures extra spaces are removed\n",
    "    # Tokenize and remove stopwords\n",
    "    text = word_tokenize(text)\n",
    "    text = [word for word in text if word not in stop_words]  # Remove stopwords\n",
    "    return ' '.join(text)\n",
    "\n",
    "# Apply cleaning function to reviews\n",
    "df['cleaned_text'] = df['reviews.text'].apply(clean_text)\n",
    "\n",
    "# Check cleaned data\n",
    "print(df[['reviews.text', 'cleaned_text']].head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Sentiment Classification (Positive, Neutral, Negative)\n",
    "We'll classify sentiment using the rating (1-5 scale):\n",
    "\n",
    "Positive: Ratings 4-5,\n",
    "Neutral: Rating 3,\n",
    "Negative: Ratings 1-2\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sentiment\n",
      "positive    62546\n",
      "neutral      2902\n",
      "negative     2510\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Label sentiment based on the rating\n",
    "def label_sentiment(rating):\n",
    "    if rating >= 4:\n",
    "        return 'positive'\n",
    "    elif rating == 3:\n",
    "        return 'neutral'\n",
    "    else:\n",
    "        return 'negative'\n",
    "\n",
    "# Apply sentiment labeling\n",
    "df['sentiment'] = df['reviews.rating'].apply(label_sentiment)\n",
    "\n",
    "# Check distribution of sentiment\n",
    "print(df['sentiment'].value_counts())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d88b0d3fb054ce5ab047b2e997e3b48",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/48.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ddd9d11f5b2e43bd83d1f39e214619f5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/483 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "43490c1400b742fbb3a17248db702d8e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6c4773d0b6024c62853159bac6b63499",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/466k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                        cleaned_text  \\\n",
      "0  order 3 one item bad quality missing backup sp...   \n",
      "1    bulk always less expensive way go products like   \n",
      "2                          well duracell price happy   \n",
      "3  seem work well name brand batteries much bette...   \n",
      "4                 batteries long lasting price great   \n",
      "\n",
      "                                           tokenized  \n",
      "0  [101, 2344, 1017, 2028, 8875, 2919, 3737, 4394...  \n",
      "1  [101, 9625, 2467, 2625, 6450, 2126, 2175, 3688...  \n",
      "2  [101, 2092, 4241, 22903, 3363, 3976, 3407, 102...  \n",
      "3  [101, 4025, 2147, 2092, 2171, 4435, 10274, 217...  \n",
      "4  [101, 10274, 2146, 9879, 3976, 2307, 102, 0, 0...  \n"
     ]
    }
   ],
   "source": [
    "# Tokenization\n",
    "from transformers import AutoTokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\")\n",
    "\n",
    "# Tokenize the dataset\n",
    "def tokenize_function(text):\n",
    "    return tokenizer(text, padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n",
    "\n",
    "# Apply tokenization to the 'cleaned_text' column\n",
    "df['tokenized'] = df['cleaned_text'].apply(lambda x: tokenize_function(x)['input_ids'].squeeze().tolist())\n",
    "\n",
    "# Convert 'tokenized' column to a list if it's a DatasetDict or tensor object\n",
    "df['tokenized'] = df['tokenized'].apply(lambda x: x.tolist() if isinstance(x, torch.Tensor) else x)\n",
    "\n",
    "# Check tokenized data\n",
    "print(df[['cleaned_text', 'tokenized']].head())\n",
    "\n",
    "\n",
    "# Split data into training and testing sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(df['tokenized'], df['sentiment'], test_size=0.2, random_state=42)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Integrating BERT for sentiment analysis "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DistilBertForSequenceClassification(\n",
       "  (distilbert): DistilBertModel(\n",
       "    (embeddings): Embeddings(\n",
       "      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
       "      (position_embeddings): Embedding(512, 768)\n",
       "      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "      (dropout): Dropout(p=0.1, inplace=False)\n",
       "    )\n",
       "    (transformer): Transformer(\n",
       "      (layer): ModuleList(\n",
       "        (0-5): 6 x TransformerBlock(\n",
       "          (attention): MultiHeadSelfAttention(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (q_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (k_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (v_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "            (out_lin): Linear(in_features=768, out_features=768, bias=True)\n",
       "          )\n",
       "          (sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "          (ffn): FFN(\n",
       "            (dropout): Dropout(p=0.1, inplace=False)\n",
       "            (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
       "            (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
       "            (activation): GELUActivation()\n",
       "          )\n",
       "          (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (pre_classifier): Linear(in_features=768, out_features=768, bias=True)\n",
       "  (classifier): Linear(in_features=768, out_features=3, bias=True)\n",
       "  (dropout): Dropout(p=0.2, inplace=False)\n",
       ")"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import AutoModelForSequenceClassification, AutoTokenizer\n",
    "import torch\n",
    "\n",
    "# Define the folder path where the model is saved\n",
    "model_folder = \"./Robo-Reviews-Project/r\"\n",
    "\n",
    "# Load the tokenizer from the saved model folder\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_folder)\n",
    "\n",
    "# Load the model from the saved model folder\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_folder)\n",
    "\n",
    "# Ensure the model is set to evaluation mode (important for inference)\n",
    "model.eval()\n",
    "\n",
    "# Define the device (GPU if available, otherwise CPU)\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "model.to(device)  # Move the model to the appropriate device\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted sentiment: positive\n"
     ]
    }
   ],
   "source": [
    "# Example function to predict sentiment using the loaded model\n",
    "def predict_sentiment(text):\n",
    "    # Tokenize the input text and move to the appropriate device\n",
    "    inputs = tokenizer(text, return_tensors=\"pt\", padding=\"max_length\", truncation=True).to(device)\n",
    "\n",
    "    # Perform inference (no gradient calculation needed for evaluation)\n",
    "    with torch.no_grad():\n",
    "        outputs = model(**inputs)\n",
    "    \n",
    "    # Get the predicted label (index of the max logit)\n",
    "    predictions = torch.argmax(outputs.logits, dim=-1).item()\n",
    "\n",
    "    # Map the label index to sentiment\n",
    "    label_map = {0: 'positive', 1: 'neutral', 2: 'negative'}\n",
    "    return label_map[predictions]\n",
    "\n",
    "# Example usage\n",
    "sample_text = \"This is a great product!\"\n",
    "predicted_sentiment = predict_sentiment(sample_text)\n",
    "print(f\"Predicted sentiment: {predicted_sentiment}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-10-17 18:23:03.243859: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2024-10-17 18:23:03.243931: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2024-10-17 18:23:03.245125: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2024-10-17 18:23:03.252188: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2024-10-17 18:23:04.191499: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1ea65297c18041f6ab8fa3d49d92008e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/67958 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from transformers import AutoModelForSequenceClassification, AutoTokenizer, Trainer, TrainingArguments\n",
    "from datasets import Dataset\n",
    "import torch\n",
    "\n",
    "# Load the tokenizer and model from your saved folder (adjust the path if necessary)\n",
    "model_folder = \"./Robo-Reviews-Project/r\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_folder)\n",
    "model = AutoModelForSequenceClassification.from_pretrained(model_folder)\n",
    "\n",
    "# Tokenize the text data\n",
    "def tokenize_function(examples):\n",
    "    return tokenizer(examples[\"cleaned_text\"], padding=\"max_length\", truncation=True)\n",
    "\n",
    "# Convert the pandas dataframe to a Huggingface Dataset\n",
    "df['label'] = df['sentiment'].map({'positive': 0, 'neutral': 1, 'negative': 2})  \n",
    "dataset = Dataset.from_pandas(df[['cleaned_text', 'label']])\n",
    "\n",
    "# Tokenize the dataset\n",
    "tokenized_dataset = dataset.map(tokenize_function, batched=True)\n",
    "\n",
    "# Split the dataset into training and test datasets\n",
    "train_test_split = tokenized_dataset.train_test_split(test_size=0.2)\n",
    "train_dataset = train_test_split['train']\n",
    "test_dataset = train_test_split['test']\n",
    "\n",
    "# Set the format for PyTorch\n",
    "train_dataset.set_format(type='torch', columns=['input_ids', 'attention_mask', 'label'])\n",
    "test_dataset.set_format(type='torch', columns=['input_ids', 'attention_mask', 'label'])\n",
    "\n",
    "# You can now use Trainer, TrainingArguments, and other functionalities as needed\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will train BERT to predict sentiments based on the review text. Since BERT is pre-trained, we will fine-tune it on the review dataset for sentiment classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving the model in folder\n",
    "output_dir = \"./Robo-Reviews-Project/r\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "output_dir = './results'\n",
    "if not os.path.exists(output_dir):\n",
    "    os.makedirs(output_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You're using a DistilBertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='3398' max='3398' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [3398/3398 47:27, Epoch 2/2]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.085200</td>\n",
       "      <td>0.062728</td>\n",
       "      <td>0.986683</td>\n",
       "      <td>0.986397</td>\n",
       "      <td>0.986338</td>\n",
       "      <td>0.986683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.056500</td>\n",
       "      <td>0.067497</td>\n",
       "      <td>0.986536</td>\n",
       "      <td>0.986329</td>\n",
       "      <td>0.986238</td>\n",
       "      <td>0.986536</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='1700' max='850' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [850/850 04:05]\n",
       "    </div>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.06749678403139114, 'eval_accuracy': 0.9865361977633902, 'eval_f1': 0.986328855985762, 'eval_precision': 0.986237538727686, 'eval_recall': 0.9865361977633902, 'eval_runtime': 122.9193, 'eval_samples_per_second': 110.577, 'eval_steps_per_second': 6.915, 'epoch': 2.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "('./results/tokenizer_config.json',\n",
       " './results/special_tokens_map.json',\n",
       " './results/vocab.txt',\n",
       " './results/added_tokens.json',\n",
       " './results/tokenizer.json')"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score, precision_recall_fscore_support\n",
    "from transformers import Trainer\n",
    "import torch\n",
    "import random\n",
    "import numpy as np\n",
    "\n",
    "# Set seed for reproducibility\n",
    "seed = 42\n",
    "torch.manual_seed(seed)\n",
    "torch.cuda.manual_seed_all(seed)  # If using a GPU\n",
    "np.random.seed(seed)\n",
    "random.seed(seed)\n",
    "\n",
    "# Step 1: Define the metrics calculation function\n",
    "# This function calculates accuracy, precision, recall, and F1-score.\n",
    "def compute_metrics(p):\n",
    "    predictions, labels = p\n",
    "    preds = predictions.argmax(-1)  # Get the predicted labels\n",
    "    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='weighted')\n",
    "    acc = accuracy_score(labels, preds)\n",
    "    return {\n",
    "        'accuracy': acc,\n",
    "        'f1': f1,\n",
    "        'precision': precision,\n",
    "        'recall': recall\n",
    "    }\n",
    "\n",
    "# Define training arguments\n",
    "training_args = TrainingArguments(\n",
    "    output_dir='./results',            # output directory\n",
    "    evaluation_strategy=\"epoch\",       # evaluate at the end of every epoch\n",
    "    learning_rate=1e-5,                # learning rate\n",
    "    per_device_train_batch_size=16,    # batch size for training\n",
    "    per_device_eval_batch_size=16,     # batch size for evaluation\n",
    "    num_train_epochs=2,                # Increase this to fine-tune longer\n",
    "    weight_decay=0.02,                 # stronger regularization to prevent overfitting\n",
    "    report_to=[\"none\"],                # avoid reporting to any platform\n",
    "    gradient_accumulation_steps=2     # simulate larger batch size if memory is limited\n",
    ")\n",
    "\n",
    "# Step 2: Define Trainer with the compute_metrics function\n",
    "trainer = Trainer(\n",
    "    model=model,                         \n",
    "    args=training_args,                  \n",
    "    train_dataset=train_dataset,         \n",
    "    eval_dataset=test_dataset,           \n",
    "    tokenizer=tokenizer,   \n",
    "    compute_metrics=compute_metrics  # Added the metrics computation function\n",
    ")\n",
    "\n",
    "# Train the model\n",
    "trainer.train()\n",
    "\n",
    "\n",
    "# Evaluate the model and print metrics\n",
    "metrics = trainer.evaluate()\n",
    "print(metrics)\n",
    "# After training, save the model and tokenizer\n",
    "trainer.save_model(output_dir)  # Saves the model, tokenizer, and config to 'output_dir'\n",
    "tokenizer.save_pretrained(output_dir)  # Save the tokenizer to the same directory\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the BERT Model:\n",
    "Evaluating the model on the test dataset to ensure the model has learned the sentiment classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Results: Loss = 0.0675, Accuracy = 0.9865, F1-Score = 0.9863, Precision = 0.9862, Recall = 0.9865\n"
     ]
    }
   ],
   "source": [
    "# Evaluate the model\n",
    "metrics = trainer.evaluate()\n",
    "print(f\"Evaluation Results: Loss = {metrics['eval_loss']:.4f}, \"\n",
    "      f\"Accuracy = {metrics['eval_accuracy']:.4f}, \"\n",
    "      f\"F1-Score = {metrics['eval_f1']:.4f}, \"\n",
    "      f\"Precision = {metrics['eval_precision']:.4f}, \"\n",
    "      f\"Recall = {metrics['eval_recall']:.4f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use BERT for Sentiment Prediction on Entire Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bert_sentiment\n",
      "positive    62840\n",
      "neutral      2675\n",
      "negative     2443\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "# Step 4: Fix the `predict_sentiment` function to ensure inputs and model are on the same device (CUDA or CPU) \n",
    "def predict_sentiment(text):\n",
    "    # Ensure everything is moved to the same device\n",
    "    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "    model.to(device)  # Move model to the appropriate device\n",
    "\n",
    "    # Tokenize and prepare inputs, move inputs to the same device\n",
    "    inputs = tokenizer(text, return_tensors=\"pt\", padding=\"max_length\", truncation=True).to(device)\n",
    "\n",
    "    # Perform prediction\n",
    "    outputs = model(**inputs)\n",
    "    predictions = torch.argmax(outputs.logits, dim=-1)\n",
    "\n",
    "    # Map predictions to sentiment labels\n",
    "    label_map = {0: 'positive', 1: 'neutral', 2: 'negative'}\n",
    "    return label_map[predictions.item()]\n",
    "\n",
    "#  Apply BERT sentiment prediction to the cleaned text (Updated Code)\n",
    "df['bert_sentiment'] = df['cleaned_text'].apply(predict_sentiment)\n",
    "\n",
    "# Check the distribution of predicted sentiments\n",
    "print(df['bert_sentiment'].value_counts())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review 1:\n",
      "Text: I order 3 of them and one of the item is bad quality. Is missing backup spring so I have to put a pcs of aluminum to make the battery work.\n",
      "Star Rating: 3.0\n",
      "Predicted Sentiment: neutral\n",
      "--------------------------------------------------\n",
      "Review 2:\n",
      "Text: Bulk is always the less expensive way to go for products like these\n",
      "Star Rating: 4.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 3:\n",
      "Text: Well they are not Duracell but for the price i am happy.\n",
      "Star Rating: 5.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 4:\n",
      "Text: Seem to work as well as name brand batteries at a much better price\n",
      "Star Rating: 5.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 5:\n",
      "Text: These batteries are very long lasting the price is great.\n",
      "Star Rating: 5.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 6:\n",
      "Text: Bought a lot of batteries for Christmas and the AmazonBasics Cell have been good. I haven't noticed a difference between the brand name batteries and the Amazon Basic brand. Just a lot easier to purchase and have arrive at the house and have on hand. Will buy again.\n",
      "Star Rating: 5.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 7:\n",
      "Text: ive not had any problame with these batteries have ordered them in the past been very pleased.\n",
      "Star Rating: 5.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 8:\n",
      "Text: Well if you are looking for cheap non-rechargeable batteries that last quite a while then these are perfect. Nothing more to say.\n",
      "Star Rating: 5.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 9:\n",
      "Text: These do not hold the amount of high power juice like energizer or duracell, but they are half the price.\n",
      "Star Rating: 3.0\n",
      "Predicted Sentiment: neutral\n",
      "--------------------------------------------------\n",
      "Review 10:\n",
      "Text: AmazonBasics AA AAA batteries have done well by me appear to have a good shelf life. I'll buy them again.\n",
      "Star Rating: 4.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 11:\n",
      "Text: I find amazon basics batteries to be equal if not superior to name brand ones. Can't believe I didn't start buying them sooner! The packages are large and the price is great too.\n",
      "Star Rating: 5.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 12:\n",
      "Text: When I first started getting the Amazon basic batteries I really liked them. With recent purchases, they do not seem to last like they had, or maybe a mixed-bag (inconsistent with some lasting better than others). I have not done any tests, but feel some other brands may last longer. However, the price is hard to beat.\n",
      "Star Rating: 3.0\n",
      "Predicted Sentiment: neutral\n",
      "--------------------------------------------------\n",
      "Review 13:\n",
      "Text: Use it for my fish tank's light at night and works great, I love how you can easily switch it off and on if you want it on while guests are there.\n",
      "Star Rating: 5.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 14:\n",
      "Text: just got em so I can't really comment on how good the do the job, good price, quick delivery but have only put two into one of my keyboards but they can go up to a year so who can say after three days\n",
      "Star Rating: 5.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 15:\n",
      "Text: we have many things that need aa battery they are great\n",
      "Star Rating: 5.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 16:\n",
      "Text: Thankful that I was able to find on Amazon for a great price and even better shipping. Arrived in perfect condition and did exactly what I needed it to. Great purchase and would purchase again.\n",
      "Star Rating: 5.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 17:\n",
      "Text: I don't know if I would buy thus brand again seems like they don't last as long as Duracell\n",
      "Star Rating: 1.0\n",
      "Predicted Sentiment: negative\n",
      "--------------------------------------------------\n",
      "Review 18:\n",
      "Text: In my opinion these did not last anywhere near as long as Duracel in things like LED candles (which is crazy) and trail cameras. Cameras were not exposed to cold temps more or less than other batteries. WE buy in bulk for the north house. Amazon Basics is great for things like sheets and beeding and towels. In my opinion the battery life, in the large package of aaa and aa size we purchased were lacking.\n",
      "Star Rating: 2.0\n",
      "Predicted Sentiment: negative\n",
      "--------------------------------------------------\n",
      "Review 19:\n",
      "Text: They don't last as long as the brand name but are good enough considering they are much cheaper.\n",
      "Star Rating: 4.0\n",
      "Predicted Sentiment: positive\n",
      "--------------------------------------------------\n",
      "Review 20:\n",
      "Text: Bought these batteries for my Christmas gifts the month of (december) only lasted like 2months toys now need replacement batteries . I also used some for my doorbell and just now needs replacement batteries. Tv Remote control is still working but these batteries don't last very long...\n",
      "Star Rating: 3.0\n",
      "Predicted Sentiment: neutral\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Print the first 20 reviews, their star ratings, and the DistilBERT predicted sentiment\n",
    "\n",
    "# Define a function to apply the model and predict the sentiment\n",
    "def predict_sentiment(text):\n",
    "    # Tokenize the input text and move to the appropriate device\n",
    "    inputs = tokenizer(text, return_tensors=\"pt\", padding=\"max_length\", truncation=True).to(device)\n",
    "\n",
    "    # Perform inference (no gradient calculation needed for evaluation)\n",
    "    with torch.no_grad():\n",
    "        outputs = model(**inputs)\n",
    "    \n",
    "    # Get the predicted label (index of the max logit)\n",
    "    predictions = torch.argmax(outputs.logits, dim=-1).item()\n",
    "\n",
    "    # Map the label index to sentiment\n",
    "    label_map = {0: 'positive', 1: 'neutral', 2: 'negative'}\n",
    "    return label_map[predictions]\n",
    "\n",
    "# Compare the first 20 reviews, star ratings, and predicted sentiments\n",
    "for idx, row in df.head(20).iterrows():\n",
    "    review_text = row['reviews.text']\n",
    "    star_rating = row['reviews.rating']\n",
    "    predicted_sentiment = predict_sentiment(row['cleaned_text'])\n",
    "    df['bert_sentiment']\n",
    "\n",
    "    print(f\"Review {idx+1}:\")\n",
    "    print(f\"Text: {review_text}\")\n",
    "    print(f\"Star Rating: {star_rating}\")\n",
    "    print(f\"Predicted Sentiment: {predicted_sentiment}\")\n",
    "    print(\"-\" * 50)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Comparison with Rating-Based Sentiment:\n",
    "After predicting the sentiment using BERT, comparing results with the rating-based labels originally used in step 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  sentiment bert_sentiment\n",
      "0   neutral        neutral\n",
      "1  positive       positive\n",
      "2  positive       positive\n",
      "3  positive       positive\n",
      "4  positive       positive\n"
     ]
    }
   ],
   "source": [
    "# Compare the BERT predicted sentiment with the rating-based sentiment\n",
    "print(df[['sentiment', 'bert_sentiment']].head())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "torch.cuda.empty_cache()  # Clear the GPU cache"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Meta-Category Classification\n",
    "Step 3: Meta-Category Classification, focused on the XLNet model and including the evaluation with Silhouette Score and Davies-Bouldin Index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                              categories  meta_category  \\\n",
      "18505  Fire Tablets,Computers/Tablets & Networking,Ta...        Tablets   \n",
      "56501  Stereos,Remote Controls,Amazon Echo,Audio Dock...  Ebook Readers   \n",
      "14165  Fire Tablets,Learning Toys,Toys,Tablets,Amazon...        Tablets   \n",
      "53946  Walmart for Business,Office Electronics,Tablet...  Ebook Readers   \n",
      "60462  Stereos,Remote Controls,Amazon Echo,Audio Dock...  Ebook Readers   \n",
      "\n",
      "       meta_category_label  \n",
      "18505                    3  \n",
      "56501                    0  \n",
      "14165                    3  \n",
      "53946                    0  \n",
      "60462                    0  \n",
      "                                              categories  meta_category  \\\n",
      "62339  Featured Brands,Electronics,Amazon Devices,Hom...  Ebook Readers   \n",
      "60428  Stereos,Remote Controls,Amazon Echo,Audio Dock...  Ebook Readers   \n",
      "20873  Fire Tablets,Tablets,Amazon Tablets,Computers ...        Tablets   \n",
      "5012   AA,AAA,Health,Electronics,Health & Household,C...      Batteries   \n",
      "12418  Amazon Echo,Home Theater & Audio,MP3 MP4 Playe...  Ebook Readers   \n",
      "\n",
      "       meta_category_label  \n",
      "62339                    0  \n",
      "60428                    0  \n",
      "20873                    3  \n",
      "5012                     1  \n",
      "12418                    0  \n",
      "Empty DataFrame\n",
      "Columns: [categories, meta_category]\n",
      "Index: []\n",
      "Empty DataFrame\n",
      "Columns: [categories, meta_category]\n",
      "Index: []\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6802c98f20194d749f2a654b6b3701ad",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/760 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc37dfb6466043539a471e3b12dd3608",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "pytorch_model.bin:   0%|          | 0.00/467M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of XLNetForSequenceClassification were not initialized from the model checkpoint at xlnet-base-cased and are newly initialized: ['sequence_summary.summary.weight', 'sequence_summary.summary.bias', 'logits_proj.weight', 'logits_proj.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training xlnet...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='6486' max='16990' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [ 6486/16990 1:25:38 < 2:18:45, 1.26 it/s, Epoch 1.91/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>F1</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.604300</td>\n",
       "      <td>0.584056</td>\n",
       "      <td>0.819968</td>\n",
       "      <td>0.788599</td>\n",
       "      <td>0.764260</td>\n",
       "      <td>0.819968</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.11/dist-packages/sklearn/metrics/_classification.py:1469: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    }
   ],
   "source": [
    "# Import the models and tokenizers for XLNET, ELECTRA, and ALBERT\n",
    "from transformers import XLNetTokenizer, XLNetForSequenceClassification\n",
    "\n",
    "import torch\n",
    "from datasets import Dataset\n",
    "from sklearn.metrics import accuracy_score, precision_recall_fscore_support\n",
    "\n",
    "\n",
    "# First, split the data into training and testing sets\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split df into df_train and df_test (80% train, 20% test)\n",
    "df_train, df_test = train_test_split(df, test_size=0.2, random_state=42)\n",
    "\n",
    "# Define the meta-categories based on broader product categories\n",
    "categories_mapping = {\n",
    "    \"Ebook Readers\": [\"kindle\", \"ereader\"],\n",
    "    \"Batteries\": [\"battery\", \"charge\", \"AAA\", \"AA\", \"alkaline\"],\n",
    "    \"Accessories\": [\"keyboard\", \"mouse\", \"laptop stand\", \"case\", \"headphones\" \"adapter\", \"speakers\", \"charger\", \"cables\", \"remote controls\", \"docker\" \"TV fires sticks\", \"docker\",],\n",
    "    \"Tablets\": [\"Ipad\", \"Kids Tablets\", \"Fire Tablets\", \"Amazon Tablets\"],\n",
    "    \"Non-Electronics\": [\"nespresso\", \"pod\", \"pet carrier\", \"coffee\"]\n",
    "}\n",
    "\n",
    "# Function to categorize products based on keywords in the 'categories' column\n",
    "def assign_category(product_category):\n",
    "    for meta_category, keywords in categories_mapping.items():\n",
    "        if any(keyword.lower() in str(product_category).lower() for keyword in keywords):\n",
    "            return meta_category\n",
    "    return 'Other'\n",
    "\n",
    "# Apply category assignment to both training and test sets\n",
    "df_train['meta_category'] = df_train['categories'].apply(assign_category)\n",
    "df_test['meta_category'] = df_test['categories'].apply(assign_category)\n",
    "\n",
    "# Map the 'meta_category' column to numerical labels\n",
    "meta_category_mapping = {category: i for i, category in enumerate(categories_mapping.keys())}\n",
    "meta_category_mapping['Other'] = len(meta_category_mapping) \n",
    "df_train['meta_category_label'] = df_train['meta_category'].map(meta_category_mapping)\n",
    "df_test['meta_category_label'] = df_test['meta_category'].map(meta_category_mapping)\n",
    "\n",
    "# Check the resulting meta-category labels\n",
    "print(df_train[['categories', 'meta_category', 'meta_category_label']].head())\n",
    "print(df_test[['categories', 'meta_category', 'meta_category_label']].head())\n",
    "\n",
    "# Check for NaN values in the labels\n",
    "print(df_train[df_train['meta_category_label'].isna()][['categories', 'meta_category']])\n",
    "print(df_test[df_test['meta_category_label'].isna()][['categories', 'meta_category']])\n",
    "\n",
    "# Convert 'tokenized' column and 'meta_category_label' into PyTorch tensors for both sets\n",
    "X_train_meta = torch.tensor(df_train['tokenized'].tolist())\n",
    "X_test_meta = torch.tensor(df_test['tokenized'].tolist())\n",
    "y_train_meta = torch.tensor(df_train['meta_category_label'].tolist(), dtype=torch.long)  # Ensure long dtype for class indices\n",
    "y_test_meta = torch.tensor(df_test['meta_category_label'].tolist(), dtype=torch.long)    # Ensure long dtype for class indices\n",
    "\n",
    "# Reshape target tensors\n",
    "y_train_meta = y_train_meta.view(-1)\n",
    "y_test_meta = y_test_meta.view(-1)\n",
    "\n",
    "# Define the models for XLNet, ELECTRA, and ALBERT\n",
    "models = {\n",
    "    \"xlnet\": XLNetForSequenceClassification.from_pretrained(\"xlnet-base-cased\", num_labels=len(meta_category_mapping))\n",
    "    # \"electra\": ElectraForSequenceClassification.from_pretrained(\"google/electra-small-discriminator\", num_labels=len(meta_category_mapping)),\n",
    "    # \"albert\": AlbertForSequenceClassification.from_pretrained(\"albert-base-v2\", num_labels=len(meta_category_mapping))\n",
    "}\n",
    "\n",
    "# Create Hugging Face datasets for training and testing\n",
    "train_dataset = Dataset.from_dict({\n",
    "    \"input_ids\": X_train_meta.tolist(),\n",
    "    \"labels\": y_train_meta.tolist()\n",
    "})\n",
    "test_dataset = Dataset.from_dict({\n",
    "    \"input_ids\": X_test_meta.tolist(),\n",
    "    \"labels\": y_test_meta.tolist()\n",
    "})\n",
    "\n",
    "# Set format for PyTorch\n",
    "train_dataset.set_format(type='torch', columns=['input_ids', 'labels'])\n",
    "test_dataset.set_format(type='torch', columns=['input_ids', 'labels'])\n",
    "\n",
    "\n",
    "# Training loop for each model\n",
    "training_args = TrainingArguments(\n",
    "    output_dir='./results',\n",
    "    evaluation_strategy=\"epoch\",\n",
    "    learning_rate=2e-5,\n",
    "    per_device_train_batch_size=8,  # Reduce the batch size to prevent memory overflow\n",
    "    per_device_eval_batch_size=8,   # Reduce evaluation batch size as well\n",
    "    num_train_epochs=5,\n",
    "    weight_decay=0.01,\n",
    "    report_to=[\"none\"],\n",
    "    gradient_accumulation_steps=2,    # Accumulate gradients over 2 steps to simulate a larger batch size\n",
    "    fp16=True                         # Enable FP16 mixed precision training\n",
    ")\n",
    "\n",
    "# Metrics calculation function (accuracy, precision, recall, F1)\n",
    "def compute_metrics(p):\n",
    "    preds = np.argmax(p.predictions, axis=1)\n",
    "    precision, recall, f1, _ = precision_recall_fscore_support(p.label_ids, preds, average='weighted')\n",
    "    accuracy = accuracy_score(p.label_ids, preds)\n",
    "    return {'accuracy': accuracy, 'f1': f1, 'precision': precision, 'recall': recall}\n",
    "\n",
    "# Train and evaluate each model\n",
    "results = {}\n",
    "for model_name, model in models.items():\n",
    "    print(f\"Training {model_name}...\")\n",
    "    \n",
    "    # Setup Trainer\n",
    "    trainer = Trainer(\n",
    "        model=model,\n",
    "        args=training_args,\n",
    "        train_dataset=train_dataset,\n",
    "        eval_dataset=test_dataset,\n",
    "        compute_metrics=compute_metrics\n",
    "    )\n",
    "    \n",
    "    # Start training\n",
    "trainer.train()\n",
    "\n",
    "# Evaluate the XLNet model\n",
    "metrics = trainer.evaluate()\n",
    "\n",
    "# Print the evaluation metrics\n",
    "print(f\"Evaluation Results: Loss = {metrics['eval_loss']:.4f}, \"\n",
    "      f\"Accuracy = {metrics['eval_accuracy']:.4f}, \"\n",
    "      f\"F1-Score = {metrics['eval_f1']:.4f}, \"\n",
    "      )\n",
    "     \n",
    "# Saving the model in folder\n",
    "output_dir = \"./Robo-Reviews-Project/r\"\n",
    "\n",
    "# After training, save the model and tokenizer\n",
    "trainer.save_model(output_dir)  # Saves the model, tokenizer, and config to 'output_dir'\n",
    "tokenizer.save_pretrained(output_dir)  # Save the tokenizer to the same directory\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: Generate Blog-Style Articles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summarization Pipeline: Uses the BART model to generate summaries for each meta-category.\n",
    "Top Products and Worst Product Identification: Selects the top 3 and worst product based on ratings and review count.\n",
    "Top Complaints Extraction: Extracts common complaints for the top 3 products from negative reviews.\n",
    "Blog Post Generation: Combines the summary, top products, differences, complaints, and the worst product into a blog-style format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3becd086044e46f9bb15b0af4f6b9c59",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.json:   0%|          | 0.00/899k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fe00400654c045d3a844f505afa3d4f4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72289bcc248747d19be76eda16ac49cb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/1.36M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f74c64ab52ad44cba3b44c0102a29e24",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/1.58k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fa2967ead4f645d3a115504759860566",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/1.63G [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48bf7773358e4df2bba8383426395354",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "generation_config.json:   0%|          | 0.00/363 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "ename": "KeyError",
     "evalue": "'meta_category'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[60], line 17\u001b[0m\n\u001b[1;32m     14\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m summary_text\n\u001b[1;32m     16\u001b[0m \u001b[38;5;66;03m# Group reviews by meta-category (remains the same)\u001b[39;00m\n\u001b[0;32m---> 17\u001b[0m grouped_reviews \u001b[38;5;241m=\u001b[39m \u001b[43mdf\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgroupby\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mmeta_category\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m[\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mcleaned_text\u001b[39m\u001b[38;5;124m'\u001b[39m]\u001b[38;5;241m.\u001b[39mapply(\u001b[38;5;28mlist\u001b[39m)\u001b[38;5;241m.\u001b[39mto_dict()\n\u001b[1;32m     19\u001b[0m \u001b[38;5;66;03m# Summarizing each meta-category (remains the same)\u001b[39;00m\n\u001b[1;32m     20\u001b[0m summaries \u001b[38;5;241m=\u001b[39m {category: summarize_reviews(reviews) \u001b[38;5;28;01mfor\u001b[39;00m category, reviews \u001b[38;5;129;01min\u001b[39;00m grouped_reviews\u001b[38;5;241m.\u001b[39mitems()}\n",
      "File \u001b[0;32m/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py:9156\u001b[0m, in \u001b[0;36mDataFrame.groupby\u001b[0;34m(self, by, axis, level, as_index, sort, group_keys, observed, dropna)\u001b[0m\n\u001b[1;32m   9153\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m level \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m by \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m   9154\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mYou have to supply one of \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mby\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m and \u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mlevel\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m-> 9156\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mDataFrameGroupBy\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   9157\u001b[0m \u001b[43m    \u001b[49m\u001b[43mobj\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m   9158\u001b[0m \u001b[43m    \u001b[49m\u001b[43mkeys\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mby\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   9159\u001b[0m \u001b[43m    \u001b[49m\u001b[43maxis\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maxis\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   9160\u001b[0m \u001b[43m    \u001b[49m\u001b[43mlevel\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlevel\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   9161\u001b[0m \u001b[43m    \u001b[49m\u001b[43mas_index\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mas_index\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   9162\u001b[0m \u001b[43m    \u001b[49m\u001b[43msort\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msort\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   9163\u001b[0m \u001b[43m    \u001b[49m\u001b[43mgroup_keys\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mgroup_keys\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   9164\u001b[0m \u001b[43m    \u001b[49m\u001b[43mobserved\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mobserved\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   9165\u001b[0m \u001b[43m    \u001b[49m\u001b[43mdropna\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mdropna\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   9166\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m/usr/local/lib/python3.11/dist-packages/pandas/core/groupby/groupby.py:1329\u001b[0m, in \u001b[0;36mGroupBy.__init__\u001b[0;34m(self, obj, keys, axis, level, grouper, exclusions, selection, as_index, sort, group_keys, observed, dropna)\u001b[0m\n\u001b[1;32m   1326\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdropna \u001b[38;5;241m=\u001b[39m dropna\n\u001b[1;32m   1328\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m grouper \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m-> 1329\u001b[0m     grouper, exclusions, obj \u001b[38;5;241m=\u001b[39m \u001b[43mget_grouper\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m   1330\u001b[0m \u001b[43m        \u001b[49m\u001b[43mobj\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1331\u001b[0m \u001b[43m        \u001b[49m\u001b[43mkeys\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1332\u001b[0m \u001b[43m        \u001b[49m\u001b[43maxis\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maxis\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1333\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlevel\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlevel\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1334\u001b[0m \u001b[43m        \u001b[49m\u001b[43msort\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43msort\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1335\u001b[0m \u001b[43m        \u001b[49m\u001b[43mobserved\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mFalse\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[38;5;28;43;01mif\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mobserved\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;129;43;01mis\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mlib\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mno_default\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43;01melse\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mobserved\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1336\u001b[0m \u001b[43m        \u001b[49m\u001b[43mdropna\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdropna\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m   1337\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1339\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m observed \u001b[38;5;129;01mis\u001b[39;00m lib\u001b[38;5;241m.\u001b[39mno_default:\n\u001b[1;32m   1340\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28many\u001b[39m(ping\u001b[38;5;241m.\u001b[39m_passed_categorical \u001b[38;5;28;01mfor\u001b[39;00m ping \u001b[38;5;129;01min\u001b[39;00m grouper\u001b[38;5;241m.\u001b[39mgroupings):\n",
      "File \u001b[0;32m/usr/local/lib/python3.11/dist-packages/pandas/core/groupby/grouper.py:1043\u001b[0m, in \u001b[0;36mget_grouper\u001b[0;34m(obj, key, axis, level, sort, observed, validate, dropna)\u001b[0m\n\u001b[1;32m   1041\u001b[0m         in_axis, level, gpr \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mFalse\u001b[39;00m, gpr, \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m   1042\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m-> 1043\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m(gpr)\n\u001b[1;32m   1044\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(gpr, Grouper) \u001b[38;5;129;01mand\u001b[39;00m gpr\u001b[38;5;241m.\u001b[39mkey \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m   1045\u001b[0m     \u001b[38;5;66;03m# Add key to exclusions\u001b[39;00m\n\u001b[1;32m   1046\u001b[0m     exclusions\u001b[38;5;241m.\u001b[39madd(gpr\u001b[38;5;241m.\u001b[39mkey)\n",
      "\u001b[0;31mKeyError\u001b[0m: 'meta_category'"
     ]
    }
   ],
   "source": [
    "# Blog Article Generation using Summarization Pipeline with BART\n",
    "from transformers import BartTokenizer, BartForConditionalGeneration\n",
    "\n",
    "# Load the BART model and tokenizer for summarization \n",
    "tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')\n",
    "model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')\n",
    "\n",
    "# Summarize reviews for each meta-category using BART \n",
    "def summarize_reviews(reviews):\n",
    "    review_text = \" \".join(reviews)  # Concatenate reviews into a single string\n",
    "    inputs = tokenizer([review_text], max_length=1024, return_tensors='pt', truncation=True)  # Tokenize and truncate if needed\n",
    "    summary_ids = model.generate(inputs['input_ids'], max_length=150, min_length=50, length_penalty=2.0, num_beams=4, early_stopping=True)\n",
    "    summary_text = tokenizer.decode(summary_ids[0], skip_special_tokens=True)  # Decode generated summary\n",
    "    return summary_text\n",
    "\n",
    "# Group reviews by meta-category \n",
    "grouped_reviews = df.groupby('meta_category')['cleaned_text'].apply(list).to_dict()\n",
    "\n",
    "# Summarizing each meta-category \n",
    "summaries = {category: summarize_reviews(reviews) for category, reviews in grouped_reviews.items()}\n",
    "\n",
    "# Identifying Top 3 Products and Worst Product in Each Category \n",
    "def get_top_products(df_category):\n",
    "    top_products = df_category.groupby('product_name').agg({\n",
    "        'reviews.rating': 'mean', 'reviews.numHelpful': 'sum', 'product_name': 'count'\n",
    "    }).sort_values(by=['reviews.rating', 'product_name'], ascending=[False, False])\n",
    "    \n",
    "    # Get the top 3 products\n",
    "    top_3 = top_products.head(3).index.tolist()\n",
    "    \n",
    "    # Get the worst product (lowest-rated product)\n",
    "    worst = top_products.tail(1).index[0]\n",
    "    \n",
    "    return top_3, worst\n",
    "\n",
    "# Extracting top complaints for the top 3 products \n",
    "def extract_top_complaints(df_category, top_3):\n",
    "    complaints = {}\n",
    "    for product in top_3:\n",
    "        product_reviews = df_category[df_category['product_name'] == product]\n",
    "        negative_reviews = product_reviews[product_reviews['sentiment'] == 'negative']\n",
    "        complaint_keywords = negative_reviews['cleaned_text'].str.cat(sep=' ').split()\n",
    "        complaint_summary = \" \".join(pd.Series(complaint_keywords).value_counts().head(3).index)\n",
    "        complaints[product] = complaint_summary\n",
    "    return complaints\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 5: Final Blog Output\n",
    "Finally, generate blog-style content based on the summaries and insights.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Blog Article Generation using Summarization Pipeline with BART\n",
    "from transformers import BartTokenizer, BartForConditionalGeneration\n",
    "\n",
    "# Load the BART model and tokenizer for summarization \n",
    "tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')\n",
    "model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')\n",
    "\n",
    "# Summarize reviews for each meta-category using BART \n",
    "def summarize_reviews(reviews):\n",
    "    review_text = \" \".join(reviews)  # Concatenate reviews into a single string\n",
    "    inputs = tokenizer([review_text], max_length=1024, return_tensors='pt', truncation=True)  # Tokenize and truncate if needed\n",
    "    summary_ids = model.generate(inputs['input_ids'], max_length=150, min_length=50, length_penalty=2.0, num_beams=4, early_stopping=True)\n",
    "    summary_text = tokenizer.decode(summary_ids[0], skip_special_tokens=True)  # Decode generated summary\n",
    "    return summary_text\n",
    "\n",
    "# Generate blog post for each meta-category \n",
    "def generate_blog_post(category, summary, top_3, worst_product, complaints):\n",
    "    blog_post = f\"Category: {category}\\n\\nSummary: {summary}\\n\\n\"\n",
    "    blog_post += \"Top 3 Products:\\n\"\n",
    "    \n",
    "    for product in top_3:\n",
    "        blog_post += f\"- {product}\\n\"\n",
    "    \n",
    "    blog_post += \"\\nKey differences:\\n- Product 1 is better for feature X\\n- Product 2 excels at feature Y\\n- Product 3 is known for durability\\n\"\n",
    "    blog_post += \"\\nTop complaints:\\n\"\n",
    "    \n",
    "    for product, complaint in complaints.items():\n",
    "        blog_post += f\"- {product}: {complaint}\\n\"\n",
    "    \n",
    "    blog_post += f\"\\nWorst product: {worst_product}\\nReason: Poor performance, durability issues.\"\n",
    "    return blog_post\n",
    "\n",
    "# Create blog posts for each meta-category \n",
    "blog_posts = {}\n",
    "\n",
    "for category, reviews in grouped_reviews.items():\n",
    "    df_category = df[df['meta_category'] == category]\n",
    "    \n",
    "    # Summarizing using BART\n",
    "    summary = summaries[category]  # BART is now used for generating summaries\n",
    "    top_3, worst_product = get_top_products(df_category)\n",
    "    complaints = extract_top_complaints(df_category, top_3)\n",
    "    \n",
    "    # Generate blog posts\n",
    "    blog_posts[category] = generate_blog_post(category, summary, top_3, worst_product, complaints)\n",
    "\n",
    "# Display blog posts (remains the same)\n",
    "for category, blog in blog_posts.items():\n",
    "    print(f\"Blog Post for {category}:\\n{blog}\\n\")\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}