inventory_management.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Inventory Management with Object Detection and SingleStore Database\n",
        "\n",
        "This notebook explores an advanced inventory management system leveraging state-of-the-art object detection techniques with the YOLO (You Only Look Once) model and integrating OpenAI's GPT-4 for vision-based object detection insights. The system is designed to automate inventory tracking, offering a sophisticated approach to recognizing, counting, and managing stock levels through visual analysis.\n",
        "\n",
        "## Objectives\n",
        "\n",
        "- **Model Training**: Utilize YOLO for training a custom object detection model tailored to identify specific inventory items within images.\n",
        "- **Object Detection**: Implement object detection on inventory images to locate and identify items using the trained YOLO model.\n",
        "- **Data Management**: Store and manage detected bounding box information using SingleStore, a high-performance, SQL-compliant database optimized for real-time analytics.\n",
        "- **Insight Generation**: Apply OpenAI's GPT-4 vision capabilities to interpret images and provide actionable inventory insights.\n",
        "\n",
        "## Technologies Used\n",
        "\n",
        "- **YOLO (You Only Look Once)**: A fast, accurate object detection framework, ideal for real-time applications.\n",
        "- **SingleStore Database**: A distributed SQL database that excels in fast data ingestion and real-time analytics.\n",
        "- **OpenAI GPT-4**: Leveraging the latest in AI to provide vision-based analysis and natural language processing.\n",
        "\n",
        "By the end of this notebook, you will have a comprehensive understanding of how to build and deploy a cutting-edge inventory management system that bridges the gap between traditional stock-keeping methods and AI-driven automation.\n"
      ],
      "metadata": {
        "id": "xpMp4V_ldNYQ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Check the GPU status and configuration with `nvidia-smi`, ensuring that the environment is ready for deep learning tasks."
      ],
      "metadata": {
        "id": "fXMmIHOUdcMp"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P9Vavqlmx-yr"
      },
      "outputs": [],
      "source": [
        "!nvidia-smi"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Environment Setup and Library Import\n",
        "\n",
        "In this section, we prepare our working environment for object detection tasks:\n",
        "\n",
        "1. **Determine the Current Working Directory**: Establish a reference to our working directory, which is crucial for relative path operations and file management throughout the notebook.\n",
        "2. **Install the Ultralytics Package**: We install a specific version of the `ultralytics` library, which provides an easy-to-use interface for working with YOLO models. This ensures compatibility and access to the latest features necessary for our object detection tasks.\n",
        "3. **Clear Output**: The output of the installation process is cleared for a cleaner notebook presentation.\n",
        "4. **Library Imports and Checks**: We import necessary libraries, including `ultralytics` for object detection and IPython utilities for better output management. Additionally, we perform system checks to ensure our environment is correctly set up for utilizing YOLO models with the `ultralytics.checks()` function.\n",
        "\n",
        "This initial setup is essential for ensuring that all subsequent operations, from model training to object detection, run smoothly in a well-prepared environment.\n"
      ],
      "metadata": {
        "id": "k9xO7mgqdfH8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "HOME = os.getcwd()\n",
        "print(HOME)"
      ],
      "metadata": {
        "id": "Wl18c64ayS-o"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install ultralytics==8.0.196\n",
        "\n",
        "from IPython import display\n",
        "display.clear_output()\n",
        "\n",
        "import ultralytics\n",
        "ultralytics.checks()"
      ],
      "metadata": {
        "id": "YbkcsSUCyWCL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from ultralytics import YOLO\n",
        "from IPython.display import display, Image"
      ],
      "metadata": {
        "id": "oluAajMxycJ8"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training the YOLO Model\n",
        "\n",
        "In this crucial step, we commence the training of our YOLO object detection model with the following parameters:\n",
        "\n",
        "- **Task**: We specify `detect` as our primary task, focusing on detecting objects within images.\n",
        "- **Mode**: Set to `train`, indicating that we are in the training phase of our model development.\n",
        "- **Model**: Utilizes `yolov8s.pt`, a pre-trained YOLOv8 small model, serving as our starting point for training. This choice balances efficiency and accuracy.\n",
        "- **Data**: Points to our dataset configuration file `data.yaml` located within the `supermarket_dataset` directory. This file contains all necessary information about our training and validation datasets.\n",
        "- **Epochs**: The model will go through 25 training epochs, allowing it to learn from the dataset iteratively.\n",
        "- **Image Size**: Sets the image resolution to `640x640` pixels, which is a common size for balancing model performance and detection accuracy.\n",
        "- **Plots**: Enabled to generate and display training progress plots, offering insights into how the model's performance evolves over time.\n",
        "\n",
        "This training phase is pivotal, as it fine-tunes the pre-trained YOLO model on our specific supermarket dataset, enabling it to recognize and locate items from the inventory effectively.\n"
      ],
      "metadata": {
        "id": "cG62yOJeeDji"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!yolo task=detect mode=train model=yolov8s.pt data={\"/content/drive/MyDrive/supermarket_dataset\"}/data.yaml epochs=25 imgsz=640 plots=True # use your own dataset path"
      ],
      "metadata": {
        "id": "H3g8MBEIyyMb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "Load the trained YOLO model and run detection on an image to obtain results, specifying a confidence threshold of 0.5."
      ],
      "metadata": {
        "id": "qlcah0OpiF3l"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = YOLO(f'/content/runs/detect/train/weights/best.pt')\n",
        "results = model(source='/content/cleaning-store.jpg', conf=0.5)"
      ],
      "metadata": {
        "id": "86vtVHf1GZja"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "After loading the YOLO model with custom-trained weights, we perform object detection on a specific image, adjusting the confidence threshold to 0.25. The detected objects are then visualized and saved as an image, showcasing the model's ability to identify items within the scene."
      ],
      "metadata": {
        "id": "CfyAgvY4iM2p"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from ultralytics import YOLO\n",
        "from PIL import Image\n",
        "\n",
        "# Load the model\n",
        "model = YOLO('/content/runs/detect/train/weights/best.pt')  # Adjust path as necessary\n",
        "\n",
        "# Perform detection\n",
        "results = model(source='/content/coke.jpg', conf=0.25)\n",
        "\n",
        "# Show the results\n",
        "for r in results:\n",
        "    im_array = r.plot()  # plot a BGR numpy array of predictions\n",
        "    im = Image.fromarray(im_array[..., ::-1])  # RGB PIL image\n",
        "    im.show()  # show image\n",
        "    im.save('results.jpg')  # save image"
      ],
      "metadata": {
        "id": "HMrMonrrJYzq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Utilizing the YOLO model, object detection is executed on an image with a specified confidence threshold of 0.25, focusing on accurately identifying objects. The output includes the bounding boxes for each detected object, which are extracted and converted into a list format for further analysis. This process effectively demonstrates how the model discerns and locates items within an image, highlighting the precise coordinates of each object's boundaries."
      ],
      "metadata": {
        "id": "aM-qzAHniS1i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from ultralytics import YOLO\n",
        "import torch\n",
        "\n",
        "\n",
        "model = YOLO('/content/runs/detect/train/weights/best.pt')  # Adjust path as necessary\n",
        "\n",
        "# Perform detection\n",
        "results = model(source='/content/coke.jpg', conf=0.25)\n",
        "\n",
        "boxes = []\n",
        "# View results\n",
        "for r in results:\n",
        "    boxes.append(r.boxes.xyxy.tolist())\n",
        "\n",
        "\n",
        "print(boxes)"
      ],
      "metadata": {
        "id": "LzDg1LrjMsJD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Integration with SingleStore for Bounding Box Data Management\n",
        "\n",
        "This section outlines the steps for integrating object detection results with SingleStore, a highly scalable SQL database, to store and manage bounding box data efficiently. The process involves several key steps, from setting up the database connection to creating a dedicated table for the bounding boxes and inserting detection results.\n",
        "\n",
        "### Installing SingleStoreDB Library\n",
        "\n",
        "First, we install the `singlestoredb` Python library, which facilitates interaction with SingleStore databases directly from Python applications. This library allows us to execute SQL commands and manage data within SingleStore seamlessly.\n"
      ],
      "metadata": {
        "id": "F2KK5qaqjQxz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install singlestoredb\n"
      ],
      "metadata": {
        "id": "pOGzuzWuOMnA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Establishing Database Connection\n",
        "\n",
        "A connection to the SingleStore database is established using the `singlestoredb.connect` method, which requires specifying connection parameters such as the username, password, and database URL. This connection enables us to execute SQL commands and interact with the database:\n"
      ],
      "metadata": {
        "id": "Rx5-PCDojcTm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import singlestoredb as s2\n",
        "conn_params = 'user:password@host:port/database' #enter your database params\n",
        "conn = s2.connect(conn_params)\n"
      ],
      "metadata": {
        "id": "nFXOzHOFOThj"
      },
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating the BoundingBoxes Table\n",
        "\n",
        "We define and execute a SQL command to create a `BoundingBoxes` table if it does not already exist. This table is designed to store the bounding box coordinates (`X1`, `Y1`, `X2`, `Y2`) along with an associated `ImageID`, which links each bounding box to a specific image in our dataset:\n"
      ],
      "metadata": {
        "id": "vfFl5T2GjoOZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# SQL command to create the BoundingBoxes table\n",
        "create_table_command = \"\"\"\n",
        "CREATE TABLE IF NOT EXISTS BoundingBoxes (\n",
        "    BoxID INT AUTO_INCREMENT PRIMARY KEY,\n",
        "    ImageID INT,\n",
        "    X1 FLOAT,\n",
        "    Y1 FLOAT,\n",
        "    X2 FLOAT,\n",
        "    Y2 FLOAT\n",
        ");\n",
        "\"\"\"\n",
        "\n",
        "\n",
        "\n",
        "# Execute the SQL command\n",
        "with s2.connect(conn_params) as conn:\n",
        "    with conn.cursor() as cursor:\n",
        "        cursor.execute(create_table_command)\n",
        "        print(\"Table 'BoundingBoxes' created successfully.\")"
      ],
      "metadata": {
        "id": "kyocCi6JQvul"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Inserting Detection Results\n",
        "\n",
        "With the `BoundingBoxes` table in place, we prepare an SQL statement to insert bounding box data for each detected object. This step involves iterating over the list of detected bounding boxes obtained from the YOLO model and inserting each box's coordinates into the table. By associating each bounding box with an `ImageID`, we create a structured and queryable record of detections:"
      ],
      "metadata": {
        "id": "rL7sKjdfjqOy"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Prepare the SQL statement for inserting bounding box data\n",
        "stmt = 'INSERT INTO BoundingBoxes (ImageID, X1, Y1, X2, Y2) VALUES (%s, %s, %s, %s, %s)'\n",
        "\n",
        "# Assuming each bounding box is associated with the same image, for example, ImageID = 1\n",
        "image_id = 1  # Adjust this based on your actual ImageID\n",
        "\n",
        "with s2.connect(conn_params) as conn:\n",
        "    conn.autocommit(True)\n",
        "    with conn.cursor() as cur:\n",
        "        # Iterate over each bounding box and insert its data\n",
        "        for box in boxes[0]:  # Assuming boxes[0] contains your bounding box data for one image\n",
        "            cur.execute(stmt, (image_id, box[0], box[1], box[2], box[3]))\n"
      ],
      "metadata": {
        "id": "vvNRPwUZO9D3"
      },
      "execution_count": 37,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "This integration not only facilitates the efficient storage and management of detection results but also lays the groundwork for further analysis and querying of object detection data within the SingleStore database. By leveraging SingleStore's capabilities, we can perform real-time analytics and gain deeper insights into the detected objects across our image dataset.\n"
      ],
      "metadata": {
        "id": "9j0hrsY1jy16"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Leveraging GPT-4 for Vision-Based Inventory Insights\n",
        "\n",
        "In this innovative approach, we utilize OpenAI's GPT-4 with vision capabilities to analyze inventory levels directly from images. By integrating GPT-4's advanced understanding of visual content, we aim to extract quantitative insights that inform inventory management decisions.\n",
        "\n",
        "### Implementation Details\n",
        "\n",
        "We employ the `openai` Python library to interface with GPT-4's API, specifically its vision model variant. The process involves sending an image URL to the model along with a prompt that instructs GPT-4 to interpret the image in the context of inventory management. The model is tasked with estimating the percentage of inventory remaining, providing a concise and actionable piece of information:\n"
      ],
      "metadata": {
        "id": "xFj4OIekkDmQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install openai"
      ],
      "metadata": {
        "id": "nrcbkkg5VwpF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from openai import OpenAI\n",
        "from google.colab import userdata\n",
        "\n",
        "client = OpenAI(api_key=userdata.get('openai_key')) # store your openai key secretly in env variables\n",
        "\n",
        "response = client.chat.completions.create(\n",
        "  model=\"gpt-4-vision-preview\",\n",
        "  messages=[\n",
        "      {\"role\": \"system\", \"content\": \"You are an inventory management bot. Your only goal is to output a percentage. This percentage is the amount of inventory left based on the input image.\"},\n",
        "    {\n",
        "      \"role\": \"user\",\n",
        "      \"content\": [\n",
        "        {\n",
        "          \"type\": \"image_url\",\n",
        "          \"image_url\": {\n",
        "            \"url\": \"https://www.redmetal.eu/wp-content/uploads/2021/08/Warehouse-Pallet-Racking-Manufacturer-07.webp\",\n",
        "          },\n",
        "        },\n",
        "      ],\n",
        "    }\n",
        "  ],\n",
        "  max_tokens=300,\n",
        ")\n",
        "\n",
        "print(response.choices[0].message)"
      ],
      "metadata": {
        "id": "tFnfYMnWVY7v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Considerations for AI-Driven Inventory Analysis\n",
        "\n",
        "- **Explainability**: Understanding the reasoning behind the AI's percentage estimation is challenging. Multimodal LLMs, especially those as complex as GPT-4, are often considered \"black boxes\" due to their opaque decision-making processes. This can make it difficult to fully trust or understand the basis of their predictions.\n",
        "\n",
        "- **Reliability**: While powerful, AI predictions can sometimes be unreliable or inconsistent, particularly in nuanced or ambiguous contexts. It's essential to validate these automated insights with real-world data and adjust strategies accordingly.\n",
        "\n",
        "- **Future Improvements**: As AI technology and vision models continue to evolve, we can expect improvements in accuracy, reliability, and explainability. These advancements will further enhance the utility of AI in inventory management and beyond.\n",
        "\n",
        "By integrating GPT-4's vision capabilities into our inventory management toolkit, we unlock new possibilities for automated, image-based analysis. However, it's crucial to approach these AI-driven insights with a critical eye, acknowledging the current limitations while looking forward to future advancements.\n"
      ],
      "metadata": {
        "id": "BAuyHPRSkTms"
      }
    }
  ]
}