2.逻辑回归.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.逻辑回归（1958年）\n",
    "\n",
    "**学习目标**\n",
    "\n",
    "1. 熟悉逻辑回归算法和模型构建方法\n",
    "\n",
    "2. 熟练使用Sigmoid激活函数\n",
    "\n",
    "3. 熟练使用二值交叉熵（BCE）损失函数\n",
    "\n",
    "****"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "逻辑回归（Logistic Regression）源于逻辑函数（Logistic Function），这里的Logistic与“逻辑”没什么关系，而是与 Log（对数）有关。逻辑函数的起源可以追溯到人口增长模型。马尔萨斯（1766-1834）提出的人口增长理论认为在无约束条件下人口会呈指数增长，但是人口增长会受到资源限制，因此不会无限增长。马尔萨斯的人口增长模型是指数增长模型。1838年，比利时数学家Verhulst（1804-1849）在阅读了马尔萨斯的理论后，提出了逻辑增长模型，用于描述人口增长的上限和饱和现象。他把该模型命名为“Logistique”，源自拉丁语“logisticus”，意为“计算的艺术”，也就是现在的Logistic。逻辑函数（Logistic Function）也被称为 Sigmoid 函数，因其形状类似于字母“S”。\n",
    "\n",
    "1944年，美国生物统计学家约瑟夫·伯克森（Joseph Berkson，1899 – 1982）首次提出了逻辑回归的基本思想，尽管当时他并没有明确使用“逻辑回归”这个术语。他研究了二分类问题，并提出了logit函数的概念，用于建模分类概率。1958年，英国统计学家大卫·考克斯（David Cox）正式提出了逻辑回归模型，并将其应用于统计学领域。考克斯的工作使得逻辑回归成为一种广泛应用的分类工具。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 逻辑回归\n",
    "\n",
    "逻辑回归虽然名字中有“回归”二字，但其实质是一种分类算法，常用于二分类问题。它通过使用logistic函数（Sigmoid函数）将线性回归的输出映射到0和1之间，从而进行概率预测。Sigmoid函数的表达式如下：\n",
    "$$\n",
    "\\sigma(x) = \\frac{1}{1 + e^{-x}}\n",
    "$$\n",
    "我们可以用matplotlib可视化出Sigmoid函数的图像：\n",
    "\n",
    "<img src=\"./images/sigmoid.png\" style=\"zoom:60%;\" />\n",
    "\n",
    "逻辑回归可以看作是一个单层的神经网络，其中只有输入层和输出层，没有隐藏层。逻辑回归主要用于简单的分类任务，它假设特征和输出之间存在线性关系。\n",
    "\n",
    "逻辑回归由于其简单性，在特征和目标变量之间关系较为线性时表现良好，但在面对复杂的非线性关系时可能表现不佳。\n",
    "\n",
    "逻辑回归常用于医疗诊断、信用评分、垃圾邮件识别等二分类问题。\n",
    "\n",
    "总的来说，逻辑回归可以看作是神经网络的一个特例，而神经网络是一种更为通用和强大的模型，能够处理更复杂的数据和任务。在实际应用中，选择逻辑回归还是神经网络取决于问题的复杂性、数据的特征以及所需的性能。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 逻辑回归的优化方法\n",
    "\n",
    "1、二值交叉熵(Binary Cross-Entropy)损失函数\n",
    "\n",
    "二值交叉熵损失函数常用于二分类问题。计算交叉熵损失函数的公式如下：\n",
    "\n",
    "$$\n",
    "\\text{BCELoss} = -\\frac{1}{n} \\sum_{i=1}^{n} \\left[ y_i \\log(p(y_i)) + (1 - y_i) \\log(1 - p(y_i)) \\right]\n",
    "$$\n",
    "\n",
    "其中，n 是样本的数量，$y_i$是样本的真实标签（正样本：1和负样本：0），$p(y_i)$是正样本的预测概率。\n",
    "\n",
    "当标签为正样本：即 $y_i$ =1 时，$p(y_i)$ 越大就与 $y_i$ 越接近，即预测越准确，loss越小；\n",
    "\n",
    "当标签为负样本：即 $y_i$ =0 时，$p(y_i)$ 越小就与 $y_i$ 越接近，即预测越准确，loss越小。\n",
    "\n",
    "最终的loss是$y_i$=0和$y_i$=1两种类别的loss相加。\n",
    "\n",
    "然而BCE Loss有一个明显的缺点，当正样本（$y_i$=1）数量远远小于负样本（$y_i$=0）的数量时，loss函数中$y_i$=0的成分就会占据主导，使得模型的预测严重偏向负样本。\n",
    "\n",
    "2、随机梯度下降(SGD)\n",
    "\n",
    "随机梯度下降(SGD)是通过迭代的方式不断更新模型的参数，使得损失函数尽可能小。SGD的基本思想是每次迭代时，随机选择一个样本，计算损失函数关于该样本的梯度，然后更新模型的参数，直到损失函数足够小。\n",
    "\n",
    "SGD的更新公式如下：\n",
    "\n",
    "$$\n",
    "\\theta_{t+1} = \\theta_t - \\alpha \\frac{\\partial L}{\\partial \\theta}\n",
    "$$\n",
    "\n",
    "其中，$\\theta$ 是模型的参数，$\\alpha$ 是学习率，$L$ 是损失函数。\n",
    "\n",
    "为了让初学者更容易理解，这里举一个单变量函数梯度下降的例子。假设该损失函数是$L(\\theta)$，$\\theta$是单变量，那么梯度$\\frac{\\partial L}{\\partial \\theta}$是导数。\n",
    "\n",
    "梯度下降的实现过程如下：\n",
    "\n",
    "（1）初始化参数：设置初始点$\\theta_0$ = 3（随机选择），设定步长$\\alpha$=0.1（超参数，需要调节），设置迭代次数如epochs = 5（超参数，需要调节）或者损失函数的阈值等；\n",
    "\n",
    "（2）计算梯度：对下图中的函数求导，得L'= $2\\theta$ - 2，此时初始点的梯度为L'(3) = 4；\n",
    "\n",
    "（3）更新参数：计算下一步的更新点可得$\\theta_1$ = 3 - 0.1 * 4 = 2.6，沿着梯度的反方向将参数由红色的初始点更新到橙色的点（此时loss值由5变成了3.56）；\n",
    "\n",
    "（4）检查是否收敛：损失函数值的变化小于某个阈值则认为算法收敛，或者达到预设的迭代次数（epochs）则停止迭代；\n",
    "\n",
    "（5）如果未满足收敛条件，返回步骤 2，继续计算梯度和更新参数，直到loss不再下降，或达到最大迭代次数。\n",
    "\n",
    "<img src=\"./images/gd-gd.png\" style=\"zoom:40%;\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 逻辑回归的应用——草莓品质分类\n",
    "\n",
    "情景：在一家名为潘达利亚的高科技农场里，草莓是他们的明星产品。这个农场位于风景如画的山谷中，拥有肥沃的土壤和清澈的溪水，为草莓提供了理想的生长环境。农场主李逍遥是一位对智慧农业充满热情的创业者，他一直致力于提高草莓的品质和产量。草莓的种植采用了最先进的农业技术，包括智能灌溉系统、自动化施肥和病虫害监测。尽管如此，草莓的品质检测仍然是一个挑战。传统的人工检测方法耗时耗力，且主观性强，难以保证一致性和准确性。李逍遥意识到，为了进一步提升农场的竞争力，他需要一种更高效、更客观的检测方法。\n",
    "\n",
    "<img src=\"./images/strawberry2.png\" style=\"zoom:50%;\" />\n",
    "\n",
    "​\t一次偶然的机会，李逍遥接触到了机器学习技术。他意识到，通过训练一个逻辑回归模型，他可以预测草莓的成熟度和品质，从而实现自动化的品质检测。这个想法让他兴奋不已，他决定将这个技术应用到他的农场中。李逍遥收集了大量的草莓样本，并记录了它们的各种特征，如颜色、大小、形状和甜度等。他还记录了每个样本的品质评级，作为模型的标签。有了这些数据，他开始构建逻辑回归模型。\n",
    "\n",
    "​\t在模型训练过程中，李逍遥遇到了各种挑战，包括数据不平衡、特征选择和模型调优等。但他坚持不懈，通过不断尝试和改进，最终成功训练出了一个准确率很高的模型。这个模型能够根据草莓的特征预测其品质，帮助农场实现了自动化的品质检测。现在，农场的草莓品质检测工作变得既快速又准确。农场的效率大大提高，草莓的品质也得到了保证。李逍遥的创新不仅提升了农场的经济效益，也为农业科技的发展做出了贡献。\n",
    "\n",
    "​\t这里有偶然获得的李逍遥当时采集的一部分数据样本，我们借助PyTorch深度学习框架来复现一下他的实现方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.导入必需的模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2.数据集的加载和预处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "（1）读取数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   SSC   TA    Esters  Score  premium\n",
      "0  7.0  7.5  3.265765      6    False\n",
      "1  6.8  6.9  3.134130      5    False\n",
      "2  6.3  7.1  3.053652      6    False\n",
      "3  6.2  7.5  2.417363      5    False\n",
      "4  5.5  7.4  3.679450      5    False\n"
     ]
    }
   ],
   "source": [
    "df = pd.read_csv(\"./datasets/strawberry/strawberry_clf.csv\")\n",
    "\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "（2）数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(180, 3)\n",
      "(180, 1)\n"
     ]
    }
   ],
   "source": [
    "# 提取特征和标签\n",
    "x_data = df[['SSC', 'TA', 'Esters']].values\n",
    "y_data = df['premium'].values.reshape(-1, 1)\n",
    "\n",
    "print(x_data.shape)\n",
    "print(y_data.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([180, 3])\n",
      "torch.Size([180, 1])\n"
     ]
    }
   ],
   "source": [
    "# 转换为张量\n",
    "X = torch.tensor(x_data, dtype=torch.float32)\n",
    "y = torch.tensor(y_data, dtype=torch.float32)\n",
    "\n",
    "print(X.shape)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "（3）数据集划分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 划分训练集和测试集,random_state用于指定随机数生成器的种子\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.模型的构建"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "（1）参数设置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "input_size = x_data.shape[1]\n",
    "num_epochs = 1000\n",
    "print(input_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "（2）构建逻辑回归模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Linear(in_features=3, out_features=1, bias=True)\n",
      "  (1): Sigmoid()\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "model = nn.Sequential(\n",
    "    nn.Linear(input_size, 1),\n",
    "    nn.Sigmoid())  # 使用sigmoid激活函数\n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "nn.Sigmoid()用于实现Sigmoid激活函数。Sigmoid函数是一种将输入压缩到0和1之间的函数，通常用于二分类问题中。\n",
    "\n",
    "这个函数的特点是当输入值非常大或非常小时，输出值会接近0或1，而在0附近，函数的梯度接近于0，这会导致梯度消失问题，在训练深层网络时可能会遇到困难。\n",
    "\n",
    "使用nn.Sigmoid()非常简单，你只需要将它作为模型的一部分，然后在前向传播时自动应用这个函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "（3）定义损失函数和优化器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.BCELoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4.模型训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [100/1000], Loss: 0.35217\n",
      "Epoch [200/1000], Loss: 0.35034\n",
      "Epoch [300/1000], Loss: 0.34868\n",
      "Epoch [400/1000], Loss: 0.34719\n",
      "Epoch [500/1000], Loss: 0.34584\n",
      "Epoch [600/1000], Loss: 0.34461\n",
      "Epoch [700/1000], Loss: 0.34350\n",
      "Epoch [800/1000], Loss: 0.34247\n",
      "Epoch [900/1000], Loss: 0.34154\n",
      "Epoch [1000/1000], Loss: 0.34068\n"
     ]
    }
   ],
   "source": [
    "for epoch in range(num_epochs):\n",
    "    model.train()\n",
    "    \n",
    "    outputs = model(X_train)\n",
    "    loss = criterion(outputs, y_train)\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    \n",
    "    if (epoch+1) % 100 == 0 or epoch == num_epochs - 1:\n",
    "        print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.5f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5.模型测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.83\n"
     ]
    }
   ],
   "source": [
    "model.eval()\n",
    "\n",
    "with torch.no_grad():\n",
    "    \n",
    "    y_pred = model(X_test)\n",
    "    # print(f'Input的形状为：{X_test.shape}')\n",
    "    # print(f'Output的形状为：{y_pred.shape}')\n",
    "    # print(f'Output的第六个元素的值为：')\n",
    "    # print(y_pred[6], y_pred[6][0], y_pred[6, 0])\n",
    "    \n",
    "    predicted = (y_pred > 0.5).float()  # 应用阈值0.5进行分类\n",
    "    # print(predicted.shape, y_test.shape)\n",
    "    # print('\\n输出所有的预测值和真实值：')\n",
    "    # for i in range(len(predicted)):\n",
    "    #     print(predicted[i], y_test[i])\n",
    "        \n",
    "    # numel()方法用于计算张量（tensor）中元素的总数\n",
    "    accuracy = (predicted == y_test).sum() / y_test.numel()  # .sum() 和 .numel() 需要一维张量来正确计算\n",
    "    \n",
    "    print(f'Accuracy: {accuracy:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本节程序调用了机器学习库scikit-learn中的模块来对数据集进行划分，我们也可以调用PyTorch中的模型划分数据集，详细代码参见：logistic_regression.py文件。\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "d2l",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}