notes/11a-clust-code.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "73d4ee48",
   "metadata": {},
   "source": [
    "Last week, we learned about the mechanics of two of the most popular clustering algorithms, k-means and DBSCAN.  Before that, we learned how to take a high dimensional data set and project it in to a lower dimensions while maintinaing the total variance in the data with PCA.  Today, we will put all this together with some code.\n",
    "\n",
    "Let's get started by importing a few libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c427ac8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.datasets import make_blobs\n",
    "from sklearn.cluster import KMeans"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc26e06b",
   "metadata": {},
   "source": [
    "## K-means"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "286c40b3",
   "metadata": {},
   "source": [
    "We will start off by making gaussian blobs.  That is a fancy way of saying that we will create point sets that are ball shaped that tend to have more points closer to the center of the blob and less as we move further away from the center.  Indeed, the density of the blob is dictated by a Gaussian distribution in which we can specify various parameters of the distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2685de9",
   "metadata": {},
   "outputs": [],
   "source": [
    "D, labels = make_blobs(n_samples=500, centers=3, cluster_std=.3, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d53c63a",
   "metadata": {},
   "source": [
    "Let's take a look at this data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c7d72b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "D.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ba08947",
   "metadata": {},
   "source": [
    "Super, so our data is in 2D, so let's take a look at it by ploting the data with the labels defining the color."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5501c67f",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(D[:,0], D[:,1], c=labels)\n",
    "plt.xlabel(\"X1\")\n",
    "plt.ylabel(\"X2\")\n",
    "plt.title('2d clusters')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c82b3ca8",
   "metadata": {},
   "source": [
    "Last week, we noted that k-means does quite well if our clusters are \"ball shaped\".  Well, these clusters look pretty ball shaped, let's see how it does!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0846100",
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters=3, init='random', max_iter=300, random_state=0)\n",
    "pred_labels = kmeans.fit_predict(D)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "408d3327",
   "metadata": {},
   "source": [
    "Great!  Now we have some labels.  Let's take a look at them!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f85a060",
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_labels[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8818c064",
   "metadata": {},
   "source": [
    "Note that we have lables 0, 1, and, 2. That makes sense since we said that we set k=3 in k-means.  Let's look at how our points clustered.  First, let's get the centers and then visuzlize using the predicted labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2d80998",
   "metadata": {},
   "outputs": [],
   "source": [
    "centers = kmeans.cluster_centers_\n",
    "centers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5df73c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(D[:,0], D[:,1], c=pred_labels)\n",
    "plt.scatter(centers[:,0], centers[:,1], s=50, c='red')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ee46f76",
   "metadata": {},
   "source": [
    "Cool!  So that worked pretty well.  The centers look to be in the centers of the clusters and note that while the colors changed, the points that are in each cluster are the same.  For now, we will eyeball (but later in the week, we will learn how to quanity the quality the result).\n",
    "\n",
    "We spoke last week of the functon that we are trying to minimize with k-means.  \n",
    "\n",
    "**Question:** *provide in words and with a formuala what the function is*.\n",
    "\n",
    "Given a clustering, we can compute the value of the function.  sklearn calls it 'inertia'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80afa624",
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans.inertia_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dec003fa",
   "metadata": {},
   "source": [
    "Note that while we are trying to minimize inertia, we have to be a bit careful with blindly minizing that number.  For example, it seems that we got the \"correct\" answer with 3 clusters, but what if we try with 4 clusters.  Do you think we can reducde the objective function?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e18bf1cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans4 = KMeans(n_clusters=4, init='k-means++', max_iter=300, random_state=0)\n",
    "pred_labels = kmeans4.fit_predict(D)\n",
    "\n",
    "centers = kmeans4.cluster_centers_\n",
    "plt.scatter(D[:,0], D[:,1], c=pred_labels)\n",
    "plt.scatter(centers[:,0], centers[:,1], s=50, c='red')\n",
    "\n",
    "print(\"Inertia of kmeans with 3 clusters\", kmeans.inertia_)\n",
    "print(\"Inertia of kmeans with 4 clusters\", kmeans4.inertia_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d50bfdc7",
   "metadata": {},
   "source": [
    "So we reduced the value of the objective function, but, it didn't really give us a \"better\" clustering.  So, how do we pick a good k?  Well for this example, we generated the data and know what the correct k is, but in general, we don't.  So, one way that in practice that we decide k is called the \"elbo method\".  We will plot the inertia for various k's and then pick the point when the inertia stops rapidly improving.  So for example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae2fe29c",
   "metadata": {},
   "outputs": [],
   "source": [
    "max_clusters = 50\n",
    "inertias = np.zeros(50)\n",
    "cluster_range = range(1, max_clusters)\n",
    "for i in cluster_range:\n",
    "    kmeans = KMeans(n_clusters=i, init='k-means++', max_iter=300, random_state=0)\n",
    "    kmeans.fit_predict(D)\n",
    "    inertias[i] = kmeans.inertia_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ae1cc32",
   "metadata": {},
   "source": [
    "And then plot the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed95826f",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(cluster_range, inertias[cluster_range], c='b', marker='.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5edfff9d",
   "metadata": {},
   "source": [
    "So, notice that we see a huge drop in the objective function value from adding clusters until we hit 3.  But after, we are only getting a small improvement.  So, we would pick 3 clusters because it is the smallest number of clusters that we can have before we stop seeing that substaintail improvement.\n",
    "\n",
    "While this elbow method doesn't have any gurantess, it is, in practice, what people tend to do becuase it works pretty well.\n",
    "\n",
    "Next, let's play around with the synthetically generated data sets.  First, let's see what happens if we use different standard deviations for our blobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da2aeed3",
   "metadata": {},
   "outputs": [],
   "source": [
    "D, labels = make_blobs(n_samples=500, centers=2, cluster_std=[.1, .8], random_state=0)\n",
    "plt.scatter(D[:,0], D[:,1], c=labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66104a19",
   "metadata": {},
   "source": [
    "and we can still cluster as we did before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa4b128a",
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters=2, init='k-means++', max_iter=300, random_state=0)\n",
    "pred_labels = kmeans.fit_predict(D)\n",
    "\n",
    "centers = kmeans.cluster_centers_\n",
    "plt.scatter(D[:,0], D[:,1], c=pred_labels)\n",
    "plt.scatter(centers[:,0], centers[:,1], s=50, c='red')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fab88c4c",
   "metadata": {},
   "source": [
    "But notice that some of the points at the top of the purple cluster are ending up in the wrong cluster!  **Question** Why do you think that is?  \n",
    "\n",
    "Let's explore this further by creating a lot of overlap between the clusters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab079f9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "D, labels = make_blobs(n_samples=500, centers=[[0,0], [.5, .5]], cluster_std=.3, random_state=0)\n",
    "plt.scatter(D[:,0], D[:,1], c=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c492e950",
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters=2, init='k-means++', max_iter=300, random_state=0)\n",
    "pred_labels = kmeans.fit_predict(D)\n",
    "\n",
    "centers = kmeans.cluster_centers_\n",
    "plt.scatter(D[:,0], D[:,1], c=pred_labels)\n",
    "plt.scatter(centers[:,0], centers[:,1], s=50, c='red')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c37bd751",
   "metadata": {},
   "source": [
    "So, what is the takeaway:\n",
    "\n",
    "1. k-means works well when our clusters are \"ball shaped\"\n",
    "1. we can use the elbow method to help us pick a good k\n",
    "2. if our clusters overlap (i.e. are not well sepearate), we are going to have some problems with points ending in the wrong cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06c5cbbe",
   "metadata": {},
   "source": [
    "## DBScan\n",
    "\n",
    "Next, let's play with DBSCAN and the iris data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a98c815",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import DBSCAN\n",
    "from sklearn.datasets import load_iris\n",
    "\n",
    "iris_data = load_iris()\n",
    "D = iris_data['data']\n",
    "labels = iris_data['target']\n",
    "n_labels = len(set(labels))\n",
    "n_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd4e4889",
   "metadata": {},
   "source": [
    "And let's use DBSCAN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60935e80",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbs = DBSCAN(eps=.3, min_samples=5)\n",
    "pred_labels = dbs.fit_predict(D)\n",
    "pred_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f70f9c55",
   "metadata": {},
   "source": [
    "So we did get 3 clusters (and the -1 are noise points).  But, recall the lables on the iris dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f73c0eeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ffc1f27",
   "metadata": {},
   "source": [
    "So that was not so good... In fact, we ended up with a ton of points as noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a787c2fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "sum(pred_labels == -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26f294e6",
   "metadata": {},
   "source": [
    "And since there are 150 samples in the iris dataset, that means we ended up with 64% of our points as noise.... so not so good.\n",
    "\n",
    "But, what if we try some different parameters?  Since we have a lot of noise points, that means our neighborhoos are too sparse.  **Question** So, what happens if we fix the number of neighbors but make the size of the neighborhood larger?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2da50c8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbs = DBSCAN(eps=.5, min_samples=5)\n",
    "pred_labels = dbs.fit_predict(D)\n",
    "\n",
    "print(\"Num clusters\", max(pred_labels)+1)\n",
    "print(\"Num noise\", sum(pred_labels == -1))\n",
    "print(\"labels\", pred_labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c30d7bb",
   "metadata": {},
   "source": [
    "So, we could probe further, but unfortunaly, we cannot make visuzliations like we did for the k-means example since the iris dataset is 4d.  What do we do?\n",
    "\n",
    "I wasn't sure so I went to the expert ... Prof Barktreuse had an idea!!  We just learned how to project our data to a lower dimension while maintaining a large portion of the total variance.... She suggested we should use PCA to try an capture over 95% of the variance!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2a65624",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "pca = PCA()\n",
    "\n",
    "D_pca = pca.fit_transform(D)\n",
    "\n",
    "var_ratio = pca.explained_variance_ratio_\n",
    "plt.plot(range(1,len(var_ratio)+1), np.cumsum(var_ratio), marker='x')\n",
    "plt.title('Iris data: fraction of total variance preserved by principal components')\n",
    "plt.xlabel('r : the number of principal components')\n",
    "plt.ylabel('f(r) : fraction of total variance preserved')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bca30350",
   "metadata": {},
   "source": [
    "We can see fro the plot that with even 2 components, we have ~98% of the variance.  So let's use those.  But before we grab those two dimensions, let's review a few ideas. First, let's look at the first two dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccb2c54c",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(D[:,0], D[:,1], c=labels)\n",
    "plt.title('iris data using first two attributes')\n",
    "plt.xlabel('X1')\n",
    "plt.ylabel('X2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b689bd1",
   "metadata": {},
   "source": [
    "Next, let's experiment with the first two components:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52226d72",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "D2_pca = D_pca[:,0:2]\n",
    "plt.scatter(D2_pca[:,0], D2_pca[:,1], c=labels)\n",
    "plt.title('PCA-transformed data using first two dimensions')\n",
    "plt.xlabel('PC1')\n",
    "plt.ylabel('PC2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d673891",
   "metadata": {},
   "source": [
    "Since we have worked with kmeans a bunch and know there are 3 culsters, let's try that first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7a85dfd",
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters=3, init='random', max_iter=300, random_state=0)\n",
    "pred_labels = kmeans.fit_predict(D2_pca)\n",
    "\n",
    "center = kmeans.cluster_centers_\n",
    "plt.scatter(D2_pca[:,0], D2_pca[:,1], c=pred_labels)\n",
    "plt.scatter(center[:, 0], center[:,1], c='red')\n",
    "plt.title('PCA-transformed iris data in 2 dimensions')\n",
    "plt.xlabel('PC1')\n",
    "plt.ylabel('PC2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c3ca665",
   "metadata": {},
   "source": [
    "So the centers look pretty good, but since the clusters are not \"ball shaped\" (they are more ellipsoidal, we get some points in the wrong cluster.  Ok, so let's try DBSCAN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "370d10e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbs = DBSCAN(eps=0.6, min_samples=5)\n",
    "pred_labels = dbs.fit_predict(D2_pca)\n",
    "plt.scatter(D2_pca[:,0], D2_pca[:,1], c=pred_labels)\n",
    "plt.title('DBSCAN results on PCA-transformed iris data in 2 dimensions')\n",
    "plt.xlabel('PC1')\n",
    "plt.ylabel('PC2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b94770f1",
   "metadata": {},
   "source": [
    "Okay, so that didn't turn out so well.  Two of our clusters were merged in to one and then we got a bunch of noise points, but we can at lease \"see\" what happended.  Let's see if we can pick some better params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f5a900a",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbs = DBSCAN(eps=0.25, min_samples=5)\n",
    "pred_labels = dbs.fit_predict(D2_pca)\n",
    "plt.scatter(D2_pca[:,0], D2_pca[:,1], c=pred_labels)\n",
    "plt.title('DBSCAN results on PCA-transformed iris data in 2 dimensions')\n",
    "plt.xlabel('PC1')\n",
    "plt.ylabel('PC2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "257e8032",
   "metadata": {},
   "source": [
    "Okay, that looks a little better, what if we get a little smaller?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "055fb5fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbs = DBSCAN(eps=0.2, min_samples=5)\n",
    "pred_labels = dbs.fit_predict(D2_pca)\n",
    "plt.scatter(D2_pca[:,0], D2_pca[:,1], c=pred_labels)\n",
    "plt.title('DBSCAN results on PCA-transformed iris data in 2 dimensions')\n",
    "plt.xlabel('PC1')\n",
    "plt.ylabel('PC2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed74972f",
   "metadata": {},
   "source": [
    "Hmmm, well that was maybe a little too small.  But, it turns out, in this representation, the iris data set is not seperable, which means we cannot draw a planer curve that separates the two classes, which means in this rep, we will not be able to do well with DBSCAN.  But, let's try a data set in which we can succeed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02349600",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "D, labels = make_moons(n_samples=500, noise=.1)\n",
    "plt.scatter(D[:,0], D[:,1], c=labels)\n",
    "plt.title('Moons data in in 2 dimensions')\n",
    "plt.xlabel('X1')\n",
    "plt.ylabel('X2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc8f20af",
   "metadata": {},
   "source": [
    "Great!  Let's see what we get with K-means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fff72ebf",
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans = KMeans(n_clusters=2, init='random', max_iter=300, random_state=0)\n",
    "pred_labels = kmeans.fit_predict(D)\n",
    "\n",
    "center = kmeans.cluster_centers_\n",
    "plt.scatter(D[:,0], D[:,1], c=pred_labels)\n",
    "plt.scatter(center[:, 0], center[:,1], c='red')\n",
    "plt.title('K-means labeled moon data in 2 dimensions')\n",
    "plt.xlabel('X1')\n",
    "plt.ylabel('X2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f879f726",
   "metadata": {},
   "source": [
    "Well, as we would expect, that was terrible.  Let's try DBSCAN!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3c683df",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbs = DBSCAN(eps=.1, min_samples=8)\n",
    "pred_labels = dbs.fit_predict(D)\n",
    "\n",
    "plt.scatter(D[:,0], D[:,1], c=pred_labels)\n",
    "plt.title('DBSCAN labeled moon data in 2 dimensions')\n",
    "plt.xlabel('X1')\n",
    "plt.ylabel('X2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c5e769d",
   "metadata": {},
   "source": [
    "Okay, so eps is too small, let's double eps and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0a5d44b",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbs = DBSCAN(eps=.2, min_samples=8)\n",
    "pred_labels = dbs.fit_predict(D)\n",
    "\n",
    "plt.scatter(D[:,0], D[:,1], c=pred_labels)\n",
    "plt.title('DBSCAN labeled moon data in 2 dimensions')\n",
    "plt.xlabel('X1')\n",
    "plt.ylabel('X2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be6f1a5d",
   "metadata": {},
   "source": [
    "Okay, that was too big.  I was about to give up hope and then Prof Barktreuse spoke up.  She said, you have an upper and lower bound, try to binary search for a good epsilon!  ... okay that sounds good!  Let's try .15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67bd8d4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbs = DBSCAN(eps=.15, min_samples=8)\n",
    "pred_labels = dbs.fit_predict(D)\n",
    "\n",
    "plt.scatter(D[:,0], D[:,1], c=pred_labels)\n",
    "plt.title('DBSCAN labeled moon data in 2 dimensions')\n",
    "plt.xlabel('X1')\n",
    "plt.ylabel('X2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "694d5652",
   "metadata": {},
   "source": [
    "Better!  Let's try a little bigger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3d068a8",
   "metadata": {},
   "outputs": [],
   "source": [
    ".15 + (.2-.15)/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "034fce09",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbs = DBSCAN(eps=.175, min_samples=8)\n",
    "pred_labels = dbs.fit_predict(D)\n",
    "\n",
    "plt.scatter(D[:,0], D[:,1], c=pred_labels)\n",
    "plt.title('DBSCAN labeled moon data in 2 dimensions')\n",
    "plt.xlabel('X1')\n",
    "plt.ylabel('X2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd9004a8",
   "metadata": {},
   "source": [
    "Even better!!  Let's try a little bigger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fa46937",
   "metadata": {},
   "outputs": [],
   "source": [
    ".175 + (.2-.175)/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59bbdab1",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbs = DBSCAN(eps=.1875, min_samples=8)\n",
    "pred_labels = dbs.fit_predict(D)\n",
    "\n",
    "plt.scatter(D[:,0], D[:,1], c=pred_labels)\n",
    "plt.title('DBSCAN labeled moon data in 2 dimensions')\n",
    "plt.xlabel('X1')\n",
    "plt.ylabel('X2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50068974",
   "metadata": {},
   "source": [
    "Seems like we flew too close to the sun.  Let's stick with .175 and ship!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc692818",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbs = DBSCAN(eps=.175, min_samples=8)\n",
    "pred_labels = dbs.fit_predict(D)\n",
    "\n",
    "plt.scatter(D[:,0], D[:,1], c=pred_labels)\n",
    "plt.title('DBSCAN labeled moon data in 2 dimensions')\n",
    "plt.xlabel('X1')\n",
    "plt.ylabel('X2')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "66eee27a",
   "metadata": {},
   "source": [
    "Note that while we binary searched for a good epsilon, there are more direct ways to find good epsilon.  Similar to with k-means, we can look for \"elbows\" and \"knees\" in various representatiosn of the data.  For more on finding elbows with DBSCAN (and some packages that can help with that analysis), see [tutorial](https://machinelearningknowledge.ai/tutorial-for-dbscan-clustering-in-python-sklearn/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53c5a031",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}