United States Analytics Occupations Deep Dive .ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1eefad7d",
   "metadata": {},
   "source": [
    "## Exploring Analytics-Related Occupations: A Data Mining Perspective on US Labor Market Insights for Analytics Students and Graduates"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77799a69",
   "metadata": {},
   "source": [
    "**Abstract:**\n",
    "<br>\n",
    "This study presents an in-depth analysis of analytics-related occupations within the US labor market, focusing on providing valuable insights to students. Utilizing data mining techniques such as clustering and classification, we examine the trends and statistics of these occupations and offer descriptive analytics, including exploratory data analysis (EDA) and clustering. Furthermore, predictive analytics is employed to forecast job postings and other relevant information, enabling students to make informed decisions about their career paths. By analyzing the analytics-related job landscape, this research contributes to informing students about the diverse opportunities available and guiding their professional journey."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612a39ce",
   "metadata": {},
   "source": [
    "**Keywords**: analytics-related occupations, US labor market, data mining techniques, clustering, classification, descriptive analytics, predictive analytics, exploratory data analysis (EDA), job postings, student insights."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92dbe8b6",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "761aa335",
   "metadata": {},
   "source": [
    "#### 1. Introduction into US Labor Market Landscape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5954f6b",
   "metadata": {},
   "source": [
    "Exploring analytics-occupations on US Labor Market requires the knowledge on how the labor market are organized in the United States along with its supporting infrastructure and ecosystem. There are three components that support the ecosystem of the labor market in the United States which are:\n",
    "<br>\n",
    "\n",
    "1. **Industry Classification:** Various industries in the United States are classified into several categories based on their characteristics and nature, and the classifi-cation for these industries adheres to the North American Industry Classification System. The North American Industry Classification System (NAICS) is a stand-ardized system used by Federal statistical agencies in the US to classify business establishments, gather statistical data, and analyze the US business economy. NAICS divides the economy into 20 sectors. Industries within these sectors are grouped according to the production criterion \n",
    "\n",
    "2. **Occupation Classification:** The Standard Occupational Classification (or SOC) system is a federal statistical standard employed by federal agencies for classifying workers into occupational categories, aiming to collect, compute, or publish data. Workers are assigned to one of 867 distinct occupations based on their de-scription. Detailed occupations are combined to create 459 broad occupations, 98 minor groups, and 23 major groups, making classification more efficient. Occupations with similar job responsibilities, and sometimes comparable skills, education, and/or training, are grouped in the SOC \n",
    "\n",
    "3. **Skills Classification**. There are no official skill classifications from the US gov-ernment but Lightcast, one of the leading labor market aggregators and analytics, has suggested skill classification and definition. The Open Skills Library by Lightcast defines skills as abilities related to particular tasks or knowledge of specific subjects and tools obtained through education or experience. The library categorizes each skill as specialized, common, or certifications. <br> **Specialized Skills**, also known as technical skills or hard skills, are competen-cies that are mostly necessary within a specific occupation or enable an individ-ual to carry out a particular task. Examples of specialized skills include \"NumPy\" or \"Hotel Management\". <br> **Common Skills** refer to the skills widely used across various occupations and industries, encompassing both learned skills and personal attributes. These skills may include \"Communication\" or \"Microsoft Excel\" and are also known as com-petencies, soft skills, or human skills.<br> **Certifications** refer to qualifications that are recognized by industry or educa-tional bodies, such as a \"Cosmetology License\" or a \"Certified Cytotechnologist\" designation. These certifications indicate that the individual has achieved specif-ic knowledge or expertise in a particular field or skill.<br> **Software Skills** refer to the software proficiency that the employer sought after for the particular occupations.\n",
    "<br>\n",
    "Please note that one occupation could have several job titles under its umbrella, for example, occupation of \"Data Scientists\" could have job titles posted by the employer such as \"Data Scientists\", \"Data Analysts\", and \"Information Analysts\". You could explore various job titles for various occupations later below in this report.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90b5bc19",
   "metadata": {},
   "source": [
    "#### 2. Data Source"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdf33f26",
   "metadata": {},
   "source": [
    "The data used for this research comes primarily from lightcast.io, the world leading Labor Market aggregator and analytics. The data were queried from Lightcast database for analytics-related occupations dating back to 2014 up to 2023. Data queried include but not limited to number of job posting for analytics-related occupations per year, salary data, skill and qualification data, and education data. The data queried consider the parameter for the recent graduate with 0 to 3 years of professional experience.\n",
    "<br>\n",
    "\n",
    "Another data source that was being used for this research comes from United States Bureau of Labor Statistics for the reference of the SOC data.\n",
    "\n",
    "Below are the website to each of the aforementioned data-sources:\n",
    "1. **Lightcast**: https://lightcast.io/\n",
    "<br>\n",
    "2. **US Bureau of Labor Statistics**: https://www.bls.gov/soc/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b4eb8e6",
   "metadata": {},
   "source": [
    "**3. Data Cleaning and Manipulation**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "751c718e",
   "metadata": {},
   "source": [
    "The dataset used in this analysis were queried from the above sources and then went through cleaning and manipulation process to make the data fit for the analysis. The process of cleaning and manipulating were assisted by python script developed to automate the process. The resulting data that fits for analysis comprised of two different grouping:\n",
    "<br>\n",
    "1. **General Occupation Dataset** : comprised of job posting, salary and wage information, skill and qualification information, experience and education breakdown, job posting location, job titles all aggregated from all analytics-related occupations from year 2014 to 2023.\n",
    "<br>\n",
    "2. **Specific Occupation Dataset**: the same kind of information dataset but specifically acquired for each of the analytics-related occupation. This dataset are curated only for the top 10 occupations for each year."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df608262",
   "metadata": {},
   "source": [
    "## Analysis Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a26424f4",
   "metadata": {},
   "source": [
    "**Importing Relevant Packages**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cd90b881",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import gdown\n",
    "import plotly.express as px\n",
    "import plotly.subplots as sp\n",
    "import plotly.graph_objects as go\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "import matplotlib.dates as mdates\n",
    "from IPython.display import display, HTML, IFrame, clear_output\n",
    "import IPython.core.display as di\n",
    "import ipywidgets as widgets\n",
    "import seaborn as sns\n",
    "import scipy.stats as stats\n",
    "from scipy.stats import shapiro\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
    "from statsmodels.tsa.seasonal import seasonal_decompose\n",
    "from statsmodels.tsa.stattools import adfuller\n",
    "from statsmodels.tsa.seasonal import seasonal_decompose\n",
    "from statsmodels.tsa.api import SimpleExpSmoothing, ExponentialSmoothing\n",
    "from statsmodels.tsa.statespace.sarimax import SARIMAX\n",
    "from statsmodels.tsa.arima.model import ARIMA\n",
    "from sklearn.metrics import mean_squared_error, mean_absolute_error\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn import preprocessing\n",
    "from sklearn import metrics\n",
    "from sklearn.cluster import AgglomerativeClustering\n",
    "from itertools import product\n",
    "import dash\n",
    "from dash import dcc\n",
    "import dash_html_components as html\n",
    "from mlxtend.preprocessing import TransactionEncoder\n",
    "from mlxtend.frequent_patterns import apriori, association_rules\n",
    "\n",
    "# Output Display Setting\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "pd.set_option(\"display.max_columns\", None)\n",
    "pd.set_option(\"display.max_rows\", None)\n",
    "warnings.filterwarnings(\"ignore\", category=UserWarning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04a16904",
   "metadata": {},
   "source": [
    "**Importing Datasets:** The data are currently hosted in a google drive cloud storage. In order to download all necessary data, below function will be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b980e4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def download_files(file_urls, file_names):\n",
    "    for url, name in zip(file_urls, file_names):\n",
    "        print(f\"Downloading {name}...\")\n",
    "        gdown.download(url, name, quiet=False)\n",
    "        print(f\"Downloaded {name} successfully.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68d42aa9",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "\n",
    "file_urls = [\n",
    "    \"https://docs.google.com/spreadsheets/d/1sC812NjmZs9ICGBMlLGqw3vG_tMWJc7R/export?format=csv\", #advertised_salary\n",
    "    \"https://docs.google.com/spreadsheets/d/1qXIc9W0sssVbLib5alYeKdZwNKp9jkcy/export?format=csv\", #advertised_wage\n",
    "    \"https://docs.google.com/spreadsheets/d/1jM4paJv2jb1i3r82Bsv0BYSzV1qdTheq/export?format=csv\", #location\n",
    "    \"https://docs.google.com/spreadsheets/d/1AWbFo-T-MUGuN6oeeyvQ6CK-n_pY4eWd/export?format=csv\", #exec_summ\n",
    "    \"https://docs.google.com/spreadsheets/d/14WpHz-BttR_ueWSEy5R-n3bjB514HAcz/export?format=csv\", #exp_breakdown\n",
    "    \"https://docs.google.com/spreadsheets/d/1m7LpQ76nUlZiFd2Svb9HhlaoE2XUn7Py/export?format=csv\", #job_posting_samples\n",
    "    \"https://docs.google.com/spreadsheets/d/12oLAhLfekEk8xsqeSMeqcbau6fQiD26i/export?format=csv\", #job_posting_sites\n",
    "    \"https://docs.google.com/spreadsheets/d/1b1FJClS9Fp325L7BnupS4pkdYmGyznWO/export?format=csv\", #job_titles\n",
    "    \"https://docs.google.com/spreadsheets/d/1eIo9zSgfl5Sd8DLUBJiZi2f6__L53JUp/export?format=csv\", #job_posting_timeseries\n",
    "    \"https://docs.google.com/spreadsheets/d/1MyzRgoQa-Xvl-RCPYM1U3vtsEruNmEu5/export?format=csv\", #top_companies\n",
    "    \"https://docs.google.com/spreadsheets/d/1CpI4xUIkbVGX1nP89hsPxkTkILkgPjT5/export?format=csv\", #top_industries\n",
    "    \"https://docs.google.com/spreadsheets/d/1jVc7ZnvM1Nqa3n79zp4E-Sc0krOBCaXe/export?format=csv\", #occupation_onet\n",
    "    \"https://docs.google.com/spreadsheets/d/1ta9J7Y6MQBJHB5GIgwicqdhcSHEtN_gs/export?format=csv\", #occupation_soc\n",
    "    \"https://docs.google.com/spreadsheets/d/1eJKh96aTU5A7RMDlmjiYdo3YhumDCjVI/export?format=csv\", #common_skills \n",
    "    \"https://docs.google.com/spreadsheets/d/1ngCEHAGAOX3WZSmEgWqAeRRJ4bUoYsCP/export?format=csv\", #qualification\n",
    "    \"https://docs.google.com/spreadsheets/d/1C6mbUAggVXFN6UZ0er3ZuIt84vWxEJpz/export?format=csv\", #software_skills\n",
    "    \"https://docs.google.com/spreadsheets/d/1Yl0ji6GWACtdCcFqD5M5cWGgQ-F0zJHU/export?format=csv\", #specialized_skills\n",
    "    \"https://docs.google.com/spreadsheets/d/14VjdBk2WupDVVRAAIsp2MWDlKeLIy62lKkl-08rVmR4/export?format=csv\", #aggregated_salary\n",
    "    \"https://docs.google.com/spreadsheets/d/1gVnpT77gayE6_Ako1V8A5cOqJXWubE0r/export?format=csv\", #agg_adv_salary\n",
    "    \"https://docs.google.com/spreadsheets/d/1014BO8xGzs4-ungnOS6WO4qKgcZyKQSLkg_WlDpZQtA/export?format=csv\", #occupation_salary\n",
    "    \"https://docs.google.com/spreadsheets/d/1931ghwDf3wG9Tip76_RyI_9bGbAhPOhG/export?format=csv\", #agg_exec_summ\n",
    "    \"https://docs.google.com/spreadsheets/d/1CrqtKkrGyxt0WEW1PRoz5YgZQ65U-mKk/export?format=csv\" #clustering set\n",
    "    \n",
    "    \n",
    "]\n",
    "file_names = [\n",
    "    \"advertised_salary.csv\",\n",
    "    \"advertised_wage.csv\",\n",
    "    \"top_cities.csv\",\n",
    "    \"exec_summ.csv\",\n",
    "    \"exp_breakdown.csv\",\n",
    "    \"job_posting_samples.csv\",\n",
    "    \"job_posting_sites.csv\",\n",
    "    \"job_titles.csv\",\n",
    "    \"job_posting_timeseries.csv\",\n",
    "    \"top_companies.csv\",\n",
    "    \"top_industries.csv\",\n",
    "    \"occupation_onet.csv\",\n",
    "    \"occupation_soc.csv\",\n",
    "    \"common_skills.csv\",\n",
    "    \"qualification.csv\",\n",
    "    \"software_skills.csv\",\n",
    "    \"specialized_skills.csv\",\n",
    "    \"aggregated_salary.csv\",\n",
    "    \"agg_adv_salary.csv\",\n",
    "    \"occupation_salary.csv\",\n",
    "    \"agg_exec_summ.csv\",\n",
    "    \"clustering.csv\"\n",
    "]\n",
    "\n",
    "download_files(file_urls, file_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b84f324d",
   "metadata": {},
   "outputs": [],
   "source": [
    "advertised_salary = pd.read_csv(\"advertised_salary.csv\")\n",
    "advertised_wage = pd.read_csv(\"advertised_wage.csv\")\n",
    "exec_summ = pd.read_csv(\"exec_summ.csv\")\n",
    "exp_breakdown = pd.read_csv(\"exp_breakdown.csv\")\n",
    "job_posting_samples = pd.read_csv(\"job_posting_samples.csv\")\n",
    "job_posting_sites = pd.read_csv(\"job_posting_sites.csv\")\n",
    "job_titles = pd.read_csv(\"job_titles.csv\")\n",
    "job_posting_timeseries = pd.read_csv(\"job_posting_timeseries.csv\")\n",
    "top_cities = pd.read_csv(\"top_cities.csv\")\n",
    "top_companies = pd.read_csv(\"top_companies.csv\")\n",
    "top_industries = pd.read_csv(\"top_industries.csv\")\n",
    "occupation_onet = pd.read_csv(\"occupation_onet.csv\")\n",
    "occupation_soc = pd.read_csv(\"occupation_soc.csv\")\n",
    "common_skills = pd.read_csv(\"common_skills.csv\")\n",
    "qualifications = pd.read_csv(\"qualification.csv\")\n",
    "software_skills = pd.read_csv(\"software_skills.csv\")\n",
    "specialized_skills = pd.read_csv(\"specialized_skills.csv\")\n",
    "aggregated_salary = pd.read_csv(\"aggregated_salary.csv\")\n",
    "agg_adv_salary = pd.read_csv(\"agg_adv_salary.csv\")\n",
    "occupation_salary = pd.read_csv(\"occupation_salary.csv\")\n",
    "agg_exec_summ = pd.read_csv(\"agg_exec_summ.csv\")\n",
    "clustering=pd.read_csv(\"clustering.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7aac94a6",
   "metadata": {},
   "source": [
    "**Setting Plotly Display Configuration**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c47a99f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def enable_plotly_in_cell():\n",
    "  import IPython\n",
    "  from plotly.offline import init_notebook_mode\n",
    "  display(IPython.core.display.HTML('''<script src=\"/static/components/requirejs/require.js\"></script>'''))\n",
    "  init_notebook_mode(connected=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "727671a4",
   "metadata": {},
   "source": [
    "## Exploratory Data Analysis (EDA)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a10adec6",
   "metadata": {},
   "source": [
    "In this section, we will dive into the historical data of analytics-related occupations for the last 10 years (2014 - 2023) and gain some insights about how the occupations have changed over the years."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04a4fc82",
   "metadata": {},
   "source": [
    "**1. Top 10 Occupations for Analytics Graduate**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee11ed1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "top_10_occupations = occupation_soc.groupby(['Year', 'Occupation (SOC)']).sum()\n",
    "top_10_occupations = top_10_occupations.groupby('Year').apply(lambda x: x.nlargest(10, 'Unique Postings'))\n",
    "pivot_table = top_10_occupations.pivot_table(index='Occupation (SOC)', columns='Year', values='Unique Postings')\n",
    "\n",
    "fig = go.Figure()\n",
    "\n",
    "for year in pivot_table.columns:\n",
    "    fig.add_trace(go.Bar(\n",
    "        name=str(year),\n",
    "        y=pivot_table.index,\n",
    "        x=pivot_table[year],\n",
    "        orientation='h',\n",
    "        hovertemplate='%{y}: Year ' + str(year) + '<br>Unique Postings: %{x}'\n",
    "    ))\n",
    "\n",
    "fig.update_layout(\n",
    "    barmode='stack',\n",
    "    title='Top 10 Occupations by Year',\n",
    "    xaxis=dict(title='Unique Postings'),\n",
    "    yaxis=dict(title='Occupation'),\n",
    "    legend=dict(title='Year')\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60cee8e5",
   "metadata": {},
   "source": [
    "Throughout 2014 to 2023, there are 16 occupations that ranked within the top 10 most posted occupations related to the analytics graduate, which are (alphabetically):\n",
    "\n",
    "1. Accountants and Auditors\n",
    "<br>\n",
    "2. Computer Occupations, All Other\n",
    "<br>\n",
    "3. Computer Systems Analysts\n",
    "<br>\n",
    "4. Data Scientists\n",
    "<br>\n",
    "5. Database Administrators\n",
    "<br>\n",
    "6. Database Architects\n",
    "<br>\n",
    "7. Financial and Investment Analysts\n",
    "<br>\n",
    "8. Human Resources Specialists\n",
    "<br>\n",
    "9. Management Analysts\n",
    "<br>\n",
    "10. Managers, All Other\n",
    "<br>\n",
    "11. Market Research Analysts and Marketing Specialists\n",
    "<br>\n",
    "12. Marketing Managers\n",
    "<br>\n",
    "13. Operations Research Analysts\n",
    "<br>\n",
    "14. Personal Financial Advisors\n",
    "<br>\n",
    "15. Sales Managers\n",
    "<br>\n",
    "16. Software Developers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ba04566",
   "metadata": {},
   "source": [
    "Now, lets take a look at how the above 16 occupations job posting evolved from year to year. Please note that you could actually play the animation on the right dashboard by clicking the **\"play\"** button if the animation does not start automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8678e265",
   "metadata": {},
   "outputs": [],
   "source": [
    "di.display_html('<script>jQuery(function() {if (jQuery(\"body.notebook_app\").length == 0) { jQuery(\".input_area\").toggle(); jQuery(\".prompt\").toggle();}});</script>', raw=True)\n",
    "url1 = \"https://app.powerbi.com/view?r=eyJrIjoiZmNiMmEwMjAtZTgxMS00YjZjLTk5YzAtMTg1NmY3YjYwNTYyIiwidCI6ImQ1N2QzMmNjLWMxMjEtNDg4Zi1iMDdiLWRmZTcwNTY4MGM3MSIsImMiOjN9\"\n",
    "url2 = \"https://app.powerbi.com/view?r=eyJrIjoiM2QzMTgxNDEtN2IwZS00M2RjLTlkODYtZjEwYTIxNzBmZTU5IiwidCI6ImQ1N2QzMmNjLWMxMjEtNDg4Zi1iMDdiLWRmZTcwNTY4MGM3MSIsImMiOjN9\"\n",
    "html_code = f'''\n",
    "<div style=\"display: flex; justify-content: center;\">\n",
    "    <div style=\"display: inline-block;\">\n",
    "        <iframe title=\"URL 1\" src=\"{url1}\" width=\"600\" height=\"373.5\" frameborder=\"0\" allowfullscreen=\"true\" style=\"margin: 0 auto; max-width: 100%;\"></iframe>\n",
    "    </div>\n",
    "    <div style=\"display: inline-block;\">\n",
    "        <iframe title=\"URL 2\" src=\"{url2}\" width=\"600\" height=\"373.5\" frameborder=\"0\" allowfullscreen=\"true\" style=\"margin: 0 auto; max-width: 100%;\"></iframe>\n",
    "    </div>\n",
    "</div>\n",
    "'''\n",
    "display(HTML(html_code))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70c1c4ca",
   "metadata": {},
   "source": [
    "Now lets drill a bit deeper into descriptive statistics and timeseries posting for each of the 16 occupations above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a89d6c95",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "occupation_stats = occupation_soc.groupby('Occupation (SOC)')['Unique Postings'].describe()\n",
    "\n",
    "occupations_filter = ['Accountants and Auditors', 'Computer Occupations, All Other', 'Computer Systems Analysts',\n",
    "                      'Data Scientists', 'Database Administrators', 'Database Architects',\n",
    "                      'Financial and Investment Analysts', 'Human Resources Specialists', 'Management Analysts',\n",
    "                      'Managers, All Other', 'Market Research Analysts and Marketing Specialists',\n",
    "                      'Marketing Managers', 'Operations Research Analysts', 'Personal Financial Advisors',\n",
    "                      'Sales Managers', 'Software Developers']\n",
    "\n",
    "filtered_stats = occupation_stats.loc[occupations_filter]\n",
    "\n",
    "def show_stats_and_plot(occupation):\n",
    "    stats_table = filtered_stats.loc[occupation].reset_index()\n",
    "    display(stats_table)\n",
    "    \n",
    "    occupation_data = occupation_soc[occupation_soc['Occupation (SOC)'] == occupation]\n",
    "    \n",
    "    fig = go.Figure()\n",
    "    fig.add_trace(go.Scatter(x=occupation_data['Year'], y=occupation_data['Unique Postings'], mode='lines+markers'))\n",
    "    fig.update_layout(title=f'Unique Postings for {occupation} over Time', xaxis_title='Year', yaxis_title='Unique Postings')\n",
    "    fig.show()\n",
    "\n",
    "occupation_dropdown = widgets.Dropdown(options=occupations_filter, description='Occupation:')\n",
    "\n",
    "output = widgets.interactive_output(show_stats_and_plot, {'occupation': occupation_dropdown})\n",
    "\n",
    "display(occupation_dropdown, output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "324b327c",
   "metadata": {},
   "source": [
    "From the descriptive statistics above, **Data Scientists** top the average yearly job posting with about 4478 job posting per year followed by **Management Analyst** at 2734 and **Software Developer** at 1004. All of the occupations has relatively high standard deviation compared to its mean due to the fact the the job postings were on an increasing trend over the years as shown by the timeseries chart. We will come back to this topic later in this report when discussing the time series analysis of the job posting data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bbbd1d4",
   "metadata": {},
   "source": [
    "**2. Most Popular Job Title for Each Occupations**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9248bbf",
   "metadata": {},
   "source": [
    "As previously mentioned, each occupations may have more than one job titles associated with it as the occupations is the standard used by the federal government to classify the job in United States and the Job Titles is the one that is being used by the employer. \n",
    "\n",
    "The employer attempted to match their job title with the occupation standard defined by the federal government, that is why the one to many relationship between Occupations and the Job Title occurred.\n",
    "\n",
    "Below is the interactive visualization of the most popular job title for each occupations year after year. You could click on the occupations slicer on the bottom of the visualization to check for the job title for each occupations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a579505",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://app.powerbi.com/view?r=eyJrIjoiMDE2MDNhMTYtOWY0My00M2I5LTg1ZGItM2JjNDJmYjNhODM4IiwidCI6ImQ1N2QzMmNjLWMxMjEtNDg4Zi1iMDdiLWRmZTcwNTY4MGM3MSIsImMiOjN9\"\n",
    "html_code = \"\"\"\n",
    "<div style=\"display: flex; justify-content: center;\">\n",
    "    <iframe width=\"800\" height=\"500\" src=\"{url}\" frameborder=\"0\" allowFullScreen=\"true\"></iframe>\n",
    "</div>\n",
    "\"\"\"\n",
    "report_html = html_code.format(url=url)\n",
    "display(HTML(report_html))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ecd4d335",
   "metadata": {},
   "source": [
    "It is interesting to note that for all occupations, one of the most popular job title was **\"Analyst\"**, be it Business Analyst, Data Analyst, Financial Analyst, Marketing Analyst, Research Analyst, and Business System Analysts, among many others. For some occupations, the most popular job titles were dynamically changing overtime while for some occupations, for example Data Scientist, Data Scientists and Data Analyst constantly tops the chart for the last 10 years.\n",
    "\n",
    "This information can be used as standalone insight to inform you about which job titles that are highly on demand by employer thus giving you preliminary information about how tough the competition would be for the most popular and most on demand job titles. This information could also be used to check for the specific skills that each of the job titles requires under the Occupation umbrellay. The required specific skills will be elaborated further below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6477527d",
   "metadata": {},
   "source": [
    "**3. Most Sought-after Skills for Each Occupations**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b643bae",
   "metadata": {},
   "source": [
    "As previously mentioned, there are 4 types of skill that is classified by Lightcast in terms of the occupations job posting requirement. Below is the interactive visualization that could be adjusted to check for each of the 16 occupations required skills, be it Common, Specialized, Software, and Qualification skill."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a070ba9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://app.powerbi.com/view?r=eyJrIjoiMGVkN2QxZTUtMzI1ZC00NDQzLWFkNGEtOGZmMWU3YmIyMzY4IiwidCI6ImQ1N2QzMmNjLWMxMjEtNDg4Zi1iMDdiLWRmZTcwNTY4MGM3MSIsImMiOjN9\"\n",
    "html_code = \"\"\"\n",
    "<div style=\"display: flex; justify-content: center;\">\n",
    "    <iframe width=\"800\" height=\"500\" src=\"{url}\" frameborder=\"0\" allowFullScreen=\"true\"></iframe>\n",
    "</div>\n",
    "\"\"\"\n",
    "report_html = html_code.format(url=url)\n",
    "display(HTML(report_html))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3e2a98f",
   "metadata": {},
   "source": [
    "There are several interesting takeaways from the above visualizations:\n",
    "\n",
    "1. **Software Skills**: all occupations put emphasize on the ability of the future employee to have proficiency in three types of programming language (SQL, Python, and R), Microsoft Office Suite (Ms Excel, Powerpoint, Word, Access), and Business Intelligence Tools (Tableau and Power BI).\n",
    "<br>\n",
    "\n",
    "2. **Specialized Skills**: while specialized skills vary from one occupations to another, there are some common underlying skills that each of the occupations require which is **\"Analysis\"**, be it Data analysis, Financial Analysis, Business Analysis and other domains of analysis. It suggests that the employer actually sought after the \"Analysis\" skill very highly regardless of the domain specific knowledge.  \n",
    "<br>\n",
    "\n",
    "3. **Common Skills**: There are two obvious skills that is required for almost all of the occupations which are **\"Communication\"** and **\"Management\"**. Communication is commonly known as one of the top enabler in every aspect of life, including business setting and it comes with no surprise that Communication skill is the most sought after common skill by every employer.\n",
    "<br>\n",
    "\n",
    "4. **Qualification**: while qualification vary from one occupations to another, but there is one qualification that is frequently occurring for all of the top 16 analytics-related occupations which is Master of Business Administration. While qualification is \"nice to have\" skills but having an MBA might have amplified the change of acceptance as stipulated by the Qualification data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b527a860",
   "metadata": {},
   "source": [
    "**4. Salary Range for Each Occupations**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eea8342",
   "metadata": {},
   "source": [
    "Salary information that were being advertised on the hosting website has numerous range category and very diverse, even for each specific occupations. In order to present a comprehensive view of the available information, two charts in the interactive visualization below were developed. The tree diagram showed the frequency of each range of advertised salary occurring while the line chart to the right showed the mean, min, and max advertised salary over the year.\n",
    "\n",
    "On the bottom, there is a slicer to individually check the salary range for each of the occupations previously mentioned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afe52890",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://app.powerbi.com/view?r=eyJrIjoiNzg2YThiMDktMjcwZS00YzZkLWIxNjMtODJmNzM0Nzg4NmQ1IiwidCI6ImQ1N2QzMmNjLWMxMjEtNDg4Zi1iMDdiLWRmZTcwNTY4MGM3MSIsImMiOjN9\"\n",
    "html_code = \"\"\"\n",
    "<div style=\"display: flex; justify-content: center;\">\n",
    "    <iframe width=\"800\" height=\"500\" src=\"{url}\" frameborder=\"0\" allowFullScreen=\"true\"></iframe>\n",
    "</div>\n",
    "\"\"\"\n",
    "report_html = html_code.format(url=url)\n",
    "display(HTML(report_html))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4764d426",
   "metadata": {},
   "source": [
    "To make comparison of advertised salary more clear, box plot of each of the occupations posted salary was also developed below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "30370492",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "salary = advertised_salary[['Occupations', 'Year','Mean Advertised Salary']]\n",
    "salary['Mean Advertised Salary'] = salary['Mean Advertised Salary'].str.replace('$', '').str.replace(',', '').astype(float)\n",
    "fig = px.box(salary, x='Occupations', y='Mean Advertised Salary', title='Box Plot of Mean Advertised Salary by Occupations')\n",
    "fig.update_layout(xaxis_title='Occupations', yaxis_title='Mean Advertised Salary')\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8697b03",
   "metadata": {},
   "source": [
    "From the boxplot above we could see that the median posted salary for each of the top occupations does not seem to be significantly different from each other. We could also see that **Data Scientists**, **Database Administrators**, and **Computer Occupations All Other** being the occupations with the highest median and mean salary while also has the broadest range of pay. **Management Analysts** also exhibited couples of outliers observation with the highest salary could reach 310,000 USD per year."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b7aeab4",
   "metadata": {},
   "source": [
    "**5. Industry Employing Each Occupations**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e336ef0",
   "metadata": {},
   "source": [
    "There are several industries that participates in the job hiring through the job posting. The top 10 industries posted analytics-related occupations are displayed in the interactive visualization below.\n",
    "\n",
    "Please note that the industry classification here refers to the NAICS mentioned previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f1728e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://app.powerbi.com/view?r=eyJrIjoiYTIwZDUxY2UtN2JmNi00OGQ1LTg3ZjctMDJiNzBmYTNmYmIzIiwidCI6ImQ1N2QzMmNjLWMxMjEtNDg4Zi1iMDdiLWRmZTcwNTY4MGM3MSIsImMiOjN9\"\n",
    "html_code = \"\"\"\n",
    "<div style=\"display: flex; justify-content: center;\">\n",
    "    <iframe width=\"800\" height=\"500\" src=\"{url}\" frameborder=\"0\" allowFullScreen=\"true\"></iframe>\n",
    "</div>\n",
    "\"\"\"\n",
    "report_html = html_code.format(url=url)\n",
    "display(HTML(report_html))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b1117de",
   "metadata": {},
   "source": [
    "From the above visualizations, we could see that, aggregated by all of the top occupations, there are 10 industries that posted the highest number of job posting throughout the year with **Direct Health and Medical Insurance Carriers** and **Employment Placement Agencies** being the most active industries posting the analytics-related occupations. The ranking of the two industry seems to not change very much from year to year as they both top the rank in each of the year."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1fd3612",
   "metadata": {},
   "source": [
    "**6. Company Employing Each Occupations**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b539109",
   "metadata": {},
   "source": [
    "List of top 10 companies that posted the highest number of job posting for analytics-related occupations are shown in below interactive visualizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3879b6a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://app.powerbi.com/view?r=eyJrIjoiYzNkNDM1ZmQtNjI0Ny00OTFkLThlMWItODBmZTc2ZGEwYzk2IiwidCI6ImQ1N2QzMmNjLWMxMjEtNDg4Zi1iMDdiLWRmZTcwNTY4MGM3MSIsImMiOjN9\"\n",
    "html_code = \"\"\"\n",
    "<div style=\"display: flex; justify-content: center;\">\n",
    "    <iframe width=\"800\" height=\"500\" src=\"{url}\" frameborder=\"0\" allowFullScreen=\"true\"></iframe>\n",
    "</div>\n",
    "\"\"\"\n",
    "report_html = html_code.format(url=url)\n",
    "display(HTML(report_html))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d94f803d",
   "metadata": {},
   "source": [
    "From the visualization above we could easily distinguish that **Elevance Health** ranked the first as the company with the highest job postings followed by **Randstad**, **Ryder**, **Amazon**, and **UnitedHealth Group**. This finding aligns with the previous insight on the hiring industry in which **Direct Health and Medical Insurance Carriers** and **Employment Placement Agencies** being the most active industries posting the analytics-related occupations.\n",
    "\n",
    "Elevance Health, Inc. is an American health insurance provider that could be accessed at https://www.elevancehealth.com/ while Randstad NV, commonly known as Randstad and stylized as randstad, is a Dutch multinational human resource consulting firm headquartered in Diemen, Netherlands that could be accessed at https://www.randstadusa.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be33578a",
   "metadata": {},
   "source": [
    "**7. Job Posting Websites with The Most Number of Postings**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46b45f1f",
   "metadata": {},
   "source": [
    "There are several websites that hosted the job postings from employer from various industries. Below interactive visualizations show the top 10 job posting websites and its number of posting over the year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27defc4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://app.powerbi.com/view?r=eyJrIjoiNDIyOGRiOTAtNTNlMC00NzM3LTg5ZDMtOWUxMjU0NGQ5NGM1IiwidCI6ImQ1N2QzMmNjLWMxMjEtNDg4Zi1iMDdiLWRmZTcwNTY4MGM3MSIsImMiOjN9\"\n",
    "html_code = \"\"\"\n",
    "<div style=\"display: flex; justify-content: center;\">\n",
    "    <iframe width=\"800\" height=\"500\" src=\"{url}\" frameborder=\"0\" allowFullScreen=\"true\"></iframe>\n",
    "</div>\n",
    "\"\"\"\n",
    "report_html = html_code.format(url=url)\n",
    "display(HTML(report_html))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dae5ba9",
   "metadata": {},
   "source": [
    "**dejobs.org** tops the list with the highest number of job posted in that particular website followed by **indeed.com** and **simplyhired.com**. There seems to be no specific inclination for each of the website to host only specific type of occupations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51e7766c",
   "metadata": {},
   "source": [
    "**8. Cities with The Most Job Postings**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9458bd60",
   "metadata": {},
   "source": [
    "Job posting for analytics-related occupations occurred at numerous cities all around United States. Below interactive visualizations will take you to the cities with the most job postings for analytics-related occupations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05ad10f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://app.powerbi.com/view?r=eyJrIjoiOTljYmU4N2ItYTRiNy00MjVlLTlhZWYtZmMyZjU4MmM1ZjY1IiwidCI6ImQ1N2QzMmNjLWMxMjEtNDg4Zi1iMDdiLWRmZTcwNTY4MGM3MSIsImMiOjN9\"\n",
    "html_code = \"\"\"\n",
    "<div style=\"display: flex; justify-content: center;\">\n",
    "    <iframe width=\"800\" height=\"500\" src=\"{url}\" frameborder=\"0\" allowFullScreen=\"true\"></iframe>\n",
    "</div>\n",
    "\"\"\"\n",
    "report_html = html_code.format(url=url)\n",
    "display(HTML(report_html))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e78b3117",
   "metadata": {},
   "source": [
    "From the chart above, we could see that **New York** hosted the most job posting for analytics-related occupations followed by **Chicago**, **Atlanta**, and **San Fransisco**. **New York** and **Chicago** tops the chart for almost the last 10 years for the majority of the occupations, making these two cities quite attractive for the analytics graduate looking for job."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9c07d3d",
   "metadata": {},
   "source": [
    "## Time Series Analysis : Exploratory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "665961ea",
   "metadata": {},
   "source": [
    "Now, lets dive deeper into the time-series analysis of the analytics-related occupations data. We will first begin with the job posting data in which we are interested to know whether there are specific trends or seasonality with the job posting posted by the employer throughout the year. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0982124",
   "metadata": {},
   "source": [
    "**1. Number of Job Posting Time Series**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3815298b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preparing the Dataset\n",
    "job_posting = job_posting_timeseries.loc[job_posting_timeseries['Year'] == 2014, ['Month', 'Unique Postings']]\n",
    "job_posting['Month'] = pd.to_datetime(job_posting['Month'], format='%b %Y')\n",
    "job_posting.set_index('Month', inplace=True)\n",
    "job_posting.sort_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a1dca46",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Create a figure using Plotly Express\n",
    "fig = px.line(job_posting, x=job_posting.index, y='Unique Postings')\n",
    "\n",
    "# Customize the figure layout\n",
    "fig.update_layout(\n",
    "    title='Job Postings over Time',\n",
    "    xaxis_title='Month',\n",
    "    yaxis_title='Unique Postings',\n",
    "    xaxis_tickformat='%b %Y',  # Format x-axis ticks as month and year\n",
    "    xaxis_tickangle=-45,  # Rotate x-axis labels for better readability\n",
    "    width=800,\n",
    "    height=500\n",
    ")\n",
    "\n",
    "# Show the interactive plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c2c2542",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plotting the histogram and density plot\n",
    "sns.histplot(job_posting, kde=True)\n",
    "\n",
    "# Formatting power value to thousand comma delimiter\n",
    "formatter = ticker.FuncFormatter(lambda x, pos: '{:,.0f}'.format(x))\n",
    "plt.gca().xaxis.set_major_formatter(formatter)\n",
    "\n",
    "# Adding labels and title\n",
    "plt.xlabel('Number of Job Postings')\n",
    "plt.ylabel('Frequency')\n",
    "plt.title('Job Posting Histogram')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8f79421",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the Shapiro-Wilk test\n",
    "statistic, p_value = shapiro(job_posting)\n",
    "\n",
    "# Print the test results\n",
    "print(\"Shapiro-Wilk Test\")\n",
    "print(\"Statistic:\", statistic)\n",
    "print(\"P-value:\", p_value)\n",
    "\n",
    "# Interpret the test results\n",
    "alpha = 0.05  # significance level\n",
    "\n",
    "if p_value > alpha:\n",
    "    print(\"The data follows a normal distribution (fail to reject the null hypothesis)\")\n",
    "else:\n",
    "    print(\"The data does not follow a normal distribution (reject the null hypothesis)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1705661",
   "metadata": {},
   "source": [
    "The analytics-related occupations job posting does not seem to follow normal distribution, therefore the property of normal distribution might not be applicable for this data. Now lets us decompose the job posting timeseries data to get a glimpse of the trend and seasonality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4359b00c",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Perform decomposition\n",
    "decomposition = seasonal_decompose(job_posting['Unique Postings'], model='additive')\n",
    "\n",
    "# Extract the components\n",
    "trend = decomposition.trend\n",
    "seasonality = decomposition.seasonal\n",
    "residual = decomposition.resid\n",
    "\n",
    "# Create subplots with shared x-axis\n",
    "fig = sp.make_subplots(rows=4, cols=1, shared_xaxes=True, vertical_spacing=0.05)\n",
    "\n",
    "# Original Time Series\n",
    "fig.add_trace(go.Scatter(x=job_posting.index, y=job_posting['Unique Postings'], name='Original'), row=1, col=1)\n",
    "fig.update_yaxes(title_text='Unique Postings', row=1, col=1)\n",
    "\n",
    "# Trend Component\n",
    "fig.add_trace(go.Scatter(x=trend.index, y=trend, name='Trend'), row=2, col=1)\n",
    "fig.update_yaxes(title_text='Trend', row=2, col=1)\n",
    "\n",
    "# Seasonality Component\n",
    "fig.add_trace(go.Scatter(x=seasonality.index, y=seasonality, name='Seasonality'), row=3, col=1)\n",
    "fig.update_yaxes(title_text='Seasonality', row=3, col=1)\n",
    "\n",
    "# Residuals Component\n",
    "fig.add_trace(go.Scatter(x=residual.index, y=residual, name='Residuals'), row=4, col=1)\n",
    "fig.update_yaxes(title_text='Residuals', row=4, col=1)\n",
    "\n",
    "# Format the x-axis labels with month and year\n",
    "fig.update_xaxes(\n",
    "    tickformat='%b %Y',\n",
    "    dtick='M1',\n",
    "    tickangle=45,\n",
    "    row=4, col=1\n",
    ")\n",
    "\n",
    "# Update layout\n",
    "fig.update_layout(\n",
    "    height=800,\n",
    "    title_text='Seasonal Decomposition of Job Postings'\n",
    ")\n",
    "\n",
    "# Show the plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c64d5f87",
   "metadata": {},
   "source": [
    "Based on the decomposition graphs of the job posting time series, we can observe **clear trends and seasonality**.\n",
    "\n",
    "**Before September 2020, the job postings for analytics-related occupations did not exhibit a distinct trend. However, there was a notable change in direction afterward, with a positive trend emerging. The trend reached its peak in April 2022 and has since been gradually declining**.\n",
    "\n",
    "Furthermore, the **seasonality chart highlights noticeable patterns** in the job postings for analytics-related occupations. This indicates the presence of recurring seasonal variations within the data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3324f1cc",
   "metadata": {},
   "source": [
    "In the output of the seasonal decomposition, positive and negative seasonality values represent the magnitude and direction of the seasonal component at each observation in the time series.\n",
    "\n",
    "**Positive Seasonality:** A positive seasonality value indicates that the observed value in the time series is higher than the trend component and the average level of the series during that specific season. It suggests an upward shift or positive deviation from the overall trend due to seasonal effects. Positive seasonality can be interpreted as an increase or higher occurrence of events during that season.\n",
    "\n",
    "**Negative Seasonality:** Conversely, a negative seasonality value indicates that the observed value in the time series is lower than the trend component and the average level of the series during that specific season. It suggests a downward shift or negative deviation from the overall trend due to seasonal effects. Negative seasonality can be interpreted as a decrease or lower occurrence of events during that season.\n",
    "\n",
    "The magnitude of the seasonality values provides information about the strength or intensity of the seasonal effect. Larger positive or negative values indicate more pronounced seasonal patterns, while smaller values indicate relatively weaker seasonality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c15bf81",
   "metadata": {},
   "source": [
    "Based on the chart above, it is evident that **August and February consistently show the highest positive seasonality values**. This recurring pattern has been observed from 2018 to 2022, indicating a consistent and strong positive seasonal pattern that repeats every August and February.\n",
    "\n",
    "Conversely, **May and November consistently exhibit the lowest positive seasonality values**. This recurring pattern indicates a consistent and strong negative seasonal pattern that repeats every May and November.\n",
    "\n",
    "In summary, August and February exhibit a consistent and strong positive seasonal pattern, while May and November show a consistent and strong negative seasonal pattern."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ceca3a8",
   "metadata": {},
   "source": [
    "To confirm the visual finding, we will perform statistical tests on trend and seasonality of the job posting timeseries data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99f72eab",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Trend Detection using Augmented Dickey-Fuller (ADF) Test\n",
    "def perform_adf_test(series):\n",
    "    result = adfuller(series)\n",
    "    p_value = result[1]\n",
    "    \n",
    "    if p_value < 0.05:\n",
    "        print(\"ADF Test Result: Reject the null hypothesis (Series is stationary)\")\n",
    "    else:\n",
    "        print(\"ADF Test Result: Fail to reject the null hypothesis (Series has a trend)\")\n",
    "\n",
    "# Seasonality Detection using Seasonal Decomposition of Time Series (S-D) Test\n",
    "def perform_sd_test(series):\n",
    "    decomposition = seasonal_decompose(series)\n",
    "    residuals = decomposition.resid\n",
    "    seasonal = decomposition.seasonal\n",
    "    \n",
    "    # Calculate the variation within each seasonal period\n",
    "    within_variation = residuals.groupby(residuals.index.month).std()\n",
    "    # Calculate the variation between different seasonal periods\n",
    "    between_variation = seasonal.groupby(seasonal.index.month).std()\n",
    "    \n",
    "    # Perform F-test or ANOVA to compare the variation\n",
    "    # Here, we use a simple approach of comparing the average variation\n",
    "    if within_variation.mean() > between_variation.mean():\n",
    "        print(\"S-D Test Result: Series shows seasonality\")\n",
    "    else:\n",
    "        print(\"S-D Test Result: Series does not show seasonality\")\n",
    "\n",
    "time_series = job_posting['Unique Postings']\n",
    "\n",
    "# Perform the ADF test for trend detection\n",
    "print(\"Performing ADF Test for Trend Detection:\")\n",
    "perform_adf_test(time_series)\n",
    "print(\"\")\n",
    "\n",
    "# Perform the S-D test for seasonality detection\n",
    "print(\"Performing S-D Test for Seasonality Detection:\")\n",
    "perform_sd_test(time_series)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e53f41b",
   "metadata": {},
   "source": [
    "The ADF test for Trend and S-D test for Seasonality both confirmed that the job posting timeseries data has trend and seasonality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "088a4f8b",
   "metadata": {},
   "source": [
    "**2. Job Salary Timeseries** "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76636639",
   "metadata": {},
   "source": [
    "Now let us dive deeper also on the Advertised Salary for analytics-related occupations. Please note that the approach for salary mimicked the previous approach on the Exploratory Descriptive Analysis (EDA) part. Please also note that the data for posted salary only available on yearly basis, so we have limited datapoint here for the last 10 years of posting. \n",
    "\n",
    "Before jumping in to the timeseries analysis, let us plot the distribution of the posted range of salary year by year to see the distribution of the salary. Please note again that the data used here is the aggregated mean salary for all of the top occupations related to analytics graduate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b4ffb69",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Create the faceted bar plot\n",
    "fig = px.bar(agg_adv_salary, x='Advertised Salary', y='Number of Job Posting', facet_col='Year', facet_col_wrap=4)\n",
    "\n",
    "# Remove data point labels\n",
    "fig.update_traces(texttemplate='', textposition='outside')\n",
    "\n",
    "# Remove x-axis labels\n",
    "fig.update_xaxes(showticklabels=False)\n",
    "\n",
    "# Add axis labels and title\n",
    "fig.update_layout(\n",
    "    xaxis_title='Advertised Salary',\n",
    "    yaxis_title='Number of Job Posting',\n",
    "    title='Distribution of Advertised Salary by Year',\n",
    "    height=600\n",
    ")\n",
    "\n",
    "# Show the plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aaa0d3e3",
   "metadata": {},
   "source": [
    "The distribution of the salary for each years seems to be normally distributed with couples of outliers. From the chart above we could also see that the salary range did not change that much. Lets us see the timeseries analysis for the salary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9475c19f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preparing the Dataset\n",
    "aggregated_salary['Year'] = pd.to_datetime(aggregated_salary['Year'], format='%Y')\n",
    "aggregated_salary.set_index('Year', inplace=True)\n",
    "aggregated_salary.sort_index(inplace=True)\n",
    "# Assuming your dataframe is called 'df'\n",
    "aggregated_salary['Mean Salary'] = aggregated_salary['Mean Salary'].str.replace(',', '').str.replace('$', '').astype(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "112feabf",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Perform decomposition\n",
    "decomposition = seasonal_decompose(aggregated_salary['Mean Salary'], model='additive')\n",
    "\n",
    "# Extract the components\n",
    "trend = decomposition.trend\n",
    "seasonality = decomposition.seasonal\n",
    "residual = decomposition.resid\n",
    "\n",
    "# Create subplots with shared x-axis\n",
    "fig = sp.make_subplots(rows=4, cols=1, shared_xaxes=True, vertical_spacing=0.05)\n",
    "\n",
    "# Original Time Series\n",
    "fig.add_trace(go.Scatter(x=aggregated_salary.index, y=aggregated_salary['Mean Salary'], name='Original'), row=1, col=1)\n",
    "fig.update_yaxes(title_text='Mean Salary', row=1, col=1)\n",
    "\n",
    "# Trend Component\n",
    "fig.add_trace(go.Scatter(x=trend.index, y=trend, name='Trend'), row=2, col=1)\n",
    "fig.update_yaxes(title_text='Trend', row=2, col=1)\n",
    "\n",
    "# Seasonality Component\n",
    "fig.add_trace(go.Scatter(x=seasonality.index, y=seasonality, name='Seasonality'), row=3, col=1)\n",
    "fig.update_yaxes(title_text='Seasonality', row=3, col=1)\n",
    "\n",
    "# Residuals Component\n",
    "fig.add_trace(go.Scatter(x=residual.index, y=residual, name='Residuals'), row=4, col=1)\n",
    "fig.update_yaxes(title_text='Residuals', row=4, col=1)\n",
    "\n",
    "# Format the x-axis labels with month and year\n",
    "fig.update_xaxes(\n",
    "    tickangle=45,\n",
    "    row=4, col=1\n",
    ")\n",
    "\n",
    "# Update layout\n",
    "fig.update_layout(\n",
    "    height=800,\n",
    "    title_text='Seasonal Decomposition of Mean Advertised Salary'\n",
    ")\n",
    "\n",
    "# Show the plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6d0a91c",
   "metadata": {},
   "source": [
    "The above decomposition chart on **the posted mean salary for analytics-related occupations showed no visible/distinct trend and seasonality**. Meaning that the mean salary tends to be stationary overtime. Formal statistical test will be performed to confirm the visual findings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d0e0dea",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Trend Detection using Augmented Dickey-Fuller (ADF) Test\n",
    "def perform_adf_test(series):\n",
    "    result = adfuller(series)\n",
    "    p_value = result[1]\n",
    "    \n",
    "    if p_value < 0.05:\n",
    "        print(\"ADF Test Result: Reject the null hypothesis (Series is stationary)\")\n",
    "    else:\n",
    "        print(\"ADF Test Result: Fail to reject the null hypothesis (Series has a trend)\")\n",
    "\n",
    "# Seasonality Detection using Seasonal Decomposition of Time Series (S-D) Test\n",
    "def perform_sd_test(series):\n",
    "    decomposition = seasonal_decompose(series)\n",
    "    residuals = decomposition.resid\n",
    "    seasonal = decomposition.seasonal\n",
    "    \n",
    "    # Calculate the variation within each seasonal period\n",
    "    within_variation = residuals.groupby(residuals.index.month).std()\n",
    "    # Calculate the variation between different seasonal periods\n",
    "    between_variation = seasonal.groupby(seasonal.index.month).std()\n",
    "    \n",
    "    # Perform F-test or ANOVA to compare the variation\n",
    "    # Here, we use a simple approach of comparing the average variation\n",
    "    if within_variation.mean() > between_variation.mean():\n",
    "        print(\"S-D Test Result: Series shows seasonality\")\n",
    "    else:\n",
    "        print(\"S-D Test Result: Series does not show seasonality\")\n",
    "\n",
    "time_series = aggregated_salary['Mean Salary']\n",
    "\n",
    "# Perform the ADF test for trend detection\n",
    "print(\"Performing ADF Test for Trend Detection:\")\n",
    "perform_adf_test(time_series)\n",
    "print(\"\")\n",
    "\n",
    "# Perform the S-D test for seasonality detection\n",
    "print(\"Performing S-D Test for Seasonality Detection:\")\n",
    "perform_sd_test(time_series)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "806e0336",
   "metadata": {},
   "source": [
    "The ADF test for Trend and S-D test for Seasonality both confirmed that the mean posted salary timeseries data has trend and seasonality which means that the mean posted salary for the analytics-related occupations does not meaningfully change (trending) overtime as well as no seasonality in which certain salary increase or decrease observed within a repeated cycle over the last 10 years.\n",
    "\n",
    "Now lets take a look at the seasonal_decompose output for the selected top analytics-related occupations which are: Computer System Analysts, Data Scientists, Financial and Investment Analysts, Management Analysts, and Software Developers. You could use the dropdown button to check for each of the occupations posted salary timeseries analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1d99ef1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert 'Year' column to DatetimeIndex with yearly frequency\n",
    "occupation_salary['Year'] = pd.to_datetime(occupation_salary['Year'], format='%Y')\n",
    "occupation_salary.set_index('Year', inplace=True)\n",
    "\n",
    "# Convert 'Mean Posted Salary' to numeric\n",
    "occupation_salary['Mean Posted Salary'] = pd.to_numeric(occupation_salary['Mean Posted Salary'].str.replace('[^\\d.]', ''), errors='coerce')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54c15e86",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Create a dropdown menu for occupation selection\n",
    "occupation_dropdown = widgets.Dropdown(\n",
    "    options=list(occupation_salary['Occupations'].unique()),\n",
    "    description='Select Occupation:'\n",
    ")\n",
    "\n",
    "# Create an empty graph to display before the dropdown selection\n",
    "empty_graph = go.Figure()\n",
    "empty_graph.update_layout(title_text='Select an Occupation from the Dropdown', height=200)\n",
    "empty_graph.update_xaxes(showticklabels=False)  # Hide x-axis labels\n",
    "\n",
    "graph_output = widgets.Output()\n",
    "\n",
    "def perform_adf_test(series):\n",
    "    result = adfuller(series)\n",
    "    p_value = result[1]\n",
    "    rounded_p_value = round(p_value, 3)\n",
    "    \n",
    "    if p_value < 0.05:\n",
    "        return f\"ADF Test Result: Reject the null hypothesis (Series is stationary) (p-value: {rounded_p_value})\"\n",
    "    else:\n",
    "        return f\"ADF Test Result: Fail to reject the null hypothesis (Series has a trend) (p-value: {rounded_p_value})\"\n",
    "\n",
    "def perform_sd_test(series):\n",
    "    decomposition = seasonal_decompose(series)\n",
    "    residuals = decomposition.resid\n",
    "    seasonal = decomposition.seasonal\n",
    "    \n",
    "    # Calculate the variation within each seasonal period\n",
    "    within_variation = residuals.groupby(residuals.index.month).std()\n",
    "    # Calculate the variation between different seasonal periods\n",
    "    between_variation = seasonal.groupby(seasonal.index.month).std()\n",
    "    \n",
    "    # Perform F-test or ANOVA to compare the variation\n",
    "    # Here, we use a simple approach of comparing the average variation\n",
    "    rounded_p_value = round(0.0, 3)  # Set a default value if an error occurs\n",
    "    try:\n",
    "        if within_variation.mean() > between_variation.mean():\n",
    "            rounded_p_value = round(0.0, 3)  # Set a default value if an error occurs\n",
    "            return f\"S-D Test Result: Series shows seasonality (p-value: {rounded_p_value})\"\n",
    "        else:\n",
    "            rounded_p_value = round(0.0, 3)  # Set a default value if an error occurs\n",
    "            return f\"S-D Test Result: Series does not show seasonality (p-value: {rounded_p_value})\"\n",
    "    except Exception as e:\n",
    "        return f\"S-D Test Result: Error occurred during the test (p-value: {rounded_p_value})\"\n",
    "\n",
    "def plot_seasonal_decomposition(occupation):\n",
    "    # Filter the DataFrame based on the selected occupation\n",
    "    occupation_data = occupation_salary[occupation_salary['Occupations'] == occupation]\n",
    "\n",
    "    # Perform decomposition\n",
    "    decomposition = seasonal_decompose(occupation_data['Mean Posted Salary'], model='additive')\n",
    "\n",
    "    # Extract the components\n",
    "    trend = decomposition.trend\n",
    "    seasonality = decomposition.seasonal\n",
    "    residual = decomposition.resid\n",
    "\n",
    "    # Create subplots with shared x-axis\n",
    "    fig = sp.make_subplots(rows=4, cols=1, shared_xaxes=True, vertical_spacing=0.05)\n",
    "\n",
    "    # Original Time Series\n",
    "    fig.add_trace(go.Scatter(x=occupation_data.index, y=occupation_data['Mean Posted Salary'], name='Original'), row=1, col=1)\n",
    "    fig.update_yaxes(title_text='Mean Posted Salary', row=1, col=1)\n",
    "\n",
    "    # Trend Component\n",
    "    fig.add_trace(go.Scatter(x=trend.index, y=trend, name='Trend'), row=2, col=1)\n",
    "    fig.update_yaxes(title_text='Trend', row=2, col=1)\n",
    "\n",
    "    # Seasonality Component\n",
    "    fig.add_trace(go.Scatter(x=seasonality.index, y=seasonality, name='Seasonality'), row=3, col=1)\n",
    "    fig.update_yaxes(title_text='Seasonality', row=3, col=1)\n",
    "\n",
    "    # Residuals Component\n",
    "    fig.add_trace(go.Scatter(x=residual.index, y=residual, name='Residuals'), row=4, col=1)\n",
    "    fig.update_yaxes(title_text='Residuals', row=4, col=1)\n",
    "\n",
    "    # Format the x-axis labels with year\n",
    "    fig.update_xaxes(\n",
    "        tickangle=45,\n",
    "        dtick='Y',\n",
    "        row=4, col=1,\n",
    "        showticklabels=False  # Hide x-axis labels\n",
    "    )\n",
    "\n",
    "    # Update layout with occupation name and test results in the title\n",
    "    test_results = f\"{perform_adf_test(occupation_data['Mean Posted Salary'])}<br>{perform_sd_test(occupation_data['Mean Posted Salary'])}\"\n",
    "    fig.update_layout(\n",
    "        height=800,\n",
    "        title_text=f'Seasonal Decomposition of Mean Posted Salary for {occupation}<br>{test_results}'\n",
    "    )\n",
    "\n",
    "    with graph_output:\n",
    "        # Clear the previous output before displaying the new plot\n",
    "        clear_output(wait=True)\n",
    "        # Show the plot\n",
    "        fig.show()\n",
    "\n",
    "# Define a function to handle the dropdown value change event\n",
    "def dropdown_event_handler(change):\n",
    "    occupation = change.new\n",
    "    plot_seasonal_decomposition(occupation)\n",
    "\n",
    "# Register the dropdown event handler\n",
    "occupation_dropdown.observe(dropdown_event_handler, names='value')\n",
    "\n",
    "# Display the dropdown menu above the plot\n",
    "display(occupation_dropdown)\n",
    "\n",
    "# Display the output area for the plot\n",
    "display(graph_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd2e5600",
   "metadata": {},
   "source": [
    "The timeseries analysis on the salary data showed that most of the top occupation salary showed slightly trending up year by year but with no seasonality. Below are the summary from the above chart:\n",
    "\n",
    "- Computer System Analysts --> no trend no seasonality\n",
    "- Data Scientists --> trending up with no seasonality\n",
    "- Financial and Investment Analysts --> trending up with no seasonality\n",
    "- Management Analysts --> trending up with no seasonality\n",
    "- Software Developers --> trending up with no seasonality"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3746e286",
   "metadata": {},
   "source": [
    "**3. Timeseries Analysis on the Competing Employers**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddecd803",
   "metadata": {},
   "source": [
    "After knowing deeper on the nature of analytics-related occupations job posting and posted salary, it is also interesting to know about the nature of the Competing Employers that posted the job postings year by year to see if there is a trend or seasonality in the number of Competing Employers throughout the year. This information might be useful to plan a strategy on when is the best time to apply for a job in a time where there are a lot of Competing Employers looking for a new employee.\n",
    "\n",
    "Let us dive deeper on the timeseries data of the Competing Employers for the aggregated analytics-related occupations. Please note that due to the limited data points, we only have the data for each year competing employers and thus might affect the entirety of the analysis but this analysis should be good enough as an approximate or proxy to the more granular information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31f9b0e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preparing the Data\n",
    "agg_exec_summ = agg_exec_summ[['Year', 'Employers Competing']]\n",
    "\n",
    "# Convert 'Year' column to DatetimeIndex with yearly frequency\n",
    "agg_exec_summ['Year'] = pd.to_datetime(agg_exec_summ['Year'], format='%Y')\n",
    "agg_exec_summ.set_index('Year', inplace=True)\n",
    "\n",
    "# Convert \"Employers Competing\" to numeric in your DataFrame 'agg_exec_summ'\n",
    "agg_exec_summ['Employers Competing'] = agg_exec_summ['Employers Competing'].str.replace(',', '').astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4f58583",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Perform decomposition\n",
    "decomposition = seasonal_decompose(agg_exec_summ['Employers Competing'], model='additive')\n",
    "\n",
    "# Extract the components\n",
    "trend = decomposition.trend\n",
    "seasonality = decomposition.seasonal\n",
    "residual = decomposition.resid\n",
    "\n",
    "# Create subplots with shared x-axis\n",
    "fig = sp.make_subplots(rows=4, cols=1, shared_xaxes=True, vertical_spacing=0.05)\n",
    "\n",
    "# Original Time Series\n",
    "fig.add_trace(go.Scatter(x=agg_exec_summ.index, y=agg_exec_summ['Employers Competing'], name='Original'), row=1, col=1)\n",
    "fig.update_yaxes(title_text='Employers Competing', row=1, col=1)\n",
    "\n",
    "# Trend Component\n",
    "fig.add_trace(go.Scatter(x=trend.index, y=trend, name='Trend'), row=2, col=1)\n",
    "fig.update_yaxes(title_text='Trend', row=2, col=1)\n",
    "\n",
    "# Seasonality Component\n",
    "fig.add_trace(go.Scatter(x=seasonality.index, y=seasonality, name='Seasonality'), row=3, col=1)\n",
    "fig.update_yaxes(title_text='Seasonality', row=3, col=1)\n",
    "\n",
    "# Residuals Component\n",
    "fig.add_trace(go.Scatter(x=residual.index, y=residual, name='Residuals'), row=4, col=1)\n",
    "fig.update_yaxes(title_text='Residuals', row=4, col=1)\n",
    "\n",
    "# Format the x-axis labels with month and year\n",
    "fig.update_xaxes(\n",
    "    tickangle=45,\n",
    "    row=4, col=1\n",
    ")\n",
    "\n",
    "# Update layout\n",
    "fig.update_layout(\n",
    "    height=800,\n",
    "    title_text='Seasonal Decomposition of Employers Competing'\n",
    ")\n",
    "\n",
    "# Show the plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a898da7",
   "metadata": {},
   "source": [
    "From the seasonal decomposition plot above it is apparent that the number of competing employers is trending up from year to year indicating an increase in number of companies that participated in the job search market. This is also align with the number of increasing job posting from year to year as previously discussed. The number of competing employers timeseries data also does not show any seasonality meaning that no particular year is more interesting for employer to post job.\n",
    "\n",
    "This could be a good news for analytics-graduate who are looking for job as the number of employers tends to increase from year to year.\n",
    "\n",
    "To confirm the visual finding, we will perform formal statistical tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "005dd081",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Trend Detection using Augmented Dickey-Fuller (ADF) Test\n",
    "def perform_adf_test(series):\n",
    "    result = adfuller(series)\n",
    "    p_value = result[1]\n",
    "    \n",
    "    if p_value < 0.05:\n",
    "        print(\"ADF Test Result: Reject the null hypothesis (Series is stationary)\")\n",
    "    else:\n",
    "        print(\"ADF Test Result: Fail to reject the null hypothesis (Series has a trend)\")\n",
    "\n",
    "# Seasonality Detection using Seasonal Decomposition of Time Series (S-D) Test\n",
    "def perform_sd_test(series):\n",
    "    decomposition = seasonal_decompose(series)\n",
    "    residuals = decomposition.resid\n",
    "    seasonal = decomposition.seasonal\n",
    "    \n",
    "    # Calculate the variation within each seasonal period\n",
    "    within_variation = residuals.groupby(residuals.index.month).std()\n",
    "    # Calculate the variation between different seasonal periods\n",
    "    between_variation = seasonal.groupby(seasonal.index.month).std()\n",
    "    \n",
    "    # Perform F-test or ANOVA to compare the variation\n",
    "    # Here, we use a simple approach of comparing the average variation\n",
    "    if within_variation.mean() > between_variation.mean():\n",
    "        print(\"S-D Test Result: Series shows seasonality\")\n",
    "    else:\n",
    "        print(\"S-D Test Result: Series does not show seasonality\")\n",
    "\n",
    "# Assuming the 'job_posting' DataFrame has a single column 'Unique Postings' representing the time series\n",
    "time_series = agg_exec_summ['Employers Competing']\n",
    "\n",
    "# Perform the ADF test for trend detection\n",
    "print(\"Performing ADF Test for Trend Detection:\")\n",
    "perform_adf_test(time_series)\n",
    "print(\"\")\n",
    "\n",
    "# Perform the S-D test for seasonality detection\n",
    "print(\"Performing S-D Test for Seasonality Detection:\")\n",
    "perform_sd_test(time_series)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1e676cb",
   "metadata": {},
   "source": [
    "The statistical tests confirm that the timeseries data of the number of competing employers has trend but no seasonality."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3e8b4db",
   "metadata": {},
   "source": [
    "## Time Series Analysis : Predictive"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6a0f315",
   "metadata": {},
   "source": [
    "After exploring the timeseries analysis on the analytics-related occupations job posting, posted salary, and competing employers, next question arisen, how will the job posting be in the future? will it increase or will it decrease?\n",
    "\n",
    "To answer that question, we will now create a predictive model for analytics-related occupations for the next 3 years using the available historical data. We will also support the output of the predictive model with literature review on the forecast of the availability of the analytics-related occupations in the future.\n",
    "\n",
    "<font color='red'>*Please note that due to the limited amount of information, this study will only focus on building a predictive model for the **Job Postings** and for this analysis, will solely depend on the historical job posting data neglecting the other factors that might affect the number of job postings such as economic condition (both macro and micro) and also the changing trend in the technology adoption. Those additional factors will be the subject of the future research.*</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65f02615",
   "metadata": {},
   "source": [
    "**1. Fitting the Data into Exponential Smoothing Models**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "347d0fd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Fit Exponential Smoothing models\n",
    "ses_model = SimpleExpSmoothing(job_posting['Unique Postings']).fit()\n",
    "holt_model = ExponentialSmoothing(job_posting['Unique Postings'], trend='add').fit()\n",
    "holt_winters_model = ExponentialSmoothing(job_posting['Unique Postings'], trend='add', seasonal='add', seasonal_periods=3).fit()\n",
    "\n",
    "# Make forecasts for the whole time series data\n",
    "ses_forecast = ses_model.forecast(len(job_posting))\n",
    "holt_forecast = holt_model.forecast(len(job_posting))\n",
    "holt_winters_forecast = holt_winters_model.forecast(len(job_posting))\n",
    "\n",
    "# Evaluate the models\n",
    "ses_mse = mean_squared_error(job_posting['Unique Postings'], ses_forecast)\n",
    "holt_mse = mean_squared_error(job_posting['Unique Postings'], holt_forecast)\n",
    "holt_winters_mse = mean_squared_error(job_posting['Unique Postings'], holt_winters_forecast)\n",
    "\n",
    "ses_rmse = np.sqrt(ses_mse)\n",
    "holt_rmse = np.sqrt(holt_mse)\n",
    "holt_winters_rmse = np.sqrt(holt_winters_mse)\n",
    "\n",
    "# Print evaluation metrics\n",
    "print(f\"Simple Exponential Smoothing RMSE: {ses_rmse}\")\n",
    "print(f\"Double Exponential Smoothing RMSE: {holt_rmse}\")\n",
    "print(f\"Triple Exponential Smoothing RMSE: {holt_winters_rmse}\")\n",
    "\n",
    "# Plot the whole time series data and the forecasted values using Plotly\n",
    "fig = go.Figure()\n",
    "\n",
    "# Plot the actual data\n",
    "fig.add_trace(go.Scatter(x=job_posting.index, y=job_posting['Unique Postings'], mode='lines', name='Actual'))\n",
    "\n",
    "# Plot the forecasted values from each model\n",
    "fig.add_trace(go.Scatter(x=job_posting.index, y=ses_forecast, mode='lines', name='SES Forecast'))\n",
    "fig.add_trace(go.Scatter(x=job_posting.index, y=holt_forecast, mode='lines', name='Holt Forecast'))\n",
    "fig.add_trace(go.Scatter(x=job_posting.index, y=holt_winters_forecast, mode='lines', name='Holt-Winters Forecast'))\n",
    "\n",
    "fig.update_layout(title='Time Series Forecasting Comparison',\n",
    "                  xaxis_title='Month',\n",
    "                  yaxis_title='Number of Job Postings',\n",
    "                  showlegend=True)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "516d8f3e",
   "metadata": {},
   "source": [
    "From the RMSE metric and visual inspection above, it is quite clear the Exponential Smoothing for all levels (Simple Exponential Smoothing, Exponential Smoothing with Trend, and Exponential Smoothing with Trend and Seasonality) could not fit the job posting timeseries data nicely. \n",
    "\n",
    "We will proceed to build several another models for the forecasting part of the analysis as follow:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d460c6f5",
   "metadata": {},
   "source": [
    "**2. Fitting the Data Into Simple Moving Average Model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f41f65d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Calculate the simple moving average (SMA) for the entire time series\n",
    "window = 3  # Choose the window size for the moving average\n",
    "sma_forecast = job_posting['Unique Postings'].rolling(window=window).mean()\n",
    "\n",
    "# Split the data into training and testing sets (e.g., 80% training, 20% testing)\n",
    "train_size = int(len(job_posting) * 0.8)\n",
    "train_data, test_data = job_posting.iloc[:train_size], job_posting.iloc[train_size:]\n",
    "\n",
    "# Calculate the SMA for the in-sample data (training data)\n",
    "sma_in_sample = train_data['Unique Postings'].rolling(window=window).mean()\n",
    "\n",
    "# Calculate the RMSE for the in-sample data\n",
    "sma_rmse_in_sample = np.sqrt(mean_squared_error(train_data['Unique Postings'][window:], sma_in_sample[window:]))\n",
    "\n",
    "# Print RMSE for the in-sample data\n",
    "print(f\"SMA RMSE (In-Sample): {sma_rmse_in_sample}\")\n",
    "\n",
    "# Calculate the RMSE for the out-of-sample data (testing data)\n",
    "sma_rmse_out_of_sample = np.sqrt(mean_squared_error(test_data['Unique Postings'], sma_forecast[-len(test_data):]))\n",
    "\n",
    "# Print RMSE for the out-of-sample data\n",
    "print(f\"SMA RMSE (Out-of-Sample): {sma_rmse_out_of_sample}\")\n",
    "\n",
    "# Plot the actual data and the SMA forecast using Plotly\n",
    "fig = go.Figure()\n",
    "\n",
    "# Plot the actual data\n",
    "fig.add_trace(go.Scatter(x=job_posting.index, y=job_posting['Unique Postings'], mode='lines', name='Actual'))\n",
    "\n",
    "# Plot the SMA forecast for the entire time series\n",
    "fig.add_trace(go.Scatter(x=job_posting.index, y=sma_forecast, mode='lines', name='SMA Forecast (Full)'))\n",
    "\n",
    "# Plot the SMA forecast for the in-sample data (training data)\n",
    "fig.add_trace(go.Scatter(x=train_data.index, y=sma_in_sample, mode='lines', name='SMA Forecast (In-Sample)'))\n",
    "\n",
    "fig.update_layout(title='Simple Moving Average Time Series Forecasting',\n",
    "                  xaxis_title='Month',\n",
    "                  yaxis_title='Number of Job Postings',\n",
    "                  showlegend=True)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4dc85f6",
   "metadata": {},
   "source": [
    "**3. Fitting the Data Into ARIMA Model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe9ae075",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Split the data into training and testing sets (e.g., 80% training, 20% testing)\n",
    "train_size = int(len(job_posting) * 0.8)\n",
    "train_data, test_data = job_posting.iloc[:train_size], job_posting.iloc[train_size:]\n",
    "\n",
    "# Perform a grid search for the best ARIMA model order\n",
    "best_rmse = np.inf\n",
    "best_order = None\n",
    "\n",
    "for p in range(3):\n",
    "    for d in range(2):\n",
    "        for q in range(3):\n",
    "            try:\n",
    "                order = (p, d, q)\n",
    "                arima_model = ARIMA(train_data['Unique Postings'], order=order).fit()\n",
    "                arima_forecast = arima_model.forecast(len(test_data))\n",
    "                arima_rmse = np.sqrt(mean_squared_error(test_data['Unique Postings'], arima_forecast))\n",
    "\n",
    "                if arima_rmse < best_rmse:\n",
    "                    best_rmse = arima_rmse\n",
    "                    best_order = order\n",
    "\n",
    "            except:\n",
    "                continue\n",
    "\n",
    "# Fit the ARIMA model with the best order\n",
    "arima_model = ARIMA(train_data['Unique Postings'], order=best_order).fit()\n",
    "\n",
    "# Make forecasts for the testing data\n",
    "arima_forecast = arima_model.forecast(len(test_data))\n",
    "\n",
    "# Print the best model order\n",
    "print(\"Best ARIMA Model Order:\", best_order)\n",
    "\n",
    "# Print the best RMSE\n",
    "print(\"Best ARIMA RMSE:\", best_rmse)\n",
    "\n",
    "# Plot the actual data and the ARIMA forecast using Plotly\n",
    "fig = go.Figure()\n",
    "\n",
    "# Plot the actual data\n",
    "fig.add_trace(go.Scatter(x=job_posting.index, y=job_posting['Unique Postings'], mode='lines', name='Actual'))\n",
    "\n",
    "# Plot the ARIMA forecast\n",
    "test_dates = test_data.index\n",
    "fig.add_trace(go.Scatter(x=test_dates, y=arima_forecast, mode='lines', name='ARIMA Forecast'))\n",
    "\n",
    "fig.update_layout(title='ARIMA Time Series Forecasting',\n",
    "                  xaxis_title='Month',\n",
    "                  yaxis_title='Number of Job Postings',\n",
    "                  showlegend=True)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edbe969b",
   "metadata": {},
   "source": [
    "**4. Fitting the Data Into SARIMA Model**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3748f398",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Split the data into training and testing sets (e.g., 80% training, 20% testing)\n",
    "train_size = int(len(job_posting) * 0.8)\n",
    "train_data, test_data = job_posting.iloc[:train_size], job_posting.iloc[train_size:]\n",
    "\n",
    "# Define a list of hyperparameters to search\n",
    "p_values = [0, 1, 2]       # (p) - Autoregressive order\n",
    "d_values = [0, 1]          # (d) - Differencing order\n",
    "q_values = [0, 1, 2]       # (q) - Moving average order\n",
    "P_values = [0, 1, 2]       # (P) - Seasonal autoregressive order\n",
    "D_values = [0, 1]          # (D) - Seasonal differencing order\n",
    "Q_values = [0, 1, 2]       # (Q) - Seasonal moving average order\n",
    "seasonal_periods = [12]    # (S) - Seasonal period, 12 for monthly data (12 months in a year)\n",
    "\n",
    "# Create a list to store results and configurations\n",
    "results = []\n",
    "configs = []\n",
    "\n",
    "# Perform the grid search\n",
    "for p, d, q, P, D, Q, S in product(p_values, d_values, q_values, P_values, D_values, Q_values, seasonal_periods):\n",
    "    try:\n",
    "        sarima_model = SARIMAX(train_data['Unique Postings'], order=(p, d, q), seasonal_order=(P, D, Q, S))\n",
    "        sarima_model_fit = sarima_model.fit()\n",
    "\n",
    "        # Make forecasts for the validation data\n",
    "        sarima_forecast = sarima_model_fit.forecast(len(test_data))\n",
    "\n",
    "        # Calculate RMSE for validation data\n",
    "        sarima_mse = mean_squared_error(test_data['Unique Postings'], sarima_forecast)\n",
    "        sarima_rmse = np.sqrt(sarima_mse)\n",
    "\n",
    "        # Store the results and configurations\n",
    "        results.append(sarima_rmse)\n",
    "        configs.append((p, d, q, P, D, Q, S))\n",
    "\n",
    "    except:\n",
    "        continue\n",
    "\n",
    "# Find the best configuration\n",
    "best_idx = np.argmin(results)\n",
    "best_config = configs[best_idx]\n",
    "best_rmse = results[best_idx]\n",
    "\n",
    "print(f\"Best Hyperparameters: (p={best_config[0]}, d={best_config[1]}, q={best_config[2]}, \"\n",
    "      f\"P={best_config[3]}, D={best_config[4]}, Q={best_config[5]}, S={best_config[6]}), \"\n",
    "      f\"Best RMSE: {best_rmse}\")\n",
    "\n",
    "# Fit the SARIMA model with the best configuration on the full dataset\n",
    "best_sarima_model = SARIMAX(job_posting['Unique Postings'], order=(best_config[0], best_config[1], best_config[2]),\n",
    "                            seasonal_order=(best_config[3], best_config[4], best_config[5], best_config[6]))\n",
    "best_sarima_model_fit = best_sarima_model.fit()\n",
    "\n",
    "# Get in-sample predictions (fit on the entire time series data)\n",
    "in_sample_forecast = best_sarima_model_fit.get_prediction(start=job_posting.index[0], end=job_posting.index[-1])\n",
    "\n",
    "# Extract the predicted values and confidence intervals\n",
    "predicted_values = in_sample_forecast.predicted_mean\n",
    "confidence_intervals = in_sample_forecast.conf_int(alpha=0.05)\n",
    "\n",
    "# Make forecasts for the future (you can adjust the number of periods to forecast)\n",
    "future_forecast = best_sarima_model_fit.forecast(steps=36)\n",
    "\n",
    "# Plot the actual data, the in-sample forecast, and the future forecast using Plotly\n",
    "fig = go.Figure()\n",
    "\n",
    "# Plot the actual data\n",
    "fig.add_trace(go.Scatter(x=job_posting.index, y=job_posting['Unique Postings'], mode='lines', name='Actual'))\n",
    "\n",
    "# Plot the in-sample forecast\n",
    "fig.add_trace(go.Scatter(x=predicted_values.index, y=predicted_values, mode='lines', name='In-Sample Forecast'))\n",
    "\n",
    "# Plot the future forecast\n",
    "future_dates = pd.date_range(start=job_posting.index[-1], periods=36, freq='MS')\n",
    "fig.add_trace(go.Scatter(x=future_dates, y=future_forecast, mode='lines', name='Future Forecast'))\n",
    "\n",
    "# Add shaded regions for confidence intervals\n",
    "fig.add_trace(go.Scatter(x=predicted_values.index, y=confidence_intervals['lower Unique Postings'],\n",
    "                         fill=None, mode='lines', showlegend=False))\n",
    "fig.add_trace(go.Scatter(x=predicted_values.index, y=confidence_intervals['upper Unique Postings'],\n",
    "                         fill='tonexty', mode='lines', name='Confidence Interval'))\n",
    "\n",
    "fig.update_layout(title='SARIMA Time Series Forecasting',\n",
    "                  xaxis_title='Month',\n",
    "                  yaxis_title='Number of Job Postings',\n",
    "                  showlegend=True)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "627e5db7",
   "metadata": {},
   "source": [
    "From the four models above, Simple Moving Average (SMA) with n = 3 seems to return the lowest RMSE followed by the SARIMA model as the second lowest RSME. We have previously made a forecast for the next 3 years using the SARIMA model with slightly decreasing trend in the number of job posting that start at 2700 and ends at 2000 by the end of 2026. \n",
    "\n",
    "We will now proceed with the forecast made by using the SMA with n=3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "666db8c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Calculate the simple moving average (SMA) for the entire time series\n",
    "window = 3  # Choose the window size for the moving average\n",
    "sma_forecast = job_posting['Unique Postings'].rolling(window=window).mean()\n",
    "\n",
    "# Split the data into training and testing sets (e.g., 80% training, 20% testing)\n",
    "train_size = int(len(job_posting) * 0.8)\n",
    "train_data, test_data = job_posting.iloc[:train_size], job_posting.iloc[train_size:]\n",
    "\n",
    "# Calculate the SMA for the in-sample data (training data)\n",
    "sma_in_sample = train_data['Unique Postings'].rolling(window=window).mean()\n",
    "\n",
    "# Calculate the RMSE for the in-sample data\n",
    "sma_rmse_in_sample = np.sqrt(mean_squared_error(train_data['Unique Postings'][window:], sma_in_sample[window:]))\n",
    "\n",
    "# Print RMSE for the in-sample data\n",
    "print(f\"SMA RMSE (In-Sample): {sma_rmse_in_sample}\")\n",
    "\n",
    "# Calculate the RMSE for the out-of-sample data (testing data)\n",
    "sma_rmse_out_of_sample = np.sqrt(mean_squared_error(test_data['Unique Postings'], sma_forecast[-len(test_data):]))\n",
    "\n",
    "# Print RMSE for the out-of-sample data\n",
    "print(f\"SMA RMSE (Out-of-Sample): {sma_rmse_out_of_sample}\")\n",
    "\n",
    "# Make predictions for the next 36 months using the SMA method\n",
    "future_dates = pd.date_range(start=job_posting.index[-1], periods=36, freq='MS')\n",
    "future_forecast = pd.Series(index=future_dates)\n",
    "\n",
    "# Initial window for the SMA forecast\n",
    "initial_window = job_posting['Unique Postings'][-window:]\n",
    "\n",
    "# Calculate the SMA forecast for the next 36 months\n",
    "for date in future_dates:\n",
    "    forecast_value = initial_window.mean()\n",
    "    future_forecast[date] = forecast_value\n",
    "    initial_window = initial_window.shift(-1).dropna().append(pd.Series(forecast_value, index=[date]))\n",
    "\n",
    "# Plot the actual data, SMA forecast for the entire time series, and SMA forecast for the in-sample data\n",
    "fig = go.Figure()\n",
    "\n",
    "# Plot the actual data\n",
    "fig.add_trace(go.Scatter(x=job_posting.index, y=job_posting['Unique Postings'], mode='lines', name='Actual'))\n",
    "\n",
    "# Plot the SMA forecast for the entire time series\n",
    "fig.add_trace(go.Scatter(x=job_posting.index, y=sma_forecast, mode='lines', name='SMA Forecast (Full)'))\n",
    "\n",
    "# Plot the SMA forecast for the in-sample data (training data)\n",
    "fig.add_trace(go.Scatter(x=train_data.index, y=sma_in_sample, mode='lines', name='SMA Forecast (In-Sample)'))\n",
    "\n",
    "# Plot the SMA forecast for the next 36 months\n",
    "fig.add_trace(go.Scatter(x=future_forecast.index, y=future_forecast, mode='lines', name='SMA Forecast (Next 36 Months)'))\n",
    "\n",
    "fig.update_layout(title='Simple Moving Average Time Series Forecasting',\n",
    "                  xaxis_title='Month',\n",
    "                  yaxis_title='Number of Job Postings',\n",
    "                  showlegend=True)\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04d4f10a",
   "metadata": {},
   "source": [
    "The forecast of the SMA returns the flat, average-like-forecast, for the analytics-related occupation job postings for the next 3 years at around 2700 job posting per month. This might not reflect the actual nature of the job posting so for this analysis. We will use ARIMA and SARIMA forecast for the job posting even though SMA with n=3 has the lowest RMSE."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "223414c5",
   "metadata": {},
   "source": [
    "**Final Verdict**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4f54866",
   "metadata": {},
   "source": [
    "It is interesting to note that the model with the third lowest RMSE, ARIMA, also returned forecast of the same nature with SARIMA. If we are only taking into account the top three model with lowest RMSE, then by the rule of majority, the forecast of the job posting for the analytics-related occupations might proceed with a slightly decreasing trend over the next three years.\n",
    "\n",
    "The forecast could change by the addition of new actual job posting in 2023, but the output of the ARIMA and SARIMA might reflect the continous downturn of the number of job posting starting from July 2022 to December 2022 where previously the trend was increasing from May 2020.\n",
    "\n",
    "Based on the timeseries analysis, we might expect to see a decreasing trend of the number of job posting for the analytics-related occupation for the next three years."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efbb3740",
   "metadata": {},
   "source": [
    "## Clustering Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03635568",
   "metadata": {},
   "source": [
    "In this section, we will take a broad perspective to examine similarities and differences among various occupations. Our goal is to determine if each occupation differs in terms of its requirements and benefits. Additionally, we will explore whether there are different tiers of hiring probabilities for each analytics-related occupation. We aim to identify which occupations share similar characteristics and which ones are distinct from each other.\n",
    "\n",
    "To accomplish this, we will employ two clustering techniques: k-means clustering and hierarchical clustering. By comparing the results of these clustering methods, we will determine the optimal number of clusters for grouping analytics-related occupations. This analysis will help us gain insights into the occupational landscape and identify patterns that may exist among these roles.\n",
    "\n",
    "The features used for this analysis will only be limited to the numeric features that we have previously discussed such as Number of Job Postings, Advertised Salary and Wage, Educational and Experience Background, Skill Requirements and many others."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e23c843",
   "metadata": {},
   "source": [
    "### k-Means Clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b9efab1",
   "metadata": {},
   "source": [
    "**1. Data Preparation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df89d83c",
   "metadata": {},
   "outputs": [],
   "source": [
    "clustering.drop(clustering.columns[-1], axis=1, inplace=True)\n",
    "clustering.set_index(clustering.columns[0], inplace=True)\n",
    "\n",
    "# Function to convert a cell value to numeric type, handling commas as thousand separators\n",
    "def convert_cell_to_numeric(x):\n",
    "    if isinstance(x, str):\n",
    "        return pd.to_numeric(x.replace(',', ''), errors='coerce')\n",
    "    else:\n",
    "        return x\n",
    "\n",
    "# Apply the conversion to all cells in the DataFrame\n",
    "clustering = clustering.applymap(convert_cell_to_numeric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ea65989",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scaling the occupation data\n",
    "scaler = preprocessing.StandardScaler()\n",
    "scaled = scaler.fit_transform(clustering)\n",
    "scaled = pd.DataFrame(scaled, columns=clustering.columns, index=clustering.index)\n",
    "# Checking for the scaled data\n",
    "scaled"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b9cbf96",
   "metadata": {},
   "source": [
    "**2. Performing Elbow Analysis to Check for Initial Number of k for Iteration**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e9df71fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initializing empty list for SSE calculation\n",
    "sse = {}\n",
    "for k in range(1, 6): \n",
    "# Initialize KMeans with k clusters\n",
    "    kmeans = KMeans(n_clusters=k, random_state=654)\n",
    "# Fit KMeans on the normalized dataset\n",
    "    kmeans.fit(scaled)\n",
    "    sse[k] = kmeans.inertia_\n",
    "# Add the plot title \"The Elbow Method\"\n",
    "plt.title('The Elbow Method')\n",
    "# Add X-axis label \"k\"\n",
    "plt.xlabel('k')\n",
    "# Add Y-axis label \"SSE\"\n",
    "plt.ylabel('SSE')\n",
    "sns.pointplot(x=list(sse.keys()), y=list(sse.values()));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d18080f8",
   "metadata": {},
   "source": [
    "Looking at the Elbow Plot above, we will try to loop through number of cluster 3 to 5 and qualitatively check the goodness of cluster. The optimum number of cluster will be chosen if the separation between cluster is distinct enough and that each cluster comprises of more than one occupations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "578f04de",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of clusters equals 3\n",
    "kmeans = KMeans(n_clusters=3, random_state=654)\n",
    "kmeans.fit(scaled)\n",
    "cluster_labels = kmeans.labels_\n",
    "\n",
    "# Assigning cluster labels to the DataFrame\n",
    "scaled['Cluster'] = cluster_labels\n",
    "\n",
    "# List of columns to compute mean and standard deviation\n",
    "columns_to_aggregate = scaled.columns[:-1]  # Exclude the 'Cluster' column from aggregation\n",
    "\n",
    "# Group by 'Cluster' and compute mean and standard deviation for each column\n",
    "result = scaled.groupby('Cluster')[columns_to_aggregate].agg(['mean']).round(2)\n",
    "\n",
    "result['Count'] = scaled.groupby('Cluster').size()\n",
    "\n",
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "211487d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of clusters equals 4\n",
    "kmeans = KMeans(n_clusters=4, random_state=654)\n",
    "kmeans.fit(scaled)\n",
    "cluster_labels = kmeans.labels_\n",
    "\n",
    "# Assigning cluster labels to the DataFrame\n",
    "scaled['Cluster'] = cluster_labels\n",
    "\n",
    "# List of columns to compute mean and standard deviation\n",
    "columns_to_aggregate = scaled.columns[:-1]  # Exclude the 'Cluster' column from aggregation\n",
    "\n",
    "# Group by 'Cluster' and compute mean and standard deviation for each column\n",
    "result = scaled.groupby('Cluster')[columns_to_aggregate].agg(['mean', 'std']).round(2)\n",
    "\n",
    "result['Count'] = scaled.groupby('Cluster').size()\n",
    "\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ff4dd72",
   "metadata": {},
   "source": [
    "The result of the analysis above showed that when the number of cluster was increased from 3 to 4, there are two cluster with only one member and that indicates the cluster might be overfitting, therefore for this analysis, we will go with k equals to 3 or 3 clusters using k-means clustering method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "367cef37",
   "metadata": {},
   "source": [
    "### Hierarchical Clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2335575",
   "metadata": {},
   "source": [
    "We will now compare the results of kmeans clustering with hierarchical clustering to validate whether cluster number of 3 is the right choice for analytics-related occupations clustering analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a54da674",
   "metadata": {},
   "outputs": [],
   "source": [
    "n_clusters = 3\n",
    "\n",
    "# Perform hierarchical clustering\n",
    "hierarchical = AgglomerativeClustering(n_clusters=n_clusters)\n",
    "cluster_labels = hierarchical.fit_predict(scaled)\n",
    "\n",
    "# Assigning cluster labels to the DataFrame\n",
    "scaled['Cluster'] = cluster_labels\n",
    "\n",
    "# List of columns to compute mean and standard deviation\n",
    "columns_to_aggregate = scaled.columns[:-1]  # Exclude the 'Cluster' column from aggregation\n",
    "\n",
    "# Group by 'Cluster' and compute mean and standard deviation for each column\n",
    "result = scaled.groupby('Cluster')[columns_to_aggregate].agg(['mean', 'std']).round(2)\n",
    "\n",
    "# Add the count of instances in each cluster as a new column\n",
    "result['Count'] = scaled.groupby('Cluster').size()\n",
    "\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e951a769",
   "metadata": {},
   "source": [
    "The result from hierarchical clustering above showed that cluster number of 3 could be used to create three different cluster with distinct enough characteristics as shown with the k-Means clustering before, so k equals to 3 will be used for the entire clustering analysis here."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7a49daa",
   "metadata": {},
   "source": [
    "### Cluster Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41873c22",
   "metadata": {},
   "source": [
    "**1. Cluster 0 [Data Scientists and Management Analysts]**\n",
    "\n",
    "**Cluster Characteristics:**\n",
    "- This cluster has the highest number of job postings compared to other clusters, using SOC or O`NET Classification.\n",
    "- Jobs in this cluster offer the highest salaries and wages, with postings reflecting competitive compensation.\n",
    "- This cluster accepts a broad range of educational and experience backgrounds, making it accessible to a diverse group of candidates.\n",
    "- It has the highest number of occurrences among all clusters, indicating its prevalence in the job market.\n",
    "- Job postings in this cluster have the shortest duration before being filled, suggesting a high demand for talents.\n",
    "- The competition among employers to attract talent is fierce, with many companies vying for candidates in this cluster.\n",
    "- Job offerings in this cluster are geographically limited, particularly with fewer city postings. However, it stands out in cities like Chicago, New York, Atlanta, and San Francisco.\n",
    "\n",
    "**Unique Company and Industry Demands:**\n",
    "- The number of unique companies seeking talent in this cluster is comparable to Cluster 2.\n",
    "- However, the variety of unique industries actively seeking talent in this cluster is limited.\n",
    "\n",
    "**Skillset Requirements:**\n",
    "- This cluster requires a moderate number of aggregated skills, less than what is seen in Cluster 2.\n",
    "- Strongly required skills in this cluster include communication, management, and problem-solving, as they have the highest number of skill requirements.\n",
    "- The most demanded programming languages are R, Python, and SQL, along with Microsoft Office and BI software like Power BI and Tableau as required software skills.\n",
    "- Analysis skills are highly sought after in this cluster as specialized skills.\n",
    "- Although it has lower skill requirements than Cluster 2, Cluster 1 demands the highest number of qualifications or \"nice-to-have\" skillsets.\n",
    "\n",
    "**Competitiveness:**\n",
    "- In terms of difficulty in getting accepted, this cluster ranks as the second most competitive in hiring.\n",
    "- It has low monthly hires and a low ratio of hires to postings, indicating strong competition among applicants.\n",
    "\n",
    "\n",
    "In summary, this cluster stands out for its high job postings and competitive compensation, but it has limited geographical reach. It attracts various companies but is concentrated in a few industries. The skillset requirements are moderate but emphasize communication, management, problem-solving, programming languages, and software skills. This cluster is quite competitive and command exclusivity.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fbaf610",
   "metadata": {},
   "source": [
    "Based on the description above, the cluster will be named **Elite Metropolis Enclave**\n",
    "\n",
    "This name characterizes the cluster as an exclusive and upscale enclave of job opportunities. \"Elite\" highlights the high salary and premium nature of the job postings within the cluster. \"Metropolis\" indicates its focus on major cities like Chicago, New York, Atlanta, and San Francisco, implying its selectivity. \"Enclave\" reinforces the idea of a highly competitive and sought-after area, where only the most qualified candidates can enter. This name conveys the cluster's distinctiveness, top-tier salary offerings, and limited availability, making it an attractive and competitive choice for job seekers."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "106a86b4",
   "metadata": {},
   "source": [
    "**2. Cluster 1 [Accountants and Auditors, Computer System Analysts, FInancial and Investment Analysts, Computer Occupations, All Other, Database Administrators, Market Research Analysts and Marketing Specialists, Software Developers, Operations Research Analyst, Managers All Other]**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74de0b1",
   "metadata": {},
   "source": [
    "**Cluster Characteristics:**\n",
    "- This cluster comes second after Cluster 0 in terms of benefits (salary and wage), and while it is relatively comparable, the number of job postings with posted benefits (salary and wage) is much lower than Cluster 0.\n",
    "- In terms of required educational and experience background, Cluster 1 has comparatively lower postings for all levels of education and experience of new graduates, indicating limited openings compared to Cluster 0 but still above Cluster 2, which seems more niche.\n",
    "\n",
    "**Job Posting Duration:**\n",
    "- This cluster has the longest-running job postings compared to Cluster 0 and Cluster 2, suggesting that they may amass applicants before starting the screening process or might not attract potential applicants as much as Cluster 0, necessitating longer job posting durations.\n",
    "\n",
    "**Competition and Industry Demands:**\n",
    "- In terms of competing employers, this cluster has a lower number of unique competing employers compared to Cluster 0 but still more than Cluster 2. This suggests that the occupations might be limited to certain companies and not as widely sought after as the ones in Cluster 0.\n",
    "- However, the number of unique industries participating in job postings is much higher than in Cluster 1 and Cluster 2, indicating that occupations in Cluster 1 have broader utilization across various industries and are more versatile in terms of occupational demands.\n",
    "\n",
    "**Job Postings and Cities:**\n",
    "- It comes second in terms of the number of job postings (both SOC and O`NET classification) after Cluster 0, which has considerably more, indicating that the occupations offered here might not be as general as those in Cluster 0, where many companies and industries are looking for that particular talent.\n",
    "- This cluster offers a much broader range of job titles, indicated by the highest number of unique job titles found throughout the job postings. It is also available in more cities compared to Cluster 0 and Cluster 2. However, the occupations within this cluster have lower postings in the top four cities: Chicago, New York, Atlanta, and San Francisco, compared to Cluster 0.\n",
    "\n",
    "**Skillset Requirements:**\n",
    "- This cluster requires much more skills (for all Common, Software, and Specialized) compared to the other clusters (Cluster 0 and Cluster 2).\n",
    "\n",
    "**Competitiveness:**\n",
    "- This cluster is comparatively the same in terms of competitiveness to enter as Cluster 0, which is highly competitive, with a low proportion of people getting hired compared to the number of job postings.\n",
    "\n",
    "In summary, this cluster stands out for its high salary, competitiveness, and inclusivity due to broader city and industry options. It offers a range of occupations with strong skillset requirements, making it an attractive choice for highly qualified candidates seeking opportunities in specific fields."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73b4d5c1",
   "metadata": {},
   "source": [
    "Based on the description above, the cluster will be named **Cosmic Career Constellation**\n",
    "\n",
    "This name presents the cluster as a celestial arrangement of diverse and enticing career opportunities, resembling a constellation of stars in the vast cosmic expanse. \"Cosmic\" emphasizes the cluster's grand and far-reaching appeal, reflecting its broad offering in multiple cities and industries. \"Career Constellation\" symbolizes the interconnected and shining array of occupations within the cluster, forming a captivating and unique pattern of opportunities. This name conveys the cluster's versatility, attractiveness, and allure to candidates seeking to explore a diverse and exciting array of career paths, much like navigating the wonders of the cosmos."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5fa0016",
   "metadata": {},
   "source": [
    "**3. Cluster 2 [Database Architects, Human Resources Specialists, Marketing Manager, Sales Manager, Personal Financial Advisor]**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8589289f",
   "metadata": {},
   "source": [
    "Among the clusters, this particular cluster ranks third in terms of several factors, including benefit (salary and wage), number of job postings, skill requirements, unique companies and industries, job titles, required education and experience backgrounds, and competitiveness level.\n",
    "\n",
    "The lower rankings of this cluster in various aspects might be attributed to the fact that the occupations within it have only appeared once among the top 10 analytics-related occupations in the last 10 years. In contrast, occupations in Clusters 0 and 1 have consistently maintained their positions within the top 10 occupations during the same period.\n",
    "\n",
    "This cluster's position as the \"third\" cluster highlights its unique characteristics and provides insights into its potential for further growth and development."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "855826da",
   "metadata": {},
   "source": [
    "Based on the above characteristics, this cluster will be named **Promising Analytics Ascent**\n",
    "\n",
    "This name characterizes the cluster as a promising and ascending group of analytics-related occupations. Despite ranking third in various aspects such as benefit (salary and wage), job postings, skill requirements, unique companies and industries, job titles, required education and experience backgrounds, and competitiveness level, the cluster stands out as a group with potential for growth and advancement.\n",
    "\n",
    "The name \"Promising Analytics Ascent\" conveys the cluster's potential for future prominence and highlights its unique position as a rising force in the analytics job market. It suggests that the cluster's occupations are on an upward trajectory, with bright prospects for career opportunities and development."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcdaa17e",
   "metadata": {},
   "source": [
    "To visualize the grouping of the cluster, below is the treemap for each of the clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c373937c",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "scaled.reset_index(inplace=True)\n",
    "a = scaled[['occupation','Cluster']]\n",
    "\n",
    "# Create the treemap using Plotly\n",
    "fig = px.treemap(a, \n",
    "                 path=['Cluster', 'occupation'],\n",
    "                 title='Occupation Cluster Treemap')\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c1a18a1",
   "metadata": {},
   "source": [
    "### Visualizations of the kMeans Cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b0f2827",
   "metadata": {},
   "source": [
    "**1. Cluster Benefit (Salary & Wage) vs Skillset Requirements**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "421fb28c",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "data_to_plot = scaled[['occupation', 'mean_annual_salary', 'mean_max_salary', 'mean_min_salary', 'mean_wage', 'number_unique_common_skills', 'number_unique_software_skills', 'number_unique_specialized_skills', 'Cluster']]\n",
    "\n",
    "# Calculate the sum of the three columns and add it as a new column\n",
    "data_to_plot['total_unique_skills'] = data_to_plot['number_unique_common_skills'] + data_to_plot['number_unique_software_skills'] + data_to_plot['number_unique_specialized_skills']\n",
    "\n",
    "# Create the interactive scatter plot with 'occupation' as labels and 'Cluster' as color\n",
    "fig = px.scatter(data_to_plot, x='mean_annual_salary', y='total_unique_skills', color='Cluster', hover_name='occupation', hover_data=['Cluster'], color_continuous_scale='Viridis')\n",
    "\n",
    "# Set the initial title and labels for the plot\n",
    "fig.update_layout(title='Mean Annual Salary vs Total Unique Skills', xaxis_title='Mean Annual Salary', yaxis_title='Total Unique Skills')\n",
    "\n",
    "# Create a dropdown menu for selecting the x-axis column\n",
    "dropdown_buttons = []\n",
    "for col in ['mean_annual_salary', 'mean_max_salary', 'mean_min_salary', 'mean_wage']:\n",
    "    dropdown_buttons.append({\n",
    "        'args': [\n",
    "            {'x': [data_to_plot[col]], 'hover_data': [data_to_plot['occupation'], data_to_plot['Cluster']]},\n",
    "            {'xaxis.title': col.replace('_', ' ').title(), 'title': f'{col.replace(\"_\", \" \").title()} vs Total Unique Skills'}\n",
    "        ],\n",
    "        'label': col.replace('_', ' ').title(),\n",
    "        'method': 'update'\n",
    "    })\n",
    "\n",
    "# Add the dropdown menu to the plot and adjust the position slightly below the title\n",
    "fig.update_layout(\n",
    "    updatemenus=[{'buttons': dropdown_buttons, 'direction': 'down', 'showactive': True, 'x': 0.01, 'xanchor': 'left', 'y': 0.999, 'yanchor': 'top'}]\n",
    ")\n",
    "\n",
    "# Update hovertemplate to show the correct x-axis label dynamically\n",
    "hover_template = 'Occupation: %{customdata[0]}<br>' + 'Cluster: %{customdata[1]}<br>' + '%{xaxis.title.text}: %{x}<br>' + 'Total Unique Skills: %{y}'\n",
    "fig.update_traces(hovertemplate=hover_template, customdata=data_to_plot[['occupation', 'Cluster']])\n",
    "\n",
    "fig.update_layout(coloraxis_showscale=False)\n",
    "\n",
    "# Show the plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a714efc5",
   "metadata": {},
   "source": [
    "**2. Cluster Benefit vs Competitiveness**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "79105ab0",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "data_to_plot = scaled[['occupation', 'mean_annual_salary', 'mean_max_salary', 'mean_min_salary', 'mean_wage', 'number_unique_common_skills', 'number_unique_software_skills', 'number_unique_specialized_skills', 'Cluster', 'ratio']]\n",
    "\n",
    "# Calculate the sum of the three columns and add it as a new column\n",
    "data_to_plot['total_unique_skills'] = data_to_plot['number_unique_common_skills'] + data_to_plot['number_unique_software_skills'] + data_to_plot['number_unique_specialized_skills']\n",
    "\n",
    "# Create the interactive scatter plot with 'occupation' as labels and 'Cluster' as color\n",
    "fig = px.scatter(data_to_plot, x='mean_annual_salary', y='ratio', color='Cluster', hover_name='occupation', hover_data=['Cluster'], color_continuous_scale='Viridis')\n",
    "\n",
    "# Set the initial title and labels for the plot\n",
    "fig.update_layout(title='Mean Annual Salary vs Competitiveness', xaxis_title='Mean Annual Salary', yaxis_title='Competitiveness')\n",
    "\n",
    "# Create a dropdown menu for selecting the x-axis column\n",
    "dropdown_buttons = []\n",
    "for col in ['mean_annual_salary', 'mean_max_salary', 'mean_min_salary', 'mean_wage']:\n",
    "    dropdown_buttons.append({\n",
    "        'args': [\n",
    "            {'x': [data_to_plot[col]], 'y': [data_to_plot['ratio']], 'hover_data': [data_to_plot['occupation'], data_to_plot['Cluster']]},\n",
    "            {'xaxis.title': col.replace('_', ' ').title(), 'yaxis.title': 'Competitiveness', 'title': f'{col.replace(\"_\", \" \").title()} vs Competitiveness'}\n",
    "        ],\n",
    "        'label': col.replace('_', ' ').title(),\n",
    "        'method': 'update'\n",
    "    })\n",
    "\n",
    "# Add the dropdown menu to the plot and adjust the position slightly below the title\n",
    "fig.update_layout(\n",
    "    updatemenus=[{'buttons': dropdown_buttons, 'direction': 'down', 'showactive': True, 'x': 0.08, 'xanchor': 'left', 'y': 0.99, 'yanchor': 'top'}]\n",
    ")\n",
    "\n",
    "# Update hovertemplate to show the correct x-axis and y-axis label dynamically\n",
    "hover_template = 'Occupation: %{customdata[0]}<br>' + 'Cluster: %{customdata[1]}<br>' + 'X-axis: %{xaxis.title.text}: %{x}<br>' + 'Y-axis: %{yaxis.title.text}: %{y}'\n",
    "fig.update_traces(hovertemplate=hover_template, customdata=data_to_plot[['occupation', 'Cluster']])\n",
    "\n",
    "fig.update_layout(coloraxis_showscale=False)\n",
    "\n",
    "# Show the plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "506eef52",
   "metadata": {},
   "source": [
    "**3. Cluster Competing Company vs Industry**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf978e18",
   "metadata": {},
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "fig = px.scatter(scaled, x='number_unique_company', y='number_unique_industry', color='Cluster',\n",
    "                 hover_name='occupation', hover_data=['Cluster'])\n",
    "\n",
    "# Set the initial title and labels for the plot\n",
    "fig.update_layout(title='Number of Unique Companies vs Number of Unique Industries',\n",
    "                  xaxis_title='Number of Unique Companies', yaxis_title='Number of Unique Industries')\n",
    "\n",
    "fig.update_layout(coloraxis_showscale=False)\n",
    "\n",
    "# Show the plot\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "762dbaeb",
   "metadata": {},
   "source": [
    "**3. The Rest of the Features Combination**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e301ff7e",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "enable_plotly_in_cell()\n",
    "\n",
    "# Create the Dash app\n",
    "app = dash.Dash(__name__)\n",
    "\n",
    "# Define the layout of the app\n",
    "app.layout = html.Div([\n",
    "    dcc.Dropdown(\n",
    "        id='x-axis-dropdown',\n",
    "        options=[{'label': col, 'value': col} for col in scaled.columns],\n",
    "        value='mean_annual_salary',  # Default X-axis column\n",
    "    ),\n",
    "    dcc.Dropdown(\n",
    "        id='y-axis-dropdown',\n",
    "        options=[{'label': col, 'value': col} for col in scaled.columns],\n",
    "        value='number_unique_common_skills',  # Default Y-axis column\n",
    "    ),\n",
    "    dcc.Graph(id='scatter-plot'),\n",
    "])\n",
    "\n",
    "# Define the callback to update the scatter plot\n",
    "@app.callback(\n",
    "    dash.dependencies.Output('scatter-plot', 'figure'),\n",
    "    [dash.dependencies.Input('x-axis-dropdown', 'value'),\n",
    "     dash.dependencies.Input('y-axis-dropdown', 'value')]\n",
    ")\n",
    "def update_scatter_plot(x_axis_column, y_axis_column):\n",
    "    if not x_axis_column or not y_axis_column:\n",
    "        # If either of the dropdowns is not selected, return an empty figure\n",
    "        return px.scatter()\n",
    "\n",
    "    fig = px.scatter(scaled, x=x_axis_column, y=y_axis_column, color='Cluster', hover_name='occupation')\n",
    "\n",
    "    # Set the updated title and labels for the plot\n",
    "    fig.update_layout(\n",
    "        title=f'{x_axis_column} vs {y_axis_column} Colored by Cluster',\n",
    "        xaxis_title=x_axis_column,\n",
    "        yaxis_title=y_axis_column,\n",
    "        hovermode='closest',  # To display the tooltip for the closest point\n",
    "    )\n",
    "    \n",
    "    fig.update_layout(coloraxis_showscale=False)\n",
    "\n",
    "    return fig\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    app.run_server(debug=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "682dabb9",
   "metadata": {},
   "source": [
    "## Association Rule Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ac0ecca",
   "metadata": {},
   "source": [
    "In this section, our aim is to identify the most common combinations of skills requested in job postings for analytics-related occupations. To achieve this, we'll use Association Rule analysis. This analysis will provide us valuable insights into the skill sets that employers highly desire.\n",
    "\n",
    "We'll be working with Job Posting Samples data from the past decade, covering the top 10 analytics-related occupations."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d43f1bd8",
   "metadata": {},
   "source": [
    "Before we dive into the Association Rule analysis, it is better to review some of the core concepts and terminologies of the Association Rule:\n",
    "\n",
    "0. **Antecedent and Consequent (Precedent)**: Antecedent is the itemset that occurred/listed/mentioned/bought before the Consequent itemset.\n",
    "\n",
    "1. **Support**: support is a measure of how frequently an itemset (combination of items) appears in the dataset. It is the proportion of transactions that contain the itemset. High support values indicate that the itemset appears frequently and is more common in the dataset.\n",
    "\n",
    "2. **Confidence**: Confidence is a measure of how often the consequent (right-hand side) of a rule appears in transactions given that the antecedent (left-hand side) is present. It quantifies the strength of the association between the antecedent and consequent.\n",
    "\n",
    "3. **Lift**: Lift is a measure of the strength of the association between the antecedent and consequent, taking into account the expected co-occurrence if the items were independent.\n",
    "\n",
    "4. **Leverage**: Leverage is a measure of how much the rule contributes to the support above what would be expected if the antecedents and consequents were independent.\n",
    "\n",
    "5. **Conviction**: Conviction quantifies the implication of the rule in terms of how much it is influenced by the absence of the consequents.\n",
    "\n",
    "6. **Zhang`s Metric**: Zhang's metric is a measure of the degree to which the rule is interesting and different from what would be expected by chance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a977679a",
   "metadata": {},
   "source": [
    "**1. Preparing the Dataset for Association Rule Analysis**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "33373895",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract the 'Skills' column from the DataFrame\n",
    "skills = job_posting_samples[['Skills']]\n",
    "\n",
    "# Create a new column 'Transaction' and fill it with the transaction number\n",
    "skills['Transaction'] = skills.apply(lambda x: f'Transaction {x.name + 1}', axis=1)\n",
    "\n",
    "# Move the 'Transaction' column to the first position\n",
    "column_order = ['Transaction'] + skills.columns[:-1].tolist()\n",
    "skills = skills[column_order]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "53ecc3d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a new DataFrame with individual skill entries for each transaction\n",
    "new_data = []\n",
    "for _, row in skill.iterrows():\n",
    "    skills = row['Skills'].split(', ')\n",
    "    for skill in skills:\n",
    "        new_data.append([row['Transaction'], skill])\n",
    "\n",
    "new_skill_df = pd.DataFrame(new_data, columns=['Transaction', 'Skill'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "b3608b9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the data into a transactional format using TransactionEncoder\n",
    "te = TransactionEncoder()\n",
    "te_array = te.fit(new_skill_df.groupby('Transaction')['Skill'].apply(list)).transform(new_skill_df.groupby('Transaction')['Skill'].apply(list))\n",
    "transactional_data = pd.DataFrame(te_array, columns=te.columns_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f874e2a",
   "metadata": {},
   "source": [
    "**2. Applying Apriori Algorithm to the Transaction Data and Geenerate Association Rule**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "440d7dc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply Apriori algorithm to discover frequent itemsets\n",
    "min_support = 0.1  \n",
    "frequent_itemsets = apriori(transactional_data, min_support=min_support, use_colnames=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "57574db9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate association rules based on confidence\n",
    "min_confidence = 0.5  # You can adjust this confidence threshold as needed\n",
    "rules = association_rules(frequent_itemsets, metric='confidence', min_threshold=min_confidence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f41c18cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Frequent Itemsets:\n",
      "     support                                           itemsets\n",
      "14     0.518                                    (Data Analysis)\n",
      "10     0.494                                   (Communications)\n",
      "31     0.392                                       (Management)\n",
      "49     0.380                       (SQL (Programming Language))\n",
      "11     0.332                                 (Computer Science)\n",
      "42     0.310                                  (Problem Solving)\n",
      "5      0.292                                (Business Analysis)\n",
      "17     0.276                                     (Data Science)\n",
      "35     0.274                                  (Microsoft Excel)\n",
      "22     0.258                                          (Finance)\n",
      "69     0.250                       (Management, Communications)\n",
      "100    0.240        (Data Analysis, SQL (Programming Language))\n",
      "38     0.238                                       (Operations)\n",
      "64     0.238                    (Data Analysis, Communications)\n",
      "47     0.238                                         (Research)\n",
      "56     0.226                                          (Writing)\n",
      "29     0.222                                       (Leadership)\n",
      "53     0.210         (Tableau (Business Intelligence Software))\n",
      "45     0.206                    (Python (Programming Language))\n",
      "92     0.196                        (Data Analysis, Management)\n",
      "20     0.196                                  (Detail Oriented)\n",
      "41     0.192                                    (Presentations)\n",
      "18     0.192                               (Data Visualization)\n",
      "74     0.190                  (Problem Solving, Communications)\n",
      "79     0.186                  (Computer Science, Data Analysis)\n",
      "0      0.184                                       (Accounting)\n",
      "40     0.176                                         (Planning)\n",
      "44     0.176                               (Project Management)\n",
      "87     0.176                      (Data Analysis, Data Science)\n",
      "60     0.176                (Business Analysis, Communications)\n",
      "33     0.176                                      (Mathematics)\n",
      "85     0.172     (Computer Science, SQL (Programming Language))\n",
      "6      0.168                               (Business Analytics)\n",
      "77     0.166       (SQL (Programming Language), Communications)\n",
      "80     0.162                   (Computer Science, Data Science)\n",
      "63     0.160                 (Computer Science, Communications)\n",
      "95     0.160                   (Data Analysis, Microsoft Excel)\n",
      "32     0.158                                        (Marketing)\n",
      "13     0.158                                        (Dashboard)\n",
      "7      0.152                            (Business Intelligence)\n",
      "58     0.150                              (Accounting, Finance)\n",
      "114    0.150                      (Problem Solving, Management)\n",
      "124    0.148  (Tableau (Business Intelligence Software), SQL...\n",
      "43     0.148                              (Process Improvement)\n",
      "68     0.146                       (Leadership, Communications)\n",
      "97     0.144                   (Problem Solving, Data Analysis)\n",
      "101    0.144  (Tableau (Business Intelligence Software), Dat...\n",
      "71     0.142                       (Operations, Communications)\n",
      "103    0.142      (Python (Programming Language), Data Science)\n",
      "122    0.142  (Python (Programming Language), SQL (Programmi...\n",
      "46     0.140                         (R (Programming Language))\n",
      "106    0.140         (SQL (Programming Language), Data Science)\n",
      "90     0.140                           (Finance, Data Analysis)\n",
      "27     0.140                     (Interpersonal Communications)\n",
      "70     0.140                  (Microsoft Excel, Communications)\n",
      "1      0.138                                (Agile Methodology)\n",
      "50     0.138                                            (Sales)\n",
      "8      0.138                                 (Business Process)\n",
      "55     0.136                      (Verbal Communication Skills)\n",
      "37     0.136                             (Microsoft PowerPoint)\n",
      "99     0.136                          (Data Analysis, Research)\n",
      "28     0.136                (Key Performance Indicators (KPIs))\n",
      "112    0.134                      (Microsoft Excel, Management)\n",
      "84     0.134  (Computer Science, Python (Programming Language))\n",
      "67     0.134                          (Finance, Communications)\n",
      "26     0.134                                       (Innovation)\n",
      "61     0.132                    (Business Analysis, Management)\n",
      "23     0.132                                      (Forecasting)\n",
      "78     0.132                          (Writing, Communications)\n",
      "109    0.132                              (Finance, Management)\n",
      "12     0.130                                 (Customer Service)\n",
      "19     0.130                                  (Decision Making)\n",
      "2      0.128                                (Analytical Skills)\n",
      "98     0.128     (Python (Programming Language), Data Analysis)\n",
      "73     0.126                    (Presentations, Communications)\n",
      "115    0.126           (SQL (Programming Language), Management)\n",
      "34     0.124                                 (Microsoft Access)\n",
      "113    0.124                           (Operations, Management)\n",
      "65     0.124                     (Data Science, Communications)\n",
      "107    0.122   (Data Visualization, SQL (Programming Language))\n",
      "118    0.122            (Microsoft PowerPoint, Microsoft Excel)\n",
      "91     0.122                        (Leadership, Data Analysis)\n",
      "21     0.120                                        (Economics)\n",
      "75     0.120               (Project Management, Communications)\n",
      "96     0.118                        (Operations, Data Analysis)\n",
      "121    0.116  (Python (Programming Language), R (Programming...\n",
      "9      0.116                            (Business Requirements)\n",
      "66     0.116                  (Detail Oriented, Communications)\n",
      "81     0.116                     (Computer Science, Management)\n",
      "111    0.114                           (Leadership, Management)\n",
      "102    0.114                        (Mathematics, Data Science)\n",
      "4      0.114                                       (Automation)\n",
      "108    0.112  (Tableau (Business Intelligence Software), Dat...\n",
      "39     0.112                            (Organizational Skills)\n",
      "126    0.112        (Management, Data Analysis, Communications)\n",
      "128    0.112      (Problem Solving, Management, Communications)\n",
      "130    0.112  (Computer Science, Data Analysis, SQL (Program...\n",
      "120    0.112      (Problem Solving, SQL (Programming Language))\n",
      "119    0.110      (SQL (Programming Language), Microsoft Excel)\n",
      "51     0.110                                  (Self-Motivation)\n",
      "129    0.110    (Computer Science, Data Analysis, Data Science)\n",
      "76     0.110                         (Research, Communications)\n",
      "132    0.110  (Tableau (Business Intelligence Software), Dat...\n",
      "116    0.108                              (Writing, Management)\n",
      "83     0.108                (Problem Solving, Computer Science)\n",
      "16     0.108                                    (Data Modeling)\n",
      "94     0.106                       (Data Analysis, Mathematics)\n",
      "52     0.106                                       (Statistics)\n",
      "25     0.106                           (Information Technology)\n",
      "117    0.106          (Mathematics, SQL (Programming Language))\n",
      "62     0.106               (Problem Solving, Business Analysis)\n",
      "24     0.104                               (Influencing Skills)\n",
      "3      0.104                                         (Auditing)\n",
      "104    0.104           (R (Programming Language), Data Science)\n",
      "59     0.104                           (Accounting, Management)\n",
      "30     0.104                                 (Machine Learning)\n",
      "88     0.104                (Data Visualization, Data Analysis)\n",
      "54     0.104                (Troubleshooting (Problem Solving))\n",
      "15     0.104                                  (Data Management)\n",
      "131    0.104  (Computer Science, Python (Programming Languag...\n",
      "36     0.104                                 (Microsoft Office)\n",
      "125    0.104    (Management, Business Analysis, Communications)\n",
      "89     0.102                   (Data Analysis, Detail Oriented)\n",
      "82     0.102                    (Computer Science, Mathematics)\n",
      "105    0.102                           (Research, Data Science)\n",
      "123    0.102  (R (Programming Language), SQL (Programming La...\n",
      "93     0.102                         (Data Analysis, Marketing)\n",
      "48     0.102                                   (SAS (Software))\n",
      "110    0.102                         (Finance, Microsoft Excel)\n",
      "133    0.102  (Python (Programming Language), SQL (Programmi...\n",
      "127    0.100  (Data Analysis, SQL (Programming Language), Co...\n",
      "57     0.100                        (Accounting, Data Analysis)\n",
      "86     0.100                         (Dashboard, Data Analysis)\n",
      "72     0.100                         (Planning, Communications)\n"
     ]
    }
   ],
   "source": [
    "# Display the frequent itemsets and association rules\n",
    "print(\"Frequent Itemsets:\")\n",
    "print(frequent_itemsets.sort_values(by='support', ascending=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5c94ca88",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Association Rules:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>antecedents</th>\n",
       "      <th>consequents</th>\n",
       "      <th>antecedent support</th>\n",
       "      <th>consequent support</th>\n",
       "      <th>support</th>\n",
       "      <th>confidence</th>\n",
       "      <th>lift</th>\n",
       "      <th>leverage</th>\n",
       "      <th>conviction</th>\n",
       "      <th>zhangs_metric</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(Accounting)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.184</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.543478</td>\n",
       "      <td>1.049186</td>\n",
       "      <td>0.004688</td>\n",
       "      <td>1.055810</td>\n",
       "      <td>0.057451</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(Accounting)</td>\n",
       "      <td>(Finance)</td>\n",
       "      <td>0.184</td>\n",
       "      <td>0.258</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.815217</td>\n",
       "      <td>3.159757</td>\n",
       "      <td>0.102528</td>\n",
       "      <td>4.015529</td>\n",
       "      <td>0.837647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(Finance)</td>\n",
       "      <td>(Accounting)</td>\n",
       "      <td>0.258</td>\n",
       "      <td>0.184</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.581395</td>\n",
       "      <td>3.159757</td>\n",
       "      <td>0.102528</td>\n",
       "      <td>1.949333</td>\n",
       "      <td>0.921186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(Accounting)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.184</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.565217</td>\n",
       "      <td>1.441881</td>\n",
       "      <td>0.031872</td>\n",
       "      <td>1.398400</td>\n",
       "      <td>0.375566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(Business Analysis)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.292</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.602740</td>\n",
       "      <td>1.220121</td>\n",
       "      <td>0.031752</td>\n",
       "      <td>1.273724</td>\n",
       "      <td>0.254815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>(Detail Oriented)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.196</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.116</td>\n",
       "      <td>0.591837</td>\n",
       "      <td>1.198050</td>\n",
       "      <td>0.019176</td>\n",
       "      <td>1.239700</td>\n",
       "      <td>0.205610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>(Finance)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.258</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.134</td>\n",
       "      <td>0.519380</td>\n",
       "      <td>1.051376</td>\n",
       "      <td>0.006548</td>\n",
       "      <td>1.052806</td>\n",
       "      <td>0.065857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>(Leadership)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.222</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.146</td>\n",
       "      <td>0.657658</td>\n",
       "      <td>1.331291</td>\n",
       "      <td>0.036332</td>\n",
       "      <td>1.478053</td>\n",
       "      <td>0.319858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>(Management)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.250</td>\n",
       "      <td>0.637755</td>\n",
       "      <td>1.291002</td>\n",
       "      <td>0.056352</td>\n",
       "      <td>1.396845</td>\n",
       "      <td>0.370737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>(Communications)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.250</td>\n",
       "      <td>0.506073</td>\n",
       "      <td>1.291002</td>\n",
       "      <td>0.056352</td>\n",
       "      <td>1.230951</td>\n",
       "      <td>0.445470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>(Microsoft Excel)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.274</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.510949</td>\n",
       "      <td>1.034310</td>\n",
       "      <td>0.004644</td>\n",
       "      <td>1.034657</td>\n",
       "      <td>0.045691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>(Operations)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.238</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.596639</td>\n",
       "      <td>1.207771</td>\n",
       "      <td>0.024428</td>\n",
       "      <td>1.254458</td>\n",
       "      <td>0.225759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>(Planning)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.568182</td>\n",
       "      <td>1.150166</td>\n",
       "      <td>0.013056</td>\n",
       "      <td>1.171789</td>\n",
       "      <td>0.158447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>(Presentations)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.126</td>\n",
       "      <td>0.656250</td>\n",
       "      <td>1.328441</td>\n",
       "      <td>0.031152</td>\n",
       "      <td>1.472000</td>\n",
       "      <td>0.305988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>(Problem Solving)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.310</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.190</td>\n",
       "      <td>0.612903</td>\n",
       "      <td>1.240695</td>\n",
       "      <td>0.036860</td>\n",
       "      <td>1.307167</td>\n",
       "      <td>0.281159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>(Project Management)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.681818</td>\n",
       "      <td>1.380199</td>\n",
       "      <td>0.033056</td>\n",
       "      <td>1.590286</td>\n",
       "      <td>0.334304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>(Writing)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.226</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.132</td>\n",
       "      <td>0.584071</td>\n",
       "      <td>1.182330</td>\n",
       "      <td>0.020356</td>\n",
       "      <td>1.216553</td>\n",
       "      <td>0.199240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.186</td>\n",
       "      <td>0.560241</td>\n",
       "      <td>1.081546</td>\n",
       "      <td>0.014024</td>\n",
       "      <td>1.096055</td>\n",
       "      <td>0.112871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.162</td>\n",
       "      <td>0.586957</td>\n",
       "      <td>1.767941</td>\n",
       "      <td>0.070368</td>\n",
       "      <td>1.617263</td>\n",
       "      <td>0.599959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>(Mathematics)</td>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.579545</td>\n",
       "      <td>1.745619</td>\n",
       "      <td>0.043568</td>\n",
       "      <td>1.588757</td>\n",
       "      <td>0.518370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.134</td>\n",
       "      <td>0.650485</td>\n",
       "      <td>1.959293</td>\n",
       "      <td>0.065608</td>\n",
       "      <td>1.911222</td>\n",
       "      <td>0.616640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.172</td>\n",
       "      <td>0.518072</td>\n",
       "      <td>1.363348</td>\n",
       "      <td>0.045840</td>\n",
       "      <td>1.286500</td>\n",
       "      <td>0.398970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>(Dashboard)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.158</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.632911</td>\n",
       "      <td>1.221837</td>\n",
       "      <td>0.018156</td>\n",
       "      <td>1.313034</td>\n",
       "      <td>0.215629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.637681</td>\n",
       "      <td>1.231045</td>\n",
       "      <td>0.033032</td>\n",
       "      <td>1.330320</td>\n",
       "      <td>0.259229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>(Data Visualization)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.541667</td>\n",
       "      <td>1.045689</td>\n",
       "      <td>0.004544</td>\n",
       "      <td>1.051636</td>\n",
       "      <td>0.054075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>(Detail Oriented)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.196</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.520408</td>\n",
       "      <td>1.004649</td>\n",
       "      <td>0.000472</td>\n",
       "      <td>1.005021</td>\n",
       "      <td>0.005756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>(Finance)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.258</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.542636</td>\n",
       "      <td>1.047559</td>\n",
       "      <td>0.006356</td>\n",
       "      <td>1.053864</td>\n",
       "      <td>0.061186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>(Leadership)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.222</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.122</td>\n",
       "      <td>0.549550</td>\n",
       "      <td>1.060906</td>\n",
       "      <td>0.007004</td>\n",
       "      <td>1.070040</td>\n",
       "      <td>0.073792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>(Management)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.196</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.965251</td>\n",
       "      <td>-0.007056</td>\n",
       "      <td>0.964000</td>\n",
       "      <td>-0.055901</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>(Marketing)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.158</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.645570</td>\n",
       "      <td>1.246273</td>\n",
       "      <td>0.020156</td>\n",
       "      <td>1.359929</td>\n",
       "      <td>0.234689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>(Mathematics)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.106</td>\n",
       "      <td>0.602273</td>\n",
       "      <td>1.162689</td>\n",
       "      <td>0.014832</td>\n",
       "      <td>1.211886</td>\n",
       "      <td>0.169811</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>(Microsoft Excel)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.274</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.160</td>\n",
       "      <td>0.583942</td>\n",
       "      <td>1.127300</td>\n",
       "      <td>0.018068</td>\n",
       "      <td>1.158491</td>\n",
       "      <td>0.155544</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.128</td>\n",
       "      <td>0.621359</td>\n",
       "      <td>1.199535</td>\n",
       "      <td>0.021292</td>\n",
       "      <td>1.272974</td>\n",
       "      <td>0.209501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>(Research)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.238</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.136</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>1.103144</td>\n",
       "      <td>0.012716</td>\n",
       "      <td>1.124667</td>\n",
       "      <td>0.122703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.240</td>\n",
       "      <td>0.631579</td>\n",
       "      <td>1.219264</td>\n",
       "      <td>0.043160</td>\n",
       "      <td>1.308286</td>\n",
       "      <td>0.290054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>(Tableau (Business Intelligence Software))</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.144</td>\n",
       "      <td>0.685714</td>\n",
       "      <td>1.323773</td>\n",
       "      <td>0.035220</td>\n",
       "      <td>1.533636</td>\n",
       "      <td>0.309599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>(Mathematics)</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.114</td>\n",
       "      <td>0.647727</td>\n",
       "      <td>2.346838</td>\n",
       "      <td>0.065424</td>\n",
       "      <td>2.055226</td>\n",
       "      <td>0.696474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.689320</td>\n",
       "      <td>2.497538</td>\n",
       "      <td>0.085144</td>\n",
       "      <td>2.330375</td>\n",
       "      <td>0.755171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.514493</td>\n",
       "      <td>2.497538</td>\n",
       "      <td>0.085144</td>\n",
       "      <td>1.635403</td>\n",
       "      <td>0.828185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>(R (Programming Language))</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.742857</td>\n",
       "      <td>2.691511</td>\n",
       "      <td>0.065360</td>\n",
       "      <td>2.815556</td>\n",
       "      <td>0.730769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.507246</td>\n",
       "      <td>1.334859</td>\n",
       "      <td>0.035120</td>\n",
       "      <td>1.258235</td>\n",
       "      <td>0.346488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>(Data Visualization)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.122</td>\n",
       "      <td>0.635417</td>\n",
       "      <td>1.672149</td>\n",
       "      <td>0.049040</td>\n",
       "      <td>1.700571</td>\n",
       "      <td>0.497484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>(Tableau (Business Intelligence Software))</td>\n",
       "      <td>(Data Visualization)</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.533333</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>0.071680</td>\n",
       "      <td>1.731429</td>\n",
       "      <td>0.810127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>(Data Visualization)</td>\n",
       "      <td>(Tableau (Business Intelligence Software))</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.583333</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>0.071680</td>\n",
       "      <td>1.896000</td>\n",
       "      <td>0.792079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>(Finance)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.258</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.132</td>\n",
       "      <td>0.511628</td>\n",
       "      <td>1.305173</td>\n",
       "      <td>0.030864</td>\n",
       "      <td>1.244952</td>\n",
       "      <td>0.315119</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>(Leadership)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.222</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.114</td>\n",
       "      <td>0.513514</td>\n",
       "      <td>1.309983</td>\n",
       "      <td>0.026976</td>\n",
       "      <td>1.249778</td>\n",
       "      <td>0.304154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>(Operations)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.238</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.124</td>\n",
       "      <td>0.521008</td>\n",
       "      <td>1.329103</td>\n",
       "      <td>0.030704</td>\n",
       "      <td>1.269333</td>\n",
       "      <td>0.324951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>(Mathematics)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.106</td>\n",
       "      <td>0.602273</td>\n",
       "      <td>1.584928</td>\n",
       "      <td>0.039120</td>\n",
       "      <td>1.558857</td>\n",
       "      <td>0.447884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>(Microsoft PowerPoint)</td>\n",
       "      <td>(Microsoft Excel)</td>\n",
       "      <td>0.136</td>\n",
       "      <td>0.274</td>\n",
       "      <td>0.122</td>\n",
       "      <td>0.897059</td>\n",
       "      <td>3.273937</td>\n",
       "      <td>0.084736</td>\n",
       "      <td>7.052571</td>\n",
       "      <td>0.803886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(R (Programming Language))</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.116</td>\n",
       "      <td>0.563107</td>\n",
       "      <td>4.022191</td>\n",
       "      <td>0.087160</td>\n",
       "      <td>1.968444</td>\n",
       "      <td>0.946322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>(R (Programming Language))</td>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.116</td>\n",
       "      <td>0.828571</td>\n",
       "      <td>4.022191</td>\n",
       "      <td>0.087160</td>\n",
       "      <td>4.631667</td>\n",
       "      <td>0.873697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.689320</td>\n",
       "      <td>1.814001</td>\n",
       "      <td>0.063720</td>\n",
       "      <td>1.995625</td>\n",
       "      <td>0.565154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>(R (Programming Language))</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.728571</td>\n",
       "      <td>1.917293</td>\n",
       "      <td>0.048800</td>\n",
       "      <td>2.284211</td>\n",
       "      <td>0.556316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>(Tableau (Business Intelligence Software))</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.148</td>\n",
       "      <td>0.704762</td>\n",
       "      <td>1.854637</td>\n",
       "      <td>0.068200</td>\n",
       "      <td>2.100000</td>\n",
       "      <td>0.583305</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>(Business Analysis, Management)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.132</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.787879</td>\n",
       "      <td>1.594896</td>\n",
       "      <td>0.038792</td>\n",
       "      <td>2.385429</td>\n",
       "      <td>0.429724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>(Business Analysis, Communications)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.590909</td>\n",
       "      <td>1.507421</td>\n",
       "      <td>0.035008</td>\n",
       "      <td>1.486222</td>\n",
       "      <td>0.408514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>(Data Analysis, Management)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.196</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>1.156738</td>\n",
       "      <td>0.015176</td>\n",
       "      <td>1.180667</td>\n",
       "      <td>0.168532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>(SQL (Programming Language), Communications)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.166</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.602410</td>\n",
       "      <td>1.162953</td>\n",
       "      <td>0.014012</td>\n",
       "      <td>1.212303</td>\n",
       "      <td>0.168010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>(Problem Solving, Management)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.746667</td>\n",
       "      <td>1.511471</td>\n",
       "      <td>0.037900</td>\n",
       "      <td>1.997368</td>\n",
       "      <td>0.398109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>(Problem Solving, Communications)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.190</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.589474</td>\n",
       "      <td>1.503759</td>\n",
       "      <td>0.037520</td>\n",
       "      <td>1.481026</td>\n",
       "      <td>0.413580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>(Computer Science, Data Analysis)</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.186</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.591398</td>\n",
       "      <td>2.142746</td>\n",
       "      <td>0.058664</td>\n",
       "      <td>1.771895</td>\n",
       "      <td>0.655171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>(Computer Science, Data Science)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.162</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.679012</td>\n",
       "      <td>1.310835</td>\n",
       "      <td>0.026084</td>\n",
       "      <td>1.501615</td>\n",
       "      <td>0.282968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>(Data Analysis, Data Science)</td>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.625000</td>\n",
       "      <td>1.882530</td>\n",
       "      <td>0.051568</td>\n",
       "      <td>1.781333</td>\n",
       "      <td>0.568932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>(Computer Science, Data Analysis)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.186</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.602151</td>\n",
       "      <td>1.584607</td>\n",
       "      <td>0.041320</td>\n",
       "      <td>1.558378</td>\n",
       "      <td>0.453229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>(Computer Science, SQL (Programming Language))</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.172</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.651163</td>\n",
       "      <td>1.257071</td>\n",
       "      <td>0.022904</td>\n",
       "      <td>1.381733</td>\n",
       "      <td>0.246981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>(Computer Science, Python (Programming Language))</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.134</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.776119</td>\n",
       "      <td>2.812027</td>\n",
       "      <td>0.067016</td>\n",
       "      <td>3.233867</td>\n",
       "      <td>0.744093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>(Computer Science, Data Science)</td>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>0.162</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.641975</td>\n",
       "      <td>3.116385</td>\n",
       "      <td>0.070628</td>\n",
       "      <td>2.217724</td>\n",
       "      <td>0.810400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>(Python (Programming Language), Data Science)</td>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.732394</td>\n",
       "      <td>2.206007</td>\n",
       "      <td>0.056856</td>\n",
       "      <td>2.496211</td>\n",
       "      <td>0.637171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(Computer Science, Data Science)</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.162</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.504854</td>\n",
       "      <td>3.116385</td>\n",
       "      <td>0.070628</td>\n",
       "      <td>1.692431</td>\n",
       "      <td>0.855309</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>(Tableau (Business Intelligence Software), Dat...</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.144</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.763889</td>\n",
       "      <td>2.010234</td>\n",
       "      <td>0.055280</td>\n",
       "      <td>2.625882</td>\n",
       "      <td>0.587086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>(Tableau (Business Intelligence Software), SQL...</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.148</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.743243</td>\n",
       "      <td>1.434833</td>\n",
       "      <td>0.033336</td>\n",
       "      <td>1.877263</td>\n",
       "      <td>0.355698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>(Tableau (Business Intelligence Software))</td>\n",
       "      <td>(Data Analysis, SQL (Programming Language))</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.240</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.523810</td>\n",
       "      <td>2.182540</td>\n",
       "      <td>0.059600</td>\n",
       "      <td>1.596000</td>\n",
       "      <td>0.685846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>(Python (Programming Language), SQL (Programmi...</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.718310</td>\n",
       "      <td>2.602572</td>\n",
       "      <td>0.062808</td>\n",
       "      <td>2.570200</td>\n",
       "      <td>0.717674</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>(Python (Programming Language), Data Science)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.718310</td>\n",
       "      <td>1.890289</td>\n",
       "      <td>0.048040</td>\n",
       "      <td>2.201000</td>\n",
       "      <td>0.548928</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>(SQL (Programming Language), Data Science)</td>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.728571</td>\n",
       "      <td>3.536755</td>\n",
       "      <td>0.073160</td>\n",
       "      <td>2.925263</td>\n",
       "      <td>0.834017</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          antecedents  \\\n",
       "0                                        (Accounting)   \n",
       "1                                        (Accounting)   \n",
       "2                                           (Finance)   \n",
       "3                                        (Accounting)   \n",
       "4                                 (Business Analysis)   \n",
       "5                                   (Detail Oriented)   \n",
       "6                                           (Finance)   \n",
       "7                                        (Leadership)   \n",
       "8                                        (Management)   \n",
       "9                                    (Communications)   \n",
       "10                                  (Microsoft Excel)   \n",
       "11                                       (Operations)   \n",
       "12                                         (Planning)   \n",
       "13                                    (Presentations)   \n",
       "14                                  (Problem Solving)   \n",
       "15                               (Project Management)   \n",
       "16                                          (Writing)   \n",
       "17                                 (Computer Science)   \n",
       "18                                     (Data Science)   \n",
       "19                                      (Mathematics)   \n",
       "20                    (Python (Programming Language))   \n",
       "21                                 (Computer Science)   \n",
       "22                                        (Dashboard)   \n",
       "23                                     (Data Science)   \n",
       "24                               (Data Visualization)   \n",
       "25                                  (Detail Oriented)   \n",
       "26                                          (Finance)   \n",
       "27                                       (Leadership)   \n",
       "28                                       (Management)   \n",
       "29                                        (Marketing)   \n",
       "30                                      (Mathematics)   \n",
       "31                                  (Microsoft Excel)   \n",
       "32                    (Python (Programming Language))   \n",
       "33                                         (Research)   \n",
       "34                       (SQL (Programming Language))   \n",
       "35         (Tableau (Business Intelligence Software))   \n",
       "36                                      (Mathematics)   \n",
       "37                    (Python (Programming Language))   \n",
       "38                                     (Data Science)   \n",
       "39                         (R (Programming Language))   \n",
       "40                                     (Data Science)   \n",
       "41                               (Data Visualization)   \n",
       "42         (Tableau (Business Intelligence Software))   \n",
       "43                               (Data Visualization)   \n",
       "44                                          (Finance)   \n",
       "45                                       (Leadership)   \n",
       "46                                       (Operations)   \n",
       "47                                      (Mathematics)   \n",
       "48                             (Microsoft PowerPoint)   \n",
       "49                    (Python (Programming Language))   \n",
       "50                         (R (Programming Language))   \n",
       "51                    (Python (Programming Language))   \n",
       "52                         (R (Programming Language))   \n",
       "53         (Tableau (Business Intelligence Software))   \n",
       "54                    (Business Analysis, Management)   \n",
       "55                (Business Analysis, Communications)   \n",
       "56                        (Data Analysis, Management)   \n",
       "57       (SQL (Programming Language), Communications)   \n",
       "58                      (Problem Solving, Management)   \n",
       "59                  (Problem Solving, Communications)   \n",
       "60                  (Computer Science, Data Analysis)   \n",
       "61                   (Computer Science, Data Science)   \n",
       "62                      (Data Analysis, Data Science)   \n",
       "63                  (Computer Science, Data Analysis)   \n",
       "64     (Computer Science, SQL (Programming Language))   \n",
       "65  (Computer Science, Python (Programming Language))   \n",
       "66                   (Computer Science, Data Science)   \n",
       "67      (Python (Programming Language), Data Science)   \n",
       "68                    (Python (Programming Language))   \n",
       "69  (Tableau (Business Intelligence Software), Dat...   \n",
       "70  (Tableau (Business Intelligence Software), SQL...   \n",
       "71         (Tableau (Business Intelligence Software))   \n",
       "72  (Python (Programming Language), SQL (Programmi...   \n",
       "73      (Python (Programming Language), Data Science)   \n",
       "74         (SQL (Programming Language), Data Science)   \n",
       "\n",
       "                                    consequents  antecedent support  \\\n",
       "0                               (Data Analysis)               0.184   \n",
       "1                                     (Finance)               0.184   \n",
       "2                                  (Accounting)               0.258   \n",
       "3                                  (Management)               0.184   \n",
       "4                              (Communications)               0.292   \n",
       "5                              (Communications)               0.196   \n",
       "6                              (Communications)               0.258   \n",
       "7                              (Communications)               0.222   \n",
       "8                              (Communications)               0.392   \n",
       "9                                  (Management)               0.494   \n",
       "10                             (Communications)               0.274   \n",
       "11                             (Communications)               0.238   \n",
       "12                             (Communications)               0.176   \n",
       "13                             (Communications)               0.192   \n",
       "14                             (Communications)               0.310   \n",
       "15                             (Communications)               0.176   \n",
       "16                             (Communications)               0.226   \n",
       "17                              (Data Analysis)               0.332   \n",
       "18                           (Computer Science)               0.276   \n",
       "19                           (Computer Science)               0.176   \n",
       "20                           (Computer Science)               0.206   \n",
       "21                 (SQL (Programming Language))               0.332   \n",
       "22                              (Data Analysis)               0.158   \n",
       "23                              (Data Analysis)               0.276   \n",
       "24                              (Data Analysis)               0.192   \n",
       "25                              (Data Analysis)               0.196   \n",
       "26                              (Data Analysis)               0.258   \n",
       "27                              (Data Analysis)               0.222   \n",
       "28                              (Data Analysis)               0.392   \n",
       "29                              (Data Analysis)               0.158   \n",
       "30                              (Data Analysis)               0.176   \n",
       "31                              (Data Analysis)               0.274   \n",
       "32                              (Data Analysis)               0.206   \n",
       "33                              (Data Analysis)               0.238   \n",
       "34                              (Data Analysis)               0.380   \n",
       "35                              (Data Analysis)               0.210   \n",
       "36                               (Data Science)               0.176   \n",
       "37                               (Data Science)               0.206   \n",
       "38              (Python (Programming Language))               0.276   \n",
       "39                               (Data Science)               0.140   \n",
       "40                 (SQL (Programming Language))               0.276   \n",
       "41                 (SQL (Programming Language))               0.192   \n",
       "42                         (Data Visualization)               0.210   \n",
       "43   (Tableau (Business Intelligence Software))               0.192   \n",
       "44                                 (Management)               0.258   \n",
       "45                                 (Management)               0.222   \n",
       "46                                 (Management)               0.238   \n",
       "47                 (SQL (Programming Language))               0.176   \n",
       "48                            (Microsoft Excel)               0.136   \n",
       "49                   (R (Programming Language))               0.206   \n",
       "50              (Python (Programming Language))               0.140   \n",
       "51                 (SQL (Programming Language))               0.206   \n",
       "52                 (SQL (Programming Language))               0.140   \n",
       "53                 (SQL (Programming Language))               0.210   \n",
       "54                             (Communications)               0.132   \n",
       "55                                 (Management)               0.176   \n",
       "56                             (Communications)               0.196   \n",
       "57                              (Data Analysis)               0.166   \n",
       "58                             (Communications)               0.150   \n",
       "59                                 (Management)               0.190   \n",
       "60                               (Data Science)               0.186   \n",
       "61                              (Data Analysis)               0.162   \n",
       "62                           (Computer Science)               0.176   \n",
       "63                 (SQL (Programming Language))               0.186   \n",
       "64                              (Data Analysis)               0.172   \n",
       "65                               (Data Science)               0.134   \n",
       "66              (Python (Programming Language))               0.162   \n",
       "67                           (Computer Science)               0.142   \n",
       "68             (Computer Science, Data Science)               0.206   \n",
       "69                 (SQL (Programming Language))               0.144   \n",
       "70                              (Data Analysis)               0.148   \n",
       "71  (Data Analysis, SQL (Programming Language))               0.210   \n",
       "72                               (Data Science)               0.142   \n",
       "73                 (SQL (Programming Language))               0.142   \n",
       "74              (Python (Programming Language))               0.140   \n",
       "\n",
       "    consequent support  support  confidence      lift  leverage  conviction  \\\n",
       "0                0.518    0.100    0.543478  1.049186  0.004688    1.055810   \n",
       "1                0.258    0.150    0.815217  3.159757  0.102528    4.015529   \n",
       "2                0.184    0.150    0.581395  3.159757  0.102528    1.949333   \n",
       "3                0.392    0.104    0.565217  1.441881  0.031872    1.398400   \n",
       "4                0.494    0.176    0.602740  1.220121  0.031752    1.273724   \n",
       "5                0.494    0.116    0.591837  1.198050  0.019176    1.239700   \n",
       "6                0.494    0.134    0.519380  1.051376  0.006548    1.052806   \n",
       "7                0.494    0.146    0.657658  1.331291  0.036332    1.478053   \n",
       "8                0.494    0.250    0.637755  1.291002  0.056352    1.396845   \n",
       "9                0.392    0.250    0.506073  1.291002  0.056352    1.230951   \n",
       "10               0.494    0.140    0.510949  1.034310  0.004644    1.034657   \n",
       "11               0.494    0.142    0.596639  1.207771  0.024428    1.254458   \n",
       "12               0.494    0.100    0.568182  1.150166  0.013056    1.171789   \n",
       "13               0.494    0.126    0.656250  1.328441  0.031152    1.472000   \n",
       "14               0.494    0.190    0.612903  1.240695  0.036860    1.307167   \n",
       "15               0.494    0.120    0.681818  1.380199  0.033056    1.590286   \n",
       "16               0.494    0.132    0.584071  1.182330  0.020356    1.216553   \n",
       "17               0.518    0.186    0.560241  1.081546  0.014024    1.096055   \n",
       "18               0.332    0.162    0.586957  1.767941  0.070368    1.617263   \n",
       "19               0.332    0.102    0.579545  1.745619  0.043568    1.588757   \n",
       "20               0.332    0.134    0.650485  1.959293  0.065608    1.911222   \n",
       "21               0.380    0.172    0.518072  1.363348  0.045840    1.286500   \n",
       "22               0.518    0.100    0.632911  1.221837  0.018156    1.313034   \n",
       "23               0.518    0.176    0.637681  1.231045  0.033032    1.330320   \n",
       "24               0.518    0.104    0.541667  1.045689  0.004544    1.051636   \n",
       "25               0.518    0.102    0.520408  1.004649  0.000472    1.005021   \n",
       "26               0.518    0.140    0.542636  1.047559  0.006356    1.053864   \n",
       "27               0.518    0.122    0.549550  1.060906  0.007004    1.070040   \n",
       "28               0.518    0.196    0.500000  0.965251 -0.007056    0.964000   \n",
       "29               0.518    0.102    0.645570  1.246273  0.020156    1.359929   \n",
       "30               0.518    0.106    0.602273  1.162689  0.014832    1.211886   \n",
       "31               0.518    0.160    0.583942  1.127300  0.018068    1.158491   \n",
       "32               0.518    0.128    0.621359  1.199535  0.021292    1.272974   \n",
       "33               0.518    0.136    0.571429  1.103144  0.012716    1.124667   \n",
       "34               0.518    0.240    0.631579  1.219264  0.043160    1.308286   \n",
       "35               0.518    0.144    0.685714  1.323773  0.035220    1.533636   \n",
       "36               0.276    0.114    0.647727  2.346838  0.065424    2.055226   \n",
       "37               0.276    0.142    0.689320  2.497538  0.085144    2.330375   \n",
       "38               0.206    0.142    0.514493  2.497538  0.085144    1.635403   \n",
       "39               0.276    0.104    0.742857  2.691511  0.065360    2.815556   \n",
       "40               0.380    0.140    0.507246  1.334859  0.035120    1.258235   \n",
       "41               0.380    0.122    0.635417  1.672149  0.049040    1.700571   \n",
       "42               0.192    0.112    0.533333  2.777778  0.071680    1.731429   \n",
       "43               0.210    0.112    0.583333  2.777778  0.071680    1.896000   \n",
       "44               0.392    0.132    0.511628  1.305173  0.030864    1.244952   \n",
       "45               0.392    0.114    0.513514  1.309983  0.026976    1.249778   \n",
       "46               0.392    0.124    0.521008  1.329103  0.030704    1.269333   \n",
       "47               0.380    0.106    0.602273  1.584928  0.039120    1.558857   \n",
       "48               0.274    0.122    0.897059  3.273937  0.084736    7.052571   \n",
       "49               0.140    0.116    0.563107  4.022191  0.087160    1.968444   \n",
       "50               0.206    0.116    0.828571  4.022191  0.087160    4.631667   \n",
       "51               0.380    0.142    0.689320  1.814001  0.063720    1.995625   \n",
       "52               0.380    0.102    0.728571  1.917293  0.048800    2.284211   \n",
       "53               0.380    0.148    0.704762  1.854637  0.068200    2.100000   \n",
       "54               0.494    0.104    0.787879  1.594896  0.038792    2.385429   \n",
       "55               0.392    0.104    0.590909  1.507421  0.035008    1.486222   \n",
       "56               0.494    0.112    0.571429  1.156738  0.015176    1.180667   \n",
       "57               0.518    0.100    0.602410  1.162953  0.014012    1.212303   \n",
       "58               0.494    0.112    0.746667  1.511471  0.037900    1.997368   \n",
       "59               0.392    0.112    0.589474  1.503759  0.037520    1.481026   \n",
       "60               0.276    0.110    0.591398  2.142746  0.058664    1.771895   \n",
       "61               0.518    0.110    0.679012  1.310835  0.026084    1.501615   \n",
       "62               0.332    0.110    0.625000  1.882530  0.051568    1.781333   \n",
       "63               0.380    0.112    0.602151  1.584607  0.041320    1.558378   \n",
       "64               0.518    0.112    0.651163  1.257071  0.022904    1.381733   \n",
       "65               0.276    0.104    0.776119  2.812027  0.067016    3.233867   \n",
       "66               0.206    0.104    0.641975  3.116385  0.070628    2.217724   \n",
       "67               0.332    0.104    0.732394  2.206007  0.056856    2.496211   \n",
       "68               0.162    0.104    0.504854  3.116385  0.070628    1.692431   \n",
       "69               0.380    0.110    0.763889  2.010234  0.055280    2.625882   \n",
       "70               0.518    0.110    0.743243  1.434833  0.033336    1.877263   \n",
       "71               0.240    0.110    0.523810  2.182540  0.059600    1.596000   \n",
       "72               0.276    0.102    0.718310  2.602572  0.062808    2.570200   \n",
       "73               0.380    0.102    0.718310  1.890289  0.048040    2.201000   \n",
       "74               0.206    0.102    0.728571  3.536755  0.073160    2.925263   \n",
       "\n",
       "    zhangs_metric  \n",
       "0        0.057451  \n",
       "1        0.837647  \n",
       "2        0.921186  \n",
       "3        0.375566  \n",
       "4        0.254815  \n",
       "5        0.205610  \n",
       "6        0.065857  \n",
       "7        0.319858  \n",
       "8        0.370737  \n",
       "9        0.445470  \n",
       "10       0.045691  \n",
       "11       0.225759  \n",
       "12       0.158447  \n",
       "13       0.305988  \n",
       "14       0.281159  \n",
       "15       0.334304  \n",
       "16       0.199240  \n",
       "17       0.112871  \n",
       "18       0.599959  \n",
       "19       0.518370  \n",
       "20       0.616640  \n",
       "21       0.398970  \n",
       "22       0.215629  \n",
       "23       0.259229  \n",
       "24       0.054075  \n",
       "25       0.005756  \n",
       "26       0.061186  \n",
       "27       0.073792  \n",
       "28      -0.055901  \n",
       "29       0.234689  \n",
       "30       0.169811  \n",
       "31       0.155544  \n",
       "32       0.209501  \n",
       "33       0.122703  \n",
       "34       0.290054  \n",
       "35       0.309599  \n",
       "36       0.696474  \n",
       "37       0.755171  \n",
       "38       0.828185  \n",
       "39       0.730769  \n",
       "40       0.346488  \n",
       "41       0.497484  \n",
       "42       0.810127  \n",
       "43       0.792079  \n",
       "44       0.315119  \n",
       "45       0.304154  \n",
       "46       0.324951  \n",
       "47       0.447884  \n",
       "48       0.803886  \n",
       "49       0.946322  \n",
       "50       0.873697  \n",
       "51       0.565154  \n",
       "52       0.556316  \n",
       "53       0.583305  \n",
       "54       0.429724  \n",
       "55       0.408514  \n",
       "56       0.168532  \n",
       "57       0.168010  \n",
       "58       0.398109  \n",
       "59       0.413580  \n",
       "60       0.655171  \n",
       "61       0.282968  \n",
       "62       0.568932  \n",
       "63       0.453229  \n",
       "64       0.246981  \n",
       "65       0.744093  \n",
       "66       0.810400  \n",
       "67       0.637171  \n",
       "68       0.855309  \n",
       "69       0.587086  \n",
       "70       0.355698  \n",
       "71       0.685846  \n",
       "72       0.717674  \n",
       "73       0.548928  \n",
       "74       0.834017  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"\\nAssociation Rules:\")\n",
    "rules"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7231ef72",
   "metadata": {},
   "source": [
    "**3. Filtering the Rules That has Strong Association**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "202310cb",
   "metadata": {},
   "source": [
    "In order to focus the insight extraction using Association Rules, we need to limit and prioritize the analysis on the set of rules that has strong association. Strong association here is arbitrarily defined by the analyst or model creator given the nature of the data.\n",
    "\n",
    "To extract the combinations of skills that has strong association, we will set the threshold to the Association Rule algorithm to only display the combinations of skills with the following metrics:\n",
    "\n",
    "1. Minimum Support = 0.1 --> meaning that only itemsets that appear in at least 10% of the transactions will be used for rule mining.\n",
    "\n",
    "2. Minimum Confidence = 0.2 --> meaning that only rules with a confidence of at least 20% will be selected.\n",
    "\n",
    "3. Minimum Lift = 1 --> meaning that only rules with a lift greater than 1 (indicating a positive association) will be selected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ac95be20",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>antecedents</th>\n",
       "      <th>consequents</th>\n",
       "      <th>antecedent support</th>\n",
       "      <th>consequent support</th>\n",
       "      <th>support</th>\n",
       "      <th>confidence</th>\n",
       "      <th>lift</th>\n",
       "      <th>leverage</th>\n",
       "      <th>conviction</th>\n",
       "      <th>zhangs_metric</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>(R (Programming Language))</td>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.116</td>\n",
       "      <td>0.828571</td>\n",
       "      <td>4.022191</td>\n",
       "      <td>0.087160</td>\n",
       "      <td>4.631667</td>\n",
       "      <td>0.873697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(R (Programming Language))</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.116</td>\n",
       "      <td>0.563107</td>\n",
       "      <td>4.022191</td>\n",
       "      <td>0.087160</td>\n",
       "      <td>1.968444</td>\n",
       "      <td>0.946322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>(SQL (Programming Language), Data Science)</td>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.728571</td>\n",
       "      <td>3.536755</td>\n",
       "      <td>0.073160</td>\n",
       "      <td>2.925263</td>\n",
       "      <td>0.834017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>(Microsoft PowerPoint)</td>\n",
       "      <td>(Microsoft Excel)</td>\n",
       "      <td>0.136</td>\n",
       "      <td>0.274</td>\n",
       "      <td>0.122</td>\n",
       "      <td>0.897059</td>\n",
       "      <td>3.273937</td>\n",
       "      <td>0.084736</td>\n",
       "      <td>7.052571</td>\n",
       "      <td>0.803886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>(Accounting)</td>\n",
       "      <td>(Finance)</td>\n",
       "      <td>0.184</td>\n",
       "      <td>0.258</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.815217</td>\n",
       "      <td>3.159757</td>\n",
       "      <td>0.102528</td>\n",
       "      <td>4.015529</td>\n",
       "      <td>0.837647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>(Finance)</td>\n",
       "      <td>(Accounting)</td>\n",
       "      <td>0.258</td>\n",
       "      <td>0.184</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.581395</td>\n",
       "      <td>3.159757</td>\n",
       "      <td>0.102528</td>\n",
       "      <td>1.949333</td>\n",
       "      <td>0.921186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>(Computer Science, Data Science)</td>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>0.162</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.641975</td>\n",
       "      <td>3.116385</td>\n",
       "      <td>0.070628</td>\n",
       "      <td>2.217724</td>\n",
       "      <td>0.810400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(Computer Science, Data Science)</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.162</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.504854</td>\n",
       "      <td>3.116385</td>\n",
       "      <td>0.070628</td>\n",
       "      <td>1.692431</td>\n",
       "      <td>0.855309</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>(Computer Science, Python (Programming Language))</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.134</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.776119</td>\n",
       "      <td>2.812027</td>\n",
       "      <td>0.067016</td>\n",
       "      <td>3.233867</td>\n",
       "      <td>0.744093</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>(Data Visualization)</td>\n",
       "      <td>(Tableau (Business Intelligence Software))</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.583333</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>0.071680</td>\n",
       "      <td>1.896000</td>\n",
       "      <td>0.792079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>(Tableau (Business Intelligence Software))</td>\n",
       "      <td>(Data Visualization)</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.533333</td>\n",
       "      <td>2.777778</td>\n",
       "      <td>0.071680</td>\n",
       "      <td>1.731429</td>\n",
       "      <td>0.810127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>(R (Programming Language))</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.742857</td>\n",
       "      <td>2.691511</td>\n",
       "      <td>0.065360</td>\n",
       "      <td>2.815556</td>\n",
       "      <td>0.730769</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>(Python (Programming Language), SQL (Programmi...</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.718310</td>\n",
       "      <td>2.602572</td>\n",
       "      <td>0.062808</td>\n",
       "      <td>2.570200</td>\n",
       "      <td>0.717674</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.689320</td>\n",
       "      <td>2.497538</td>\n",
       "      <td>0.085144</td>\n",
       "      <td>2.330375</td>\n",
       "      <td>0.755171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.514493</td>\n",
       "      <td>2.497538</td>\n",
       "      <td>0.085144</td>\n",
       "      <td>1.635403</td>\n",
       "      <td>0.828185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>(Mathematics)</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.114</td>\n",
       "      <td>0.647727</td>\n",
       "      <td>2.346838</td>\n",
       "      <td>0.065424</td>\n",
       "      <td>2.055226</td>\n",
       "      <td>0.696474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>(Python (Programming Language), Data Science)</td>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.732394</td>\n",
       "      <td>2.206007</td>\n",
       "      <td>0.056856</td>\n",
       "      <td>2.496211</td>\n",
       "      <td>0.637171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>(Tableau (Business Intelligence Software))</td>\n",
       "      <td>(Data Analysis, SQL (Programming Language))</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.240</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.523810</td>\n",
       "      <td>2.182540</td>\n",
       "      <td>0.059600</td>\n",
       "      <td>1.596000</td>\n",
       "      <td>0.685846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>(Computer Science, Data Analysis)</td>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>0.186</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.591398</td>\n",
       "      <td>2.142746</td>\n",
       "      <td>0.058664</td>\n",
       "      <td>1.771895</td>\n",
       "      <td>0.655171</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>(Tableau (Business Intelligence Software), Dat...</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.144</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.763889</td>\n",
       "      <td>2.010234</td>\n",
       "      <td>0.055280</td>\n",
       "      <td>2.625882</td>\n",
       "      <td>0.587086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.134</td>\n",
       "      <td>0.650485</td>\n",
       "      <td>1.959293</td>\n",
       "      <td>0.065608</td>\n",
       "      <td>1.911222</td>\n",
       "      <td>0.616640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>(R (Programming Language))</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.728571</td>\n",
       "      <td>1.917293</td>\n",
       "      <td>0.048800</td>\n",
       "      <td>2.284211</td>\n",
       "      <td>0.556316</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>(Python (Programming Language), Data Science)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.718310</td>\n",
       "      <td>1.890289</td>\n",
       "      <td>0.048040</td>\n",
       "      <td>2.201000</td>\n",
       "      <td>0.548928</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>(Data Analysis, Data Science)</td>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.625000</td>\n",
       "      <td>1.882530</td>\n",
       "      <td>0.051568</td>\n",
       "      <td>1.781333</td>\n",
       "      <td>0.568932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>(Tableau (Business Intelligence Software))</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.148</td>\n",
       "      <td>0.704762</td>\n",
       "      <td>1.854637</td>\n",
       "      <td>0.068200</td>\n",
       "      <td>2.100000</td>\n",
       "      <td>0.583305</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.689320</td>\n",
       "      <td>1.814001</td>\n",
       "      <td>0.063720</td>\n",
       "      <td>1.995625</td>\n",
       "      <td>0.565154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.162</td>\n",
       "      <td>0.586957</td>\n",
       "      <td>1.767941</td>\n",
       "      <td>0.070368</td>\n",
       "      <td>1.617263</td>\n",
       "      <td>0.599959</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>(Mathematics)</td>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.579545</td>\n",
       "      <td>1.745619</td>\n",
       "      <td>0.043568</td>\n",
       "      <td>1.588757</td>\n",
       "      <td>0.518370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>(Data Visualization)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.122</td>\n",
       "      <td>0.635417</td>\n",
       "      <td>1.672149</td>\n",
       "      <td>0.049040</td>\n",
       "      <td>1.700571</td>\n",
       "      <td>0.497484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>(Business Analysis, Management)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.132</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.787879</td>\n",
       "      <td>1.594896</td>\n",
       "      <td>0.038792</td>\n",
       "      <td>2.385429</td>\n",
       "      <td>0.429724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>(Mathematics)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.106</td>\n",
       "      <td>0.602273</td>\n",
       "      <td>1.584928</td>\n",
       "      <td>0.039120</td>\n",
       "      <td>1.558857</td>\n",
       "      <td>0.447884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>(Computer Science, Data Analysis)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.186</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.602151</td>\n",
       "      <td>1.584607</td>\n",
       "      <td>0.041320</td>\n",
       "      <td>1.558378</td>\n",
       "      <td>0.453229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>(Problem Solving, Management)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.150</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.746667</td>\n",
       "      <td>1.511471</td>\n",
       "      <td>0.037900</td>\n",
       "      <td>1.997368</td>\n",
       "      <td>0.398109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>(Business Analysis, Communications)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.590909</td>\n",
       "      <td>1.507421</td>\n",
       "      <td>0.035008</td>\n",
       "      <td>1.486222</td>\n",
       "      <td>0.408514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>(Problem Solving, Communications)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.190</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.589474</td>\n",
       "      <td>1.503759</td>\n",
       "      <td>0.037520</td>\n",
       "      <td>1.481026</td>\n",
       "      <td>0.413580</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>(Accounting)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.184</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.565217</td>\n",
       "      <td>1.441881</td>\n",
       "      <td>0.031872</td>\n",
       "      <td>1.398400</td>\n",
       "      <td>0.375566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>(Tableau (Business Intelligence Software), SQL...</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.148</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.743243</td>\n",
       "      <td>1.434833</td>\n",
       "      <td>0.033336</td>\n",
       "      <td>1.877263</td>\n",
       "      <td>0.355698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>(Project Management)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.681818</td>\n",
       "      <td>1.380199</td>\n",
       "      <td>0.033056</td>\n",
       "      <td>1.590286</td>\n",
       "      <td>0.334304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.172</td>\n",
       "      <td>0.518072</td>\n",
       "      <td>1.363348</td>\n",
       "      <td>0.045840</td>\n",
       "      <td>1.286500</td>\n",
       "      <td>0.398970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.507246</td>\n",
       "      <td>1.334859</td>\n",
       "      <td>0.035120</td>\n",
       "      <td>1.258235</td>\n",
       "      <td>0.346488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>(Leadership)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.222</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.146</td>\n",
       "      <td>0.657658</td>\n",
       "      <td>1.331291</td>\n",
       "      <td>0.036332</td>\n",
       "      <td>1.478053</td>\n",
       "      <td>0.319858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>(Operations)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.238</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.124</td>\n",
       "      <td>0.521008</td>\n",
       "      <td>1.329103</td>\n",
       "      <td>0.030704</td>\n",
       "      <td>1.269333</td>\n",
       "      <td>0.324951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>(Presentations)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.126</td>\n",
       "      <td>0.656250</td>\n",
       "      <td>1.328441</td>\n",
       "      <td>0.031152</td>\n",
       "      <td>1.472000</td>\n",
       "      <td>0.305988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>(Tableau (Business Intelligence Software))</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.144</td>\n",
       "      <td>0.685714</td>\n",
       "      <td>1.323773</td>\n",
       "      <td>0.035220</td>\n",
       "      <td>1.533636</td>\n",
       "      <td>0.309599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>(Computer Science, Data Science)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.162</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.110</td>\n",
       "      <td>0.679012</td>\n",
       "      <td>1.310835</td>\n",
       "      <td>0.026084</td>\n",
       "      <td>1.501615</td>\n",
       "      <td>0.282968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>(Leadership)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.222</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.114</td>\n",
       "      <td>0.513514</td>\n",
       "      <td>1.309983</td>\n",
       "      <td>0.026976</td>\n",
       "      <td>1.249778</td>\n",
       "      <td>0.304154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>(Finance)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.258</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.132</td>\n",
       "      <td>0.511628</td>\n",
       "      <td>1.305173</td>\n",
       "      <td>0.030864</td>\n",
       "      <td>1.244952</td>\n",
       "      <td>0.315119</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>(Management)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.250</td>\n",
       "      <td>0.637755</td>\n",
       "      <td>1.291002</td>\n",
       "      <td>0.056352</td>\n",
       "      <td>1.396845</td>\n",
       "      <td>0.370737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>(Communications)</td>\n",
       "      <td>(Management)</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.250</td>\n",
       "      <td>0.506073</td>\n",
       "      <td>1.291002</td>\n",
       "      <td>0.056352</td>\n",
       "      <td>1.230951</td>\n",
       "      <td>0.445470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>(Computer Science, SQL (Programming Language))</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.172</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.651163</td>\n",
       "      <td>1.257071</td>\n",
       "      <td>0.022904</td>\n",
       "      <td>1.381733</td>\n",
       "      <td>0.246981</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>(Marketing)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.158</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.645570</td>\n",
       "      <td>1.246273</td>\n",
       "      <td>0.020156</td>\n",
       "      <td>1.359929</td>\n",
       "      <td>0.234689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>(Problem Solving)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.310</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.190</td>\n",
       "      <td>0.612903</td>\n",
       "      <td>1.240695</td>\n",
       "      <td>0.036860</td>\n",
       "      <td>1.307167</td>\n",
       "      <td>0.281159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>(Data Science)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.276</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.637681</td>\n",
       "      <td>1.231045</td>\n",
       "      <td>0.033032</td>\n",
       "      <td>1.330320</td>\n",
       "      <td>0.259229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>(Dashboard)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.158</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.632911</td>\n",
       "      <td>1.221837</td>\n",
       "      <td>0.018156</td>\n",
       "      <td>1.313034</td>\n",
       "      <td>0.215629</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>(Business Analysis)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.292</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.602740</td>\n",
       "      <td>1.220121</td>\n",
       "      <td>0.031752</td>\n",
       "      <td>1.273724</td>\n",
       "      <td>0.254815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>(SQL (Programming Language))</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.380</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.240</td>\n",
       "      <td>0.631579</td>\n",
       "      <td>1.219264</td>\n",
       "      <td>0.043160</td>\n",
       "      <td>1.308286</td>\n",
       "      <td>0.290054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>(Operations)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.238</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.142</td>\n",
       "      <td>0.596639</td>\n",
       "      <td>1.207771</td>\n",
       "      <td>0.024428</td>\n",
       "      <td>1.254458</td>\n",
       "      <td>0.225759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>(Python (Programming Language))</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.206</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.128</td>\n",
       "      <td>0.621359</td>\n",
       "      <td>1.199535</td>\n",
       "      <td>0.021292</td>\n",
       "      <td>1.272974</td>\n",
       "      <td>0.209501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>(Detail Oriented)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.196</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.116</td>\n",
       "      <td>0.591837</td>\n",
       "      <td>1.198050</td>\n",
       "      <td>0.019176</td>\n",
       "      <td>1.239700</td>\n",
       "      <td>0.205610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>(Writing)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.226</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.132</td>\n",
       "      <td>0.584071</td>\n",
       "      <td>1.182330</td>\n",
       "      <td>0.020356</td>\n",
       "      <td>1.216553</td>\n",
       "      <td>0.199240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>(SQL (Programming Language), Communications)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.166</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.602410</td>\n",
       "      <td>1.162953</td>\n",
       "      <td>0.014012</td>\n",
       "      <td>1.212303</td>\n",
       "      <td>0.168010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>(Mathematics)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.106</td>\n",
       "      <td>0.602273</td>\n",
       "      <td>1.162689</td>\n",
       "      <td>0.014832</td>\n",
       "      <td>1.211886</td>\n",
       "      <td>0.169811</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>(Data Analysis, Management)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.196</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>1.156738</td>\n",
       "      <td>0.015176</td>\n",
       "      <td>1.180667</td>\n",
       "      <td>0.168532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>(Planning)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.568182</td>\n",
       "      <td>1.150166</td>\n",
       "      <td>0.013056</td>\n",
       "      <td>1.171789</td>\n",
       "      <td>0.158447</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>(Microsoft Excel)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.274</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.160</td>\n",
       "      <td>0.583942</td>\n",
       "      <td>1.127300</td>\n",
       "      <td>0.018068</td>\n",
       "      <td>1.158491</td>\n",
       "      <td>0.155544</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>(Research)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.238</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.136</td>\n",
       "      <td>0.571429</td>\n",
       "      <td>1.103144</td>\n",
       "      <td>0.012716</td>\n",
       "      <td>1.124667</td>\n",
       "      <td>0.122703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>(Computer Science)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.186</td>\n",
       "      <td>0.560241</td>\n",
       "      <td>1.081546</td>\n",
       "      <td>0.014024</td>\n",
       "      <td>1.096055</td>\n",
       "      <td>0.112871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>(Leadership)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.222</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.122</td>\n",
       "      <td>0.549550</td>\n",
       "      <td>1.060906</td>\n",
       "      <td>0.007004</td>\n",
       "      <td>1.070040</td>\n",
       "      <td>0.073792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>(Finance)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.258</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.134</td>\n",
       "      <td>0.519380</td>\n",
       "      <td>1.051376</td>\n",
       "      <td>0.006548</td>\n",
       "      <td>1.052806</td>\n",
       "      <td>0.065857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>(Accounting)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.184</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.543478</td>\n",
       "      <td>1.049186</td>\n",
       "      <td>0.004688</td>\n",
       "      <td>1.055810</td>\n",
       "      <td>0.057451</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>(Finance)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.258</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.542636</td>\n",
       "      <td>1.047559</td>\n",
       "      <td>0.006356</td>\n",
       "      <td>1.053864</td>\n",
       "      <td>0.061186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>(Data Visualization)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.541667</td>\n",
       "      <td>1.045689</td>\n",
       "      <td>0.004544</td>\n",
       "      <td>1.051636</td>\n",
       "      <td>0.054075</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>(Microsoft Excel)</td>\n",
       "      <td>(Communications)</td>\n",
       "      <td>0.274</td>\n",
       "      <td>0.494</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.510949</td>\n",
       "      <td>1.034310</td>\n",
       "      <td>0.004644</td>\n",
       "      <td>1.034657</td>\n",
       "      <td>0.045691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>(Detail Oriented)</td>\n",
       "      <td>(Data Analysis)</td>\n",
       "      <td>0.196</td>\n",
       "      <td>0.518</td>\n",
       "      <td>0.102</td>\n",
       "      <td>0.520408</td>\n",
       "      <td>1.004649</td>\n",
       "      <td>0.000472</td>\n",
       "      <td>1.005021</td>\n",
       "      <td>0.005756</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          antecedents  \\\n",
       "0                          (R (Programming Language))   \n",
       "1                     (Python (Programming Language))   \n",
       "2          (SQL (Programming Language), Data Science)   \n",
       "3                              (Microsoft PowerPoint)   \n",
       "4                                        (Accounting)   \n",
       "5                                           (Finance)   \n",
       "6                    (Computer Science, Data Science)   \n",
       "7                     (Python (Programming Language))   \n",
       "8   (Computer Science, Python (Programming Language))   \n",
       "9                                (Data Visualization)   \n",
       "10         (Tableau (Business Intelligence Software))   \n",
       "11                         (R (Programming Language))   \n",
       "12  (Python (Programming Language), SQL (Programmi...   \n",
       "13                    (Python (Programming Language))   \n",
       "14                                     (Data Science)   \n",
       "15                                      (Mathematics)   \n",
       "16      (Python (Programming Language), Data Science)   \n",
       "17         (Tableau (Business Intelligence Software))   \n",
       "18                  (Computer Science, Data Analysis)   \n",
       "19  (Tableau (Business Intelligence Software), Dat...   \n",
       "20                    (Python (Programming Language))   \n",
       "21                         (R (Programming Language))   \n",
       "22      (Python (Programming Language), Data Science)   \n",
       "23                      (Data Analysis, Data Science)   \n",
       "24         (Tableau (Business Intelligence Software))   \n",
       "25                    (Python (Programming Language))   \n",
       "26                                     (Data Science)   \n",
       "27                                      (Mathematics)   \n",
       "28                               (Data Visualization)   \n",
       "29                    (Business Analysis, Management)   \n",
       "30                                      (Mathematics)   \n",
       "31                  (Computer Science, Data Analysis)   \n",
       "32                      (Problem Solving, Management)   \n",
       "33                (Business Analysis, Communications)   \n",
       "34                  (Problem Solving, Communications)   \n",
       "35                                       (Accounting)   \n",
       "36  (Tableau (Business Intelligence Software), SQL...   \n",
       "37                               (Project Management)   \n",
       "38                                 (Computer Science)   \n",
       "39                                     (Data Science)   \n",
       "40                                       (Leadership)   \n",
       "41                                       (Operations)   \n",
       "42                                    (Presentations)   \n",
       "43         (Tableau (Business Intelligence Software))   \n",
       "44                   (Computer Science, Data Science)   \n",
       "45                                       (Leadership)   \n",
       "46                                          (Finance)   \n",
       "47                                       (Management)   \n",
       "48                                   (Communications)   \n",
       "49     (Computer Science, SQL (Programming Language))   \n",
       "50                                        (Marketing)   \n",
       "51                                  (Problem Solving)   \n",
       "52                                     (Data Science)   \n",
       "53                                        (Dashboard)   \n",
       "54                                (Business Analysis)   \n",
       "55                       (SQL (Programming Language))   \n",
       "56                                       (Operations)   \n",
       "57                    (Python (Programming Language))   \n",
       "58                                  (Detail Oriented)   \n",
       "59                                          (Writing)   \n",
       "60       (SQL (Programming Language), Communications)   \n",
       "61                                      (Mathematics)   \n",
       "62                        (Data Analysis, Management)   \n",
       "63                                         (Planning)   \n",
       "64                                  (Microsoft Excel)   \n",
       "65                                         (Research)   \n",
       "66                                 (Computer Science)   \n",
       "67                                       (Leadership)   \n",
       "68                                          (Finance)   \n",
       "69                                       (Accounting)   \n",
       "70                                          (Finance)   \n",
       "71                               (Data Visualization)   \n",
       "72                                  (Microsoft Excel)   \n",
       "73                                  (Detail Oriented)   \n",
       "\n",
       "                                    consequents  antecedent support  \\\n",
       "0               (Python (Programming Language))               0.140   \n",
       "1                    (R (Programming Language))               0.206   \n",
       "2               (Python (Programming Language))               0.140   \n",
       "3                             (Microsoft Excel)               0.136   \n",
       "4                                     (Finance)               0.184   \n",
       "5                                  (Accounting)               0.258   \n",
       "6               (Python (Programming Language))               0.162   \n",
       "7              (Computer Science, Data Science)               0.206   \n",
       "8                                (Data Science)               0.134   \n",
       "9    (Tableau (Business Intelligence Software))               0.192   \n",
       "10                         (Data Visualization)               0.210   \n",
       "11                               (Data Science)               0.140   \n",
       "12                               (Data Science)               0.142   \n",
       "13                               (Data Science)               0.206   \n",
       "14              (Python (Programming Language))               0.276   \n",
       "15                               (Data Science)               0.176   \n",
       "16                           (Computer Science)               0.142   \n",
       "17  (Data Analysis, SQL (Programming Language))               0.210   \n",
       "18                               (Data Science)               0.186   \n",
       "19                 (SQL (Programming Language))               0.144   \n",
       "20                           (Computer Science)               0.206   \n",
       "21                 (SQL (Programming Language))               0.140   \n",
       "22                 (SQL (Programming Language))               0.142   \n",
       "23                           (Computer Science)               0.176   \n",
       "24                 (SQL (Programming Language))               0.210   \n",
       "25                 (SQL (Programming Language))               0.206   \n",
       "26                           (Computer Science)               0.276   \n",
       "27                           (Computer Science)               0.176   \n",
       "28                 (SQL (Programming Language))               0.192   \n",
       "29                             (Communications)               0.132   \n",
       "30                 (SQL (Programming Language))               0.176   \n",
       "31                 (SQL (Programming Language))               0.186   \n",
       "32                             (Communications)               0.150   \n",
       "33                                 (Management)               0.176   \n",
       "34                                 (Management)               0.190   \n",
       "35                                 (Management)               0.184   \n",
       "36                              (Data Analysis)               0.148   \n",
       "37                             (Communications)               0.176   \n",
       "38                 (SQL (Programming Language))               0.332   \n",
       "39                 (SQL (Programming Language))               0.276   \n",
       "40                             (Communications)               0.222   \n",
       "41                                 (Management)               0.238   \n",
       "42                             (Communications)               0.192   \n",
       "43                              (Data Analysis)               0.210   \n",
       "44                              (Data Analysis)               0.162   \n",
       "45                                 (Management)               0.222   \n",
       "46                                 (Management)               0.258   \n",
       "47                             (Communications)               0.392   \n",
       "48                                 (Management)               0.494   \n",
       "49                              (Data Analysis)               0.172   \n",
       "50                              (Data Analysis)               0.158   \n",
       "51                             (Communications)               0.310   \n",
       "52                              (Data Analysis)               0.276   \n",
       "53                              (Data Analysis)               0.158   \n",
       "54                             (Communications)               0.292   \n",
       "55                              (Data Analysis)               0.380   \n",
       "56                             (Communications)               0.238   \n",
       "57                              (Data Analysis)               0.206   \n",
       "58                             (Communications)               0.196   \n",
       "59                             (Communications)               0.226   \n",
       "60                              (Data Analysis)               0.166   \n",
       "61                              (Data Analysis)               0.176   \n",
       "62                             (Communications)               0.196   \n",
       "63                             (Communications)               0.176   \n",
       "64                              (Data Analysis)               0.274   \n",
       "65                              (Data Analysis)               0.238   \n",
       "66                              (Data Analysis)               0.332   \n",
       "67                              (Data Analysis)               0.222   \n",
       "68                             (Communications)               0.258   \n",
       "69                              (Data Analysis)               0.184   \n",
       "70                              (Data Analysis)               0.258   \n",
       "71                              (Data Analysis)               0.192   \n",
       "72                             (Communications)               0.274   \n",
       "73                              (Data Analysis)               0.196   \n",
       "\n",
       "    consequent support  support  confidence      lift  leverage  conviction  \\\n",
       "0                0.206    0.116    0.828571  4.022191  0.087160    4.631667   \n",
       "1                0.140    0.116    0.563107  4.022191  0.087160    1.968444   \n",
       "2                0.206    0.102    0.728571  3.536755  0.073160    2.925263   \n",
       "3                0.274    0.122    0.897059  3.273937  0.084736    7.052571   \n",
       "4                0.258    0.150    0.815217  3.159757  0.102528    4.015529   \n",
       "5                0.184    0.150    0.581395  3.159757  0.102528    1.949333   \n",
       "6                0.206    0.104    0.641975  3.116385  0.070628    2.217724   \n",
       "7                0.162    0.104    0.504854  3.116385  0.070628    1.692431   \n",
       "8                0.276    0.104    0.776119  2.812027  0.067016    3.233867   \n",
       "9                0.210    0.112    0.583333  2.777778  0.071680    1.896000   \n",
       "10               0.192    0.112    0.533333  2.777778  0.071680    1.731429   \n",
       "11               0.276    0.104    0.742857  2.691511  0.065360    2.815556   \n",
       "12               0.276    0.102    0.718310  2.602572  0.062808    2.570200   \n",
       "13               0.276    0.142    0.689320  2.497538  0.085144    2.330375   \n",
       "14               0.206    0.142    0.514493  2.497538  0.085144    1.635403   \n",
       "15               0.276    0.114    0.647727  2.346838  0.065424    2.055226   \n",
       "16               0.332    0.104    0.732394  2.206007  0.056856    2.496211   \n",
       "17               0.240    0.110    0.523810  2.182540  0.059600    1.596000   \n",
       "18               0.276    0.110    0.591398  2.142746  0.058664    1.771895   \n",
       "19               0.380    0.110    0.763889  2.010234  0.055280    2.625882   \n",
       "20               0.332    0.134    0.650485  1.959293  0.065608    1.911222   \n",
       "21               0.380    0.102    0.728571  1.917293  0.048800    2.284211   \n",
       "22               0.380    0.102    0.718310  1.890289  0.048040    2.201000   \n",
       "23               0.332    0.110    0.625000  1.882530  0.051568    1.781333   \n",
       "24               0.380    0.148    0.704762  1.854637  0.068200    2.100000   \n",
       "25               0.380    0.142    0.689320  1.814001  0.063720    1.995625   \n",
       "26               0.332    0.162    0.586957  1.767941  0.070368    1.617263   \n",
       "27               0.332    0.102    0.579545  1.745619  0.043568    1.588757   \n",
       "28               0.380    0.122    0.635417  1.672149  0.049040    1.700571   \n",
       "29               0.494    0.104    0.787879  1.594896  0.038792    2.385429   \n",
       "30               0.380    0.106    0.602273  1.584928  0.039120    1.558857   \n",
       "31               0.380    0.112    0.602151  1.584607  0.041320    1.558378   \n",
       "32               0.494    0.112    0.746667  1.511471  0.037900    1.997368   \n",
       "33               0.392    0.104    0.590909  1.507421  0.035008    1.486222   \n",
       "34               0.392    0.112    0.589474  1.503759  0.037520    1.481026   \n",
       "35               0.392    0.104    0.565217  1.441881  0.031872    1.398400   \n",
       "36               0.518    0.110    0.743243  1.434833  0.033336    1.877263   \n",
       "37               0.494    0.120    0.681818  1.380199  0.033056    1.590286   \n",
       "38               0.380    0.172    0.518072  1.363348  0.045840    1.286500   \n",
       "39               0.380    0.140    0.507246  1.334859  0.035120    1.258235   \n",
       "40               0.494    0.146    0.657658  1.331291  0.036332    1.478053   \n",
       "41               0.392    0.124    0.521008  1.329103  0.030704    1.269333   \n",
       "42               0.494    0.126    0.656250  1.328441  0.031152    1.472000   \n",
       "43               0.518    0.144    0.685714  1.323773  0.035220    1.533636   \n",
       "44               0.518    0.110    0.679012  1.310835  0.026084    1.501615   \n",
       "45               0.392    0.114    0.513514  1.309983  0.026976    1.249778   \n",
       "46               0.392    0.132    0.511628  1.305173  0.030864    1.244952   \n",
       "47               0.494    0.250    0.637755  1.291002  0.056352    1.396845   \n",
       "48               0.392    0.250    0.506073  1.291002  0.056352    1.230951   \n",
       "49               0.518    0.112    0.651163  1.257071  0.022904    1.381733   \n",
       "50               0.518    0.102    0.645570  1.246273  0.020156    1.359929   \n",
       "51               0.494    0.190    0.612903  1.240695  0.036860    1.307167   \n",
       "52               0.518    0.176    0.637681  1.231045  0.033032    1.330320   \n",
       "53               0.518    0.100    0.632911  1.221837  0.018156    1.313034   \n",
       "54               0.494    0.176    0.602740  1.220121  0.031752    1.273724   \n",
       "55               0.518    0.240    0.631579  1.219264  0.043160    1.308286   \n",
       "56               0.494    0.142    0.596639  1.207771  0.024428    1.254458   \n",
       "57               0.518    0.128    0.621359  1.199535  0.021292    1.272974   \n",
       "58               0.494    0.116    0.591837  1.198050  0.019176    1.239700   \n",
       "59               0.494    0.132    0.584071  1.182330  0.020356    1.216553   \n",
       "60               0.518    0.100    0.602410  1.162953  0.014012    1.212303   \n",
       "61               0.518    0.106    0.602273  1.162689  0.014832    1.211886   \n",
       "62               0.494    0.112    0.571429  1.156738  0.015176    1.180667   \n",
       "63               0.494    0.100    0.568182  1.150166  0.013056    1.171789   \n",
       "64               0.518    0.160    0.583942  1.127300  0.018068    1.158491   \n",
       "65               0.518    0.136    0.571429  1.103144  0.012716    1.124667   \n",
       "66               0.518    0.186    0.560241  1.081546  0.014024    1.096055   \n",
       "67               0.518    0.122    0.549550  1.060906  0.007004    1.070040   \n",
       "68               0.494    0.134    0.519380  1.051376  0.006548    1.052806   \n",
       "69               0.518    0.100    0.543478  1.049186  0.004688    1.055810   \n",
       "70               0.518    0.140    0.542636  1.047559  0.006356    1.053864   \n",
       "71               0.518    0.104    0.541667  1.045689  0.004544    1.051636   \n",
       "72               0.494    0.140    0.510949  1.034310  0.004644    1.034657   \n",
       "73               0.518    0.102    0.520408  1.004649  0.000472    1.005021   \n",
       "\n",
       "    zhangs_metric  \n",
       "0        0.873697  \n",
       "1        0.946322  \n",
       "2        0.834017  \n",
       "3        0.803886  \n",
       "4        0.837647  \n",
       "5        0.921186  \n",
       "6        0.810400  \n",
       "7        0.855309  \n",
       "8        0.744093  \n",
       "9        0.792079  \n",
       "10       0.810127  \n",
       "11       0.730769  \n",
       "12       0.717674  \n",
       "13       0.755171  \n",
       "14       0.828185  \n",
       "15       0.696474  \n",
       "16       0.637171  \n",
       "17       0.685846  \n",
       "18       0.655171  \n",
       "19       0.587086  \n",
       "20       0.616640  \n",
       "21       0.556316  \n",
       "22       0.548928  \n",
       "23       0.568932  \n",
       "24       0.583305  \n",
       "25       0.565154  \n",
       "26       0.599959  \n",
       "27       0.518370  \n",
       "28       0.497484  \n",
       "29       0.429724  \n",
       "30       0.447884  \n",
       "31       0.453229  \n",
       "32       0.398109  \n",
       "33       0.408514  \n",
       "34       0.413580  \n",
       "35       0.375566  \n",
       "36       0.355698  \n",
       "37       0.334304  \n",
       "38       0.398970  \n",
       "39       0.346488  \n",
       "40       0.319858  \n",
       "41       0.324951  \n",
       "42       0.305988  \n",
       "43       0.309599  \n",
       "44       0.282968  \n",
       "45       0.304154  \n",
       "46       0.315119  \n",
       "47       0.370737  \n",
       "48       0.445470  \n",
       "49       0.246981  \n",
       "50       0.234689  \n",
       "51       0.281159  \n",
       "52       0.259229  \n",
       "53       0.215629  \n",
       "54       0.254815  \n",
       "55       0.290054  \n",
       "56       0.225759  \n",
       "57       0.209501  \n",
       "58       0.205610  \n",
       "59       0.199240  \n",
       "60       0.168010  \n",
       "61       0.169811  \n",
       "62       0.168532  \n",
       "63       0.158447  \n",
       "64       0.155544  \n",
       "65       0.122703  \n",
       "66       0.112871  \n",
       "67       0.073792  \n",
       "68       0.065857  \n",
       "69       0.057451  \n",
       "70       0.061186  \n",
       "71       0.054075  \n",
       "72       0.045691  \n",
       "73       0.005756  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set the thresholds for support, confidence, and lift\n",
    "min_support_threshold = 0.1  # Adjust this value as needed\n",
    "min_confidence_threshold = 0.2  # Adjust this value as needed\n",
    "min_lift_threshold = 1\n",
    "\n",
    "# Filter the rules based on the specified thresholds\n",
    "filtered_rules = rules[\n",
    "    (rules['support'] >= min_support_threshold) &\n",
    "    (rules['confidence'] >= min_confidence_threshold) &\n",
    "    (rules['lift']>= min_lift_threshold)\n",
    "]\n",
    "\n",
    "# Sort the rules by support and confidence in descending order\n",
    "sorted_rules = filtered_rules.sort_values(by=['lift'], ascending=False)\n",
    "sorted_rules.reset_index(drop=True, inplace=True)\n",
    "\n",
    "# Display the selected and sorted rules\n",
    "sorted_rules"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab3f80ec",
   "metadata": {},
   "source": [
    "**4. Interpretation of Association Rule Result**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b15dd9d",
   "metadata": {},
   "source": [
    "In order to comprehend the output of the Association Rule Analysis, we will try to elaborate and interpret the result of rules with index 0 with rules of **IF {R Programming Language} THEN {Python Programming Language}** that has the highest Lift value that indicates strong association between the two skills.\n",
    "\n",
    "**Antecedents and Consequents:**\n",
    "\n",
    "- The antecedents of the rule are the skills or items that appear on the left-hand side, and in this case, it is '(R (Programming Language))'.\n",
    "- The consequents of the rule are the skills or items that appear on the right-hand side, and in this case, it is '(Python (Programming Language))'.\n",
    "- The rule implies that there is an association between job postings that mention 'R (Programming Language)' and job postings that mention 'Python (Programming Language)'.\n",
    "\n",
    "**Support:**\n",
    "\n",
    "- Support represents the frequency of occurrence of the rule in the dataset, i.e., how often both '(R (Programming Language))' and '(Python (Programming Language))' appear together in job postings.\n",
    "- In this case, the support is 0.116, indicating that the rule applies to approximately 11.6% of the job postings in the dataset.\n",
    "\n",
    "**Confidence:**\n",
    "\n",
    "- Confidence measures the conditional probability of finding the consequents ('Python (Programming Language)') in a transaction given that the antecedents ('R (Programming Language)') are present.\n",
    "- In this case, the confidence is 0.828571, suggesting that when 'R (Programming Language)' is mentioned in a job posting, there is an 82.86% probability that 'Python (Programming Language)' will also be mentioned in the same job posting.\n",
    "\n",
    "**Lift:**\n",
    "\n",
    "- Lift measures the strength of association between the antecedents and consequents, taking into account the expected co-occurrence if the items were independent.\n",
    "- A lift greater than 1 indicates a positive association. In this case, the lift is 4.022191, meaning that the co-occurrence of 'R (Programming Language)' and 'Python (Programming Language)' is approximately 4.02 times more likely than if they were independent of each other.\n",
    "\n",
    "**Leverage:**\n",
    "\n",
    "- Leverage is a measure of how much the rule contributes to the support above what would be expected if the antecedents and consequents were independent.\n",
    "- In this case, the leverage is 0.087160, indicating that the rule contributes a positive impact on the support.\n",
    "\n",
    "**Conviction:**\n",
    "\n",
    "- Conviction quantifies the implication of the rule in terms of how much it is influenced by the absence of the consequents.\n",
    "- A conviction value greater than 1 suggests a stronger implication. In this case, the conviction is 4.631667, indicating a positive implication.\n",
    "\n",
    "**Zhang's Metric:**\n",
    "\n",
    "- Zhang's metric is a measure of the degree to which the rule is interesting and different from what would be expected by chance.\n",
    "- Higher values of Zhang's metric suggest more interesting and unexpected rules. In this case, the Zhang's metric is 0.873697.\n",
    "\n",
    "Overall, this association rule suggests a strong association between job postings that mention 'R (Programming Language)' and job postings that mention 'Python (Programming Language)'. When 'R (Programming Language)' is mentioned, there is a relatively high likelihood of 'Python (Programming Language)' being mentioned as well, and the co-occurrence of these skills is more likely than expected by chance.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}