야놀자크롤링주석정리.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "야놀자크롤링주석정리.ipynb",
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMll1y3bmib3+eg4HVwTde4",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dmscornjs/Hotelreview/blob/main/%EC%95%BC%EB%86%80%EC%9E%90%ED%81%AC%EB%A1%A4%EB%A7%81%EC%A3%BC%EC%84%9D%EC%A0%95%EB%A6%AC.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 야놀자 크롤링 제출용 정리"
      ],
      "metadata": {
        "id": "vY_Mdn_YAvh8"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "본 코드는 코랩 환경 작성되었습니다."
      ],
      "metadata": {
        "id": "M1jp8cy9A8uR"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 코랩에서 selenium을 사용하기 위해 설치"
      ],
      "metadata": {
        "id": "GQMTuGttA2ON"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EglD-HqLAfJO"
      },
      "outputs": [],
      "source": [
        "!pip install selenium\n",
        "!apt-get update\n",
        "!apt install chromium-chromedriver"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 필요한 패키지 불러오기"
      ],
      "metadata": {
        "id": "Bk7L22JHBixL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from selenium import webdriver\n",
        "import time\n",
        "import pandas as pd\n",
        "from bs4 import BeautifulSoup\n",
        "import datetime\n",
        "from datetime import timedelta"
      ],
      "metadata": {
        "id": "4G4vRcYfBUis"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        " 야놀자 사이트에서 호텔로 구분되는 것 중 지역 선택에서 서울 전체로 구분되는 호텔들을 모아 그 리뷰를 전부 수집하였다."
      ],
      "metadata": {
        "id": "YXNdPxp-Byix"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 호텔별 URL을 가져오기"
      ],
      "metadata": {
        "id": "AW6Dj1gXCyi8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# url 함수\n",
        "def make_url(Url):\n",
        "  chrome_options = webdriver.ChromeOptions()\n",
        "  chrome_options.add_argument('--headless')\n",
        "  chrome_options.add_argument('--no-sandbox')\n",
        "  chrome_options.add_argument('--disable-dev-shm-usage')\n",
        "  driver = webdriver.Chrome('chromedriver', chrome_options=chrome_options)\n",
        "  driver.get(Url)\n",
        "\n",
        "  html = driver.page_source\n",
        "  soup = BeautifulSoup(html, 'html.parser')\n",
        "\n",
        "  # 무한 스크롤\n",
        "  import time\n",
        "  time.sleep(1)\n",
        "\n",
        "  SCROLL_PAUSE_TIME = 1.5\n",
        "\n",
        "  # Get scroll height\n",
        "  last_height = driver.execute_script(\"return document.body.scrollHeight\")\n",
        "\n",
        "  while True:\n",
        "      # Scroll down to bottom\n",
        "      driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
        "\n",
        "      # Wait to load page\n",
        "      time.sleep(SCROLL_PAUSE_TIME)\n",
        "      driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight-50);\")\n",
        "      time.sleep(SCROLL_PAUSE_TIME)\n",
        "\n",
        "      # Calculate new scroll height and compare with last scroll height\n",
        "      new_height = driver.execute_script(\"return document.body.scrollHeight\")\n",
        "\n",
        "      if new_height == last_height:\n",
        "          break\n",
        "\n",
        "      last_height = new_height  \n",
        "\n",
        "  data = soup.select(\"section > div > div > div > a\")\n",
        "\n",
        "  for d in data:\n",
        "    hotelurl.append(d.attrs['href'])\n",
        "  driver.close()\n",
        "  "
      ],
      "metadata": {
        "id": "7V20LnZcBXif"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "hotelurl = []\n",
        "make_url(\"https://www.yanolja.com/hotel/r-900582?advert=AREA&topAdvertiseMore=1\")"
      ],
      "metadata": {
        "id": "4WRchQYYCSom"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 리뷰 수집"
      ],
      "metadata": {
        "id": "W8o2VRLGEVrs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def yanolja_crawl(url):\n",
        "    global df # 전역 변수를 지역 범위에서 적용\n",
        "    #Url = \"https://www.yanolja.com\"+url\n",
        "    driver.get(url)\n",
        "\n",
        "\n",
        "    hotel = driver.find_element_by_css_selector('section.PlaceDetailTitle_titleContainer__3sGdf h1').text\n",
        "    hotel = hotel.replace(\"[★숙박대전] \", \"\")\n",
        "\n",
        "    # 리뷰 페이지 클릭 (Click review page)\n",
        "    driver.find_element_by_css_selector('section.PlaceDetailTitle_titleContainer__3sGdf > a').click()\n",
        "\n",
        "    # 무한 스크롤\n",
        "    import time\n",
        "    time.sleep(1)\n",
        "\n",
        "    SCROLL_PAUSE_TIME = 1.5\n",
        "\n",
        "    # Get scroll height\n",
        "    last_height = driver.execute_script(\"return document.body.scrollHeight\")\n",
        "\n",
        "    while True:\n",
        "        # Scroll down to bottom\n",
        "        driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight);\")\n",
        "\n",
        "        # Wait to load page\n",
        "        time.sleep(SCROLL_PAUSE_TIME)\n",
        "        driver.execute_script(\"window.scrollTo(0, document.body.scrollHeight-50);\")\n",
        "        time.sleep(SCROLL_PAUSE_TIME)\n",
        "\n",
        "        # Calculate new scroll height and compare with last scroll height\n",
        "        new_height = driver.execute_script(\"return document.body.scrollHeight\")\n",
        "\n",
        "        if new_height == last_height:\n",
        "            break\n",
        "\n",
        "        last_height = new_height\n",
        "\n",
        "    # 스크롤 끝나고 페이지 elements 가져오기\n",
        "    raw = driver.page_source # 페이지의 elements 모두 가져오기\n",
        "    html = BeautifulSoup(raw, \"html.parser\")\n",
        "\n",
        "    # 리뷰 텍스트 선택자: div.item-wrap\n",
        "    reviews = html.select(\"div.item-wrap\")\n",
        "\n",
        "    columns = ['hotel', 'star', 'review', 'date']\n",
        "\n",
        "    for r in reviews:\n",
        "        review = r.select_one(\"p\")\n",
        "        date = r.select_one('time')\n",
        "\n",
        "        stars = r.select('.container.score')\n",
        "        for s in stars:\n",
        "            star_count = 5 - len(s.select(\".ico.gray\"))\n",
        "\n",
        "        df2 = df2.append(pd.DataFrame([[hotel, star_count, review.text, date.text]], columns = columns),ignore_index=True)\n",
        "    \n",
        "def change_date(x):\n",
        "    now = datetime.datetime.today()\n",
        "    today = now.strftime(\"%Y. %m. %d\")\n",
        "    yesterday = (now - timedelta(days=1)).strftime(\"%Y. %m. %d\")\n",
        "    _2days_ago = (now - timedelta(days = 2)).strftime(\"%Y. %m. %d\")\n",
        "    _3days_ago = (now - timedelta(days = 3)).strftime(\"%Y. %m. %d\")\n",
        "    hour = int(datetime.datetime.today().strftime(\"%H\"))\n",
        "    if \"시간 전\" in x:\n",
        "        x = int(x.replace(\"시간 전\", \"\"))\n",
        "        if hour - x > 0: return today\n",
        "        else: return yesterday\n",
        "    elif \"일 전\" in x:\n",
        "        if \"1\" in x: return yesterday\n",
        "        elif \"2\" in x: return _2days_ago\n",
        "        else: return _3days_ago\n",
        "    else: return x"
      ],
      "metadata": {
        "id": "1ZMuTwb5ChbK"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "chrome_options = webdriver.ChromeOptions()\n",
        "chrome_options.add_argument('--headless')\n",
        "chrome_options.add_argument('--no-sandbox')\n",
        "chrome_options.add_argument('--disable-dev-shm-usage')\n",
        "driver = webdriver.Chrome('chromedriver', chrome_options=chrome_options)\n",
        "for url in hotelurl:\n",
        "    Url = \"https://www.yanolja.com\"+url\n",
        "    yanolja_crawl(Url)\n",
        "\n",
        "driver.close()"
      ],
      "metadata": {
        "id": "zeo2cYc_D3Jk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#날짜 형식 변환\n",
        "df['real_date'] = df['date'].apply(change_date)"
      ],
      "metadata": {
        "id": "6YZtqm2aEO_t"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.describe()"
      ],
      "metadata": {
        "id": "lF47cAOyH08c"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.to_csv(\"야놀자호텔리뷰.csv\")"
      ],
      "metadata": {
        "id": "G0buiEvAHwx-"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}