cookbooks/01-Website_Crawler_using_Spider.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <img src=\"https://socialify.git.ci/julep-ai/julep/image?description=1&descriptionEditable=Build%20AI%20agents%20and%20workflows%20with%20a%20simple%20API&font=Source%20Code%20Pro&logo=https%3A%2F%2Fraw.githubusercontent.com%2Fjulep-ai%2Fjulep%2Fdev%2F.github%2Fjulep-logo.svg&owner=1&pattern=Solid&stargazers=1&theme=Auto\" alt=\"julep\" width=\"640\" height=\"320\" />\n",
    "</div>\n",
    "\n",
    "## Task Definition: Spider Crawler Integration without Agents\n",
    "\n",
    "### Overview\n",
    "\n",
    "This task is designed to generate a simple Spider Crawler and integrate it with the workflow to scrape data from the web.\n",
    "\n",
    "### Task Flow\n",
    "\n",
    "1. **Input**: The user provides a URL to crawl.\n",
    "\n",
    "2. **Spider Tool Integration**: \n",
    "   - Create a spider tool that can crawl the web and extract data from the given URL.\n",
    "\n",
    "3. **Output**: The final output is the extracted data from the given URL.\n",
    "\n",
    "This task leverages the Spider Crawler tool to extract data from the given URL."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation\n",
    "\n",
    "To recreate the notebook and see the code implementation for this task, you can access the Google Colab notebook using the link below:\n",
    "\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/julep-ai/julep/blob/dev/cookbooks/01-Website_Crawler_using_Spider.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "\n",
    "### Additional Information\n",
    "\n",
    "For more details about the task or if you have any questions, please don't hesitate to contact the author:\n",
    "\n",
    "**Author:** Julep AI  \n",
    "**Contact:** [hey@julep.ai](mailto:hey@julep.ai) or  <a href=\"https://discord.com/invite/JTSBGRZrzj\" rel=\"dofollow\">Discord</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Installing the Julep Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install julep -U --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### NOTE:\n",
    "\n",
    "- UUIDs are generated for both the agent and task to uniquely identify them within the system.\n",
    "- Once created, these UUIDs should remain unchanged for simplicity.\n",
    "- Altering a UUID will result in the system treating it as a new agent or task.\n",
    "- If a UUID is changed, the original agent or task will continue to exist in the system alongside the new one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Global UUID is generated for agent and task\n",
    "import uuid\n",
    "\n",
    "AGENT_UUID = uuid.uuid4()\n",
    "TASK_UUID = uuid.uuid4() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Julep Client with the API Key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from julep import Client\n",
    "\n",
    "api_key = \"\" # Your API key here\n",
    "\n",
    "# Create a client\n",
    "client = Client(api_key=api_key, environment=\"dev\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating an \"agent\"\n",
    "\n",
    "Agent is the object to which LLM settings, like model, temperature along with tools are scoped to.\n",
    "\n",
    "To learn more about the agent, please refer to the [documentation](https://github.com/julep-ai/julep/blob/dev/docs/julep-concepts.md#agent)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Defining the agent\n",
    "name = \"Jarvis\"\n",
    "about = \"The original AI conscious the Iron Man.\"\n",
    "default_settings = {\n",
    "    \"temperature\": 0.7,\n",
    "    \"top_p\": 1,\n",
    "    \"min_p\": 0.01,\n",
    "    \"presence_penalty\": 0,\n",
    "    \"frequency_penalty\": 0,\n",
    "    \"length_penalty\": 1.0,\n",
    "    \"max_tokens\": 150,\n",
    "}\n",
    "\n",
    "\n",
    "# Create the agent\n",
    "agent = client.agents.create_or_update(\n",
    "    agent_id=AGENT_UUID,\n",
    "    name=name,\n",
    "    about=about,\n",
    "    model=\"gpt-4o\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining a Task\n",
    "\n",
    "Tasks in Julep are Github Actions style workflows that define long-running, multi-step actions. \n",
    "You can use them to conduct complex actions by defining them step-by-step. They have access to all Julep integrations.\n",
    "\n",
    "To learn more about tasks, visit [Julep documentation](https://github.com/julep-ai/julep/blob/dev/docs/julep-concepts.md#task)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a Task which uses a agent to generate a sarcastic response to a given text using a DuckDuckGo search tool.\n",
    "\n",
    "More on how to define a task can be found [here](https://github.com/julep-ai/julep/blob/dev/docs/julep-concepts.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# defining the task definition\n",
    "task_def = yaml.safe_load(\"\"\"\n",
    "name: Agent Crawler\n",
    "\n",
    "tools:\n",
    "- name: spider_crawler\n",
    "  type: integration\n",
    "  integration:\n",
    "    provider: spider\n",
    "    setup:\n",
    "      spider_api_key: \"your_spider_api_key\"\n",
    "\n",
    "main:\n",
    "- tool: spider_crawler\n",
    "  arguments:\n",
    "    url: '\"https://spider.cloud\"'\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creating/Updating a task to generate a sarcastic response to a given text using a Intergation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating the task object\n",
    "task = client.tasks.create_or_update(\n",
    "    task_id=TASK_UUID,\n",
    "    agent_id=AGENT_UUID,\n",
    "    **task_def\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating an Execution\n",
    "\n",
    "An execution is a single run of a task. It is a way to run a task with a specific set of inputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creates a execution worflow for the Task defined in the yaml file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating an execution object\n",
    "execution = client.executions.create(\n",
    "    task_id=TASK_UUID,\n",
    "    input={}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# getting the execution details\n",
    "execution = client.executions.get(execution.id)\n",
    "#printing the output\n",
    "execution.output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retrieves and lists all the steps of a defined task that have been executed up to that point in time. Unlike streaming, this function does not continuously monitor the execution; it only provides a snapshot of the steps completed so far without displaying real-time updates as the task progresses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.executions.transitions.list(execution_id=execution.id).items"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continuously monitor and stream the steps of a defined task. It retrieves and displays the transitions or execution steps of the task identified by execution.id in real-time, showing each step sequentially until the task is either completed or an error causes it to terminate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.executions.transitions.stream(execution_id=execution.id)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ai",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}