data_generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "359710f6",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "* &nbsp;\n",
    "\t* [Dataset Creation:](#Dataset-Creation:)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "119059f6",
   "metadata": {},
   "source": [
    "## Dataset Creation:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9875436",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How can we generate the dataset?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "abd9cb8a",
   "metadata": {
    "format": "column",
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.10773651 0.71644464 0.94283329 0.85951229 0.2807103  0.16025629\n",
      " 0.88060116 0.23190625 0.43986917 0.55574045]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "import pandas as pd\n",
    "print(np.random.random(10))\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "495d3bb6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "76933f99",
   "metadata": {},
   "source": [
    "So we need to generate a dataset that has:\n",
    "\n",
    "- Symmetrical relations\n",
    "- Entities\n",
    "- Logic task\n",
    "- Redudant information for second dataset\n",
    "\n",
    "So the first step is going to be to define the set of relations, the axioms of symmetry between them and rules of compositionality and recursiveness. Then we can think about the actual generation by expanding a simple example into a more general case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "aff5e6e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0b0252e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "r = ['P', 'S'] # set of symmetrical relations\n",
    "\n",
    "r = np.asarray(r)\n",
    "e = np.array(list(string.ascii_lowercase)) # this will be the set of entities \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29ac368a",
   "metadata": {},
   "source": [
    "Now, we can define the axioms and the rules to use to generate our dataset\n",
    "\n",
    "we want elements of this type $S(a,b)$\n",
    "\n",
    "And our first axiom is that $S(a, b) == P(b, a)$\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "56ea22b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "i = np.random.choice(r)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "17ba80c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "t = '{}({}, {})'.format(i, np.random.choice(e), np.random.choice(e))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "09f7c0ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'S(t, d)'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "caa58445",
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_rules(relation):\n",
    "    if 'S'  in relation:\n",
    "\n",
    "        x = list(relation)\n",
    "        x[0] = 'P'\n",
    "        x[2], x[5] = x[5], x[2]  # Swap using tuple unpacking\n",
    "        return ''.join(x)\n",
    "       \n",
    "    elif 'P' in relation:\n",
    "    # Convert to list only if necessary to make changes\n",
    "        x = list(relation)\n",
    "        x[0] = 'S'\n",
    "        x[2], x[5] = x[5], x[2]  # Swap using tuple unpacking\n",
    "        return ''.join(x)\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "5a11d2a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'P(d, t)'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "check_rules(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "52926489",
   "metadata": {},
   "outputs": [],
   "source": [
    "e = list(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "418b634b",
   "metadata": {},
   "source": [
    "Now let's try to generate a k number of relations from a subset of the entities "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "9c68647a",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_trees(k):\n",
    "    sub_e = np.asarray(random.sample(e, k))\n",
    "    \n",
    "    inst = []\n",
    "    for i in range (0, (int(k / 2))):\n",
    "        t = '{}({}, {})'.format(np.random.choice(r), np.random.choice(sub_e), np.random.choice(sub_e))\n",
    "        inst.append(t)\n",
    "        \n",
    "    return inst\n",
    "\n",
    "        \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "fb3e079b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['P(l, p)', 'S(p, l)']"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_trees(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "b5309223",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "S(y, y)\n",
      "P(y, y)\n",
      "P(w, v)\n",
      "S(v, w)\n",
      "P(v, s)\n",
      "S(s, v)\n"
     ]
    }
   ],
   "source": [
    "for i in generate_trees(6):\n",
    "    print(i)\n",
    "    f = check_rules(i)\n",
    "    print(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11125212",
   "metadata": {},
   "source": [
    "To create a consistent knowledge base we use the following approach:\n",
    "\n",
    "- A subset of the all entities is taken\n",
    "- The depth of the tree is decided\n",
    "- The first relationship is generated, defining the first layer of the tree\n",
    "- From this point each other relation is created by using the constraints that come with the previous relations\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ef256d5",
   "metadata": {},
   "source": [
    "So, suppose that the tree has been created by the first relation: $S(a, b)$ where it means that a is the son of b.\n",
    "\n",
    "It follows that there are two levels (parent and son) and the following relations must account for this. \n",
    "So, if the following relation is of the type $B(b, c)$, whihc means that b is the brother of c, there will be an addition fo the same hierarchical level of b, and that will constrain relationship of the same type $B()$ between the levels. \n",
    "\n",
    "So a relation of the type $B(a, d)$ is admissible, but after it, we cannot say that $B(d, b)$, because the relation brother is only accetable between a previous entity and a new one or between entities that share the same branch."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e838d414",
   "metadata": {},
   "source": [
    "*Definition of the algorithm*\n",
    "\n",
    "- Define number of the relations in the I\n",
    "- Check if entities used is empty\n",
    "- If it is, select a random entites from set of entities\n",
    "- Select second entity (random between used, if U is > 2, and total entites)\n",
    "- Check for constraints between the two\n",
    "- First relation between the two, add constraints to C\n",
    "- Repeat step 2, in this case we select the first entity from the one previously used in order to have a chain of relations\n",
    "\n",
    "\n",
    "When k match the size of our I, we need to select an hypothesis to check: \n",
    "\n",
    "\n",
    "Other idea, select two entites, define their distance into the concept space (number of instances?)\n",
    "So, let's say k=3, a and b: I can connect them with the minimal number of relations knowing that their distance should be 2 (k-1). Thus, I start from a which cannot be connected directly to b as it would be dist = 1.\n",
    "\n",
    "a will be in a random relation: example: P(a, x) so a is the parent of x. Now I now that from this one there must be another connection between x and b to get the final relation: it cannot be on the same level, so it can be B(x, b). Now we have the connections (we can create them redundant by using the check formula. So, now we must create the H, which woudl be \n",
    "\n",
    "\n",
    "Define in advance the following for the algorithm to work: final relation to check, step to obtained. \n",
    "\n",
    "\n",
    "Task: given relation say whether is true or not ? what can the model use?\n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "46a24887",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_task():\n",
    "    t = '{}({}, {})'.format(np.random.choice(r), np.random.choice(e), np.random.choice(e))\n",
    "    return t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e5790c60",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initial array\n",
    "\n",
    "\n",
    "def add_elements(arr, elements):\n",
    "    position = 1\n",
    "    for element in elements:\n",
    "        arr.insert(position, element)\n",
    "        position += 1\n",
    "    return arr\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fb3a4818",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_inst(k):\n",
    "    task = generate_task()\n",
    "    # create a list of the entities necessary to use for the main chain\n",
    "    l = [task[2], task[5]]\n",
    "    used = []\n",
    "    \n",
    "    ent = np.random.choice(e, int(k/2)) # here we must control to avoid redundancy between entities\n",
    "    \n",
    "    add_elements(l, ent)\n",
    "    \n",
    "    # Iterate in entities, starts from head and second entity create first relation, add. Must stop to the last entity\n",
    "    \n",
    "    \n",
    "    # we need to keep in mind the task constraint for the construction of the data: keep track of the depth\n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    # Generate noise:\n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    print(task)\n",
    "    return(l)\n",
    "    \n",
    "\n",
    "\n",
    "\n",
    "    \n",
    "    \n",
    "    # start from \n",
    "    \n",
    "        \n",
    "        \n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "18a39875",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P(r, r)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['r', 'v', 'y', 'f', 'r']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate_inst(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baa2b84e",
   "metadata": {},
   "source": [
    "### Try new method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9b80d119",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task: P(b, i)\n",
      "Direct Chain and Task: B(h, f) | B(c, x) | B(b, g) | P(n, p) | S(l, s) | P(q, o) | B(g, c) | B(x, i)\n",
      "Direct Binary Vector: 01100011\n",
      "Redundant Chain and Task: P(n, p) | B(g, c) | B(c, g) | B(b, g) | B(i, x) | S(l, s) | B(x, c) | B(x, i) | B(h, f) | B(c, x) | P(q, o) | B(g, b)\n",
      "Redundant Binary Vector: 011110110101\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import string\n",
    "\n",
    "def generate_chain(task_relation, entity1, entity2, k):\n",
    "    entities = list(string.ascii_lowercase)\n",
    "    chain = []\n",
    "    current_entity = entity1\n",
    "    remaining_entities = [e for e in entities if e not in {entity1, entity2}]\n",
    "\n",
    "    # Define possible next relations based on the current relation\n",
    "    relation_followup = {\n",
    "        'P': ['P', 'B'],\n",
    "        'S': ['S', 'B'],\n",
    "        'B': ['P', 'S', 'B']\n",
    "    }\n",
    "\n",
    "    # Generate the chain of relations\n",
    "    for _ in range(k - 1):\n",
    "        if not remaining_entities:\n",
    "            break  # Prevent empty list operations\n",
    "        next_entity = random.choice(remaining_entities)\n",
    "        remaining_entities.remove(next_entity)\n",
    "        next_relation = random.choice(relation_followup[task_relation])\n",
    "        chain.append((next_relation, current_entity, next_entity))\n",
    "        current_entity = next_entity\n",
    "        task_relation = next_relation\n",
    "\n",
    "    # Final link in the chain\n",
    "    chain.append((task_relation, current_entity, entity2))\n",
    "    return chain\n",
    "\n",
    "def add_noise(chain, entities, noise_count):\n",
    "    noise = []\n",
    "    used_entities = {e for _, e1, e2 in chain for e in {e1, e2}}\n",
    "    available_entities = [e for e in entities if e not in used_entities]\n",
    "\n",
    "    while noise_count > 0 and len(available_entities) >= 2:\n",
    "        e1, e2 = random.sample(available_entities, 2)\n",
    "        relation = random.choice(['P', 'S', 'B'])\n",
    "        noise.append((relation, e1, e2))\n",
    "        available_entities.remove(e1)\n",
    "        available_entities.remove(e2)\n",
    "        noise_count -= 1\n",
    "\n",
    "    return noise\n",
    "\n",
    "def generate_redundant_info(direct_chain):\n",
    "    redundant_chain = []\n",
    "    for relation, e1, e2 in direct_chain:\n",
    "        redundant_chain.append((relation, e1, e2))\n",
    "        if relation == 'P':\n",
    "            redundant_chain.append(('S', e2, e1))\n",
    "        elif relation == 'S':\n",
    "            redundant_chain.append(('P', e2, e1))\n",
    "        elif relation == 'B':\n",
    "            redundant_chain.append(('B', e2, e1))\n",
    "    return redundant_chain\n",
    "\n",
    "def generate_binary_vector(direct_chain, combined_chain):\n",
    "    return [1 if item in direct_chain else 0 for item in combined_chain]\n",
    "\n",
    "def generate_task_and_chain():\n",
    "    entities = list(string.ascii_lowercase)\n",
    "    relations = ['P', 'S', 'B']\n",
    "    k = random.randint(3, 5)\n",
    "    noise_count = random.randint(2, 4)\n",
    "    entity1, entity2 = random.sample(entities, 2)\n",
    "    task_relation = random.choice(relations)\n",
    "\n",
    "    # Generate the direct chain and redundant chain\n",
    "    direct_chain = generate_chain(task_relation, entity1, entity2, k)\n",
    "    redundant_chain = generate_redundant_info(direct_chain)\n",
    "    # Generate noise and combine with direct and redundant chains\n",
    "    noise = add_noise(direct_chain, entities, noise_count)\n",
    "    combined_direct_chain = direct_chain + noise\n",
    "    combined_redundant_chain = redundant_chain + noise\n",
    "    random.shuffle(combined_direct_chain)\n",
    "    random.shuffle(combined_redundant_chain)\n",
    "    # Generate binary vectors\n",
    "    binary_vector_direct = generate_binary_vector(direct_chain, combined_direct_chain)\n",
    "    binary_vector_redundant = generate_binary_vector(redundant_chain, combined_redundant_chain)\n",
    "\n",
    "    # Format output\n",
    "    direct_chain_repr = \" | \".join(f\"{r}({a}, {b})\" for r, a, b in combined_direct_chain)\n",
    "    redundant_chain_repr = \" | \".join(f\"{r}({a}, {b})\" for r, a, b in combined_redundant_chain)\n",
    "    print(\"Task:\", f\"{task_relation}({entity1}, {entity2})\")\n",
    "    print(\"Direct Chain and Task:\", direct_chain_repr)\n",
    "    print(\"Direct Binary Vector:\", ''.join(map(str, binary_vector_direct)))\n",
    "    print(\"Redundant Chain and Task:\", redundant_chain_repr)\n",
    "    print(\"Redundant Binary Vector:\", ''.join(map(str, binary_vector_redundant)))\n",
    "\n",
    "# Example usage\n",
    "generate_task_and_chain()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "143f054b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Task: P(z, g)\n",
      "Direct Chain and Task: P(w, b) | P(p, v) | S(q, s) | B(i, j) | P(z, t) | P(b, g) | P(t, w)\n",
      "Direct Binary Vector: 1000111\n",
      "Redundant Chain and Task: S(q, s) | P(t, w) | B(i, j) | P(p, v) | P(w, b) | S(w, t) | S(t, z) | P(b, g) | S(g, b) | S(b, w) | P(z, t)\n",
      "Redundant Binary Vector: 01001001001\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import string\n",
    "\n",
    "def generate_chain(task_relation, entity1, entity2, k):\n",
    "    entities = list(string.ascii_lowercase)\n",
    "    chain = []\n",
    "    current_entity = entity1\n",
    "    remaining_entities = [e for e in entities if e not in {entity1, entity2}]\n",
    "\n",
    "    # Generate the chain of relations\n",
    "    for _ in range(k - 1):\n",
    "        if not remaining_entities:\n",
    "            break\n",
    "        next_entity = random.choice(remaining_entities)\n",
    "        remaining_entities.remove(next_entity)\n",
    "        chain.append((task_relation, current_entity, next_entity))\n",
    "        current_entity = next_entity\n",
    "\n",
    "    # Final link in the chain\n",
    "    chain.append((task_relation, current_entity, entity2))\n",
    "    return chain\n",
    "\n",
    "def add_noise(chain, entities, noise_count):\n",
    "    noise = []\n",
    "    used_entities = {e for _, e1, e2 in chain for e in {e1, e2}}\n",
    "    available_entities = [e for e in entities if e not in used_entities]\n",
    "\n",
    "    while noise_count > 0 and len(available_entities) >= 2:\n",
    "        e1, e2 = random.sample(available_entities, 2)\n",
    "        relation = random.choice(['P', 'S', 'B'])\n",
    "        noise.append((relation, e1, e2))\n",
    "        available_entities.remove(e1)\n",
    "        available_entities.remove(e2)\n",
    "        noise_count -= 1\n",
    "\n",
    "    return noise\n",
    "\n",
    "def generate_redundant_info(direct_chain):\n",
    "    redundant_chain = []\n",
    "    for relation, e1, e2 in direct_chain:\n",
    "        redundant_chain.append((relation, e1, e2))\n",
    "        if relation == 'P':\n",
    "            redundant_chain.append(('S', e2, e1))\n",
    "        elif relation == 'S':\n",
    "            redundant_chain.append(('P', e2, e1))\n",
    "        elif relation == 'B':\n",
    "            redundant_chain.append(('B', e2, e1))\n",
    "    return redundant_chain\n",
    "\n",
    "def generate_binary_vector(direct_chain, combined_chain):\n",
    "    direct_set = set(direct_chain)\n",
    "    return [1 if item in direct_set else 0 for item in combined_chain]\n",
    "\n",
    "def generate_task_and_chain():\n",
    "    entities = list(string.ascii_lowercase)\n",
    "    relations = ['P', 'S', 'B']\n",
    "    k = random.randint(3, 5)\n",
    "    noise_count = random.randint(2, 4)\n",
    "    entity1, entity2 = random.sample(entities, 2)\n",
    "    task_relation = random.choice(relations)\n",
    "\n",
    "    direct_chain = generate_chain(task_relation, entity1, entity2, k)\n",
    "    redundant_chain = generate_redundant_info(direct_chain)\n",
    "    noise = add_noise(direct_chain, entities, noise_count)\n",
    "    \n",
    "    combined_direct_chain = direct_chain + noise\n",
    "    random.shuffle(combined_direct_chain)\n",
    "    binary_vector_direct = generate_binary_vector(direct_chain, combined_direct_chain)\n",
    "    \n",
    "    combined_redundant_chain = redundant_chain + noise\n",
    "    random.shuffle(combined_redundant_chain)\n",
    "    binary_vector_redundant = generate_binary_vector(direct_chain, combined_redundant_chain)  # Use direct chain to mark\n",
    "\n",
    "    print(\"Task:\", f\"{task_relation}({entity1}, {entity2})\")\n",
    "    print(\"Direct Chain and Task:\", \" | \".join(f\"{r}({a}, {b})\" for r, a, b in combined_direct_chain))\n",
    "    print(\"Direct Binary Vector:\", ''.join(map(str, binary_vector_direct)))\n",
    "    print(\"Redundant Chain and Task:\", \" | \".join(f\"{r}({a}, {b})\" for r, a, b in combined_redundant_chain))\n",
    "    print(\"Redundant Binary Vector:\", ''.join(map(str, binary_vector_redundant)))\n",
    "\n",
    "generate_task_and_chain()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f61f1f44",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Direct Dataset Example:\n",
      "{'input': '[B(f, n)] B(a, n) | B(f, w) | B(w, a) | P(y, z) | B(l, u) | S(b, m)', 'output': [1, 1, 1, 0, 0, 0]}\n",
      "\n",
      "Redundant Dataset Example:\n",
      "{'input': '[B(f, n)] B(a, n) | B(a, w) | S(b, m) | B(w, a) | B(n, a) | B(w, f) | B(f, w) | P(y, z) | B(l, u)', 'output': [1, 1, 0, 1, 1, 1, 1, 0, 0]}\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import string\n",
    "\n",
    "def generate_chain(task_relation, entity1, entity2, k):\n",
    "    entities = list(string.ascii_lowercase)\n",
    "    chain = []\n",
    "    current_entity = entity1\n",
    "    remaining_entities = [e for e in entities if e not in {entity1, entity2}]\n",
    "\n",
    "    # Generate the chain of relations\n",
    "    for _ in range(k - 1):\n",
    "        if not remaining_entities:\n",
    "            break\n",
    "        next_entity = random.choice(remaining_entities)\n",
    "        remaining_entities.remove(next_entity)\n",
    "        chain.append(f\"{task_relation}({current_entity}, {next_entity})\")\n",
    "        current_entity = next_entity\n",
    "\n",
    "    # Final link in the chain\n",
    "    chain.append(f\"{task_relation}({current_entity}, {entity2})\")\n",
    "    return chain\n",
    "\n",
    "def add_noise(chain, entities, noise_count):\n",
    "    noise = []\n",
    "    used_entities = {e for rel in chain for e in rel[rel.find('(')+1:rel.find(')')].split(', ')}\n",
    "    available_entities = [e for e in entities if e not in used_entities]\n",
    "\n",
    "    while noise_count > 0 and len(available_entities) >= 2:\n",
    "        e1, e2 = random.sample(available_entities, 2)\n",
    "        relation = random.choice(['P', 'S', 'B'])\n",
    "        noise.append(f\"{relation}({e1}, {e2})\")\n",
    "        available_entities.remove(e1)\n",
    "        available_entities.remove(e2)\n",
    "        noise_count -= 1\n",
    "\n",
    "    return noise\n",
    "\n",
    "def generate_redundant_info(direct_chain):\n",
    "    redundant_chain = []\n",
    "    for rel in direct_chain:\n",
    "        rel_type, entities = rel[0], rel[2:-1].split(',')\n",
    "        e1, e2 = entities[0].strip(), entities[1].strip()\n",
    "        redundant_chain.append(rel)\n",
    "        if rel_type == 'P':\n",
    "            redundant_chain.append(f\"S({e2}, {e1})\")\n",
    "        elif rel_type == 'S':\n",
    "            redundant_chain.append(f\"P({e2}, {e1})\")\n",
    "        elif rel_type == 'B':\n",
    "            redundant_chain.append(f\"B({e2}, {e1})\")\n",
    "    return redundant_chain\n",
    "\n",
    "def generate_binary_vector(direct_chain, combined_chain):\n",
    "    direct_set = set(direct_chain)\n",
    "    return [1 if item in direct_set else 0 for item in combined_chain]\n",
    "\n",
    "def generate_datasets(k):\n",
    "    entities = list(string.ascii_lowercase)\n",
    "    relations = ['P', 'S', 'B']\n",
    "    direct_data = []\n",
    "    redundant_data = []\n",
    "\n",
    "    for _ in range(k):\n",
    "        entity1, entity2 = random.sample(entities, 2)\n",
    "        task_relation = random.choice(relations)\n",
    "        k_chain = random.randint(3, 5)\n",
    "        noise_count = random.randint(2, 4)\n",
    "\n",
    "        direct_chain = generate_chain(task_relation, entity1, entity2, k_chain)\n",
    "        redundant_chain = generate_redundant_info(direct_chain)\n",
    "        noise = add_noise(direct_chain, entities, noise_count)\n",
    "\n",
    "        combined_direct_chain = direct_chain + noise\n",
    "        random.shuffle(combined_direct_chain)\n",
    "        binary_vector_direct = generate_binary_vector(direct_chain, combined_direct_chain)\n",
    "\n",
    "        combined_redundant_chain = redundant_chain + noise\n",
    "        random.shuffle(combined_redundant_chain)\n",
    "        binary_vector_redundant = generate_binary_vector(redundant_chain, combined_redundant_chain)\n",
    "\n",
    "        task_str = f\"{task_relation}({entity1}, {entity2})\"\n",
    "        direct_data.append({\n",
    "            'input': f\"[{task_str}] \" + \" | \".join(combined_direct_chain),\n",
    "            'output': binary_vector_direct\n",
    "        })\n",
    "\n",
    "        redundant_data.append({\n",
    "            'input': f\"[{task_str}] \" + \" | \".join(combined_redundant_chain),\n",
    "            'output': binary_vector_redundant\n",
    "        })\n",
    "\n",
    "    return direct_data, redundant_data\n",
    "\n",
    "# Generate datasets\n",
    "k = 10  # Number of instances to generate\n",
    "direct_data, redundant_data = generate_datasets(k)\n",
    "\n",
    "# Example: Printing the first instance of each type for verification\n",
    "print(\"Direct Dataset Example:\")\n",
    "print(direct_data[0])\n",
    "print(\"\\nRedundant Dataset Example:\")\n",
    "print(redundant_data[0])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "795d67d5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Direct Dataset Example:\n",
      "{'input': '[B(p, g)] P(q, t) | B(y, j) | P(e, b) | P(w, s) | B(r, n) | B(u, g) | B(n, u) | B(p, r)', 'output': [0, 0, 0, 0, 1, 1, 1, 1]}\n",
      "\n",
      "Redundant Dataset Example:\n",
      "{'input': '[B(p, g)] B(u, n) | P(w, s) | B(n, r) | B(n, u) | B(r, n) | P(e, b) | B(u, g) | B(p, r) | B(r, p) | B(y, j) | P(q, t) | B(g, u)', 'output': [0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0]}\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import string\n",
    "\n",
    "def generate_chain(task_relation, entity1, entity2, k):\n",
    "    entities = list(string.ascii_lowercase)\n",
    "    chain = []\n",
    "    current_entity = entity1\n",
    "    remaining_entities = [e for e in entities if e not in {entity1, entity2}]\n",
    "\n",
    "    for _ in range(k - 1):\n",
    "        if not remaining_entities:\n",
    "            break\n",
    "        next_entity = random.choice(remaining_entities)\n",
    "        remaining_entities.remove(next_entity)\n",
    "        chain.append(f\"{task_relation}({current_entity}, {next_entity})\")\n",
    "        current_entity = next_entity\n",
    "\n",
    "    chain.append(f\"{task_relation}({current_entity}, {entity2})\")\n",
    "    return chain\n",
    "\n",
    "def add_noise(chain, entities, noise_count):\n",
    "    noise = []\n",
    "    used_entities = {e for rel in chain for e in rel[rel.find('(')+1:rel.find(')')].split(', ')}\n",
    "    available_entities = [e for e in entities if e not in used_entities]\n",
    "\n",
    "    while noise_count > 0 and len(available_entities) >= 2:\n",
    "        e1, e2 = random.sample(available_entities, 2)\n",
    "        relation = random.choice(['P', 'S', 'B'])\n",
    "        noise.append(f\"{relation}({e1}, {e2})\")\n",
    "        available_entities.remove(e1)\n",
    "        available_entities.remove(e2)\n",
    "        noise_count -= 1\n",
    "\n",
    "    return noise\n",
    "\n",
    "def generate_redundant_info(direct_chain):\n",
    "    redundant_chain = []\n",
    "    for rel in direct_chain:\n",
    "        relation, pair = rel.split('(')\n",
    "        e1, e2 = pair[:-1].split(', ')\n",
    "        redundant_chain.append(rel)\n",
    "        if relation == 'P':\n",
    "            redundant_chain.append(f\"S({e2}, {e1})\")\n",
    "        elif relation == 'S':\n",
    "            redundant_chain.append(f\"P({e2}, {e1})\")\n",
    "        elif relation == 'B':\n",
    "            redundant_chain.append(f\"B({e2}, {e1})\")\n",
    "    return redundant_chain\n",
    "\n",
    "def generate_binary_vector(direct_chain, combined_chain):\n",
    "    direct_set = set(direct_chain)\n",
    "    return [1 if item in direct_set else 0 for item in combined_chain]\n",
    "\n",
    "def generate_datasets(k):\n",
    "    entities = list(string.ascii_lowercase)\n",
    "    relations = ['P', 'S', 'B']\n",
    "    direct_data = []\n",
    "    redundant_data = []\n",
    "\n",
    "    for _ in range(k):\n",
    "        entity1, entity2 = random.sample(entities, 2)\n",
    "        task_relation = random.choice(relations)\n",
    "        k_chain = random.randint(3, 5)\n",
    "        noise_count = random.randint(2, 4)\n",
    "\n",
    "        direct_chain = generate_chain(task_relation, entity1, entity2, k_chain)\n",
    "        redundant_chain = generate_redundant_info(direct_chain)\n",
    "        noise = add_noise(direct_chain, entities, noise_count)\n",
    "        \n",
    "        combined_direct_chain = direct_chain + noise\n",
    "        random.shuffle(combined_direct_chain)\n",
    "        binary_vector_direct = generate_binary_vector(direct_chain, combined_direct_chain)\n",
    "        \n",
    "        combined_redundant_chain = redundant_chain + noise\n",
    "        random.shuffle(combined_redundant_chain)\n",
    "        binary_vector_redundant = generate_binary_vector(direct_chain, combined_redundant_chain)\n",
    "\n",
    "        task_str = f\"{task_relation}({entity1}, {entity2})\"\n",
    "        direct_data.append({\n",
    "            'input': f\"[{task_str}] \" + \" | \".join(combined_direct_chain),\n",
    "            'output': binary_vector_direct\n",
    "        })\n",
    "\n",
    "        redundant_data.append({\n",
    "            'input': f\"[{task_str}] \" + \" | \".join(combined_redundant_chain),\n",
    "            'output': binary_vector_redundant\n",
    "        })\n",
    "\n",
    "    return direct_data, redundant_data\n",
    "\n",
    "# Generate datasets\n",
    "k = 10000  # Number of instances to generate\n",
    "direct_data, redundant_data = generate_datasets(k)\n",
    "\n",
    "# Example: Printing the first instance of each type for verification\n",
    "print(\"Direct Dataset Example:\")\n",
    "print(direct_data[0])\n",
    "print(\"\\nRedundant Dataset Example:\")\n",
    "print(redundant_data[0])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "6d81d9ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "def convert_to_dataframe(data):\n",
    "    # Extract 'input' and 'output' from the data to form a DataFrame\n",
    "    df = pd.DataFrame(data)\n",
    "    return df\n",
    "\n",
    "def save_to_csv(df, filename):\n",
    "    # Save the DataFrame to a CSV file\n",
    "    df.to_csv(filename, index=False)\n",
    "    print(f\"Data saved to {filename}\")\n",
    "\n",
    "# Convert data to DataFrame\n",
    "direct_df = convert_to_dataframe(direct_data)\n",
    "redundant_df = convert_to_dataframe(redundant_data)\n",
    "\n",
    "# Optionally, save to CSV files\n",
    "#save_to_csv(direct_df, 'direct_data.csv')\n",
    "#save_to_csv(redundant_df, 'redundant_data.csv')\n",
    "\n",
    "# The DataFrames can now be used to fine-tune a language model.\n",
    "# Here, 'direct_df' and 'redundant_df' are ready for use.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "3b3d27f3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>output</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[S(l, p)] S(y, f) | P(a, c) | S(f, p) | S(l, v...</td>\n",
       "      <td>[1, 0, 1, 1, 0, 1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[S(y, p)] S(n, p) | S(x, c) | S(r, v) | B(b, q...</td>\n",
       "      <td>[1, 0, 1, 0, 0, 1, 1, 1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[S(w, q)] S(w, g) | P(d, y) | S(r, q) | S(g, r...</td>\n",
       "      <td>[1, 0, 1, 1, 0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[B(l, n)] B(y, k) | S(w, x) | B(s, v) | B(v, n...</td>\n",
       "      <td>[0, 0, 1, 1, 1, 1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[B(g, o)] P(e, f) | B(g, t) | S(v, r) | B(j, o...</td>\n",
       "      <td>[0, 1, 0, 1, 1]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               input                    output\n",
       "0  [S(l, p)] S(y, f) | P(a, c) | S(f, p) | S(l, v...        [1, 0, 1, 1, 0, 1]\n",
       "1  [S(y, p)] S(n, p) | S(x, c) | S(r, v) | B(b, q...  [1, 0, 1, 0, 0, 1, 1, 1]\n",
       "2  [S(w, q)] S(w, g) | P(d, y) | S(r, q) | S(g, r...           [1, 0, 1, 1, 0]\n",
       "3  [B(l, n)] B(y, k) | S(w, x) | B(s, v) | B(v, n...        [0, 0, 1, 1, 1, 1]\n",
       "4  [B(g, o)] P(e, f) | B(g, t) | S(v, r) | B(j, o...           [0, 1, 0, 1, 1]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "direct_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "32a09426",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input</th>\n",
       "      <th>output</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[S(l, p)] S(v, y) | P(p, f) | P(a, c) | S(l, v...</td>\n",
       "      <td>[1, 0, 0, 1, 0, 1, 0, 0, 1, 0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[S(y, p)] S(r, v) | S(x, c) | P(v, r) | P(r, i...</td>\n",
       "      <td>[1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[S(w, q)] P(q, r) | P(g, w) | S(f, u) | S(w, g...</td>\n",
       "      <td>[0, 0, 0, 1, 1, 0, 1, 0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[B(l, n)] B(y, k) | B(s, v) | B(n, v) | S(w, x...</td>\n",
       "      <td>[0, 1, 0, 0, 1, 0, 1, 0, 1, 0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[B(g, o)] B(o, j) | P(e, f) | B(j, t) | B(t, g...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 1, 1, 1]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               input  \\\n",
       "0  [S(l, p)] S(v, y) | P(p, f) | P(a, c) | S(l, v...   \n",
       "1  [S(y, p)] S(r, v) | S(x, c) | P(v, r) | P(r, i...   \n",
       "2  [S(w, q)] P(q, r) | P(g, w) | S(f, u) | S(w, g...   \n",
       "3  [B(l, n)] B(y, k) | B(s, v) | B(n, v) | S(w, x...   \n",
       "4  [B(g, o)] B(o, j) | P(e, f) | B(j, t) | B(t, g...   \n",
       "\n",
       "                                    output  \n",
       "0           [1, 0, 0, 1, 0, 1, 0, 0, 1, 0]  \n",
       "1  [1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0]  \n",
       "2                 [0, 0, 0, 1, 1, 0, 1, 0]  \n",
       "3           [0, 1, 0, 0, 1, 0, 1, 0, 1, 0]  \n",
       "4                 [0, 0, 0, 0, 0, 1, 1, 1]  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "redundant_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "cd4ebc43",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data saved to direct_data.csv\n",
      "Data saved to redundant_data.csv\n"
     ]
    }
   ],
   "source": [
    "save_to_csv(direct_df, 'direct_data.csv')\n",
    "save_to_csv(redundant_df, 'redundant_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c3b67a9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python (langmodel)",
   "language": "python",
   "name": "langmodel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}