SummarizeGenAI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b74c8c43-8c2a-4a13-8ac6-12202f7c4701",
   "metadata": {},
   "source": [
    "# Meeting Notes Summarizer: AWS Summarize\n",
    "\n",
    "#### This code represents using SageMaker, and HuggingFace, to summarize the transcripts from a given meeting, and organizing them for further reference.\n",
    "\n",
    "#### GOALS:\n",
    "\n",
    "1. Integrate HuggingFace Text2Text generation Flan-T5-large model to take into input the transcripts from the meeting, and generate an organized text, based on the requirements of the conductor of the meeting\n",
    "\n",
    "2. Integrate a Speech to text converter to convert speech and points from different speakers in the meeting in a live document for our model to refer to and train our data on."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84224fc7-9276-4515-a2e5-a9e115846a0e",
   "metadata": {},
   "source": [
    "### STEP 1: Integrate the hugging face model: flan-t5-large"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e275f9bb-69a0-4af7-a59b-de5ccf9dcd13",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## model_id = \"google/flan-t5-large\"\n",
    "\n",
    "model_id = \"google/flan-t5-base\"\n",
    "\n",
    "## Represents getting the dataset ID that we will refer to: https://huggingface.co/datasets/lytang/MeetingBank-transcript\n",
    "dataset_id = \"lytang/MeetingBank-transcript\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1544058-d60b-4444-937a-f4f1956d1df1",
   "metadata": {},
   "source": [
    "#### This data set above has the following information (Columns):\n",
    "\n",
    "1. meeting_id (string): Represents the meeting ID of the given meeting we want to analyze transcripts from.\n",
    "\n",
    "2. source (string): Represents the source, which contains the speech from the given speaker that we will use as our target.\n",
    "\n",
    "3. type (string): Represents our label, which describes the purpose of the given meeting, also that can be extracted from the title of the meeting. \t\n",
    "\n",
    "4. reference (string): Not needed for our dataset training.\n",
    "\n",
    "5. city (string): Represents the city/location of the meeting - not necessarily needed for our goals of this project."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2835506-49b9-4b90-9912-7878f14efd17",
   "metadata": {},
   "source": [
    "### STEP 2: SETUP"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0e20e6d-2763-4028-8121-c23ae2a92154",
   "metadata": {},
   "source": [
    "#### Represents setting up and installing the transformers dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a2b2baeb-9f69-4f7d-90f7-75e3660b4fe6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: pip in /opt/conda/lib/python3.7/site-packages (23.2.1)\n",
      "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install --upgrade pip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "27fa6e37-8124-4be2-a996-0b739dcc5844",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip -q install transformers datasets sagemaker --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1b01237b-6867-43aa-b531-9a00984e6a17",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip -q install widgetsnbextension ipywidgets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28194218-1b8c-47ed-8535-593a1d108ca9",
   "metadata": {},
   "source": [
    "#### Represents setting up the sagemaker session and setting the default bucket for storing the training data in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f4a9b4f6-f198-403e-9098-1a125b8bf70b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.173.0\n"
     ]
    }
   ],
   "source": [
    "## Importing sagemaker\n",
    "import sagemaker\n",
    "\n",
    "## Visualizing the version of sagemaker that we are operating with \n",
    "print(sagemaker.__version__)\n",
    "\n",
    "## Represents initializing the session\n",
    "sess = sagemaker.Session()\n",
    "\n",
    "## Represents setting up our default bucket\n",
    "bucket = sess.default_bucket()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "613409ec-04f6-4d2f-9976-a6eab453396a",
   "metadata": {},
   "source": [
    "#### Now, we will import the tranformers and datasets libraries from HuggingFace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fddd1e06-e4f1-4662-9c73-3092e5dd9021",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4.30.2\n",
      "2.13.1\n"
     ]
    }
   ],
   "source": [
    "## Importing the transformers library\n",
    "import transformers\n",
    "\n",
    "## Importing the datasets library from hugging face\n",
    "import datasets\n",
    "\n",
    "## Represents showing the versions of both ofn the libraries that we are working with\n",
    "print (transformers.__version__)\n",
    "print (datasets.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c20b0fe3-f815-416d-a4ca-7b4e710c3485",
   "metadata": {},
   "source": [
    "### STEP 3: PREPROCESSING DATA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09af471e-489f-438b-b122-2025848116ed",
   "metadata": {},
   "source": [
    "#### Now, we will LOAD the DATASET from the dataset_id we are going to be using to train our model on"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f1be74e0-07cc-41a1-ba36-a3a950cfea4e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset csv (/root/.cache/huggingface/datasets/lytang___csv/lytang--MeetingBank-transcript-1a764c51490b6b19/0.0.0/eea64c71ca8b46dd3f537ed218fc9bf495d5707789152eb2764f5c78fa66d59d)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f811d9d2d50b4da09b3afdf37a36f6e8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['meeting_id', 'source', 'type', 'reference', 'city'],\n",
       "        num_rows: 5169\n",
       "    })\n",
       "    validation: Dataset({\n",
       "        features: ['meeting_id', 'source', 'type', 'reference', 'city'],\n",
       "        num_rows: 861\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['meeting_id', 'source', 'type', 'reference', 'city'],\n",
       "        num_rows: 862\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Importing the dataset ibraries\n",
    "from datasets import load_dataset, load_from_disk\n",
    "\n",
    "dataset = load_dataset(dataset_id)\n",
    "\n",
    "## Print the dataset information\n",
    "dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d7d2609-a9ed-43f8-bacc-c0fe2b1a83e8",
   "metadata": {},
   "source": [
    "### STEP 4: PREPROCESS THE DATASET"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6cabce80-2778-40ec-90bc-6b1b16d73427",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Represents importing the Autotokenizer library to convert the data into tokens, to output the same after our training\n",
    "from transformers import AutoTokenizer\n",
    "\n",
    "## Represents initializing the tokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "\n",
    "## Represents the prefix\n",
    "prefix = \"summarize: \"\n",
    "\n",
    "## Initializing the input max length\n",
    "input_max_length = 2000\n",
    "\n",
    "## Initializing the output max length\n",
    "output_max_length = 150\n",
    "\n",
    "## Represents the preprocessing function \n",
    "def preprocess_function(examples):\n",
    "    \n",
    "    ## Represents taking in all of the inputs from the document/transcripts\n",
    "    inputs = [prefix + transcript for transcript in examples[\"source\"]]\n",
    "    \n",
    "    ## Represents converting those tokens for inputting into our model\n",
    "    model_inputs = tokenizer(inputs, max_length = input_max_length, truncation=True)\n",
    "    \n",
    "    ## Now, we will set the labels we need from the examples\n",
    "    labels = tokenizer(\n",
    "        text_target = examples[\"type\"], max_length = output_max_length, truncation=True\n",
    "    )\n",
    "    \n",
    "    ## Now, we will take the model inputs and output them\n",
    "    model_inputs[\"labels\"] = labels[\"input_ids\"]\n",
    "    \n",
    "    return model_inputs\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c7f6da4f-0b73-4022-b877-f45f0c2cf2e9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/lytang___csv/lytang--MeetingBank-transcript-1a764c51490b6b19/0.0.0/eea64c71ca8b46dd3f537ed218fc9bf495d5707789152eb2764f5c78fa66d59d/cache-704b52d273e963cd.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/lytang___csv/lytang--MeetingBank-transcript-1a764c51490b6b19/0.0.0/eea64c71ca8b46dd3f537ed218fc9bf495d5707789152eb2764f5c78fa66d59d/cache-e9cb19c15b5760eb.arrow\n",
      "Loading cached processed dataset at /root/.cache/huggingface/datasets/lytang___csv/lytang--MeetingBank-transcript-1a764c51490b6b19/0.0.0/eea64c71ca8b46dd3f537ed218fc9bf495d5707789152eb2764f5c78fa66d59d/cache-e5aa10435d78532e.arrow\n"
     ]
    }
   ],
   "source": [
    "## Now, we will run this on our dataset with the map function in one go\n",
    "\n",
    "tokenized_dataset = dataset.map(\n",
    "    preprocess_function, batched=True, remove_columns=['meeting_id', 'source', 'type', 'reference', 'city']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0308edc-a893-4671-806f-416b23428d5c",
   "metadata": {},
   "source": [
    "### STEP 5: UPLOADING THE DATA TO S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1a0004e4-d083-4abe-b94d-eebb6b7c4d0e",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-988564344122/huggingface/summarize_transcripts\n",
      "s3://sagemaker-us-east-1-988564344122/huggingface/summarize_transcripts/train\n",
      "s3://sagemaker-us-east-1-988564344122/huggingface/summarize_transcripts/validation\n"
     ]
    }
   ],
   "source": [
    "## Represents importing the S3file system from the datasets library\n",
    "from datasets.filesystems import S3FileSystem\n",
    "\n",
    "## Represents initializing the file system\n",
    "s3 = S3FileSystem()\n",
    "\n",
    "s3_prefix = \"huggingface/summarize_transcripts\"\n",
    "\n",
    "## Represents getting the input paths of the dataset, training and validation datasets\n",
    "dataset_input_path = \"s3://{}/{}\".format(bucket, s3_prefix)\n",
    "train_input_path = \"{}/train\".format(dataset_input_path)\n",
    "valid_input_path = \"{}/validation\".format(dataset_input_path)\n",
    "\n",
    "## Represents printing out these paths\n",
    "print(dataset_input_path)\n",
    "print(train_input_path)\n",
    "print(valid_input_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e593fcb9-cb0a-4f11-970c-54517694b4db",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py:1429: FutureWarning: 'fs' was deprecated in favor of 'storage_options' in version 2.8.0 and will be removed in 3.0.0.\n",
      "You can remove this warning by passing 'storage_options=fs.storage_options' instead.\n",
      "  FutureWarning,\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Saving the dataset (0/1 shards):   0%|          | 0/5169 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Saving the dataset (0/1 shards):   0%|          | 0/862 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Now, because we integrated our datasets to the s3 bucket, we can save the tokenized datasets to the disk\n",
    "\n",
    "tokenized_dataset[\"train\"].save_to_disk(train_input_path, fs=s3)\n",
    "tokenized_dataset[\"test\"].save_to_disk(valid_input_path, fs=s3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2548b72-57f6-43e4-9925-6ff985535490",
   "metadata": {},
   "source": [
    "### STEP 6: FINE-TUNE ON SAGEMAKER BY USING A HUGGING FACE DEEP LEARNING CONTAINER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ee73d541-1ef9-4eca-b239-d4548f6d766c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36margparse\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mlogging\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mos\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mevaluate\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mnumpy\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mnp\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mdatasets\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m load_from_disk\u001b[37m\u001b[39;49;00m\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtransformers\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m (\u001b[37m\u001b[39;49;00m\n",
      "    AutoModelForSeq2SeqLM,\u001b[37m\u001b[39;49;00m\n",
      "    AutoTokenizer,\u001b[37m\u001b[39;49;00m\n",
      "    DataCollatorForSeq2Seq,\u001b[37m\u001b[39;49;00m\n",
      "    Seq2SeqTrainer,\u001b[37m\u001b[39;49;00m\n",
      "    Seq2SeqTrainingArguments,\u001b[37m\u001b[39;49;00m\n",
      ")\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "rouge = evaluate.load(\u001b[33m\"\u001b[39;49;00m\u001b[33mrouge\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mcompute_metrics\u001b[39;49;00m(eval_pred):\u001b[37m\u001b[39;49;00m\n",
      "    predictions, labels = eval_pred\u001b[37m\u001b[39;49;00m\n",
      "    decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=\u001b[34mTrue\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "    labels = np.where(labels != -\u001b[34m100\u001b[39;49;00m, labels, tokenizer.pad_token_id)\u001b[37m\u001b[39;49;00m\n",
      "    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=\u001b[34mTrue\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "    result = rouge.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=\u001b[34mTrue\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) \u001b[34mfor\u001b[39;49;00m pred \u001b[35min\u001b[39;49;00m predictions]\u001b[37m\u001b[39;49;00m\n",
      "    result[\u001b[33m\"\u001b[39;49;00m\u001b[33mgen_len\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m] = np.mean(prediction_lens)\u001b[37m\u001b[39;49;00m\n",
      "    \u001b[34mreturn\u001b[39;49;00m {k: \u001b[36mround\u001b[39;49;00m(v, \u001b[34m4\u001b[39;49;00m) \u001b[34mfor\u001b[39;49;00m k, v \u001b[35min\u001b[39;49;00m result.items()}\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "\u001b[34mif\u001b[39;49;00m \u001b[31m__name__\u001b[39;49;00m == \u001b[33m\"\u001b[39;49;00m\u001b[33m__main__\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\u001b[37m\u001b[39;49;00m\n",
      "    parser = argparse.ArgumentParser()\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    \u001b[37m# hyperparameters are passed as command-line arguments to the script.\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--model-name\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--learning-rate\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=\u001b[34m5e-5\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--epochs\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m3\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--train-batch-size\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m2\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--eval-batch-size\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m8\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--evaluation-strategy\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=\u001b[33m\"\u001b[39;49;00m\u001b[33mepoch\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--save-strategy\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=\u001b[33m\"\u001b[39;49;00m\u001b[33mno\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--save-steps\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m500\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    \u001b[37m# Data, model, and output directories\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--output-data-dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_OUTPUT_DATA_DIR\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--model-dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_MODEL_DIR\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--train-dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_CHANNEL_TRAIN\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\u001b[37m\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--valid-dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_CHANNEL_VALID\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    args, _ = parser.parse_known_args()\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    \u001b[37m# load datasets\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "    train_dataset = load_from_disk(args.train_dir)\u001b[37m\u001b[39;49;00m\n",
      "    valid_dataset = load_from_disk(args.valid_dir)\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    logger = logging.getLogger(\u001b[31m__name__\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "    logger.info(\u001b[33mf\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\u001b[33mtraining set: \u001b[39;49;00m\u001b[33m{\u001b[39;49;00mtrain_dataset\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "    logger.info(\u001b[33mf\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\u001b[33mvalidation set: \u001b[39;49;00m\u001b[33m{\u001b[39;49;00mvalid_dataset\u001b[33m}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    \u001b[37m# download model from model hub\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "    model = AutoModelForSeq2SeqLM.from_pretrained(args.model_name)\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    \u001b[37m# download the tokenizer too, which will be saved in the model artifact\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "    \u001b[37m# and used at prediction time\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "    tokenizer = AutoTokenizer.from_pretrained(args.model_name)\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    data_collator = DataCollatorForSeq2Seq(tokenizer=tokenizer, model=model)\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    \u001b[37m# define training args\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "    training_args = Seq2SeqTrainingArguments(\u001b[37m\u001b[39;49;00m\n",
      "        output_dir=args.model_dir,\u001b[37m\u001b[39;49;00m\n",
      "        num_train_epochs=args.epochs,\u001b[37m\u001b[39;49;00m\n",
      "        per_device_train_batch_size=args.train_batch_size,\u001b[37m\u001b[39;49;00m\n",
      "        per_device_eval_batch_size=args.eval_batch_size,\u001b[37m\u001b[39;49;00m\n",
      "        save_strategy=args.save_strategy,\u001b[37m\u001b[39;49;00m\n",
      "        save_steps=args.save_steps,\u001b[37m\u001b[39;49;00m\n",
      "        evaluation_strategy=args.evaluation_strategy,\u001b[37m\u001b[39;49;00m\n",
      "        logging_dir=\u001b[33mf\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\u001b[33m{\u001b[39;49;00margs.output_data_dir\u001b[33m}\u001b[39;49;00m\u001b[33m/logs\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "        learning_rate=\u001b[36mfloat\u001b[39;49;00m(args.learning_rate),\u001b[37m\u001b[39;49;00m\n",
      "        predict_with_generate=\u001b[34mTrue\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "        fp16=\u001b[34mTrue\u001b[39;49;00m,\u001b[37m\u001b[39;49;00m\n",
      "    )\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    \u001b[37m# create trainer\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "    trainer = Seq2SeqTrainer(\u001b[37m\u001b[39;49;00m\n",
      "        model=model,\u001b[37m\u001b[39;49;00m\n",
      "        args=training_args,\u001b[37m\u001b[39;49;00m\n",
      "        tokenizer=tokenizer,\u001b[37m\u001b[39;49;00m\n",
      "        train_dataset=train_dataset,\u001b[37m\u001b[39;49;00m\n",
      "        eval_dataset=valid_dataset,\u001b[37m\u001b[39;49;00m\n",
      "        data_collator=data_collator,\u001b[37m\u001b[39;49;00m\n",
      "        compute_metrics=compute_metrics,\u001b[37m\u001b[39;49;00m\n",
      "    )\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    \u001b[37m# train model\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "    trainer.train()\u001b[37m\u001b[39;49;00m\n",
      "\u001b[37m\u001b[39;49;00m\n",
      "    \u001b[37m# Saves the model to s3\u001b[39;49;00m\u001b[37m\u001b[39;49;00m\n",
      "    trainer.save_model(args.model_dir)\u001b[37m\u001b[39;49;00m\n"
     ]
    }
   ],
   "source": [
    "!pygmentize train.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "94484224-9c0c-488a-9dcb-6f1adead95e4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Defining the hyperparameters\n",
    "hyperparameters = {\n",
    "    \"epochs\": 1, \n",
    "    \"learning_rate\":  0.0001 , \n",
    "    \"train-batch-size\": 1, \n",
    "    \"eval-batch-size\": 3, \n",
    "    \"model-name\": model_id,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7061e5e5-e1f1-461b-85bd-777abec15066",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "## Now, we will create the hugging face estimator, passing in the script, requirements\n",
    "\n",
    "from sagemaker.huggingface import HuggingFace\n",
    "\n",
    "huggingface_estimator = HuggingFace(\n",
    "    \n",
    "    ## Getting the role of the sagemaker\n",
    "    role = sagemaker.get_execution_role(), \n",
    "    \n",
    "    ## Represents fine-tuning our script\n",
    "    entry_point=\"train.py\",\n",
    "    dependencies=[\"requirements.txt\"],\n",
    "    hyperparameters=hyperparameters, \n",
    "    \n",
    "    ## Represents the infrastructure\n",
    "    transformers_version = \"4.26.0\",\n",
    "    pytorch_version = \"1.13.1\",\n",
    "    py_version = \"py39\",\n",
    "    instance_type=\"ml.p3.16xlarge\",\n",
    "    instance_count=1, \n",
    "    distribution={\"smdistributed\": {\"dataparallel\": {\"enabled\": True}}},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d4667fcd-bd7d-48fe-bd92-e981f1e1d6b6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sagemaker in /opt/conda/lib/python3.7/site-packages (2.173.0)\n",
      "Requirement already satisfied: attrs<24,>=23.1.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker) (23.1.0)\n",
      "Requirement already satisfied: boto3<2.0,>=1.26.131 in /opt/conda/lib/python3.7/site-packages (from sagemaker) (1.26.154)\n",
      "Requirement already satisfied: cloudpickle==2.2.1 in /opt/conda/lib/python3.7/site-packages (from sagemaker) (2.2.1)\n",
      "Requirement already satisfied: google-pasta in /opt/conda/lib/python3.7/site-packages (from sagemaker) (0.2.0)\n",
      "Requirement already satisfied: numpy<2.0,>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker) (1.21.6)\n",
      "Requirement already satisfied: protobuf<5.0,>=3.12 in /opt/conda/lib/python3.7/site-packages (from sagemaker) (3.20.3)\n",
      "Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /opt/conda/lib/python3.7/site-packages (from sagemaker) (1.0.1)\n",
      "Requirement already satisfied: importlib-metadata<7.0,>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker) (6.6.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker) (23.1)\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.7/site-packages (from sagemaker) (1.3.5)\n",
      "Requirement already satisfied: pathos in /opt/conda/lib/python3.7/site-packages (from sagemaker) (0.3.0)\n",
      "Requirement already satisfied: schema in /opt/conda/lib/python3.7/site-packages (from sagemaker) (0.7.5)\n",
      "Requirement already satisfied: PyYAML~=6.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker) (6.0)\n",
      "Requirement already satisfied: jsonschema in /opt/conda/lib/python3.7/site-packages (from sagemaker) (3.2.0)\n",
      "Requirement already satisfied: platformdirs in /opt/conda/lib/python3.7/site-packages (from sagemaker) (3.5.3)\n",
      "Requirement already satisfied: tblib==1.7.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker) (1.7.0)\n",
      "Requirement already satisfied: botocore<1.30.0,>=1.29.154 in /opt/conda/lib/python3.7/site-packages (from boto3<2.0,>=1.26.131->sagemaker) (1.29.154)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3<2.0,>=1.26.131->sagemaker) (1.0.1)\n",
      "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from boto3<2.0,>=1.26.131->sagemaker) (0.6.1)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata<7.0,>=1.4.0->sagemaker) (2.2.0)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata<7.0,>=1.4.0->sagemaker) (4.6.3)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from google-pasta->sagemaker) (1.14.0)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema->sagemaker) (0.15.7)\n",
      "Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from jsonschema->sagemaker) (65.5.1)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from pandas->sagemaker) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.7/site-packages (from pandas->sagemaker) (2019.3)\n",
      "Requirement already satisfied: ppft>=1.7.6.6 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker) (1.7.6.6)\n",
      "Requirement already satisfied: dill>=0.3.6 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker) (0.3.6)\n",
      "Requirement already satisfied: pox>=0.3.2 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker) (0.3.2)\n",
      "Requirement already satisfied: multiprocess>=0.70.14 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker) (0.70.14)\n",
      "Requirement already satisfied: contextlib2>=0.5.5 in /opt/conda/lib/python3.7/site-packages (from schema->sagemaker) (0.6.0.post1)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.7/site-packages (from botocore<1.30.0,>=1.29.154->boto3<2.0,>=1.26.131->sagemaker) (1.26.16)\n",
      "\u001b[33mDEPRECATION: pyodbc 4.0.0-unsupported has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pyodbc or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install -U sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "3d39fd22-a98f-4782-9a17-76097582b572",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n",
      "INFO:sagemaker.image_uris:image_uri is not presented, retrieving image_uri based on instance_type, framework etc.\n",
      "INFO:sagemaker:Creating training-job with name: huggingface-pytorch-training-2023-07-31-02-55-44-513\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using provided s3_resource\n",
      "2023-07-31 02:55:44 Starting - Starting the training job......\n",
      "2023-07-31 02:56:41 Starting - Preparing the instances for training.........\n",
      "2023-07-31 02:57:50 Downloading - Downloading input data...\n",
      "2023-07-31 02:58:21 Training - Downloading the training image............\n",
      "2023-07-31 03:00:36 Training - Training image download completed. Training in progress.....\u001b[34mbash: cannot set terminal process group (-1): Inappropriate ioctl for device\u001b[0m\n",
      "\u001b[34mbash: no job control in this shell\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:09,550 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:09,617 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:09,630 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:09,632 sagemaker_pytorch_container.training INFO     Invoking SMDataParallel\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:09,632 sagemaker_pytorch_container.training INFO     Invoking user training script.\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:09,871 sagemaker-training-toolkit INFO     Installing dependencies from requirements.txt:\u001b[0m\n",
      "\u001b[34m/opt/conda/bin/python3.9 -m pip install -r requirements.txt\u001b[0m\n",
      "\u001b[34mCollecting evaluate\u001b[0m\n",
      "\u001b[34mDownloading evaluate-0.4.0-py3-none-any.whl (81 kB)\u001b[0m\n",
      "\u001b[34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.4/81.4 kB 4.3 MB/s eta 0:00:00\u001b[0m\n",
      "\u001b[34mCollecting rouge_score\u001b[0m\n",
      "\u001b[34mDownloading rouge_score-0.1.2.tar.gz (17 kB)\u001b[0m\n",
      "\u001b[34mPreparing metadata (setup.py): started\u001b[0m\n",
      "\u001b[34mPreparing metadata (setup.py): finished with status 'done'\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: tqdm>=4.62.1 in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (4.64.1)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: dill in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (0.3.6)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: multiprocess in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (0.70.14)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: requests>=2.19.0 in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (2.28.2)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: datasets>=2.0.0 in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (2.9.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: xxhash in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (3.2.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: fsspec[http]>=2021.05.0 in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (2023.1.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: packaging in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (23.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: responses<0.19 in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (0.18.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: huggingface-hub>=0.7.0 in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (0.12.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (1.23.5)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: pandas in /opt/conda/lib/python3.9/site-packages (from evaluate->-r requirements.txt (line 1)) (1.5.3)\u001b[0m\n",
      "\u001b[34mCollecting absl-py\u001b[0m\n",
      "\u001b[34mDownloading absl_py-1.4.0-py3-none-any.whl (126 kB)\u001b[0m\n",
      "\u001b[34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 126.5/126.5 kB 18.6 MB/s eta 0:00:00\u001b[0m\n",
      "\u001b[34mCollecting nltk\u001b[0m\n",
      "\u001b[34mDownloading nltk-3.8.1-py3-none-any.whl (1.5 MB)\u001b[0m\n",
      "\u001b[34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 70.3 MB/s eta 0:00:00\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: six>=1.14.0 in /opt/conda/lib/python3.9/site-packages (from rouge_score->-r requirements.txt (line 2)) (1.16.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: aiohttp in /opt/conda/lib/python3.9/site-packages (from datasets>=2.0.0->evaluate->-r requirements.txt (line 1)) (3.8.4)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.9/site-packages (from datasets>=2.0.0->evaluate->-r requirements.txt (line 1)) (5.4.1)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: pyarrow>=6.0.0 in /opt/conda/lib/python3.9/site-packages (from datasets>=2.0.0->evaluate->-r requirements.txt (line 1)) (11.0.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: filelock in /opt/conda/lib/python3.9/site-packages (from huggingface-hub>=0.7.0->evaluate->-r requirements.txt (line 1)) (3.9.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.9/site-packages (from huggingface-hub>=0.7.0->evaluate->-r requirements.txt (line 1)) (4.4.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.9/site-packages (from requests>=2.19.0->evaluate->-r requirements.txt (line 1)) (3.4)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.9/site-packages (from requests>=2.19.0->evaluate->-r requirements.txt (line 1)) (2.1.1)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.9/site-packages (from requests>=2.19.0->evaluate->-r requirements.txt (line 1)) (2022.12.7)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.9/site-packages (from requests>=2.19.0->evaluate->-r requirements.txt (line 1)) (1.26.14)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: joblib in /opt/conda/lib/python3.9/site-packages (from nltk->rouge_score->-r requirements.txt (line 2)) (1.2.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: click in /opt/conda/lib/python3.9/site-packages (from nltk->rouge_score->-r requirements.txt (line 2)) (8.1.2)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: regex>=2021.8.3 in /opt/conda/lib/python3.9/site-packages (from nltk->rouge_score->-r requirements.txt (line 2)) (2022.10.31)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: python-dateutil>=2.8.1 in /opt/conda/lib/python3.9/site-packages (from pandas->evaluate->-r requirements.txt (line 1)) (2.8.2)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.9/site-packages (from pandas->evaluate->-r requirements.txt (line 1)) (2022.7.1)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /opt/conda/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->evaluate->-r requirements.txt (line 1)) (4.0.2)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->evaluate->-r requirements.txt (line 1)) (1.3.3)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->evaluate->-r requirements.txt (line 1)) (1.8.2)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->evaluate->-r requirements.txt (line 1)) (6.0.4)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->evaluate->-r requirements.txt (line 1)) (22.2.0)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.9/site-packages (from aiohttp->datasets>=2.0.0->evaluate->-r requirements.txt (line 1)) (1.3.1)\u001b[0m\n",
      "\u001b[34mBuilding wheels for collected packages: rouge_score\u001b[0m\n",
      "\u001b[34mBuilding wheel for rouge_score (setup.py): started\u001b[0m\n",
      "\u001b[34mBuilding wheel for rouge_score (setup.py): finished with status 'done'\u001b[0m\n",
      "\u001b[34mCreated wheel for rouge_score: filename=rouge_score-0.1.2-py3-none-any.whl size=24935 sha256=629c9b8694790ec091bfbd9580a79e960801cec9e80156388837c6f1214f6012\u001b[0m\n",
      "\u001b[34mStored in directory: /root/.cache/pip/wheels/9b/3d/39/09558097d3119ca0a4d462df68f22c6f3c1b345ac63a09b86e\u001b[0m\n",
      "\u001b[34mSuccessfully built rouge_score\u001b[0m\n",
      "\u001b[34mInstalling collected packages: nltk, absl-py, rouge_score, evaluate\u001b[0m\n",
      "\u001b[34mSuccessfully installed absl-py-1.4.0 evaluate-0.4.0 nltk-3.8.1 rouge_score-0.1.2\u001b[0m\n",
      "\u001b[34mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n",
      "\u001b[34m[notice] A new release of pip is available: 23.0 -> 23.2.1\u001b[0m\n",
      "\u001b[34m[notice] To update, run: pip install --upgrade pip\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,812 sagemaker-training-toolkit INFO     Waiting for the process to finish and give a return code.\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,813 sagemaker-training-toolkit INFO     Done waiting for a return code. Received 0 from exiting process.\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,882 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,963 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,976 sagemaker-training-toolkit INFO     Starting MPI run as worker node.\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,976 sagemaker-training-toolkit INFO     Creating SSH daemon.\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,978 sagemaker-training-toolkit INFO     Waiting for MPI workers to establish their SSH connections\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,979 sagemaker-training-toolkit INFO     Network interface name: eth0\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,979 sagemaker-training-toolkit INFO     Host: ['algo-1']\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,980 sagemaker-training-toolkit INFO     sagemaker_communication_backend: None\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,980 sagemaker-training-toolkit WARNING  Missing library /opt/conda/lib/libsmddp.so for SMDDP collective\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,980 sagemaker-training-toolkit WARNING  The system is not configured to run SMDDP collectives optimizedfor AWS infrastructure.Please use the latest SageMaker Deep Learning Container (DLC) to enable SMDDP Collectives support.\u001b[0m\n",
      "\u001b[34mContinuing model training with default NCCL communication backend.\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,980 sagemaker-training-toolkit INFO     instance type: ml.p3.16xlarge\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:15,981 sagemaker-training-toolkit INFO     Env Hosts: ['algo-1'] Hosts: ['algo-1'] process_per_hosts: 8 num_processes: 8\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:16,048 sagemaker-training-toolkit INFO     No Neurons detected (normal if no neurons installed)\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:16,062 sagemaker-training-toolkit INFO     Invoking user script\u001b[0m\n",
      "\u001b[34mTraining Env:\u001b[0m\n",
      "\u001b[34m{\n",
      "    \"additional_framework_parameters\": {\n",
      "        \"sagemaker_distributed_dataparallel_custom_mpi_options\": \"\",\n",
      "        \"sagemaker_distributed_dataparallel_enabled\": true,\n",
      "        \"sagemaker_instance_type\": \"ml.p3.16xlarge\"\n",
      "    },\n",
      "    \"channel_input_dirs\": {\n",
      "        \"train\": \"/opt/ml/input/data/train\",\n",
      "        \"valid\": \"/opt/ml/input/data/valid\"\n",
      "    },\n",
      "    \"current_host\": \"algo-1\",\n",
      "    \"current_instance_group\": \"homogeneousCluster\",\n",
      "    \"current_instance_group_hosts\": [\n",
      "        \"algo-1\"\n",
      "    ],\n",
      "    \"current_instance_type\": \"ml.p3.16xlarge\",\n",
      "    \"distribution_hosts\": [\n",
      "        \"algo-1\"\n",
      "    ],\n",
      "    \"distribution_instance_groups\": [\n",
      "        \"homogeneousCluster\"\n",
      "    ],\n",
      "    \"framework_module\": \"sagemaker_pytorch_container.training:main\",\n",
      "    \"hosts\": [\n",
      "        \"algo-1\"\n",
      "    ],\n",
      "    \"hyperparameters\": {\n",
      "        \"epochs\": 1,\n",
      "        \"eval-batch-size\": 3,\n",
      "        \"learning_rate\": 0.0001,\n",
      "        \"model-name\": \"google/flan-t5-base\",\n",
      "        \"train-batch-size\": 1\n",
      "    },\n",
      "    \"input_config_dir\": \"/opt/ml/input/config\",\n",
      "    \"input_data_config\": {\n",
      "        \"train\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        },\n",
      "        \"valid\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        }\n",
      "    },\n",
      "    \"input_dir\": \"/opt/ml/input\",\n",
      "    \"instance_groups\": [\n",
      "        \"homogeneousCluster\"\n",
      "    ],\n",
      "    \"instance_groups_dict\": {\n",
      "        \"homogeneousCluster\": {\n",
      "            \"instance_group_name\": \"homogeneousCluster\",\n",
      "            \"instance_type\": \"ml.p3.16xlarge\",\n",
      "            \"hosts\": [\n",
      "                \"algo-1\"\n",
      "            ]\n",
      "        }\n",
      "    },\n",
      "    \"is_hetero\": false,\n",
      "    \"is_master\": true,\n",
      "    \"is_modelparallel_enabled\": null,\n",
      "    \"is_smddpmprun_installed\": true,\n",
      "    \"job_name\": \"huggingface-pytorch-training-2023-07-31-02-55-44-513\",\n",
      "    \"log_level\": 20,\n",
      "    \"master_hostname\": \"algo-1\",\n",
      "    \"model_dir\": \"/opt/ml/model\",\n",
      "    \"module_dir\": \"s3://sagemaker-us-east-1-988564344122/huggingface-pytorch-training-2023-07-31-02-55-44-513/source/sourcedir.tar.gz\",\n",
      "    \"module_name\": \"train\",\n",
      "    \"network_interface_name\": \"eth0\",\n",
      "    \"num_cpus\": 64,\n",
      "    \"num_gpus\": 8,\n",
      "    \"num_neurons\": 0,\n",
      "    \"output_data_dir\": \"/opt/ml/output/data\",\n",
      "    \"output_dir\": \"/opt/ml/output\",\n",
      "    \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
      "    \"resource_config\": {\n",
      "        \"current_host\": \"algo-1\",\n",
      "        \"current_instance_type\": \"ml.p3.16xlarge\",\n",
      "        \"current_group_name\": \"homogeneousCluster\",\n",
      "        \"hosts\": [\n",
      "            \"algo-1\"\n",
      "        ],\n",
      "        \"instance_groups\": [\n",
      "            {\n",
      "                \"instance_group_name\": \"homogeneousCluster\",\n",
      "                \"instance_type\": \"ml.p3.16xlarge\",\n",
      "                \"hosts\": [\n",
      "                    \"algo-1\"\n",
      "                ]\n",
      "            }\n",
      "        ],\n",
      "        \"network_interface_name\": \"eth0\"\n",
      "    },\n",
      "    \"user_entry_point\": \"train.py\"\u001b[0m\n",
      "\u001b[34m}\u001b[0m\n",
      "\u001b[34mEnvironment variables:\u001b[0m\n",
      "\u001b[34mSM_HOSTS=[\"algo-1\"]\u001b[0m\n",
      "\u001b[34mSM_NETWORK_INTERFACE_NAME=eth0\u001b[0m\n",
      "\u001b[34mSM_HPS={\"epochs\":1,\"eval-batch-size\":3,\"learning_rate\":0.0001,\"model-name\":\"google/flan-t5-base\",\"train-batch-size\":1}\u001b[0m\n",
      "\u001b[34mSM_USER_ENTRY_POINT=train.py\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_PARAMS={\"sagemaker_distributed_dataparallel_custom_mpi_options\":\"\",\"sagemaker_distributed_dataparallel_enabled\":true,\"sagemaker_instance_type\":\"ml.p3.16xlarge\"}\u001b[0m\n",
      "\u001b[34mSM_RESOURCE_CONFIG={\"current_group_name\":\"homogeneousCluster\",\"current_host\":\"algo-1\",\"current_instance_type\":\"ml.p3.16xlarge\",\"hosts\":[\"algo-1\"],\"instance_groups\":[{\"hosts\":[\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.p3.16xlarge\"}],\"network_interface_name\":\"eth0\"}\u001b[0m\n",
      "\u001b[34mSM_INPUT_DATA_CONFIG={\"train\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"valid\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}}\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DATA_DIR=/opt/ml/output/data\u001b[0m\n",
      "\u001b[34mSM_CHANNELS=[\"train\",\"valid\"]\u001b[0m\n",
      "\u001b[34mSM_CURRENT_HOST=algo-1\u001b[0m\n",
      "\u001b[34mSM_CURRENT_INSTANCE_TYPE=ml.p3.16xlarge\u001b[0m\n",
      "\u001b[34mSM_CURRENT_INSTANCE_GROUP=homogeneousCluster\u001b[0m\n",
      "\u001b[34mSM_CURRENT_INSTANCE_GROUP_HOSTS=[\"algo-1\"]\u001b[0m\n",
      "\u001b[34mSM_INSTANCE_GROUPS=[\"homogeneousCluster\"]\u001b[0m\n",
      "\u001b[34mSM_INSTANCE_GROUPS_DICT={\"homogeneousCluster\":{\"hosts\":[\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.p3.16xlarge\"}}\u001b[0m\n",
      "\u001b[34mSM_DISTRIBUTION_INSTANCE_GROUPS=[\"homogeneousCluster\"]\u001b[0m\n",
      "\u001b[34mSM_IS_HETERO=false\u001b[0m\n",
      "\u001b[34mSM_MODULE_NAME=train\u001b[0m\n",
      "\u001b[34mSM_LOG_LEVEL=20\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main\u001b[0m\n",
      "\u001b[34mSM_INPUT_DIR=/opt/ml/input\u001b[0m\n",
      "\u001b[34mSM_INPUT_CONFIG_DIR=/opt/ml/input/config\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DIR=/opt/ml/output\u001b[0m\n",
      "\u001b[34mSM_NUM_CPUS=64\u001b[0m\n",
      "\u001b[34mSM_NUM_GPUS=8\u001b[0m\n",
      "\u001b[34mSM_NUM_NEURONS=0\u001b[0m\n",
      "\u001b[34mSM_MODEL_DIR=/opt/ml/model\u001b[0m\n",
      "\u001b[34mSM_MODULE_DIR=s3://sagemaker-us-east-1-988564344122/huggingface-pytorch-training-2023-07-31-02-55-44-513/source/sourcedir.tar.gz\u001b[0m\n",
      "\u001b[34mSM_TRAINING_ENV={\"additional_framework_parameters\":{\"sagemaker_distributed_dataparallel_custom_mpi_options\":\"\",\"sagemaker_distributed_dataparallel_enabled\":true,\"sagemaker_instance_type\":\"ml.p3.16xlarge\"},\"channel_input_dirs\":{\"train\":\"/opt/ml/input/data/train\",\"valid\":\"/opt/ml/input/data/valid\"},\"current_host\":\"algo-1\",\"current_instance_group\":\"homogeneousCluster\",\"current_instance_group_hosts\":[\"algo-1\"],\"current_instance_type\":\"ml.p3.16xlarge\",\"distribution_hosts\":[\"algo-1\"],\"distribution_instance_groups\":[\"homogeneousCluster\"],\"framework_module\":\"sagemaker_pytorch_container.training:main\",\"hosts\":[\"algo-1\"],\"hyperparameters\":{\"epochs\":1,\"eval-batch-size\":3,\"learning_rate\":0.0001,\"model-name\":\"google/flan-t5-base\",\"train-batch-size\":1},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"train\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"valid\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"instance_groups\":[\"homogeneousCluster\"],\"instance_groups_dict\":{\"homogeneousCluster\":{\"hosts\":[\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.p3.16xlarge\"}},\"is_hetero\":false,\"is_master\":true,\"is_modelparallel_enabled\":null,\"is_smddpmprun_installed\":true,\"job_name\":\"huggingface-pytorch-training-2023-07-31-02-55-44-513\",\"log_level\":20,\"master_hostname\":\"algo-1\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-us-east-1-988564344122/huggingface-pytorch-training-2023-07-31-02-55-44-513/source/sourcedir.tar.gz\",\"module_name\":\"train\",\"network_interface_name\":\"eth0\",\"num_cpus\":64,\"num_gpus\":8,\"num_neurons\":0,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_group_name\":\"homogeneousCluster\",\"current_host\":\"algo-1\",\"current_instance_type\":\"ml.p3.16xlarge\",\"hosts\":[\"algo-1\"],\"instance_groups\":[{\"hosts\":[\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.p3.16xlarge\"}],\"network_interface_name\":\"eth0\"},\"user_entry_point\":\"train.py\"}\u001b[0m\n",
      "\u001b[34mSM_USER_ARGS=[\"--epochs\",\"1\",\"--eval-batch-size\",\"3\",\"--learning_rate\",\"0.0001\",\"--model-name\",\"google/flan-t5-base\",\"--train-batch-size\",\"1\"]\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_TRAIN=/opt/ml/input/data/train\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_VALID=/opt/ml/input/data/valid\u001b[0m\n",
      "\u001b[34mSM_HP_EPOCHS=1\u001b[0m\n",
      "\u001b[34mSM_HP_EVAL-BATCH-SIZE=3\u001b[0m\n",
      "\u001b[34mSM_HP_LEARNING_RATE=0.0001\u001b[0m\n",
      "\u001b[34mSM_HP_MODEL-NAME=google/flan-t5-base\u001b[0m\n",
      "\u001b[34mSM_HP_TRAIN-BATCH-SIZE=1\u001b[0m\n",
      "\u001b[34mPYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python39.zip:/opt/conda/lib/python3.9:/opt/conda/lib/python3.9/lib-dynload:/opt/conda/lib/python3.9/site-packages\u001b[0m\n",
      "\u001b[34mInvoking script with the following command:\u001b[0m\n",
      "\u001b[34mmpirun --host algo-1 -np 8 --allow-run-as-root --tag-output --oversubscribe -mca btl_tcp_if_include eth0 -mca oob_tcp_if_include eth0 -mca plm_rsh_no_tree_spawn 1 -mca pml ob1 -mca btl ^openib -mca orte_abort_on_non_zero_status 1 -mca btl_vader_single_copy_mechanism none -mca plm_rsh_num_concurrent 1 -x NCCL_SOCKET_IFNAME=eth0 -x NCCL_DEBUG=INFO -x LD_LIBRARY_PATH -x PATH -x SMDATAPARALLEL_USE_SINGLENODE=1 -x FI_PROVIDER=efa -x RDMAV_FORK_SAFE=1 -x LD_PRELOAD=/opt/conda/lib/python3.9/site-packages/gethostname.cpython-39-x86_64-linux-gnu.so -x USE_SMDDP_COLLECTIVES=0 smddprun /opt/conda/bin/python3.9 -m mpi4py train.py --epochs 1 --eval-batch-size 3 --learning_rate 0.0001 --model-name google/flan-t5-base --train-batch-size 1\u001b[0m\n",
      "\u001b[34m[2023-07-31 03:01:18.261: W smdistributed/modelparallel/torch/nn/predefined_hooks.py:78] Found unsupported HuggingFace version 4.26.0 for automated tensor parallelism. HuggingFace modules will not be automatically distributed. You can use smp.tp_register_with_module API to register desired modules for tensor parallelism, or directly instantiate an smp.nn.DistributedModule. Supported HuggingFace transformers versions for automated tensor parallelism: ['4.17.0', '4.20.1', '4.21.0']\u001b[0m\n",
      "\u001b[34m2023-07-31 03:01:18,268 root         INFO     Using NamedTuple = typing._NamedTuple instead.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:curl: /opt/conda/lib/libcurl.so.4: no version information available (required by curl)\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:curl: /opt/conda/lib/libcurl.so.4: no version information available (required by curl)\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:curl: /opt/conda/lib/libcurl.so.4: no version information available (required by curl)\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading builder script:   0%|          | 0.00/6.27k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading builder script: 100%|██████████| 6.27k/6.27k [00:00<00:00, 7.85MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:#015Downloading builder script:   0%|          | 0.00/6.27k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:#015Downloading builder script: 100%|██████████| 6.27k/6.27k [00:00<00:00, 7.05MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stderr>:#015Downloading builder script:   0%|          | 0.00/6.27k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stderr>:#015Downloading builder script: 100%|██████████| 6.27k/6.27k [00:00<00:00, 4.65MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stderr>:#015Downloading builder script:   0%|          | 0.00/6.27k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stderr>:#015Downloading builder script: 100%|██████████| 6.27k/6.27k [00:00<00:00, 4.90MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stderr>:#015Downloading builder script:   0%|          | 0.00/6.27k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stderr>:#015Downloading builder script: 100%|██████████| 6.27k/6.27k [00:00<00:00, 4.77MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)lve/main/config.json:   0%|          | 0.00/1.40k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)lve/main/config.json: 100%|██████████| 1.40k/1.40k [00:00<00:00, 170kB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stderr>:#015Downloading builder script:   0%|          | 0.00/6.27k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stderr>:#015Downloading builder script: 100%|██████████| 6.27k/6.27k [00:00<00:00, 4.55MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:   0%|          | 0.00/990M [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stderr>:#015Downloading builder script:   0%|          | 0.00/6.27k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stderr>:#015Downloading builder script: 100%|██████████| 6.27k/6.27k [00:00<00:00, 7.75MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:   4%|▍         | 41.9M/990M [00:00<00:02, 373MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  10%|▉         | 94.4M/990M [00:00<00:02, 377MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  14%|█▍        | 136M/990M [00:00<00:02, 376MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  19%|█▉        | 189M/990M [00:00<00:01, 404MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  24%|██▍       | 241M/990M [00:00<00:01, 420MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  30%|██▉       | 294M/990M [00:00<00:01, 428MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  35%|███▍      | 346M/990M [00:00<00:01, 435MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  40%|████      | 398M/990M [00:00<00:01, 438MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  46%|████▌     | 451M/990M [00:01<00:01, 442MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  51%|█████     | 503M/990M [00:01<00:01, 443MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  56%|█████▌    | 556M/990M [00:01<00:00, 444MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  61%|██████▏   | 608M/990M [00:01<00:00, 444MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  67%|██████▋   | 661M/990M [00:01<00:00, 446MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  72%|███████▏  | 713M/990M [00:01<00:00, 446MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  77%|███████▋  | 765M/990M [00:01<00:00, 446MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  83%|████████▎ | 818M/990M [00:01<00:00, 446MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  88%|████████▊ | 870M/990M [00:02<00:00, 446MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  93%|█████████▎| 923M/990M [00:02<00:00, 447MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";:  98%|█████████▊| 975M/990M [00:02<00:00, 445MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)\"pytorch_model.bin\";: 100%|██████████| 990M/990M [00:02<00:00, 434MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:#015Downloading (…)neration_config.json:   0%|          | 0.00/147 [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:#015Downloading (…)neration_config.json: 100%|██████████| 147/147 [00:00<00:00, 18.6kB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)okenizer_config.json:   0%|          | 0.00/2.54k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:#015Downloading (…)okenizer_config.json: 100%|██████████| 2.54k/2.54k [00:00<00:00, 694kB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015Downloading (…)\"spiece.model\";:   0%|          | 0.00/792k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015Downloading (…)\"spiece.model\";: 100%|██████████| 792k/792k [00:00<00:00, 124MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015Downloading (…)/main/tokenizer.json:   0%|          | 0.00/2.42M [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015Downloading (…)/main/tokenizer.json: 100%|██████████| 2.42M/2.42M [00:00<00:00, 51.9MB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:#015Downloading (…)cial_tokens_map.json:   0%|          | 0.00/2.20k [00:00<?, ?B/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:#015Downloading (…)cial_tokens_map.json: 100%|██████████| 2.20k/2.20k [00:00<00:00, 809kB/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:DDP Mode\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:DDP Mode\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:DDP Mode\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:DDP Mode\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:DDP Mode\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:DDP Mode\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:DDP Mode\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:DDP Mode\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Bootstrap : Using eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO cudaDriverVersion 12000\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:NCCL version 2.14.3+cuda11.7\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO cudaDriverVersion 12000\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO cudaDriverVersion 12000\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO cudaDriverVersion 12000\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO cudaDriverVersion 12000\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO cudaDriverVersion 12000\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO cudaDriverVersion 12000\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO cudaDriverVersion 12000\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] find_ofi_provider:608 NCCL WARN NET/OFI Couldn't find any optimal provider\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] ofi_init:1355 NCCL WARN NET/OFI aws-ofi-nccl initialization failed\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO NET/Socket : Using [0]eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Bootstrap : Using eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] find_ofi_provider:608 NCCL WARN NET/OFI Couldn't find any optimal provider\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] ofi_init:1355 NCCL WARN NET/OFI aws-ofi-nccl initialization failed\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO NCCL_IB_DISABLE set by environment to 1.\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO NET/Socket : Using [0]eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Bootstrap : Using eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] find_ofi_provider:608 NCCL WARN NET/OFI Couldn't find any optimal provider\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] ofi_init:1355 NCCL WARN NET/OFI aws-ofi-nccl initialization failed\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO NCCL_IB_DISABLE set by environment to 1.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO NET/Socket : Using [0]eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Bootstrap : Using eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] find_ofi_provider:608 NCCL WARN NET/OFI Couldn't find any optimal provider\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] ofi_init:1355 NCCL WARN NET/OFI aws-ofi-nccl initialization failed\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO NCCL_IB_DISABLE set by environment to 1.\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO NET/Socket : Using [0]eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Bootstrap : Using eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] find_ofi_provider:608 NCCL WARN NET/OFI Couldn't find any optimal provider\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] ofi_init:1355 NCCL WARN NET/OFI aws-ofi-nccl initialization failed\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO NCCL_IB_DISABLE set by environment to 1.\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO NET/Socket : Using [0]eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Bootstrap : Using eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] find_ofi_provider:608 NCCL WARN NET/OFI Couldn't find any optimal provider\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] ofi_init:1355 NCCL WARN NET/OFI aws-ofi-nccl initialization failed\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO NCCL_IB_DISABLE set by environment to 1.\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO NET/Socket : Using [0]eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Bootstrap : Using eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] find_ofi_provider:608 NCCL WARN NET/OFI Couldn't find any optimal provider\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] ofi_init:1355 NCCL WARN NET/OFI aws-ofi-nccl initialization failed\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO NCCL_IB_DISABLE set by environment to 1.\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO NET/Socket : Using [0]eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Bootstrap : Using eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v6 symbol.\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin symbol (v4 or v5).\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO NET/OFI Using aws-ofi-nccl 1.4.0aws\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO NET/OFI Setting FI_EFA_FORK_SAFE environment variable to 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] find_ofi_provider:608 NCCL WARN NET/OFI Couldn't find any optimal provider\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] ofi_init:1355 NCCL WARN NET/OFI aws-ofi-nccl initialization failed\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO NET/Socket : Using [0]eth0:10.2.83.153<0>\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Using network Socket\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Trees [0] 1/-1/-1->2->3 [1] 1/-1/-1->2->3 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] -1/-1/-1->2->6 [5] 6/-1/-1->2->0 [6] 1/-1/-1->2->3 [7] 1/-1/-1->2->3 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] -1/-1/-1->2->6 [11] 6/-1/-1->2->0\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Trees [0] 6/-1/-1->5->1 [1] 6/-1/-1->5->1 [2] 1/-1/-1->5->6 [3] 1/-1/-1->5->6 [4] 4/-1/-1->5->7 [5] 7/-1/-1->5->4 [6] 6/-1/-1->5->1 [7] 6/-1/-1->5->1 [8] 1/-1/-1->5->6 [9] 1/-1/-1->5->6 [10] 4/-1/-1->5->7 [11] 7/-1/-1->5->4\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 00/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 01/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 02/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 03/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 04/12 :    0   1   3   7   5   4   6   2\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 05/12 :    0   2   6   4   5   7   3   1\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 06/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 07/12 :    0   3   2   1   5   6   7   4\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 08/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 09/12 :    0   4   7   6   5   1   2   3\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 10/12 :    0   1   3   7   5   4   6   2\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 11/12 :    0   2   6   4   5   7   3   1\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Trees [0] 3/-1/-1->0->-1 [1] 3/-1/-1->0->-1 [2] 4/-1/-1->0->-1 [3] 4/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 2/-1/-1->0->-1 [6] 3/-1/-1->0->-1 [7] 3/-1/-1->0->-1 [8] 4/-1/-1->0->-1 [9] 4/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 2/-1/-1->0->-1\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Trees [0] -1/-1/-1->4->7 [1] -1/-1/-1->4->7 [2] 7/-1/-1->4->0 [3] 7/-1/-1->4->0 [4] 6/-1/-1->4->5 [5] 5/-1/-1->4->6 [6] -1/-1/-1->4->7 [7] -1/-1/-1->4->7 [8] 7/-1/-1->4->0 [9] 7/-1/-1->4->0 [10] 6/-1/-1->4->5 [11] 5/-1/-1->4->6\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Trees [0] 2/-1/-1->3->0 [1] 2/-1/-1->3->0 [2] -1/-1/-1->3->2 [3] -1/-1/-1->3->2 [4] 7/-1/-1->3->1 [5] 1/-1/-1->3->7 [6] 2/-1/-1->3->0 [7] 2/-1/-1->3->0 [8] -1/-1/-1->3->2 [9] -1/-1/-1->3->2 [10] 7/-1/-1->3->1 [11] 1/-1/-1->3->7\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Trees [0] 4/-1/-1->7->6 [1] 4/-1/-1->7->6 [2] 6/-1/-1->7->4 [3] 6/-1/-1->7->4 [4] 5/-1/-1->7->3 [5] 3/-1/-1->7->5 [6] 4/-1/-1->7->6 [7] 4/-1/-1->7->6 [8] 6/-1/-1->7->4 [9] 6/-1/-1->7->4 [10] 5/-1/-1->7->3 [11] 3/-1/-1->7->5\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 5/-1/-1->6->7 [3] 5/-1/-1->6->7 [4] 2/-1/-1->6->4 [5] 4/-1/-1->6->2 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 5/-1/-1->6->7 [9] 5/-1/-1->6->7 [10] 2/-1/-1->6->4 [11] 4/-1/-1->6->2\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Trees [0] 5/-1/-1->1->2 [1] 5/-1/-1->1->2 [2] 2/-1/-1->1->5 [3] 2/-1/-1->1->5 [4] 3/-1/-1->1->0 [5] -1/-1/-1->1->3 [6] 5/-1/-1->1->2 [7] 5/-1/-1->1->2 [8] 2/-1/-1->1->5 [9] 2/-1/-1->1->5 [10] 3/-1/-1->1->0 [11] -1/-1/-1->1->3\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 05/0 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 04/0 : 0[170] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 11/0 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 10/0 : 0[170] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 00/0 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 02/0 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 01/0 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 03/0 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 00/0 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 02/0 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 06/0 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 08/0 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 07/0 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 03/0 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 09/0 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 01/0 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 08/0 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 06/0 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 09/0 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 04/0 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 07/0 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 05/0 : 0[170] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 10/0 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 11/0 : 0[170] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 04/0 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 05/0 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 10/0 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 11/0 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 00/0 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 02/0 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 01/0 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 03/0 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 05/0 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 06/0 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 04/0 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 08/0 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 11/0 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 02/0 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 07/0 : 0[170] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 00/0 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 09/0 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 10/0 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 03/0 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 01/0 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 08/0 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 06/0 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 09/0 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 07/0 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 05/0 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 04/0 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 11/0 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 10/0 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 04/0 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 05/0 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 10/0 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 11/0 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 02/0 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 00/0 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 03/0 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 01/0 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 08/0 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 06/0 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 02/0 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 09/0 : 0[170] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 00/0 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 07/0 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 03/0 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 01/0 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 08/0 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 06/0 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 09/0 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 07/0 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 05/0 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 04/0 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 11/0 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 10/0 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 00/0 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 02/0 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 01/0 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 03/0 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 06/0 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 08/0 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 07/0 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 09/0 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 00/0 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 05/0 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 04/0 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 02/0 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 01/0 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 11/0 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 10/0 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 03/0 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 06/0 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 08/0 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 07/0 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 09/0 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Connected all rings\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 04/0 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Connected all rings\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 10/0 : 4[1b0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Connected all rings\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Connected all rings\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Connected all rings\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 00/0 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Connected all rings\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 01/0 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Connected all rings\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Connected all rings\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 06/0 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 02/0 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 07/0 : 1[180] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 03/0 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 08/0 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 00/0 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 09/0 : 5[1c0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 02/0 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 01/0 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 03/0 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 06/0 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 05/0 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 08/0 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 07/0 : 2[190] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 11/0 : 4[1b0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 09/0 : 6[1d0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 05/0 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 11/0 : 1[180] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 04/0 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 00/0 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 10/0 : 5[1c0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 01/0 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 05/0 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 04/0 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 05/0 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 11/0 : 3[1a0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 06/0 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 10/0 : 2[190] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 11/0 : 6[1d0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 07/0 : 4[1b0] -> 7[1e0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 02/0 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 03/0 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 08/0 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 04/0 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 05/0 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 09/0 : 1[180] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 00/0 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 10/0 : 6[1d0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 11/0 : 2[190] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 01/0 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 06/0 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 02/0 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 07/0 : 5[1c0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 03/0 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 08/0 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 04/0 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 09/0 : 4[1b0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 10/0 : 7[1e0] -> 3[1a0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 02/0 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 00/0 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 03/0 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 01/0 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 06/0 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 08/0 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 07/0 : 3[1a0] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 09/0 : 7[1e0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 04/0 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 05/0 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 10/0 : 3[1a0] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 11/0 : 7[1e0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 02/0 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 00/0 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 03/0 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 01/0 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 08/0 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 06/0 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 09/0 : 3[1a0] -> 2[190] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 02/0 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 04/0 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 07/0 : 7[1e0] -> 6[1d0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 00/0 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 05/0 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 03/0 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 01/0 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 10/0 : 1[180] -> 0[170] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 11/0 : 5[1c0] -> 4[1b0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 08/0 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 06/0 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 09/0 : 2[190] -> 1[180] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 07/0 : 6[1d0] -> 5[1c0] via P2P/IPC\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Connected all trees\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Connected all trees\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Connected all trees\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 08/1 : 7[1e0] -> 0[170] via P2P/indirect/4[1b0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Connected all trees\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 08/1 : 3[1a0] -> 4[1b0] via P2P/indirect/0[170]\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Connected all trees\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Connected all trees\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Connected all trees\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 04/1 : 2[190] -> 4[1b0] via P2P/indirect/0[170]\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Connected all trees\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO 12 coll channels, 16 p2p channels, 2 p2p channels per peer\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 04/1 : 6[1d0] -> 0[170] via P2P/indirect/4[1b0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 09/1 : 7[1e0] -> 0[170] via P2P/indirect/4[1b0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 09/1 : 3[1a0] -> 4[1b0] via P2P/indirect/0[170]\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 04/1 : 7[1e0] -> 1[180] via P2P/indirect/5[1c0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 05/1 : 6[1d0] -> 0[170] via P2P/indirect/4[1b0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 04/1 : 3[1a0] -> 5[1c0] via P2P/indirect/1[180]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 05/1 : 2[190] -> 4[1b0] via P2P/indirect/0[170]\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 05/1 : 7[1e0] -> 1[180] via P2P/indirect/5[1c0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 05/1 : 3[1a0] -> 5[1c0] via P2P/indirect/1[180]\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 12/1 : 7[1e0] -> 2[190] via P2P/indirect/3[1a0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 12/1 : 3[1a0] -> 6[1d0] via P2P/indirect/7[1e0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO Channel 13/1 : 7[1e0] -> 2[190] via P2P/indirect/3[1a0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 12/1 : 1[180] -> 4[1b0] via P2P/indirect/0[170]\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 12/1 : 6[1d0] -> 1[180] via P2P/indirect/5[1c0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO Channel 13/1 : 3[1a0] -> 6[1d0] via P2P/indirect/7[1e0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 12/1 : 5[1c0] -> 0[170] via P2P/indirect/4[1b0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 13/1 : 1[180] -> 4[1b0] via P2P/indirect/0[170]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 12/1 : 2[190] -> 5[1c0] via P2P/indirect/1[180]\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 13/1 : 6[1d0] -> 1[180] via P2P/indirect/5[1c0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 10/1 : 4[1b0] -> 1[180] via P2P/indirect/5[1c0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 13/1 : 5[1c0] -> 0[170] via P2P/indirect/4[1b0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 13/1 : 2[190] -> 5[1c0] via P2P/indirect/1[180]\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 11/1 : 4[1b0] -> 1[180] via P2P/indirect/5[1c0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 10/1 : 0[170] -> 5[1c0] via P2P/indirect/1[180]\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 10/1 : 6[1d0] -> 3[1a0] via P2P/indirect/2[190]\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 10/1 : 1[180] -> 6[1d0] via P2P/indirect/5[1c0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO Channel 11/1 : 6[1d0] -> 3[1a0] via P2P/indirect/2[190]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 11/1 : 0[170] -> 5[1c0] via P2P/indirect/1[180]\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 11/1 : 1[180] -> 6[1d0] via P2P/indirect/5[1c0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 10/1 : 5[1c0] -> 2[190] via P2P/indirect/1[180]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 10/1 : 2[190] -> 7[1e0] via P2P/indirect/6[1d0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 06/1 : 4[1b0] -> 2[190] via P2P/indirect/6[1d0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO Channel 11/1 : 2[190] -> 7[1e0] via P2P/indirect/6[1d0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 06/1 : 1[180] -> 7[1e0] via P2P/indirect/3[1a0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 11/1 : 5[1c0] -> 2[190] via P2P/indirect/1[180]\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO Channel 07/1 : 1[180] -> 7[1e0] via P2P/indirect/3[1a0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 07/1 : 4[1b0] -> 2[190] via P2P/indirect/6[1d0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 06/1 : 0[170] -> 6[1d0] via P2P/indirect/4[1b0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 07/1 : 0[170] -> 6[1d0] via P2P/indirect/4[1b0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 06/1 : 5[1c0] -> 3[1a0] via P2P/indirect/1[180]\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO Channel 07/1 : 5[1c0] -> 3[1a0] via P2P/indirect/1[180]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 14/1 : 0[170] -> 7[1e0] via P2P/indirect/4[1b0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 14/1 : 4[1b0] -> 3[1a0] via P2P/indirect/0[170]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO Channel 15/1 : 0[170] -> 7[1e0] via P2P/indirect/4[1b0]\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO Channel 15/1 : 4[1b0] -> 3[1a0] via P2P/indirect/0[170]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:algo-1:139:139 [3] NCCL INFO comm 0x55867936ee80 rank 3 nranks 8 cudaDev 3 busId 1a0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:algo-1:133:133 [2] NCCL INFO comm 0x55ef913d1dc0 rank 2 nranks 8 cudaDev 2 busId 190 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:algo-1:146:146 [7] NCCL INFO comm 0x562a051c3d50 rank 7 nranks 8 cudaDev 7 busId 1e0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:algo-1:141:141 [4] NCCL INFO comm 0x558877f395c0 rank 4 nranks 8 cudaDev 4 busId 1b0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:algo-1:598:598 [0] NCCL INFO comm 0x56421b483b40 rank 0 nranks 8 cudaDev 0 busId 170 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Running smdistributed.dataparallel v1.7.0\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:SMDDP: Single node mode\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:algo-1:143:143 [5] NCCL INFO comm 0x5611b5a0dad0 rank 5 nranks 8 cudaDev 5 busId 1c0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:algo-1:130:130 [1] NCCL INFO comm 0x561b6a4fc560 rank 1 nranks 8 cudaDev 1 busId 180 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:algo-1:145:145 [6] NCCL INFO comm 0x55ba309d6e60 rank 6 nranks 8 cudaDev 6 busId 1d0 - Init COMPLETE\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Using cuda_amp half precision backend\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Using cuda_amp half precision backend\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:/opt/conda/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:  warnings.warn(\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:/opt/conda/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  warnings.warn(\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:/opt/conda/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:  warnings.warn(\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stderr>:/opt/conda/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stderr>:  warnings.warn(\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stderr>:/opt/conda/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stderr>:  warnings.warn(\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stderr>:/opt/conda/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stderr>:  warnings.warn(\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stderr>:/opt/conda/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stderr>:  warnings.warn(\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stderr>:/opt/conda/lib/python3.9/site-packages/transformers/optimization.py:306: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stderr>:  warnings.warn(\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:***** Running training *****\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  Num examples = 5169\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  Num Epochs = 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:***** Running training *****\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  Num examples = 5169\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  Num Epochs = 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  Instantaneous batch size per device = 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  Instantaneous batch size per device = 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  Total train batch size (w. parallel, distributed & accumulation) = 8\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  Gradient Accumulation steps = 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  Total train batch size (w. parallel, distributed & accumulation) = 8\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  Gradient Accumulation steps = 1\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  Total optimization steps = 647\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  Total optimization steps = 647\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  Number of trainable parameters = 247577856\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  Number of trainable parameters = 247577856\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  0%|          | 0/647 [00:00<?, ?it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:[2023-07-31 03:01:37.806: W smdistributed/modelparallel/torch/nn/predefined_hooks.py:78] Found unsupported HuggingFace version 4.26.0 for automated tensor parallelism. HuggingFace modules will not be automatically distributed. You can use smp.tp_register_with_module API to register desired modules for tensor parallelism, or directly instantiate an smp.nn.DistributedModule. Supported HuggingFace transformers versions for automated tensor parallelism: ['4.17.0', '4.20.1', '4.21.0']\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:[2023-07-31 03:01:37.806: W smdistributed/modelparallel/torch/nn/predefined_hooks.py:78] Found unsupported HuggingFace version 4.26.0 for automated tensor parallelism. HuggingFace modules will not be automatically distributed. You can use smp.tp_register_with_module API to register desired modules for tensor parallelism, or directly instantiate an smp.nn.DistributedModule. Supported HuggingFace transformers versions for automated tensor parallelism: ['4.17.0', '4.20.1', '4.21.0']\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:[2023-07-31 03:01:37.806: W smdistributed/modelparallel/torch/nn/predefined_hooks.py:78] Found unsupported HuggingFace version 4.26.0 for automated tensor parallelism. HuggingFace modules will not be automatically distributed. You can use smp.tp_register_with_module API to register desired modules for tensor parallelism, or directly instantiate an smp.nn.DistributedModule. Supported HuggingFace transformers versions for automated tensor parallelism: ['4.17.0', '4.20.1', '4.21.0']\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:[2023-07-31 03:01:37.806: W smdistributed/modelparallel/torch/nn/predefined_hooks.py:78] Found unsupported HuggingFace version 4.26.0 for automated tensor parallelism. HuggingFace modules will not be automatically distributed. You can use smp.tp_register_with_module API to register desired modules for tensor parallelism, or directly instantiate an smp.nn.DistributedModule. Supported HuggingFace transformers versions for automated tensor parallelism: ['4.17.0', '4.20.1', '4.21.0']\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:[2023-07-31 03:01:37.807: W smdistributed/modelparallel/torch/nn/predefined_hooks.py:78] Found unsupported HuggingFace version 4.26.0 for automated tensor parallelism. HuggingFace modules will not be automatically distributed. You can use smp.tp_register_with_module API to register desired modules for tensor parallelism, or directly instantiate an smp.nn.DistributedModule. Supported HuggingFace transformers versions for automated tensor parallelism: ['4.17.0', '4.20.1', '4.21.0']\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:[2023-07-31 03:01:37.809: W smdistributed/modelparallel/torch/nn/predefined_hooks.py:78] Found unsupported HuggingFace version 4.26.0 for automated tensor parallelism. HuggingFace modules will not be automatically distributed. You can use smp.tp_register_with_module API to register desired modules for tensor parallelism, or directly instantiate an smp.nn.DistributedModule. Supported HuggingFace transformers versions for automated tensor parallelism: ['4.17.0', '4.20.1', '4.21.0']\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:[2023-07-31 03:01:37.809: W smdistributed/modelparallel/torch/nn/predefined_hooks.py:78] Found unsupported HuggingFace version 4.26.0 for automated tensor parallelism. HuggingFace modules will not be automatically distributed. You can use smp.tp_register_with_module API to register desired modules for tensor parallelism, or directly instantiate an smp.nn.DistributedModule. Supported HuggingFace transformers versions for automated tensor parallelism: ['4.17.0', '4.20.1', '4.21.0']\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stderr>:INFO:root:Using NamedTuple = typing._NamedTuple instead.\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stderr>:INFO:root:Using NamedTuple = typing._NamedTuple instead.\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:INFO:root:Using NamedTuple = typing._NamedTuple instead.\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:INFO:root:Using NamedTuple = typing._NamedTuple instead.\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stderr>:INFO:root:Using NamedTuple = typing._NamedTuple instead.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stderr>:INFO:root:Using NamedTuple = typing._NamedTuple instead.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:INFO:root:Using NamedTuple = typing._NamedTuple instead.\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:[2023-07-31 03:01:37.817: W smdistributed/modelparallel/torch/nn/predefined_hooks.py:78] Found unsupported HuggingFace version 4.26.0 for automated tensor parallelism. HuggingFace modules will not be automatically distributed. You can use smp.tp_register_with_module API to register desired modules for tensor parallelism, or directly instantiate an smp.nn.DistributedModule. Supported HuggingFace transformers versions for automated tensor parallelism: ['4.17.0', '4.20.1', '4.21.0']\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stderr>:INFO:root:Using NamedTuple = typing._NamedTuple instead.\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:[2023-07-31 03:01:37.848 algo-1:146 INFO utils.py:28] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:[2023-07-31 03:01:37.848 algo-1:139 INFO utils.py:28] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:[2023-07-31 03:01:37.848 algo-1:130 INFO utils.py:28] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:[2023-07-31 03:01:37.849 algo-1:133 INFO utils.py:28] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:[2023-07-31 03:01:37.849 algo-1:143 INFO utils.py:28] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:[2023-07-31 03:01:37.852 algo-1:598 INFO utils.py:28] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:[2023-07-31 03:01:37.852 algo-1:141 INFO utils.py:28] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:[2023-07-31 03:01:37.864 algo-1:145 INFO utils.py:28] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:133 INFO profiler_config_parser.py:111] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:143 INFO profiler_config_parser.py:111] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:598 INFO profiler_config_parser.py:111] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:130 INFO profiler_config_parser.py:111] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:146 INFO profiler_config_parser.py:111] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:139 INFO profiler_config_parser.py:111] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:141 INFO profiler_config_parser.py:111] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:130 INFO json_config.py:92] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:146 INFO json_config.py:92] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:139 INFO json_config.py:92] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:133 INFO json_config.py:92] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:141 INFO json_config.py:92] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:[2023-07-31 03:01:37.896 algo-1:598 INFO json_config.py:92] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:143 INFO json_config.py:92] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:130 INFO hook.py:206] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:146 INFO hook.py:206] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:133 INFO hook.py:206] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:139 INFO hook.py:206] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:141 INFO hook.py:206] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:598 INFO hook.py:206] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:143 INFO hook.py:206] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:133 INFO hook.py:259] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:146 INFO hook.py:259] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:130 INFO hook.py:259] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:146 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:130 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:139 INFO hook.py:259] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:133 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:141 INFO hook.py:259] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:[2023-07-31 03:01:37.897 algo-1:598 INFO hook.py:259] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:[2023-07-31 03:01:37.898 algo-1:139 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:[2023-07-31 03:01:37.898 algo-1:141 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:[2023-07-31 03:01:37.898 algo-1:598 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:[2023-07-31 03:01:37.898 algo-1:143 INFO hook.py:259] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:[2023-07-31 03:01:37.898 algo-1:143 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stderr>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stdout>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stdout>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stderr>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stdout>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stderr>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stdout>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stdout>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stdout>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stderr>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:[2023-07-31 03:01:37.912 algo-1:145 INFO profiler_config_parser.py:111] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:[2023-07-31 03:01:37.913 algo-1:145 INFO json_config.py:92] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:[2023-07-31 03:01:37.913 algo-1:145 INFO hook.py:206] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:[2023-07-31 03:01:37.914 algo-1:145 INFO hook.py:259] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:[2023-07-31 03:01:37.914 algo-1:145 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stderr>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stdout>:You're using a T5TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  0%|          | 1/647 [00:01<14:11,  1.32s/it]\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stderr>:INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stderr>:INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stderr>:INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stderr>:INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stderr>:INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  0%|          | 2/647 [00:01<08:01,  1.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  0%|          | 3/647 [00:01<05:49,  1.84it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  1%|          | 4/647 [00:02<04:47,  2.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  1%|          | 5/647 [00:02<04:13,  2.53it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  1%|          | 6/647 [00:02<03:53,  2.75it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  1%|          | 7/647 [00:03<03:38,  2.92it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  1%|          | 8/647 [00:03<03:29,  3.05it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  1%|▏         | 9/647 [00:03<03:23,  3.14it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  2%|▏         | 10/647 [00:04<03:19,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  2%|▏         | 11/647 [00:04<03:16,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  2%|▏         | 12/647 [00:04<03:14,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  2%|▏         | 13/647 [00:04<03:13,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  2%|▏         | 14/647 [00:05<03:11,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  2%|▏         | 15/647 [00:05<03:10,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  2%|▏         | 16/647 [00:05<03:18,  3.17it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  3%|▎         | 17/647 [00:06<03:16,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  3%|▎         | 18/647 [00:06<03:13,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  3%|▎         | 19/647 [00:06<03:13,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  3%|▎         | 20/647 [00:07<03:11,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  3%|▎         | 21/647 [00:07<03:11,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  3%|▎         | 22/647 [00:07<03:12,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  4%|▎         | 23/647 [00:08<03:11,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  4%|▎         | 24/647 [00:08<03:17,  3.16it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  4%|▍         | 25/647 [00:08<03:13,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  4%|▍         | 26/647 [00:08<03:12,  3.23it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  4%|▍         | 27/647 [00:09<03:11,  3.23it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  4%|▍         | 28/647 [00:09<03:09,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  4%|▍         | 29/647 [00:09<03:09,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  5%|▍         | 30/647 [00:10<03:08,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  5%|▍         | 31/647 [00:10<03:07,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  5%|▍         | 32/647 [00:10<03:06,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  5%|▌         | 33/647 [00:11<03:05,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  5%|▌         | 34/647 [00:11<03:04,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  5%|▌         | 35/647 [00:11<03:04,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  6%|▌         | 36/647 [00:11<02:52,  3.54it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  6%|▌         | 37/647 [00:12<02:56,  3.45it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  6%|▌         | 38/647 [00:12<02:57,  3.42it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  6%|▌         | 39/647 [00:12<03:06,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  6%|▌         | 40/647 [00:13<03:04,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  6%|▋         | 41/647 [00:13<03:03,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  6%|▋         | 42/647 [00:13<03:03,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  7%|▋         | 43/647 [00:14<03:11,  3.15it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  7%|▋         | 44/647 [00:14<03:15,  3.09it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  7%|▋         | 45/647 [00:14<03:19,  3.02it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  7%|▋         | 46/647 [00:15<03:20,  2.99it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  7%|▋         | 47/647 [00:15<03:20,  2.99it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  7%|▋         | 48/647 [00:15<03:14,  3.08it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  8%|▊         | 49/647 [00:16<03:08,  3.17it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  8%|▊         | 50/647 [00:16<03:05,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  8%|▊         | 51/647 [00:16<03:04,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  8%|▊         | 52/647 [00:17<03:02,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  8%|▊         | 53/647 [00:17<03:02,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  8%|▊         | 54/647 [00:17<03:01,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  9%|▊         | 55/647 [00:17<03:00,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  9%|▊         | 56/647 [00:18<03:05,  3.18it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  9%|▉         | 57/647 [00:18<03:04,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  9%|▉         | 58/647 [00:18<03:02,  3.22it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  9%|▉         | 59/647 [00:19<03:01,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  9%|▉         | 60/647 [00:19<03:06,  3.15it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  9%|▉         | 61/647 [00:19<03:10,  3.07it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 10%|▉         | 62/647 [00:20<03:05,  3.15it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 10%|▉         | 63/647 [00:20<03:03,  3.19it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 10%|▉         | 64/647 [00:20<02:59,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 10%|█         | 65/647 [00:21<02:58,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 10%|█         | 66/647 [00:21<02:57,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 10%|█         | 67/647 [00:21<02:57,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 11%|█         | 68/647 [00:21<02:55,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 11%|█         | 69/647 [00:22<02:55,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 11%|█         | 70/647 [00:22<02:54,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 11%|█         | 71/647 [00:22<02:53,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 11%|█         | 72/647 [00:23<02:52,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 11%|█▏        | 73/647 [00:23<02:52,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 11%|█▏        | 74/647 [00:23<02:52,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 12%|█▏        | 75/647 [00:24<02:52,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 12%|█▏        | 76/647 [00:24<02:51,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 12%|█▏        | 77/647 [00:24<02:51,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 12%|█▏        | 78/647 [00:24<02:50,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 12%|█▏        | 79/647 [00:25<02:51,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 12%|█▏        | 80/647 [00:25<02:52,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 13%|█▎        | 81/647 [00:25<02:52,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 13%|█▎        | 82/647 [00:26<02:58,  3.17it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 13%|█▎        | 83/647 [00:26<02:56,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 13%|█▎        | 84/647 [00:26<02:53,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 13%|█▎        | 85/647 [00:27<02:51,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 13%|█▎        | 86/647 [00:27<02:50,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 13%|█▎        | 87/647 [00:27<02:49,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 14%|█▎        | 88/647 [00:28<02:49,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 14%|█▍        | 89/647 [00:28<02:49,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 14%|█▍        | 90/647 [00:28<02:47,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 14%|█▍        | 91/647 [00:28<02:53,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 14%|█▍        | 92/647 [00:29<02:51,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 14%|█▍        | 93/647 [00:29<02:50,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 15%|█▍        | 94/647 [00:29<02:56,  3.13it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 15%|█▍        | 95/647 [00:30<02:52,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 15%|█▍        | 96/647 [00:30<02:50,  3.23it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 15%|█▍        | 97/647 [00:30<02:49,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 15%|█▌        | 98/647 [00:31<02:47,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 15%|█▌        | 99/647 [00:31<02:46,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 15%|█▌        | 100/647 [00:31<02:45,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 16%|█▌        | 101/647 [00:32<02:45,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 16%|█▌        | 102/647 [00:32<02:44,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 16%|█▌        | 103/647 [00:32<02:44,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 16%|█▌        | 104/647 [00:32<02:43,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 16%|█▌        | 105/647 [00:33<02:42,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 16%|█▋        | 106/647 [00:33<02:41,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 17%|█▋        | 107/647 [00:33<02:41,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 17%|█▋        | 108/647 [00:34<02:42,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 17%|█▋        | 109/647 [00:34<02:42,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 17%|█▋        | 110/647 [00:34<02:41,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 17%|█▋        | 111/647 [00:35<02:40,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 17%|█▋        | 112/647 [00:35<02:40,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 17%|█▋        | 113/647 [00:35<02:39,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 18%|█▊        | 114/647 [00:35<02:39,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 18%|█▊        | 115/647 [00:36<02:39,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 18%|█▊        | 116/647 [00:36<02:40,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 18%|█▊        | 117/647 [00:36<02:40,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 18%|█▊        | 118/647 [00:37<02:39,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 18%|█▊        | 119/647 [00:37<02:38,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 19%|█▊        | 120/647 [00:37<02:38,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 19%|█▊        | 121/647 [00:38<02:38,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 19%|█▉        | 122/647 [00:38<02:38,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 19%|█▉        | 123/647 [00:38<02:37,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 19%|█▉        | 124/647 [00:38<02:37,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 19%|█▉        | 125/647 [00:39<02:36,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 19%|█▉        | 126/647 [00:39<02:36,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 20%|█▉        | 127/647 [00:39<02:36,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 20%|█▉        | 128/647 [00:40<02:35,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 20%|█▉        | 129/647 [00:40<02:35,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 20%|██        | 130/647 [00:40<02:26,  3.52it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 20%|██        | 131/647 [00:40<02:29,  3.44it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 20%|██        | 132/647 [00:41<02:30,  3.42it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 21%|██        | 133/647 [00:41<02:31,  3.39it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 21%|██        | 134/647 [00:41<02:31,  3.39it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 21%|██        | 135/647 [00:42<02:37,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 21%|██        | 136/647 [00:42<02:36,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 21%|██        | 137/647 [00:42<02:35,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 21%|██▏       | 138/647 [00:43<02:34,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 21%|██▏       | 139/647 [00:43<02:33,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 22%|██▏       | 140/647 [00:43<02:33,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 22%|██▏       | 141/647 [00:44<02:32,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 22%|██▏       | 142/647 [00:44<02:31,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 22%|██▏       | 143/647 [00:44<02:31,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 22%|██▏       | 144/647 [00:44<02:30,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 22%|██▏       | 145/647 [00:45<02:30,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 23%|██▎       | 146/647 [00:45<02:36,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 23%|██▎       | 147/647 [00:45<02:34,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 23%|██▎       | 148/647 [00:46<02:33,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 23%|██▎       | 149/647 [00:46<02:31,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 23%|██▎       | 150/647 [00:46<02:30,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 23%|██▎       | 151/647 [00:47<02:29,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 23%|██▎       | 152/647 [00:47<02:29,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 24%|██▎       | 153/647 [00:47<02:29,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 24%|██▍       | 154/647 [00:47<02:28,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 24%|██▍       | 155/647 [00:48<02:27,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 24%|██▍       | 156/647 [00:48<02:27,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 24%|██▍       | 157/647 [00:48<02:26,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 24%|██▍       | 158/647 [00:49<02:25,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 25%|██▍       | 159/647 [00:49<02:24,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 25%|██▍       | 160/647 [00:49<02:24,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 25%|██▍       | 161/647 [00:50<02:23,  3.38it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 25%|██▌       | 162/647 [00:50<02:23,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 25%|██▌       | 163/647 [00:50<02:23,  3.38it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 25%|██▌       | 164/647 [00:50<02:23,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 26%|██▌       | 165/647 [00:51<02:27,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 26%|██▌       | 166/647 [00:51<02:26,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 26%|██▌       | 167/647 [00:51<02:25,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 26%|██▌       | 168/647 [00:52<02:25,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 26%|██▌       | 169/647 [00:52<02:24,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 26%|██▋       | 170/647 [00:52<02:24,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 26%|██▋       | 171/647 [00:53<02:30,  3.16it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 27%|██▋       | 172/647 [00:53<02:27,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 27%|██▋       | 173/647 [00:53<02:26,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 27%|██▋       | 174/647 [00:54<02:24,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 27%|██▋       | 175/647 [00:54<02:23,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 27%|██▋       | 176/647 [00:54<02:22,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 27%|██▋       | 177/647 [00:54<02:21,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 28%|██▊       | 178/647 [00:55<02:20,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 28%|██▊       | 179/647 [00:55<02:20,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 28%|██▊       | 180/647 [00:55<02:19,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 28%|██▊       | 181/647 [00:56<02:19,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 28%|██▊       | 182/647 [00:56<02:19,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 28%|██▊       | 183/647 [00:56<02:19,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 28%|██▊       | 184/647 [00:57<02:18,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 29%|██▊       | 185/647 [00:57<02:16,  3.40it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 29%|██▊       | 186/647 [00:57<02:16,  3.38it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 29%|██▉       | 187/647 [00:57<02:16,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 29%|██▉       | 188/647 [00:58<02:16,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 29%|██▉       | 189/647 [00:58<02:16,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 29%|██▉       | 190/647 [00:58<02:16,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 30%|██▉       | 191/647 [00:59<02:16,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 30%|██▉       | 192/647 [00:59<02:16,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 30%|██▉       | 193/647 [00:59<02:19,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 30%|██▉       | 194/647 [01:00<02:18,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 30%|███       | 195/647 [01:00<02:17,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 30%|███       | 196/647 [01:00<02:20,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 30%|███       | 197/647 [01:00<02:18,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 31%|███       | 198/647 [01:01<02:17,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 31%|███       | 199/647 [01:01<02:16,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 31%|███       | 200/647 [01:01<02:17,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 31%|███       | 201/647 [01:02<02:15,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 31%|███       | 202/647 [01:02<02:14,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 31%|███▏      | 203/647 [01:02<02:14,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 32%|███▏      | 204/647 [01:03<02:13,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 32%|███▏      | 205/647 [01:03<02:13,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 32%|███▏      | 206/647 [01:03<02:12,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 32%|███▏      | 207/647 [01:03<02:12,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 32%|███▏      | 208/647 [01:04<02:12,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 32%|███▏      | 209/647 [01:04<02:11,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 32%|███▏      | 210/647 [01:04<02:12,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 33%|███▎      | 211/647 [01:05<02:11,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 33%|███▎      | 212/647 [01:05<02:12,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 33%|███▎      | 213/647 [01:05<02:10,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 33%|███▎      | 214/647 [01:06<02:10,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 33%|███▎      | 215/647 [01:06<02:09,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 33%|███▎      | 216/647 [01:06<02:09,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 34%|███▎      | 217/647 [01:06<02:08,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 34%|███▎      | 218/647 [01:07<02:08,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 34%|███▍      | 219/647 [01:07<02:07,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 34%|███▍      | 220/647 [01:07<02:06,  3.38it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 34%|███▍      | 221/647 [01:08<02:12,  3.22it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 34%|███▍      | 222/647 [01:08<02:09,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 34%|███▍      | 223/647 [01:08<02:08,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 35%|███▍      | 224/647 [01:09<02:07,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 35%|███▍      | 225/647 [01:09<02:11,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 35%|███▍      | 226/647 [01:09<02:10,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 35%|███▌      | 227/647 [01:10<02:08,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 35%|███▌      | 228/647 [01:10<02:07,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 35%|███▌      | 229/647 [01:10<02:06,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 36%|███▌      | 230/647 [01:10<02:04,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 36%|███▌      | 231/647 [01:11<02:04,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 36%|███▌      | 232/647 [01:11<02:04,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 36%|███▌      | 233/647 [01:11<02:03,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 36%|███▌      | 234/647 [01:12<02:03,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 36%|███▋      | 235/647 [01:12<02:03,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 36%|███▋      | 236/647 [01:12<02:03,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 37%|███▋      | 237/647 [01:13<02:03,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 37%|███▋      | 238/647 [01:13<02:03,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 37%|███▋      | 239/647 [01:13<02:03,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 37%|███▋      | 240/647 [01:13<02:03,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 37%|███▋      | 241/647 [01:14<02:03,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 37%|███▋      | 242/647 [01:14<02:02,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 38%|███▊      | 243/647 [01:14<02:02,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 38%|███▊      | 244/647 [01:15<02:00,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 38%|███▊      | 245/647 [01:15<02:01,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 38%|███▊      | 246/647 [01:15<02:00,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 38%|███▊      | 247/647 [01:16<02:02,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 38%|███▊      | 248/647 [01:16<02:00,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 38%|███▊      | 249/647 [01:16<02:04,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 39%|███▊      | 250/647 [01:16<02:02,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 39%|███▉      | 251/647 [01:17<02:02,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 39%|███▉      | 252/647 [01:17<02:01,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 39%|███▉      | 253/647 [01:17<01:59,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 39%|███▉      | 254/647 [01:18<01:58,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 39%|███▉      | 255/647 [01:18<01:58,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 40%|███▉      | 256/647 [01:18<01:57,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 40%|███▉      | 257/647 [01:19<01:57,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 40%|███▉      | 258/647 [01:19<01:57,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 40%|████      | 259/647 [01:19<01:57,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 40%|████      | 260/647 [01:20<01:57,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 40%|████      | 261/647 [01:20<01:55,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 40%|████      | 262/647 [01:20<01:55,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 41%|████      | 263/647 [01:20<01:53,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 41%|████      | 264/647 [01:21<01:53,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 41%|████      | 265/647 [01:21<01:54,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 41%|████      | 266/647 [01:21<01:53,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 41%|████▏     | 267/647 [01:22<01:53,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 41%|████▏     | 268/647 [01:22<01:53,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 42%|████▏     | 269/647 [01:22<01:53,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 42%|████▏     | 270/647 [01:23<01:54,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 42%|████▏     | 271/647 [01:23<01:53,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 42%|████▏     | 272/647 [01:23<01:52,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 42%|████▏     | 273/647 [01:23<01:52,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 42%|████▏     | 274/647 [01:24<01:51,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 43%|████▎     | 275/647 [01:24<01:52,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 43%|████▎     | 276/647 [01:24<01:51,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 43%|████▎     | 277/647 [01:25<01:52,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 43%|████▎     | 278/647 [01:25<01:54,  3.22it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 43%|████▎     | 279/647 [01:25<01:53,  3.23it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 43%|████▎     | 280/647 [01:26<01:56,  3.16it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 43%|████▎     | 281/647 [01:26<01:57,  3.11it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 44%|████▎     | 282/647 [01:26<01:56,  3.13it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 44%|████▎     | 283/647 [01:27<01:54,  3.18it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 44%|████▍     | 284/647 [01:27<01:53,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 44%|████▍     | 285/647 [01:27<01:51,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 44%|████▍     | 286/647 [01:27<01:49,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 44%|████▍     | 287/647 [01:28<01:48,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 45%|████▍     | 288/647 [01:28<01:48,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 45%|████▍     | 289/647 [01:28<01:48,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 45%|████▍     | 290/647 [01:29<01:47,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 45%|████▍     | 291/647 [01:29<01:47,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 45%|████▌     | 292/647 [01:29<01:46,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 45%|████▌     | 293/647 [01:30<01:46,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 45%|████▌     | 294/647 [01:30<01:45,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 46%|████▌     | 295/647 [01:30<01:45,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 46%|████▌     | 296/647 [01:30<01:45,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 46%|████▌     | 297/647 [01:31<01:44,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 46%|████▌     | 298/647 [01:31<01:45,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 46%|████▌     | 299/647 [01:31<01:44,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 46%|████▋     | 300/647 [01:32<01:44,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 47%|████▋     | 301/647 [01:32<01:43,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 47%|████▋     | 302/647 [01:32<01:43,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 47%|████▋     | 303/647 [01:33<01:42,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 47%|████▋     | 304/647 [01:33<01:42,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 47%|████▋     | 305/647 [01:33<01:41,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 47%|████▋     | 306/647 [01:33<01:41,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 47%|████▋     | 307/647 [01:34<01:41,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 48%|████▊     | 308/647 [01:34<01:41,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 48%|████▊     | 309/647 [01:34<01:40,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 48%|████▊     | 310/647 [01:35<01:40,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 48%|████▊     | 311/647 [01:35<01:40,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 48%|████▊     | 312/647 [01:35<01:39,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 48%|████▊     | 313/647 [01:36<01:39,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 49%|████▊     | 314/647 [01:36<01:39,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 49%|████▊     | 315/647 [01:36<01:39,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 49%|████▉     | 316/647 [01:36<01:39,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 49%|████▉     | 317/647 [01:37<01:39,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 49%|████▉     | 318/647 [01:37<01:39,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 49%|████▉     | 319/647 [01:37<01:39,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 49%|████▉     | 320/647 [01:38<01:38,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 50%|████▉     | 321/647 [01:38<01:38,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 50%|████▉     | 322/647 [01:38<01:37,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 50%|████▉     | 323/647 [01:39<01:37,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 50%|█████     | 324/647 [01:39<01:36,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 50%|█████     | 325/647 [01:39<01:35,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 50%|█████     | 326/647 [01:39<01:37,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 51%|█████     | 327/647 [01:40<01:37,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 51%|█████     | 328/647 [01:40<01:36,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 51%|█████     | 329/647 [01:40<01:36,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 51%|█████     | 330/647 [01:41<01:35,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 51%|█████     | 331/647 [01:41<01:34,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 51%|█████▏    | 332/647 [01:41<01:34,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 51%|█████▏    | 333/647 [01:42<01:33,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 52%|█████▏    | 334/647 [01:42<01:33,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 52%|█████▏    | 335/647 [01:42<01:33,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 52%|█████▏    | 336/647 [01:42<01:32,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 52%|█████▏    | 337/647 [01:43<01:32,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 52%|█████▏    | 338/647 [01:43<01:32,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 52%|█████▏    | 339/647 [01:43<01:32,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 53%|█████▎    | 340/647 [01:44<01:31,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 53%|█████▎    | 341/647 [01:44<01:31,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 53%|█████▎    | 342/647 [01:44<01:31,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 53%|█████▎    | 343/647 [01:45<01:31,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 53%|█████▎    | 344/647 [01:45<01:31,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 53%|█████▎    | 345/647 [01:45<01:30,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 53%|█████▎    | 346/647 [01:45<01:34,  3.19it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 54%|█████▎    | 347/647 [01:46<01:33,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 54%|█████▍    | 348/647 [01:46<01:32,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 54%|█████▍    | 349/647 [01:46<01:31,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 54%|█████▍    | 350/647 [01:47<01:30,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 54%|█████▍    | 351/647 [01:47<01:30,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 54%|█████▍    | 352/647 [01:47<01:29,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 55%|█████▍    | 353/647 [01:48<01:30,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 55%|█████▍    | 354/647 [01:48<01:30,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 55%|█████▍    | 355/647 [01:48<01:29,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 55%|█████▌    | 356/647 [01:49<01:29,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 55%|█████▌    | 357/647 [01:49<01:28,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 55%|█████▌    | 358/647 [01:49<01:28,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 55%|█████▌    | 359/647 [01:49<01:27,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 56%|█████▌    | 360/647 [01:50<01:31,  3.15it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 56%|█████▌    | 361/647 [01:50<01:29,  3.19it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 56%|█████▌    | 362/647 [01:50<01:30,  3.14it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 56%|█████▌    | 363/647 [01:51<01:29,  3.19it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 56%|█████▋    | 364/647 [01:51<01:28,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 56%|█████▋    | 365/647 [01:51<01:30,  3.13it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 57%|█████▋    | 366/647 [01:52<01:27,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 57%|█████▋    | 367/647 [01:52<01:25,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 57%|█████▋    | 368/647 [01:52<01:24,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 57%|█████▋    | 369/647 [01:53<01:24,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 57%|█████▋    | 370/647 [01:53<01:20,  3.44it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 57%|█████▋    | 371/647 [01:53<01:20,  3.41it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 57%|█████▋    | 372/647 [01:53<01:21,  3.38it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 58%|█████▊    | 373/647 [01:54<01:21,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 58%|█████▊    | 374/647 [01:54<01:21,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 58%|█████▊    | 375/647 [01:54<01:21,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 58%|█████▊    | 376/647 [01:55<01:20,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 58%|█████▊    | 377/647 [01:55<01:20,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 58%|█████▊    | 378/647 [01:55<01:20,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 59%|█████▊    | 379/647 [01:56<01:20,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 59%|█████▊    | 380/647 [01:56<01:19,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 59%|█████▉    | 381/647 [01:56<01:19,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 59%|█████▉    | 382/647 [01:56<01:18,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 59%|█████▉    | 383/647 [01:57<01:18,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 59%|█████▉    | 384/647 [01:57<01:21,  3.22it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 60%|█████▉    | 385/647 [01:57<01:20,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 60%|█████▉    | 386/647 [01:58<01:19,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 60%|█████▉    | 387/647 [01:58<01:18,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 60%|█████▉    | 388/647 [01:58<01:17,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 60%|██████    | 389/647 [01:59<01:16,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 60%|██████    | 390/647 [01:59<01:16,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 60%|██████    | 391/647 [01:59<01:17,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 61%|██████    | 392/647 [01:59<01:16,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 61%|██████    | 393/647 [02:00<01:16,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 61%|██████    | 394/647 [02:00<01:15,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 61%|██████    | 395/647 [02:00<01:14,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 61%|██████    | 396/647 [02:01<01:15,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 61%|██████▏   | 397/647 [02:01<01:15,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 62%|██████▏   | 398/647 [02:01<01:15,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 62%|██████▏   | 399/647 [02:02<01:15,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 62%|██████▏   | 400/647 [02:02<01:14,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 62%|██████▏   | 401/647 [02:02<01:13,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 62%|██████▏   | 402/647 [02:02<01:13,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 62%|██████▏   | 403/647 [02:03<01:12,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 62%|██████▏   | 404/647 [02:03<01:13,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 63%|██████▎   | 405/647 [02:03<01:13,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 63%|██████▎   | 406/647 [02:04<01:13,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 63%|██████▎   | 407/647 [02:04<01:12,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 63%|██████▎   | 408/647 [02:04<01:12,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 63%|██████▎   | 409/647 [02:05<01:12,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 63%|██████▎   | 410/647 [02:05<01:12,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 64%|██████▎   | 411/647 [02:05<01:11,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 64%|██████▎   | 412/647 [02:05<01:10,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 64%|██████▍   | 413/647 [02:06<01:10,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 64%|██████▍   | 414/647 [02:06<01:10,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 64%|██████▍   | 415/647 [02:06<01:10,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 64%|██████▍   | 416/647 [02:07<01:12,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 64%|██████▍   | 417/647 [02:07<01:08,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 65%|██████▍   | 418/647 [02:07<01:08,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 65%|██████▍   | 419/647 [02:08<01:07,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 65%|██████▍   | 420/647 [02:08<01:07,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 65%|██████▌   | 421/647 [02:08<01:07,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 65%|██████▌   | 422/647 [02:08<01:06,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 65%|██████▌   | 423/647 [02:09<01:06,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 66%|██████▌   | 424/647 [02:09<01:06,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 66%|██████▌   | 425/647 [02:09<01:06,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 66%|██████▌   | 426/647 [02:10<01:05,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 66%|██████▌   | 427/647 [02:10<01:05,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 66%|██████▌   | 428/647 [02:10<01:05,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 66%|██████▋   | 429/647 [02:11<01:05,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 66%|██████▋   | 430/647 [02:11<01:04,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 67%|██████▋   | 431/647 [02:11<01:05,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 67%|██████▋   | 432/647 [02:11<01:04,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 67%|██████▋   | 433/647 [02:12<01:04,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 67%|██████▋   | 434/647 [02:12<01:03,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 67%|██████▋   | 435/647 [02:12<01:03,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 67%|██████▋   | 436/647 [02:13<01:03,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 68%|██████▊   | 437/647 [02:13<01:02,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 68%|██████▊   | 438/647 [02:13<01:02,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 68%|██████▊   | 439/647 [02:14<01:02,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 68%|██████▊   | 440/647 [02:14<01:01,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 68%|██████▊   | 441/647 [02:14<01:01,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 68%|██████▊   | 442/647 [02:14<01:01,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 68%|██████▊   | 443/647 [02:15<01:00,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 69%|██████▊   | 444/647 [02:15<01:00,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 69%|██████▉   | 445/647 [02:15<01:00,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 69%|██████▉   | 446/647 [02:16<01:00,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 69%|██████▉   | 447/647 [02:16<01:01,  3.23it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 69%|██████▉   | 448/647 [02:16<01:01,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 69%|██████▉   | 449/647 [02:17<01:00,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 70%|██████▉   | 450/647 [02:17<01:00,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 70%|██████▉   | 451/647 [02:17<00:58,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 70%|██████▉   | 452/647 [02:17<00:58,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 70%|███████   | 453/647 [02:18<00:58,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 70%|███████   | 454/647 [02:18<00:58,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 70%|███████   | 455/647 [02:18<00:58,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 70%|███████   | 456/647 [02:19<00:57,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 71%|███████   | 457/647 [02:19<00:56,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 71%|███████   | 458/647 [02:19<00:56,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 71%|███████   | 459/647 [02:20<00:56,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 71%|███████   | 460/647 [02:20<00:56,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 71%|███████▏  | 461/647 [02:20<00:55,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 71%|███████▏  | 462/647 [02:20<00:55,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 72%|███████▏  | 463/647 [02:21<00:55,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 72%|███████▏  | 464/647 [02:21<00:54,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 72%|███████▏  | 465/647 [02:21<00:54,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 72%|███████▏  | 466/647 [02:22<00:56,  3.23it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 72%|███████▏  | 467/647 [02:22<00:55,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 72%|███████▏  | 468/647 [02:22<00:54,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 72%|███████▏  | 469/647 [02:23<00:53,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 73%|███████▎  | 470/647 [02:23<00:53,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 73%|███████▎  | 471/647 [02:23<00:52,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 73%|███████▎  | 472/647 [02:23<00:52,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 73%|███████▎  | 473/647 [02:24<00:52,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 73%|███████▎  | 474/647 [02:24<00:51,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 73%|███████▎  | 475/647 [02:24<00:52,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 74%|███████▎  | 476/647 [02:25<00:53,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 74%|███████▎  | 477/647 [02:25<00:52,  3.22it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 74%|███████▍  | 478/647 [02:25<00:52,  3.23it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 74%|███████▍  | 479/647 [02:26<00:51,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 74%|███████▍  | 480/647 [02:26<00:50,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 74%|███████▍  | 481/647 [02:26<00:50,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 74%|███████▍  | 482/647 [02:27<00:50,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 75%|███████▍  | 483/647 [02:27<00:50,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 75%|███████▍  | 484/647 [02:27<00:49,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 75%|███████▍  | 485/647 [02:27<00:49,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 75%|███████▌  | 486/647 [02:28<00:49,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 75%|███████▌  | 487/647 [02:28<00:49,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 75%|███████▌  | 488/647 [02:28<00:48,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 76%|███████▌  | 489/647 [02:29<00:48,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 76%|███████▌  | 490/647 [02:29<00:47,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 76%|███████▌  | 491/647 [02:29<00:47,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 76%|███████▌  | 492/647 [02:30<00:46,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 76%|███████▌  | 493/647 [02:30<00:46,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 76%|███████▋  | 494/647 [02:30<00:46,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 77%|███████▋  | 495/647 [02:31<00:45,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 77%|███████▋  | 496/647 [02:31<00:45,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 77%|███████▋  | 497/647 [02:31<00:44,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 77%|███████▋  | 498/647 [02:31<00:44,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 77%|███████▋  | 499/647 [02:32<00:46,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 77%|███████▋  | 500/647 [02:32<00:45,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:{'loss': 0.0, 'learning_rate': 2.418856259659969e-05, 'epoch': 0.77}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 77%|███████▋  | 500/647 [02:32<00:45,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 77%|███████▋  | 501/647 [02:32<00:45,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 78%|███████▊  | 502/647 [02:33<00:44,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 78%|███████▊  | 503/647 [02:33<00:44,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 78%|███████▊  | 504/647 [02:33<00:43,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 78%|███████▊  | 505/647 [02:34<00:43,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 78%|███████▊  | 506/647 [02:34<00:43,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 78%|███████▊  | 507/647 [02:34<00:43,  3.22it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 79%|███████▊  | 508/647 [02:35<00:43,  3.23it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 79%|███████▊  | 509/647 [02:35<00:42,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 79%|███████▉  | 510/647 [02:35<00:41,  3.26it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 79%|███████▉  | 511/647 [02:35<00:41,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 79%|███████▉  | 512/647 [02:36<00:40,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 79%|███████▉  | 513/647 [02:36<00:40,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 79%|███████▉  | 514/647 [02:36<00:39,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 80%|███████▉  | 515/647 [02:37<00:39,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 80%|███████▉  | 516/647 [02:37<00:40,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 80%|███████▉  | 517/647 [02:37<00:40,  3.17it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 80%|████████  | 518/647 [02:38<00:40,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 80%|████████  | 519/647 [02:38<00:39,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 80%|████████  | 520/647 [02:38<00:38,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 81%|████████  | 521/647 [02:38<00:38,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 81%|████████  | 522/647 [02:39<00:37,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 81%|████████  | 523/647 [02:39<00:37,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 81%|████████  | 524/647 [02:39<00:37,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 81%|████████  | 525/647 [02:40<00:36,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 81%|████████▏ | 526/647 [02:40<00:36,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 81%|████████▏ | 527/647 [02:40<00:35,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 82%|████████▏ | 528/647 [02:41<00:35,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 82%|████████▏ | 529/647 [02:41<00:35,  3.36it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 82%|████████▏ | 530/647 [02:41<00:34,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 82%|████████▏ | 531/647 [02:41<00:34,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 82%|████████▏ | 532/647 [02:42<00:34,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 82%|████████▏ | 533/647 [02:42<00:33,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 83%|████████▎ | 534/647 [02:42<00:34,  3.27it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 83%|████████▎ | 535/647 [02:43<00:34,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 83%|████████▎ | 536/647 [02:43<00:33,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 83%|████████▎ | 537/647 [02:43<00:33,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 83%|████████▎ | 538/647 [02:44<00:32,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 83%|████████▎ | 539/647 [02:44<00:32,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 83%|████████▎ | 540/647 [02:44<00:32,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 84%|████████▎ | 541/647 [02:44<00:31,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 84%|████████▍ | 542/647 [02:45<00:31,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 84%|████████▍ | 543/647 [02:45<00:31,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 84%|████████▍ | 544/647 [02:45<00:31,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 84%|████████▍ | 545/647 [02:46<00:30,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 84%|████████▍ | 546/647 [02:46<00:30,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 85%|████████▍ | 547/647 [02:46<00:29,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 85%|████████▍ | 548/647 [02:47<00:29,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 85%|████████▍ | 549/647 [02:47<00:29,  3.34it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 85%|████████▌ | 550/647 [02:47<00:29,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 85%|████████▌ | 551/647 [02:47<00:28,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 85%|████████▌ | 552/647 [02:48<00:28,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 85%|████████▌ | 553/647 [02:48<00:28,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 86%|████████▌ | 554/647 [02:48<00:28,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 86%|████████▌ | 555/647 [02:49<00:27,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 86%|████████▌ | 556/647 [02:49<00:27,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 86%|████████▌ | 557/647 [02:49<00:27,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 86%|████████▌ | 558/647 [02:50<00:26,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 86%|████████▋ | 559/647 [02:50<00:26,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 87%|████████▋ | 560/647 [02:50<00:26,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 87%|████████▋ | 561/647 [02:51<00:26,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 87%|████████▋ | 562/647 [02:51<00:25,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 87%|████████▋ | 563/647 [02:51<00:26,  3.20it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 87%|████████▋ | 564/647 [02:51<00:25,  3.22it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 87%|████████▋ | 565/647 [02:52<00:26,  3.13it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 87%|████████▋ | 566/647 [02:52<00:25,  3.17it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 88%|████████▊ | 567/647 [02:52<00:24,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 88%|████████▊ | 568/647 [02:53<00:24,  3.19it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 88%|████████▊ | 569/647 [02:53<00:24,  3.21it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 88%|████████▊ | 570/647 [02:53<00:23,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 88%|████████▊ | 571/647 [02:54<00:23,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 88%|████████▊ | 572/647 [02:54<00:22,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 89%|████████▊ | 573/647 [02:54<00:22,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 89%|████████▊ | 574/647 [02:55<00:22,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 89%|████████▉ | 575/647 [02:55<00:21,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 89%|████████▉ | 576/647 [02:55<00:21,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 89%|████████▉ | 577/647 [02:55<00:21,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 89%|████████▉ | 578/647 [02:56<00:20,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 89%|████████▉ | 579/647 [02:56<00:20,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 90%|████████▉ | 580/647 [02:56<00:20,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 90%|████████▉ | 581/647 [02:57<00:19,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 90%|████████▉ | 582/647 [02:57<00:19,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 90%|█████████ | 583/647 [02:57<00:18,  3.37it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 90%|█████████ | 584/647 [02:58<00:18,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 90%|█████████ | 585/647 [02:58<00:19,  3.23it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 91%|█████████ | 586/647 [02:58<00:18,  3.24it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 91%|█████████ | 587/647 [02:58<00:18,  3.25it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 91%|█████████ | 588/647 [02:59<00:17,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 91%|█████████ | 589/647 [02:59<00:17,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 91%|█████████ | 590/647 [02:59<00:17,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 91%|█████████▏| 591/647 [03:00<00:16,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 91%|█████████▏| 592/647 [03:00<00:16,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 92%|█████████▏| 593/647 [03:00<00:16,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 92%|█████████▏| 594/647 [03:01<00:15,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 92%|█████████▏| 595/647 [03:01<00:15,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 92%|█████████▏| 596/647 [03:01<00:15,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 92%|█████████▏| 597/647 [03:01<00:15,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 92%|█████████▏| 598/647 [03:02<00:14,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 93%|█████████▎| 599/647 [03:02<00:14,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 93%|█████████▎| 600/647 [03:02<00:13,  3.52it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 93%|█████████▎| 601/647 [03:03<00:13,  3.45it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 93%|█████████▎| 602/647 [03:03<00:13,  3.39it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 93%|█████████▎| 603/647 [03:03<00:13,  3.35it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 93%|█████████▎| 604/647 [03:04<00:12,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 94%|█████████▎| 605/647 [03:04<00:12,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 94%|█████████▎| 606/647 [03:04<00:12,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 94%|█████████▍| 607/647 [03:04<00:12,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 94%|█████████▍| 608/647 [03:05<00:11,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 94%|█████████▍| 609/647 [03:05<00:11,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 94%|█████████▍| 610/647 [03:05<00:11,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 94%|█████████▍| 611/647 [03:06<00:10,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 95%|█████████▍| 612/647 [03:06<00:10,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 95%|█████████▍| 613/647 [03:06<00:10,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 95%|█████████▍| 614/647 [03:07<00:10,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 95%|█████████▌| 615/647 [03:07<00:09,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 95%|█████████▌| 616/647 [03:07<00:09,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 95%|█████████▌| 617/647 [03:08<00:09,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 96%|█████████▌| 618/647 [03:08<00:08,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 96%|█████████▌| 619/647 [03:08<00:08,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 96%|█████████▌| 620/647 [03:08<00:08,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 96%|█████████▌| 621/647 [03:09<00:07,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 96%|█████████▌| 622/647 [03:09<00:07,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 96%|█████████▋| 623/647 [03:09<00:07,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 96%|█████████▋| 624/647 [03:10<00:07,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 97%|█████████▋| 625/647 [03:10<00:06,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 97%|█████████▋| 626/647 [03:10<00:06,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 97%|█████████▋| 627/647 [03:11<00:06,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 97%|█████████▋| 628/647 [03:11<00:05,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 97%|█████████▋| 629/647 [03:11<00:05,  3.33it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 97%|█████████▋| 630/647 [03:11<00:05,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 98%|█████████▊| 631/647 [03:12<00:04,  3.32it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 98%|█████████▊| 632/647 [03:12<00:04,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 98%|█████████▊| 633/647 [03:12<00:04,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 98%|█████████▊| 634/647 [03:13<00:03,  3.31it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 98%|█████████▊| 635/647 [03:13<00:03,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 98%|█████████▊| 636/647 [03:13<00:03,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 98%|█████████▊| 637/647 [03:14<00:03,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 99%|█████████▊| 638/647 [03:14<00:02,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 99%|█████████▉| 639/647 [03:14<00:02,  3.30it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 99%|█████████▉| 640/647 [03:14<00:02,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 99%|█████████▉| 641/647 [03:15<00:01,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 99%|█████████▉| 642/647 [03:15<00:01,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 99%|█████████▉| 643/647 [03:15<00:01,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015100%|█████████▉| 644/647 [03:16<00:00,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015100%|█████████▉| 645/647 [03:16<00:00,  3.29it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015100%|█████████▉| 646/647 [03:16<00:00,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015100%|██████████| 647/647 [03:17<00:00,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:***** Running Evaluation *****\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  Num examples = 862\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  Num examples = 862\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  Batch size = 3\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  Batch size = 3\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  0%|          | 0/36 [00:00<?, ?it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  6%|▌         | 2/36 [00:01<00:17,  1.93it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015  8%|▊         | 3/36 [00:02<00:24,  1.34it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 11%|█         | 4/36 [00:03<00:27,  1.16it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 14%|█▍        | 5/36 [00:04<00:28,  1.09it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 17%|█▋        | 6/36 [00:05<00:28,  1.06it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 19%|█▉        | 7/36 [00:06<00:28,  1.03it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 22%|██▏       | 8/36 [00:07<00:27,  1.02it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 25%|██▌       | 9/36 [00:08<00:26,  1.01it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 28%|██▊       | 10/36 [00:09<00:25,  1.01it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 31%|███       | 11/36 [00:10<00:24,  1.00it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 33%|███▎      | 12/36 [00:11<00:23,  1.00it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 36%|███▌      | 13/36 [00:12<00:22,  1.00it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 39%|███▉      | 14/36 [00:13<00:22,  1.00s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 42%|████▏     | 15/36 [00:14<00:20,  1.00it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 44%|████▍     | 16/36 [00:15<00:19,  1.00it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 47%|████▋     | 17/36 [00:16<00:18,  1.01it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 50%|█████     | 18/36 [00:17<00:17,  1.01it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 53%|█████▎    | 19/36 [00:18<00:16,  1.00it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 56%|█████▌    | 20/36 [00:19<00:16,  1.00s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 58%|█████▊    | 21/36 [00:20<00:15,  1.00s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 61%|██████    | 22/36 [00:21<00:14,  1.00s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 64%|██████▍   | 23/36 [00:22<00:13,  1.00s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 67%|██████▋   | 24/36 [00:23<00:12,  1.00s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 69%|██████▉   | 25/36 [00:24<00:11,  1.00s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 72%|███████▏  | 26/36 [00:25<00:10,  1.00s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 75%|███████▌  | 27/36 [00:26<00:09,  1.01s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 78%|███████▊  | 28/36 [00:27<00:08,  1.01s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 81%|████████  | 29/36 [00:28<00:07,  1.01s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 83%|████████▎ | 30/36 [00:29<00:06,  1.00s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 86%|████████▌ | 31/36 [00:30<00:05,  1.00s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 89%|████████▉ | 32/36 [00:31<00:04,  1.00s/it]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 92%|█████████▏| 33/36 [00:32<00:02,  1.00it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 94%|█████████▍| 34/36 [00:33<00:01,  1.00it/s]#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015 97%|█████████▋| 35/36 [00:34<00:00,  1.00it/s][1,mpirank:0,algo-1]<stderr>:#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Generate config GenerationConfig {\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"decoder_start_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"eos_token_id\": 1,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"pad_token_id\": 0,\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:  \"transformers_version\": \"4.26.0\"\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015100%|██████████| 36/36 [00:35<00:00,  1.00it/s][1,mpirank:0,algo-1]<stderr>:#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:5,algo-1]<stderr>:INFO:absl:Using default tokenizer.\u001b[0m\n",
      "\u001b[34m[1,mpirank:1,algo-1]<stderr>:INFO:absl:Using default tokenizer.\u001b[0m\n",
      "\u001b[34m[1,mpirank:7,algo-1]<stderr>:INFO:absl:Using default tokenizer.\u001b[0m\n",
      "\u001b[34m[1,mpirank:4,algo-1]<stderr>:INFO:absl:Using default tokenizer.\u001b[0m\n",
      "\u001b[34m[1,mpirank:6,algo-1]<stderr>:INFO:absl:Using default tokenizer.\u001b[0m\n",
      "\u001b[34m[1,mpirank:3,algo-1]<stderr>:INFO:absl:Using default tokenizer.\u001b[0m\n",
      "\u001b[34m[1,mpirank:2,algo-1]<stderr>:INFO:absl:Using default tokenizer.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:INFO:absl:Using default tokenizer.\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015                                                 #015\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015                                               #015#033[A\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:{'eval_loss': nan, 'eval_rouge1': 0.0465, 'eval_rouge2': 0.0033, 'eval_rougeL': 0.0456, 'eval_rougeLsum': 0.0454, 'eval_gen_len': 13.4617, 'eval_runtime': 38.0329, 'eval_samples_per_second': 22.665, 'eval_steps_per_second': 0.947, 'epoch': 1.0}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015100%|██████████| 647/647 [03:55<00:00,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015100%|██████████| 36/36 [00:37<00:00,  1.00it/s]#033[A[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015                                               [1,mpirank:0,algo-1]<stderr>:#033[A[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Training completed. Do not forget to share your model on huggingface.co/models =)\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Training completed. Do not forget to share your model on huggingface.co/models =)\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:{'train_runtime': 235.146, 'train_samples_per_second': 21.982, 'train_steps_per_second': 2.751, 'train_loss': 0.0, 'epoch': 1.0}\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015100%|██████████| 647/647 [03:55<00:00,  3.28it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:#015100%|██████████| 647/647 [03:55<00:00,  2.75it/s]\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Saving model checkpoint to /opt/ml/model\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Saving model checkpoint to /opt/ml/model\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Configuration saved in /opt/ml/model/config.json\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Configuration saved in /opt/ml/model/config.json\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Configuration saved in /opt/ml/model/generation_config.json\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Configuration saved in /opt/ml/model/generation_config.json\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Model weights saved in /opt/ml/model/pytorch_model.bin\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Model weights saved in /opt/ml/model/pytorch_model.bin\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:tokenizer config file saved in /opt/ml/model/tokenizer_config.json\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:tokenizer config file saved in /opt/ml/model/tokenizer_config.json\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Special tokens file saved in /opt/ml/model/special_tokens_map.json\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Special tokens file saved in /opt/ml/model/special_tokens_map.json\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stderr>:Copy vocab file to /opt/ml/model/spiece.model\u001b[0m\n",
      "\u001b[34m[1,mpirank:0,algo-1]<stdout>:Copy vocab file to /opt/ml/model/spiece.model\u001b[0m\n",
      "\u001b[34m2023-07-31 03:05:41,737 sagemaker-training-toolkit INFO     Waiting for the process to finish and give a return code.\u001b[0m\n",
      "\u001b[34m2023-07-31 03:05:41,737 sagemaker-training-toolkit INFO     Done waiting for a return code. Received 0 from exiting process.\u001b[0m\n",
      "\u001b[34m2023-07-31 03:05:41,738 sagemaker-training-toolkit INFO     Begin writing status file from leader node to worker nodes\u001b[0m\n",
      "\u001b[34m2023-07-31 03:06:11,768 sagemaker-training-toolkit INFO     Finished writing status file from leader node to worker nodes\u001b[0m\n",
      "\u001b[34m2023-07-31 03:06:11,768 sagemaker-training-toolkit INFO     Reporting training SUCCESS\u001b[0m\n",
      "\n",
      "2023-07-31 03:06:38 Uploading - Uploading generated training model\n",
      "2023-07-31 03:07:49 Completed - Training job completed\n",
      "Training seconds: 599\n",
      "Billable seconds: 599\n"
     ]
    }
   ],
   "source": [
    "## Now, we will fit the model and start the training job\n",
    "huggingface_estimator.fit({\"train\": train_input_path, \"valid\": valid_input_path})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0db69f23-0801-4313-9939-68c2e2fa6117",
   "metadata": {},
   "source": [
    "## DEPLOY ON SAGEMAKER WITH HUGGING FACE DEEP LEARNING CONTAINER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d206c02a-72bf-4e02-bdbf-f8409159f710",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: huggingface-pytorch-training-2023-07-31-03-10-39-147\n",
      "INFO:sagemaker:Creating endpoint-config with name huggingface-pytorch-training-2023-07-31-03-10-39-147\n",
      "INFO:sagemaker:Creating endpoint with name huggingface-pytorch-training-2023-07-31-03-10-39-147\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------!"
     ]
    }
   ],
   "source": [
    "huggingface_predictor = huggingface_estimator.deploy(\n",
    "    initial_instance_count=1, instance_type=\"ml.g4dn.xlarge\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34bcee66-a781-44e2-9db8-59729b4b207f",
   "metadata": {},
   "source": [
    "### Now, let's use an example to see how a meeting transcript can be summarized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "10a05ed2-b0e7-4015-b0d8-59d3a5dd968a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'inputs': 'summarize: : {\\'meeting_id\\': \\'LongBeachCC_03222022_22-0281\\', \\'source\\': \"Speaker 0: Thank you. We have a first item up is item 17.\\\\nSpeaker 2: Item 17 Communication from Councilwoman Zendaya\\'s Recommendation to direct city attorney to draft resolution to advocate changes to the California retail food code.\\\\nSpeaker 0: There\\'s a motion in a second. I think we have come to consider how do you want to do public comment first or do you want to address, please? There\\'s one member of the public.\\\\nSpeaker 2: Mankind could.\\\\nSpeaker 6: Try again. My name is Cameron Coon and I started a catering service at the worst possible time in January of 2020. In order to survive the pandemic. My partner, Juan Fernandez, and I took our pop up cafe intended for film set, and we became street vendors. During the past year, we\\'ve served a cappuccino to Robert Garcia, a peppermint mocha to Rex Richardson. A Ice Americano to Suzy Price\\'s husband and at least two hot chocolates to Mary\\'s. And each of those beverages, harmless as they were, were prepared and served under the threat of criminality by the city of Long Beach. When the Health Department first made it clear to us that there was no way to permit our coffee cart unless it had a built in NSF certified three compartment. Think we were baffled. What about all the other street vendors? The fruit stand one is fortunate to find in a food desert who often provide children with their only healthy option for an after school snack. The iconic SoCal taco stand opened late on a Saturday night to help sober up the hungry bar crawlers, the mostly posse day laborers who spend long hours out in the elements supporting their families while providing the rest of us with an authentic cultural experience that nourishes us. And vitalize is our cityscapes. We demanded to know How do all of those other vendors get permits? Long Beach doesn\\'t give street vendor permit. The health department can see that they\\'re all illegal. We sometimes do sweeps where we go around the city to issue citations, make arrests and impound their cards. This exchange took place right after the first winter peak of COVID, while an airborne virus was overflowing our hospitals and killing droves of Americans. Most people still weren\\'t vaccinated then and everyone is praying they want to get deathly ill from the long lines at Starbucks. We were serving coffee and tea safely outdoors, being told that there was no realistic or lawful way to continue doing so. Learning about these policies during the peak of the pandemic is why we have been pushing Long Beach to legalize street vendors. By their very nature, they reduce community transmission of COVID. To treat them as criminals has always been wrong. But to do so during a pandemic and in the name of public health as an injustice that we just cannot accept. Thank you to Councilwoman Mary and De Haas for bringing these crucial commonsense reforms to Long Beach, extending the long overdue dignity and legitimacy to our community\\'s valuable and vulnerable offenders. Thank you to State Senator Lena Gonzalez, who recently introduced SB nine seven to legislation that will create a pathway to permitting for vendors throughout California. And thank you to all the city council members who vote today in support of these efforts. Anyone can get involved at k street vendors. Dot org. Thank you for your time. Appreciate it.\\\\nSpeaker 0: Thank you very much. But Councilman and de House.\\\\nSpeaker 1: Thank you, Mayor. This is a very, very personal item. I, I really feel that our street vendors and sidewalk vendors are an integral part of our Latino culture, especially here in the United States, in California, per se. A lot of us can attest to remembering growing up and anxiously, anxiously waiting for that horn to go off by the window to see our favorite alloted or our class battle game. I know I have my favorite, whether it\\'s coffee at the beach or whether it\\'s, you know, and a lot of other spots, though, at the park. It\\'s an 8.4. It\\'s a point right now more than any other time that we create a pathway to permits for street vendors and sidewalk vendors. Our street vendors and sidewalk vendors are hardworking people trying to make a living. In many of the Latino communities, you grow up learning to respect and value your street vendors. And I hope that this report, this item, will help us all not only respect them, but also recognize their importance and provide ways to help them sell their products with dignity. I\\'m hoping that my colleagues can support that, support me and allow this to report to happen. It is very important that we take action now. I know that during this pandemic we have had a lot of violence against some of our street vendors, and that\\'s because they don\\'t feel protected. You know, they won\\'t report when they are violated against because they don\\'t feel protected. So I think that now is the time to find ways to help them. One of the questions that I was going to ask staff of is, I know that there\\'s also in in the requests, there\\'s also a request for a moratorium on citations. And so I think that\\'s a very good thing. I also want to make sure that we leave a little room for those that are doing the bad actor benders. I know that, you know, the majority of the street vendors are good. They have good product, they have clean products. And that\\'s why, you know, their purpose is to keep getting repeatedly customers back. So. So one of the things is like I would like to see like maybe incorporated in there something that where if someone is a bad actor that we can give them some type of, you know, rules or regulations that they have to follow. Of course, maybe making sure that everything is is actually, you know, gone through everything is explored before we go to giving them a citation. Is that something we could do?\\\\nSpeaker 8: Yes. So I think that would be a very good modification. We would like the ability in those instances where we have really public safety, health issues and people who are not complying do still have some enforcement tools. But we certainly understand the sentiment of what the council is trying to achieve here.\\\\nSpeaker 1: Thank you. I do think that that\\'s important. And I do think that, you know, we\\'re on the right path. So with that, I encourage my colleagues to support this item.\\\\nSpeaker 0: Thank you, Councilman Ciro.\\\\nSpeaker 1: You Mayor and thank you council member Sunday has from bringing this item forward. I think that it\\'s timely as we\\'re in this road to recovery for them from the pandemic and our street vendors are part of our economy and they\\'re important part and part of our community and business as well. And I just want to say that I also want to thank Senator Lena Gonzalez for bringing forward this bill, SB nine seven to that council members recommending that we bring forth a resolution to support this. I also want to thank her, as well as Councilmember Yarrawonga and Vice Mayor Richardson, for supporting an item I brought for last time, which was safety and security program for street vendors. And it\\'s also timely at this phase of that will be bringing out more information. But one of it is that the street vendors are eligible applicants for the small business relief grant that\\'s been released as well and other support that nonprofits will be able to give them so that they\\'re able to navigate what other ways they can do to continue to recover as well as protect themselves. And so I\\'m really I think this is all timely so that we can really, in the end of the day, consider how do we support our street vendors so they\\'re part of our economy and that they\\'re able to do business without fear and that they\\'re able to also make sure that they\\'re taking care of their family in a way that is healthy. So I just I think this is timely and thank you for bringing the resolution that the item to support. Thank you.\\\\nSpeaker 0: Thank you, Councilmember Superman.\\\\nSpeaker 8: Thank you, Mayor. So the recommendation, what I\\'ve heard is unclear. So if the city attorney can draft this resolution, he must know far more than I. So I guess I\\'ll direct this to staff. And I heard in particular from the Zakaria Business Improvement District. Now, to be clear, I no longer represent that area. It\\'s mostly in districts three and two at this time. But we have a long history serving for over six years in that area of complaints. So I\\'ll just tell you what they asked me, and that is, does the recommendation pertain only to sellers of food? Does it pretend to sidewalk sellers of cell phones, Dodger shirts, stuffed animals and other tchotchkes as well as food trucks? The recommendation states to adopt a resolution implementing a moratorium on issuing citations to sidewalk vendors for vending related to activities, including for selling goods without otherwise necessary permits. This seems to allow all sidewalk vendors to continue on public sidewalks without business permits, health permits or bid fees. During the pandemic, existing brick and mortar business owners have jumped through expensive hoops to set up parklets for dining and retailers of goods have been prohibited to have parklets to sell their goods. Yet they continue to get billed for city and business improvement district fees, plus now late fees. So that those are the questions from one business improvement district. And my concern would be on the health aspects here, if we can\\'t cite, then do any health regulations apply? Is our health department involved at all, in particular cross-contamination of utensils, handwashing, etc.? So it\\'s a long question, but do have a response from staff.\\\\nSpeaker 3: Councilmember Supernormal start with with the as I understand the requested action and the item as it sits before you at this time is a recommendation for us to draft a resolution which is supporting potential state legislation regarding.\\\\nSpeaker 8: Regarding food vending and.\\\\nSpeaker 3: Food safety. The as I understood the there was also a request or discussion of a possible moratorium on enforcement that would be brought back to the Council after it\\'s crafted for council consideration and adoption at a later date. We couldn\\'t do that this evening. So that is, as I understand, the motion on the floor.\\\\nSpeaker 8: Okay. So if I could just follow up. So so we don\\'t know the parameters of the state legislation at this time. We\\'re not we\\'re not clear on on what that will be. That that I certainly don\\'t have that in front of me, you know? Correct. All right. Thank you.\\\\nSpeaker 0: Vice Mayor Richardson.\\\\nSpeaker 5: Thank you. I wanted to just first lay my support to moving in this general direction. I think when we saw food trucks begin to emerge, it was out of necessity. There were barriers to entry for restaurants and food trucks were more flexible for folks to be able to do business. And, you know, unless necessity is the mother of invention, I think we\\'re seeing a lot of innovative food carts coming up. And I think, you know, it\\'s a pathway into economic progress. And I think it was a Cameron. Cameron, you did that really cool coffee cart at our District nine holiday party. Right? That was incredible. All the things we\\'re able to pack into that cart. It was affordable. People loved it. That was a great experience. And so I think, you know, generally we have to maintain health, obviously, but whatever we can do to change our practices, to remove barriers of entry and let people get into the space where they can start with a cart, maybe to a truck, maybe to a shop. That\\'s the things we want to support. So Council Amazon Day has some I\\'m certainly proud to support this. I think your new you know that\\'s that\\'s motion a bit to say you know you\\'re not going to exclude any citations for any violations you\\'re saying use that as a last resort. Is that correct? It sounded like that\\'s what you where you were going. And I think that makes a lot of sense. Say, hey, be flexible, but if you know someone\\'s doing something that\\'s not healthy or not safe, we still have the ability to enforce our laws. But I think generally the sentiment of where you\\'re going is we need to figure out a more flexible way to allow street vending to exist in a way that\\'s still thoughtful to other businesses and thoughtful to the pedestrian experience and all of those things. But I\\'ll tell you, I mean, I was impressed, Cameron, what you were able to do and you know, I would love to see, you know, this just breed a whole lot of innovation, innovative coffee carts and fruit carts that are you know, we can you know, Long Beach has the coolest carts. I don\\'t know, like it could be a cool environment that we have something that adds to our pedestrian experience. So I just want to offer my thoughts and offer my support for this item. Thank you.\\\\nSpeaker 0: Thank you, Councilman Orson.\\\\nSpeaker 4: I thank you. And I appreciate the item in the discussion. I love my little team and any time I have the opportunity to to see them. But I also will just say that I share some of the the the the confusion, but also many of the sentiments. I think we should be encouraging entrepreneurism. We want to create a pathway to business and we want to make sure that we\\'re being balanced and fair across the board. I know for a fact that when food trucks were introduced into a couple of my business districts and the areas that I represent, it created a little bit of a friction with some of the restaurants. Right. They said we\\'re paying business licensing. We are we\\'re paying high rents. And, you know, I\\'m selling. Pizza and somebody is selling the same product in a truck outside of my business. You know, I think in crafting this and moving forward, we need to look at parameters that that recognize the ability for hopefully everybody to thrive and not to create those those type of conflicts. And so I\\'m really concerned about about that. If we don\\'t manage this in a manner or craft this in a manner moving forward, that that kind of recognizes that. Also, I wanted to just point out in the staff report, Councilmembers and Day has brought forward, it said that certain outdated provisions in state law are impeding the city from achieving the goal of fully integrating sidewalk food vendors into the economy. Do we know what those provisions are? Can we speak to those and any specific stuff? Councilmember.\\\\nSpeaker 8: But I\\'ll defer to the council members in the House to talk about the item. But then I can answer some questions on where we are with street vendors versus food trucks and state law and how the municipal code currently is.\\\\nSpeaker 4: I was curious about the provisions that are outdated that are creating those barriers.\\\\nSpeaker 0: And then what I.\\\\nSpeaker 8: Think that\\'s referring to is a number of things have happened in the state of California related to street vendors. A couple of years ago, a new state law passed that basically said that cities cannot prohibit unreasonably street vendors on their sidewalks. It was really designed with the food in mind, but it\\'s also been unclear, and that\\'s why we get some of those cell phone services and the ones that are, you know, that are out on our streets right now. The city\\'s municipal code does prohibit those things, but we are superseded by state law. So we have essentially suspended those provisions. We don\\'t enforce on those PD is not gone and enforce those in a very long time. But we still have a requirement as a health jurisdiction to enforce state health law so we don\\'t create those regulations . Those are at the state level. And Kelly Up is on the line. If you\\'d like to ask some questions. And so we have essentially pulled back from the PD enforcement of that. But we do still need to go out and make sure people are safe. So I think that some of the the disparity and then we also have our street, our food trucks, which fall under a different section in Long Beach. And those are actually permitted through L.A. County. And we\\'re currently going through two different efforts right now. First, to reform food vendors or I\\'m sorry, the food trucks and that process is underway and we have a consultant helping us. And then we funded an effort that the council just gave us money for to really look comprehensively at street vendors and modernizing our municipal code and trying to make it more streamlined, which this item speaks to.\\\\nSpeaker 4: And so that speaks to my next question. And thank you. What does the current permit processing process look like for street vendors? I mean, and why is there a barrier?\\\\nSpeaker 8: So I ask Kelly Colby to talk about the health aspects of that. And I do not believe they are permitted from a from a business licensing perspective. We don\\'t require business licensing. That\\'s part of what the state does not want us to do. And until we get a new set of municipal code to really talk about the time, place and manner, we we don\\'t have that on the on the books, but we do from a food perspective. So Kelly.\\\\nSpeaker 9: Yes. Good evening, uh, councilmember. So, um, you know, as the city manager has discussed, there\\'s a set of state requirements, their health and safety code that we have to hold all food providers to, and so an applicant can apply for a health permit. It\\'s all laid out. Um, we\\'ve worked on documentation. We spent a lot of time working with our food cart vendors as well to help them through that process. Um, if they\\'re interested, as we\\'ve been working on an education first process first and then they apply for those food permits and then we work with them to bring their cart to code based on what the state requires us to hold them accountable for. And so in that process we go through that and then they actually so food carts can be permitted. There are many permitted food carts there. There are the stickers are placed on the cart itself. Um, and then we work with them from that process. So there is a process in place and all that information is on online.\\\\nSpeaker 4: So I guess I\\'ll wait for the item to come back to determine and understand what the streamline would look like. But I guess the next question is, is there a time frame in terms of what a typical permitting process looks like for a food cart?\\\\nSpeaker 8: I\\'m I\\'m not sure I can answer the time of of like for the individual. But this is a really complicated issue for us. It has been something the council has brought up a couple of times, and it\\'s in our work plan, but it\\'s been pushed back a little bit. COVID really hit us hard, and as we try to figure this out, this is one that crosses like probably four or five different departments. A lot of these are on the public right away that are in front of a business. Some people feel like they should be regulated like a business, but they\\'re not bricks and mortar. There\\'s development services involved. PD gets all the calls because people call in for PD, but it\\'s not really a public safety response. And then there\\'s health. So we have ask you for funding. I believe it was 50 grand or something in the last amount and the last left over the last year and funding to really bring on a consultant that will help us put all that together. And we\\'re adding on to their scope of work because they\\'re doing food vendors right now are food trucks right now. So food trucks is coming to the council probably in the next month or two months we\\'ll get a handle on food trucks and what qualifies as a food truck and where they\\'ll go. And then we plan to tackle this issue and answer a lot of these questions.\\\\nSpeaker 0: Councilman Price.\\\\nSpeaker 1: You and Councilman Austin kind of touched upon it. But this is an issue in the bid specifically, and that\\'s what Councilman Superman pointed out from one of our bid directors. What is the existing law in terms of our ability to. Allow or not allow street vendors.\\\\nSpeaker 8: So generally, I believe we are not allowed to we\\'re not allowed to restrict them unless we have a municipal code in place. And even then, you can basically say this is like your area. You cannot be larger than a certain amount, but you\\'re not able to just outright prohibit them, if that\\'s correct. Mr. City Attorney That\\'s correct.\\\\nSpeaker 3: Councilmember I don\\'t have the the the actual state law in front of me. I can certainly get that and forward it up to your office. But I believe that\\'s correct what the city manager said.\\\\nSpeaker 1: But is it limited? Is it limited to food or can they sell cell phones and other things on the sidewalk?\\\\nSpeaker 8: So we believe the original intent of that legislation was more on the food vendors as what people talked about. But the legislation, it appears to be pretty broad that services are included as well as other things. So that would be some state legislation we would be very supportive of as staff is to help clearly define what they mean by those types of street vendors. Because we when we talk about street vendors, we talk about food and the person selling, you know, the tacos to make a living. But we often see some really aggressive corporations that are basically using our city streets to sell cell phones and insurance and those types of things.\\\\nSpeaker 1: And then so if so, obviously, because at state law, we we need to follow it. But if we enacted our own municipal code, could it be more restrictive than the state law and disallow for certain types of sales?\\\\nSpeaker 8: I think we need to look into that. My my initial impression is, no, you\\'re allowed to, within the state law, basically define how it\\'s done and where it\\'s done, but not actual being more restrictive. But that is those are the types of questions that we\\'re really hoping to get some professional assistance with and pull that report together\\\\nSpeaker 1: . Okay, that\\'s great. And I look forward to that coming back to us so that we know kind of what our options are and that we are doing our best to advocate for the bids who have obviously some concerns regarding bid fees and assessment fees that they have to pay that others might not have to, which as a as a business owner in a bid who has to pay the assessment fee, I understand in terms of the food vendors, I understand that this item tonight asks for a moratorium. But what happens if there is a major health concern, like an unsanitary one particular unsanitary food vendor, for example?\\\\nSpeaker 8: Yeah. So first, you\\'re not changing anything tonight. That request is to come back with a resolution and allow us some time to craft that. I think Councilmember Zendejas addressed that with some change in her motion from what\\'s printed, which is to allow for in those health situations for us to to still have enforcement action, that it\\'s not a blanket moratorium. And you, Kelly, can probably answer questions about specific incidents, but we always try education first. And so this would just be memorializing that, that education, education, education. But if there are clear health and safety issues where we\\'re required by the state code to protect public health.\\\\nSpeaker 1: Okay. Thank you. And and I missed that change in Councilman Van de House motion, so I appreciate that clarification. Well, I do have a question. If we were made aware of a particular food vendor that might have unsanitary conditions, let\\'s just say hypothetically, and we didn\\'t do anything about it and someone were to get food poisoning or sick as a result. Would we be liable? The city.\\\\nSpeaker 3: That. I mean, it depends on the notice and what we knew and when we knew. And yes, but we would certainly need to take action on that. And I think that the Councilmembers Amendment addresses that. Right. I mean, the city needs to adopt and follow the health regulations.\\\\nSpeaker 1: That\\'s great. I appreciate that amendment. I think it\\'s great and prudent. So thank you.\\\\nSpeaker 0: Councilman and de house.\\\\nSpeaker 1: In. Thank you, Mayor. I just wanted to make that clarification that. And this item is requesting for us to do a report on just food. So just wanted to make sure that we did that. And I also wanted to to thank Cameron for for being here today and, yes, for having a spectacular coffee cart. If you haven\\'t seen it, you have to see it. It\\'s just so amazing. We were able to enjoy it at the pike when one evening after caroling through our neighborhoods. And that was really, really awesome. And I want, you know, other street vendors to be able to do that. One of the things that we wanted to do in that we had we had ice cream vendors at our DIA de los Muertos stuff. So we were trying to include them in all of our events. And I think that\\'s very important to be able to have a pathway to permits so that we can actually be able to bring on these small businesses and be able to support them in this way, especially now in our recovery time. I think it\\'s more important than ever to be able to examine and really, really look at any and every opportunity that we can have to help people advance themselves. Of course, doing it in a very healthy and and good way. So thank you. Thank you. And I hope everyone will support this item tonight.\\\\nSpeaker 0: Vice Mayor Richardson.\\\\nSpeaker 5: Just just a just a quick question. Some thoughts as you do this study. So I know that there\\'s a vendor in my neighborhood who always works the same corner. I also know that there are vendors that move citywide. So as we think about it, maybe there could be some either some franchise system or, you know, some who have a broader license to move citywide, to have a higher standard or whatever. Or if you work a certain area, you have to have some relationship to that local business district or whatever. But, you know, as this conversations happened, I\\'m thinking about maybe there may be different tiers of permitting for like, let\\'s say someone has more that more than one car. How do you handle that? Just just some things that I\\'m thinking about as we have this conversation, but thanks a lot of things is very helpful.\\\\nSpeaker 0: Councilmember Hooper now.\\\\nSpeaker 8: Thank you. I would just like to return to the comments of the of Kelly Collopy. It\\'s just not clicking with me. It\\'s been a lot of years since I\\'ve been food safety certified and from the corporate side, not a restaurant operator. But what strikes me is that you can assess a cart, let\\'s say, and give it a grade and improve it. But that\\'s not how health inspections work. They go to the operation. For instance, you can get dinged for improper hand-washing procedures within the restaurant if they don\\'t witness that. Now we\\'re talking about a facility, a cart without a handwashing sink. So I\\'ll tell you, I\\'m mystified. And for that reason, I have a hard time supporting something. I just don\\'t know where this is going. Part an expression. I think we\\'ve got the cart ahead of the horse and we need to straighten this out with health before I could support it. Thank you.\\\\nSpeaker 0: Thank you. There\\'s a motion and a second. I believe we did public comment already on this item. So, members, please go out and cast your votes. The machine\\'s next item is item 18, please.\", \\'type\\': \\'Agenda Item\\', \\'reference\\': \\'Recommendation to direct City Attorney to draft resolution to have the City’s legislative advocates in Sacramento, in collaboration with the City Manager and all relevant City departments, to support legislation to enact changes to the California Retail Food Code that will make it more sensitive to the context of sidewalk food vending while ensuring food safety.\\', \\'city\\': \\'LongBeachCC\\'}'}\n"
     ]
    }
   ],
   "source": [
    "test_transcript = {\"inputs\": f\"{prefix}: {dataset['test'][20]}\"}\n",
    "print(test_transcript)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "34ebc0c9-61e0-43ba-9e24-7f4f6424e1a8",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'generated_text': 'The machine\\'s next item is item 18, please.\"'}]\n"
     ]
    }
   ],
   "source": [
    "prediction = huggingface_predictor.predict(test_transcript)\n",
    "print(prediction)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "717bce45-d640-474a-8752-f2a5475ef0d1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13ba2c25-d3bc-4b53-8ecf-262c7f9e092f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.g4dn.xlarge",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}