Interactive_ANN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5796054b",
   "metadata": {},
   "source": [
    "\n",
    "<p align=\"center\">\n",
    "    <img src=\"https://github.com/GeostatsGuy/GeostatsPy/blob/master/TCG_color_logo.png?raw=true\" width=\"220\" height=\"240\" />\n",
    "\n",
    "</p>\n",
    "\n",
    "### Interactive Simple Machine Learning Artificial Neural Network (ANN)\n",
    "\n",
    "\n",
    "#### Michael Pyrcz, Professor, The University of Texas at Austin \n",
    "\n",
    "##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e41c1943",
   "metadata": {},
   "source": [
    "#### Neural Networks\n",
    "\n",
    "Machine learning method for supervised learning for classification and regression analysis.  Here are some key aspects of support vector machines.\n",
    "\n",
    "**Basic Design** *\"...a computing system made up of a number of simple, highly interconnected processing elements, which process information by their dynamic state response to external inputs.\"* Caudill (1989). \n",
    "\n",
    "**Nature-inspire Computing** based on the neuronal structure in the brain, including many interconnected simple, processing units, known as nodes that are capable of complicated emergent pattern detection due to a large number of nodes and interconnectivity.\n",
    "\n",
    "**Training and Testing** just like and other predictive model (e.g. linear regression, decision trees and support vector machines) we perform training to fit parameters and testing to tune hyperparameters.  Here we observe the error with training and testing datasets, but do not demonstrate tuning of the hyperparameters.  \n",
    "\n",
    "**Parameters** are the weights applied to each connection and a bias term applied to each node.  For a single node in an artificial neural network, this includes the slope terms, $\\beta_i$, and the bias term, $\\beta_{0}$.\n",
    "\n",
    "\\begin{equation}\n",
    "Y = \\sum_{i=1}^m \\beta_i X + \\beta_0\n",
    "\\end{equation}\n",
    "\n",
    "it can be seen that the number of parameters increases rapidly as we increase the number of nodes and the connectivity between the nodes.\n",
    "\n",
    "**Layers** the typical artificial neural net is structured with an **input layer**, with one node for each $m$ predictor feature, $X_1,\\ldots,X_m$. There is an **ouput layer**, with one node for each $r$ response feature, $Y_1,\\ldots,Y_r$.  There may be one or more layers of nodes between the input and output layers, known as **hidden layer(s)**.  \n",
    "\n",
    "**Connections** are the linkages between the nodes in adjacent layers.  For example, in a fully connected artificial neural network, all the input nodes are connected to all of the nodes in the first layer, then all of the nodes in the first layer are connected to the next layer and so forth. Each connection includes a weight parameter as indicated above.\n",
    "\n",
    "**Nodes** receive the weighted signal from the connected previous layer nodes, sum and then apply this result to the **activation** function in the node. Some example activation functions include:\n",
    "\n",
    "* **Binary** the node fires or not.  This is represented by a Heaviside step function.\n",
    "\n",
    "* **Identify** the input is passed to the output $f(x) = x$\n",
    "\n",
    "* **Linear** the node passes a signal that increases linearly with the weighted input.\n",
    "\n",
    "* **Logistic** also known as sigmoid or soft step $f(x) = \\frac{1}{1+e^{-x}}$\n",
    "\n",
    "the node output is the nonlinear activiation function applied to the linearly weighted inputs. This is fed to all nodes in the next layer.\n",
    "\n",
    "**Training Cycles** - the presentation of a batch of data, forward application of the current prediction model to make estimates, calculation of error and then backpropagation of error to correct the artificial neural network parameters to reduce the error over all of the batches.\n",
    "\n",
    "**Batch** is the set of training data for each training cycle of forward prediction and back propagation of error, drawn to train for each iteration. There is a trade-off, a larger batch results in more computational time per iteration, but a more accurate estimate of the error to adjust the weights.  Smaller batches result in a nosier estimate of the error, but faster epochs, this results in faster learning and even possibly more robust models.\n",
    "\n",
    "**Epochs** - is a set of training cycles, batches covering all available training data.  \n",
    "\n",
    "**Local Minimums** - if one calculated the error hypersurface over the range of model parameters it would be hyparabolic, there is a global minimium error solution.  But this error hyper surface is rough and it is possible to be stuck in a local minimum. **Learning Rate** and **Mommentum** coefficients are introduced to avoid getting stuck in local minimums.\n",
    "\n",
    "* **Mommentum** is a hyperparameter to control the use of information from the weight update over the last epoch for consideration in the current epoch.  This can be accomplished with an update vector, $v_i$, a mommentum parameter, $\\alpha$, to calculate the current weight update, $v_{i+1}$ given the new update $\\theta_{i+1}$.\n",
    "\n",
    "\\begin{equation}\n",
    "v_{i+1} = \\alpha v_i + \\theta_{i+1}\n",
    "\\end{equation}\n",
    "\n",
    "* **Learning Rate** is a hyperparameter that controls the adjustment of the weights in response to the gradient indicated by backpropagation of error \n",
    "\n",
    "##### Applications to subsurface modeling\n",
    "\n",
    "We demonstrate the estimation of normal score transformed porosity from depth.  This would be useful for building a vertical trend model. \n",
    "\n",
    "* modeling the complicated relationship between porosity and depth.\n",
    "\n",
    "#### Limitations of Neural Network Estimation\n",
    "\n",
    "Since we demonstrate the use of an artificial neural network to estimate porosity from sparsely sampled data over depth, we should comment on limitations of our artificial neural networks for this estimation problem:\n",
    "\n",
    "* does not honor the well data\n",
    "\n",
    "* does not honor the histogram of the data\n",
    "\n",
    "* does not honor spatial correlation \n",
    "\n",
    "* does not honor the multivariate relationship\n",
    "\n",
    "* generally low interpretability models\n",
    "\n",
    "* requires a large number of data for effective training\n",
    "\n",
    "* high model complexity with high model variance\n",
    "\n",
    "#### Import Required Packages\n",
    "\n",
    "We will also need some standard packages. These should have been installed with Anaconda 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "877ffa62",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.ticker import (MultipleLocator, AutoMinorLocator, AutoLocator) # control of axes ticks\n",
    "plt.rc('axes', axisbelow=True)                            # set axes and grids in the background for all plots\n",
    "from ipywidgets import interactive                        # widgets and interactivity\n",
    "from ipywidgets import widgets                            \n",
    "from ipywidgets import Layout\n",
    "from ipywidgets import Label\n",
    "from ipywidgets import VBox, HBox\n",
    "import math\n",
    "seed = 13"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a27fea49",
   "metadata": {},
   "source": [
    "If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.  \n",
    "\n",
    "#### Declare Functions\n",
    "\n",
    "I just added a convenience function for adding major and minor gridlines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8abbc843",
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_grid():\n",
    "    plt.gca().grid(True, which='major',linewidth = 1.0); plt.gca().grid(True, which='minor',linewidth = 0.2) # add y grids\n",
    "    plt.gca().tick_params(which='major',length=7); plt.gca().tick_params(which='minor', length=4)\n",
    "    plt.gca().xaxis.set_minor_locator(AutoMinorLocator()); plt.gca().yaxis.set_minor_locator(AutoMinorLocator()) # turn on minor ticks   \n",
    "\n",
    "def calculate_angle_rads(x1, y1, x2, y2):\n",
    "    dx = x2 - x1     # Calculate the differences\n",
    "    dy = y2 - y1\n",
    "    angle_rads = math.atan2(dy, dx)    # Calculate the angle in radians\n",
    "    #angle_degrees = math.degrees(angle_radians)    # Convert the angle to degrees\n",
    "    return angle_rads\n",
    "    \n",
    "def offset(pto, distance, angle_deg): # modified from ChatGPT 4.o generated\n",
    "    angle_rads = math.radians(angle_deg) # Convert angle from degrees to radians\n",
    "    x_new = pto[0] + distance * math.cos(angle_rads) # Calculate the new coordinates\n",
    "    y_new = pto[1] + distance * math.sin(angle_rads)\n",
    "    return np.array((x_new, y_new))\n",
    "\n",
    "def offsetx(xo, distance, angle_deg): # modified from ChatGPT 4.o generated\n",
    "    angle_rads = math.radians(angle_deg) # Convert angle from degrees to radians\n",
    "    x_new = xo + distance * math.cos(angle_rads) # Calculate the new coordinates\n",
    "    return np.array((xo, x_new))\n",
    "\n",
    "def offset_arrx(xo, distance, angle_deg,size): # modified from ChatGPT 4.o generated\n",
    "    angle_rads = math.radians(angle_deg) # Convert angle from degrees to radians\n",
    "    x_new = xo + distance * math.cos(angle_rads) # Calculate the new coordinates\n",
    "    x_arr = x_new + size * math.cos(angle_rads+2.48) # Calculate the new coordinates\n",
    "    return np.array((x_new, x_arr))\n",
    "\n",
    "def offsety(yo, distance, angle_deg): # modified from ChatGPT 4.o generated\n",
    "    angle_rads = math.radians(angle_deg) # Convert angle from degrees to radians\n",
    "    y_new = yo + distance * math.sin(angle_rads) # Calculate the new coordinates\n",
    "    return np.array((yo, y_new))\n",
    "\n",
    "def offset_arry(yo, distance, angle_deg,size): # modified from ChatGPT 4.o generated\n",
    "    angle_rads = math.radians(angle_deg) # Convert angle from degrees to radians\n",
    "    y_new = yo + distance * math.sin(angle_rads) # Calculate the new coordinates\n",
    "    y_arr = y_new + size * math.sin(angle_rads+2.48) # Calculate the new coordinates\n",
    "    return np.array((y_new, y_arr))\n",
    "\n",
    "def lint(x1, y1, x2, y2, t):\n",
    "    # Calculate the interpolated coordinates\n",
    "    x = x1 + t * (x2 - x1)\n",
    "    y = y1 + t * (y2 - y1)\n",
    "    return np.array((x, y))\n",
    "\n",
    "def lintx(x1, y1, x2, y2, t):\n",
    "    # Calculate the interpolated coordinates\n",
    "    x = x1 + t * (x2 - x1)\n",
    "    return x\n",
    "\n",
    "def linty(x1, y1, x2, y2, t):\n",
    "    # Calculate the interpolated coordinates\n",
    "    y = y1 + t * (y2 - y1)\n",
    "    return y\n",
    "\n",
    "def lint_intx(x1, y1, x2, y2, ts, te):\n",
    "    # Calculate the interpolated coordinates\n",
    "    xs = x1 + ts * (x2 - x1)\n",
    "    xe = x1 + te * (x2 - x1)\n",
    "    return np.array((xs,xe))\n",
    "\n",
    "def lint_inty(x1, y1, x2, y2, ts, te):\n",
    "    # Calculate the interpolated coordinates\n",
    "    ys = y1 + ts * (y2 - y1)\n",
    "    ye = y1 + te * (y2 - y1)\n",
    "    return np.array((ys,ye))\n",
    "\n",
    "def lint_int_arrx(x1, y1, x2, y2, ts, te, size):\n",
    "    # Calculate the interpolated coordinates\n",
    "    xe = x1 + te * (x2 - x1)\n",
    "    line_angle_rads = calculate_angle_rads(x1, y1, x2, y2)\n",
    "    x_arr = xe + size * math.cos(line_angle_rads+2.48) # Calculate the new coordinates\n",
    "    return np.array((xe,x_arr))\n",
    "\n",
    "def lint_int_arry(x1, y1, x2, y2, ts, te, size):\n",
    "    # Calculate the interpolated coordinates\n",
    "    ye = y1 + te * (y2 - y1)\n",
    "    line_angle_rads = calculate_angle_rads(x1, y1, x2, y2)\n",
    "    y_arr = ye + size * math.sin(line_angle_rads+2.48) # Calculate the new coordinates\n",
    "    return np.array((ye,y_arr))\n",
    "\n",
    "def as_si(x, ndp): # from xnx on StackOverflow https://stackoverflow.com/questions/31453422/displaying-numbers-with-x-instead-of-e-scientific-notation-in-matplotlib \n",
    "    s = '{x:0.{ndp:d}e}'.format(x=x, ndp=ndp)\n",
    "    m, e = s.split('e')\n",
    "    return r'{m:s}\\times 10^{{{e:d}}}'.format(m=m, e=int(e))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf37b61d",
   "metadata": {},
   "source": [
    "#### Interactive Artificial Neural Network (ANN)\n",
    "\n",
    "I coded an AAN from scratch and included these interactive inputs:\n",
    "\n",
    "* **$n_{epoch}$** - number of cycles through the data, coupled forward prediction and back propagation\n",
    "\n",
    "* **$\\eta$** - learning rate, applied to the weight and bias derivatives\n",
    "\n",
    "* **$S$** - random number seed\n",
    "\n",
    "* **Weights** - show the network weights\n",
    "\n",
    "* **Forward Pass** - show the contributions and node outputs\n",
    "\n",
    "* **Back Propogation** - show the error partial derivatives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d270c12b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# widgets and dashboard\n",
    "l = widgets.Text(value='                                             Machine Learning Simple Artificial Neural Network, Prof. Michael Pyrcz, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))\n",
    "\n",
    "nepoch = widgets.IntSlider(min=0, max = 100, value=0, step = 1, description = '$n_{epoch}$',orientation='horizontal', style = {'description_width': 'initial'},\n",
    "                           continuous_update=False,layout=Layout(width='300px', height='30px'))\n",
    "\n",
    "lr = widgets.FloatLogSlider(min=-2, max = 0, value=0.1, step = 1.0, description = '$\\eta$',orientation='horizontal', style = {'description_width': 'initial'},\n",
    "                           continuous_update=False,layout=Layout(width='300px', height='30px'),readout_format='.2f')\n",
    "\n",
    "seed = widgets.IntSlider(min=0, max = 100, value=1, step = 1, description = '$S$',orientation='horizontal', style = {'description_width': 'initial'},\n",
    "                           continuous_update=False,layout=Layout(width='300px', height='30px'))\n",
    "\n",
    "weights = widgets.Checkbox(value=True,description='Weights',disabled=False,layout=Layout(width='300px', height='30px'))\n",
    "\n",
    "forward = widgets.Checkbox(value=True,description='Forward Pass',disabled=False,layout=Layout(width='300px', height='30px'))\n",
    "\n",
    "back = widgets.Checkbox(value=True,description='Back Propogation',disabled=False,layout=Layout(width='300px', height='30px'))\n",
    "\n",
    "ui1 = widgets.HBox([nepoch,lr,seed],)\n",
    "ui2 = widgets.HBox([weights,forward,back],)\n",
    "ui = widgets.VBox([l,ui1,ui2],)\n",
    "\n",
    "def run_plot(nepoch,lr,weights,forward,back,seed):                       # make data, fit models and plot\n",
    "    min_lw = 0.5; node_r = 0.2; min_node_r = 0.20\n",
    "    lw = 2.0 - min_lw; min_alpha = 0.1\n",
    "    alpha = 1.0 - min_alpha\n",
    "    iepoch = nepoch\n",
    "    np.random.seed(seed=seed)\n",
    "    x1 = 0.5; x2 = 0.2; x3 = 0.7; y = 0.3 # training data\n",
    "    \n",
    "    np.random.seed(seed=seed)\n",
    "    \n",
    "    nepoch = 1000\n",
    "    \n",
    "    y4 = np.zeros(nepoch); y5 = np.zeros(nepoch); y6 = np.zeros(nepoch)\n",
    "    \n",
    "    w14 = np.zeros(nepoch); w24 = np.zeros(nepoch); w34 = np.zeros(nepoch)\n",
    "    w15 = np.zeros(nepoch); w25 = np.zeros(nepoch); w35 = np.zeros(nepoch)\n",
    "    w46 = np.zeros(nepoch); w56 = np.zeros(nepoch)\n",
    "    \n",
    "    dw14 = np.zeros(nepoch); dw24 = np.zeros(nepoch); dw34 = np.zeros(nepoch)\n",
    "    dw15 = np.zeros(nepoch); dw25 = np.zeros(nepoch); dw35 = np.zeros(nepoch)\n",
    "    dw46 = np.zeros(nepoch); dw56 = np.zeros(nepoch)\n",
    "    \n",
    "    db4 = np.zeros(nepoch); db5 = np.zeros(nepoch); db6 = np.zeros(nepoch)\n",
    "    \n",
    "    b4 = np.zeros(nepoch); b5 = np.zeros(nepoch); b6 = np.zeros(nepoch)\n",
    "    y4 = np.zeros(nepoch); y5 = np.zeros(nepoch); y6 = np.zeros(nepoch)\n",
    "    y4in = np.zeros(nepoch); y5in = np.zeros(nepoch); y6in = np.zeros(nepoch)\n",
    "    d4 = np.zeros(nepoch); d5 = np.zeros(nepoch); d6 = np.zeros(nepoch)\n",
    "    d1 = np.zeros(nepoch); d2 = np.zeros(nepoch); d3 = np.zeros(nepoch)\n",
    "    \n",
    "    # initialize the weights - Xavier Weight Initialization \n",
    "    lower, upper = -(1.0 / np.sqrt(3.0)), (1.0 / np.sqrt(3.0)) # lower and upper bound for the weights, uses inputs to node\n",
    "    #lower, upper = -(sqrt(6.0) / sqrt(3.0 + 2.0)), (sqrt(6.0) / sqrt(3.0 + 2.0)) # Normalized Xavier weights, integrates ouputs also\n",
    "    w14[0] = lower + np.random.random() * (upper - lower); \n",
    "    w24[0] = lower + np.random.random() * (upper - lower); \n",
    "    w34[0] = lower + np.random.random() * (upper - lower);\n",
    "    w15[0] = lower + np.random.random() * (upper - lower); \n",
    "    w25[0] = lower + np.random.random() * (upper - lower); \n",
    "    w35[0] = lower + np.random.random() * (upper - lower);\n",
    "    \n",
    "    lower, upper = -(1.0 / np.sqrt(2.0)), (1.0 / np.sqrt(2.0))\n",
    "    #lower, upper = -(sqrt(6.0) / sqrt(2.0 + 1.0)), (sqrt(6.0) / sqrt(2.0 + 1.0)) # Normalized Xavier weights, integrates ouputs also\n",
    "\n",
    "    w46[0] = lower + np.random.random() * (upper - lower); \n",
    "    w56[0] = lower + np.random.random() * (upper - lower);     \n",
    "\n",
    "    #b4[0] = np.random.random(); b5[0] = np.random.random(); b6[0] = np.random.random()\n",
    "    b4[0] = (np.random.random()-0.5)*0.5; b5[0] = (np.random.random()-0.5)*0.5; b6[0] = (np.random.random()-0.5)*0.5; # small random value    \n",
    "\n",
    "    for i in range(0,nepoch):\n",
    "    \n",
    "    # forward pass of model\n",
    "        y4in[i] = w14[i]*x1 + w24[i]*x2 + w34[i]*x3 + b4[i]; \n",
    "        y4[i] = 1.0/(1 + math.exp(-1*y4in[i]))\n",
    "        \n",
    "        y5in[i] = w15[i]*x1 + w25[i]*x2 + w35[i]*x3 + b5[i]\n",
    "        y5[i] = 1.0/(1 + math.exp(-1*y5in[i]))\n",
    "        \n",
    "        y6in[i] = w46[i]*y4[i] + w56[i]*y5[i] + b6[i]\n",
    "        y6[i] = y6in[i]\n",
    "    #    y6[i] = 1.0/(1 + math.exp(-1*y6in[i])) # sgimoid / logistic activation at o6 \n",
    "    \n",
    "    # back propagate the error through the nodes\n",
    "    #    d6[i] = y6[i]*(1-y6[i])*(y-y6[i]) # sgimoid / logistic activation at o6 \n",
    "        d6[i] = (y-y6[i]) # identity activation o at o6\n",
    "        d5[i] = y5[i]*(1-y5[i])*w56[i]*d6[i]; d4[i] = y4[i]*(1-y4[i])*w46[i]*d6[i]\n",
    "        d1[i] = w14[i]*d4[i] + w15[i]*d5[i]; d2[i] = w24[i]*d4[i] + w25[i]*d5[i]; d3[i] = w34[i]*d4[i] + w35[i]*d5[i] # identity and 2 paths\n",
    "    \n",
    "    # calculate the change in weights\n",
    "        if i < nepoch - 1:\n",
    "            dw14[i] = lr*d4[i]*x1; dw24[i] = lr*d4[i]*x2; dw34[i] = lr*d4[i]*x3 \n",
    "            dw15[i] = lr*d5[i]*x1; dw25[i] = lr*d5[i]*x2; dw35[i] = lr*d5[i]*x3\n",
    "            dw46[i] = lr*d6[i]*y4[i]; dw56[i] = lr*d6[i]*y5[i] \n",
    "            \n",
    "            db4[i] = lr*d4[i]; db5[i] = lr*d5[i]; db6[i] = lr*d6[i];\n",
    "    \n",
    "            w14[i+1] = w14[i] + dw14[i]; w24[i+1] = w24[i] + dw24[i]; w34[i+1] = w34[i] + dw34[i] \n",
    "            w15[i+1] = w15[i] + dw15[i]; w25[i+1] = w25[i] + dw25[i]; w35[i+1] = w35[i] + dw35[i] \n",
    "            w46[i+1] = w46[i] + dw46[i]; w56[i+1] = w56[i] + dw56[i]\n",
    "    \n",
    "            b4[i+1] = b4[i] + db4[i]; b5[i+1] = b5[i] + db5[i]; b6[i+1] = b6[i] + db6[i] \n",
    "    \n",
    "    dx = -0.21; dy = -0.09; edge = 1.0\n",
    "    \n",
    "    o6x = 17; o6y =5; h5x = 10; h5y = 3.5; h4x = 10; h4y = 6.5\n",
    "    i1x = 3; i1y = 9.0; i2x = 3; i2y = 5; i3x = 3; i3y = 1.0; buffer = 0.5\n",
    "    \n",
    "    max_x = np.max(np.abs([x1,x2,x3]))\n",
    "    max_y = np.max(np.abs([y4[iepoch],y5[iepoch],y6[iepoch]]))\n",
    "    max_dO = np.max(np.abs([d1[iepoch],d2[iepoch],d3[iepoch]]))\n",
    "    max_d = np.max(np.abs([d1[iepoch],d2[iepoch],d3[iepoch],d4[iepoch],d5[iepoch],d6[iepoch]]))\n",
    "    max_d_trans = 1.0/abs(math.log(max_d)+0.00001)\n",
    "    max_signal = np.max(np.abs([x1*w14[iepoch],x2*w24[iepoch],x3*w34[iepoch],\n",
    "                                x1*w15[iepoch],x2*w25[iepoch],x3*w35[iepoch],\n",
    "                                y4[iepoch],y5[iepoch],y6[iepoch]]))\n",
    "    \n",
    "    plt.subplot(111)\n",
    "    plt.gca().set_axis_off()\n",
    "    \n",
    "    if (weights == True and forward == False and back == False) or (weights == False and forward == True and back == False):\n",
    "        \n",
    "        circle_i1 = plt.Circle((i1x,i1y), node_r*abs(x1)/max_x+min_node_r, fill=False, edgecolor = 'black',lw=2,zorder=100); plt.annotate(r' $I_1$',(i1x+dx,i1y+dy),zorder=110) \n",
    "        circle_i1b = plt.Circle((i1x,i1y), node_r*1.5*abs(x1)/max_x+min_node_r, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_i1); plt.gca().add_patch(circle_i1b)\n",
    "    \n",
    "        circle_i2 = plt.Circle((i2x,i2y), node_r*abs(x2)/max_x+min_node_r, fill=False, edgecolor = 'black',lw=2,zorder=100); plt.annotate(r' $I_2$',(i2x+dx,i2y+dy),zorder=110) \n",
    "        circle_i2b = plt.Circle((i2x,i2y), node_r*1.5*abs(x2)/max_x+min_node_r, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_i2); plt.gca().add_patch(circle_i2b)\n",
    "    \n",
    "        circle_i3 = plt.Circle((i3x,i3y), node_r*abs(x3)/max_x+min_node_r, fill=False, edgecolor = 'black',lw=2,zorder=100); plt.annotate(r' $I_3$',(i3x+dx,i3y+dy),zorder=110) \n",
    "        circle_i3b = plt.Circle((i3x,i3y), node_r*1.5*abs(x3)/max_x+min_node_r, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_i3); plt.gca().add_patch(circle_i3b)\n",
    "        \n",
    "        circle_h4 = plt.Circle((h4x,h4y), node_r*abs(y4[iepoch])/max_y+min_node_r,fill=True,facecolor = 'red',alpha=0.5,edgecolor = 'black',lw=2,zorder=100); plt.annotate(r'$H_4$',(h4x+dx,h4y+dy),zorder=110)\n",
    "        circle_h4o = plt.Circle((h4x,h4y), node_r*abs(y4[iepoch])/max_y+min_node_r,fill=False,alpha=1.0,edgecolor = 'black',lw=2,zorder=105) \n",
    "        circle_h4b = plt.Circle((h4x,h4y), node_r*1.5*abs(y4[iepoch])/max_y+min_node_r, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_h4); plt.gca().add_patch(circle_h4b); plt.gca().add_patch(circle_h4o)\n",
    "    \n",
    "        circle_h5 = plt.Circle((h5x,h5y), node_r*abs(y5[iepoch])/max_y+min_node_r, fill=True,facecolor = 'blue',alpha=0.5,edgecolor = 'black',lw=2,zorder=100); plt.annotate(r'$H_5$',(h5x+dx,h5y+dy),zorder=110)\n",
    "        circle_h5o = plt.Circle((h5x,h5y), node_r*abs(y5[iepoch])/max_y+min_node_r, fill=False,alpha=1.0,edgecolor = 'black',lw=2,zorder=105) \n",
    "        circle_h5b = plt.Circle((h5x,h5y), node_r*1.5*abs(y5[iepoch])/max_y+min_node_r, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10);\n",
    "        plt.gca().add_patch(circle_h5); plt.gca().add_patch(circle_h5b); plt.gca().add_patch(circle_h5o)\n",
    "    \n",
    "        circle_o6 = plt.Circle((o6x,o6y), node_r*abs(y6[iepoch])/max_y+min_node_r,fill=True,facecolor = 'orange',alpha=0.5,edgecolor = 'black',lw=2,zorder=100); plt.annotate(r'$O_6$',(o6x+dx,o6y+dy),zorder=110)\n",
    "        circle_o6o = plt.Circle((o6x,o6y), node_r*abs(y6[iepoch])/max_y+min_node_r,fill=False,alpha=1.0,edgecolor = 'black',lw=2,zorder=105)\n",
    "        circle_o6b = plt.Circle((o6x,o6y), node_r*1.5*abs(y6[iepoch])/max_y+min_node_r, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10);\n",
    "        plt.gca().add_patch(circle_o6); plt.gca().add_patch(circle_o6b); plt.gca().add_patch(circle_o6o)\n",
    "  \n",
    "    if weights == False and forward == False and back == True:\n",
    "        i1_r = (1.0/abs(math.log(abs(d1[iepoch]))+0.00001)/max_d_trans)*node_r+min_node_r\n",
    "        circle_i1 = plt.Circle((i1x,i1y), i1_r, fill=False, edgecolor = 'black',lw=2,zorder=100); plt.annotate(r' $I_1$',(i1x+dx,i1y+dy),zorder=110) \n",
    "        circle_i1b = plt.Circle((i1x,i1y), i1_r*1.5, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_i1); plt.gca().add_patch(circle_i1b)\n",
    "\n",
    "        i2_r = (1.0/abs(math.log(abs(d2[iepoch]))+0.00001)/max_d_trans)*node_r+min_node_r\n",
    "        circle_i2 = plt.Circle((i2x,i2y), i2_r, fill=False, edgecolor = 'black',lw=2,zorder=100); plt.annotate(r' $I_2$',(i2x+dx,i2y+dy),zorder=110) \n",
    "        circle_i2b = plt.Circle((i2x,i2y), i2_r*1.5, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_i2); plt.gca().add_patch(circle_i2b)\n",
    "    \n",
    "        i3_r = (1.0/abs(math.log(abs(d3[iepoch]))+0.00001)/max_d_trans)*node_r+min_node_r\n",
    "        circle_i3 = plt.Circle((i3x,i3y), i3_r, fill=False, edgecolor = 'black',lw=2,zorder=100); plt.annotate(r' $I_3$',(i3x+dx,i3y+dy),zorder=110) \n",
    "        circle_i3b = plt.Circle((i3x,i3y), i3_r*1.5*abs(x3)/max_x+min_node_r, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_i3); plt.gca().add_patch(circle_i3b)\n",
    "        \n",
    "        h4_r = (1.0/abs(math.log(abs(d4[iepoch]))+0.00001)/max_d_trans)*node_r+min_node_r\n",
    "        circle_h4 = plt.Circle((h4x,h4y), h4_r,fill=True,facecolor = 'red',alpha=0.5,edgecolor = 'black',lw=2,zorder=100); plt.annotate(r'$H_4$',(h4x+dx,h4y+dy),zorder=110)\n",
    "        circle_h4o = plt.Circle((h4x,h4y), h4_r,fill=False,alpha=1.0,edgecolor = 'black',lw=2,zorder=105) \n",
    "        circle_h4b = plt.Circle((h4x,h4y), h4_r*1.5, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_h4); plt.gca().add_patch(circle_h4b); plt.gca().add_patch(circle_h4o)\n",
    "    \n",
    "        h5_r = (1.0/abs(math.log(abs(d5[iepoch]))+0.00001)/max_d_trans)*node_r+min_node_r\n",
    "        circle_h5 = plt.Circle((h5x,h5y), h5_r, fill=True,facecolor = 'blue',alpha=0.5,edgecolor = 'black',lw=2,zorder=100); plt.annotate(r'$H_5$',(h5x+dx,h5y+dy),zorder=110)\n",
    "        circle_h5o = plt.Circle((h5x,h5y), h5_r, fill=False,alpha=1.0,edgecolor = 'black',lw=2,zorder=105) \n",
    "        circle_h5b = plt.Circle((h5x,h5y), h5_r*1.5, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10);\n",
    "        plt.gca().add_patch(circle_h5); plt.gca().add_patch(circle_h5b); plt.gca().add_patch(circle_h5o)\n",
    "    \n",
    "        h6_r = (1.0/abs(math.log(abs(d6[iepoch]))+0.00001)/max_d_trans)*node_r+min_node_r\n",
    "        circle_o6 = plt.Circle((o6x,o6y), h6_r,fill=True,facecolor = 'orange',alpha=0.5,edgecolor = 'black',lw=2,zorder=100); plt.annotate(r'$O_6$',(o6x+dx,o6y+dy),zorder=110)\n",
    "        circle_o6o = plt.Circle((o6x,o6y), h6_r,fill=False,alpha=1.0,edgecolor = 'black',lw=2,zorder=105)\n",
    "        circle_o6b = plt.Circle((o6x,o6y), h6_r*1.5, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10);\n",
    "        plt.gca().add_patch(circle_o6); plt.gca().add_patch(circle_o6b); plt.gca().add_patch(circle_o6o)\n",
    "   \n",
    "    if (weights == True and forward == True and back == True) or (weights == False and forward == False and back == False):\n",
    "\n",
    "        circle_i1 = plt.Circle((i1x,i1y), node_r, fill=False, edgecolor = 'black',lw=2,zorder=100); plt.annotate(r' $I_1$',(i1x+dx,i1y+dy),zorder=110) \n",
    "        circle_i1b = plt.Circle((i1x,i1y), node_r*1.5, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_i1); plt.gca().add_patch(circle_i1b)\n",
    "    \n",
    "        circle_i2 = plt.Circle((i2x,i2y), node_r, fill=False, edgecolor = 'black',lw=2,zorder=100); plt.annotate(r' $I_2$',(i2x+dx,i2y+dy),zorder=110) \n",
    "        circle_i2b = plt.Circle((i2x,i2y), node_r*1.5, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_i2); plt.gca().add_patch(circle_i2b)\n",
    "    \n",
    "        circle_i3 = plt.Circle((i3x,i3y), node_r, fill=False, edgecolor = 'black',lw=2,zorder=100); plt.annotate(r' $I_3$',(i3x+dx,i3y+dy),zorder=110) \n",
    "        circle_i3b = plt.Circle((i3x,i3y), node_r*1.5, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_i3); plt.gca().add_patch(circle_i3b)\n",
    "        \n",
    "        circle_h4 = plt.Circle((h4x,h4y), node_r,fill=True,facecolor = 'red',alpha=0.5,edgecolor = 'black',lw=2,zorder=100); plt.annotate(r'$H_4$',(h4x+dx,h4y+dy),zorder=110)\n",
    "        circle_h4o = plt.Circle((h4x,h4y), node_r,fill=False,alpha=1.0,edgecolor = 'black',lw=2,zorder=105) \n",
    "        circle_h4b = plt.Circle((h4x,h4y), node_r*1.5, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10)\n",
    "        plt.gca().add_patch(circle_h4); plt.gca().add_patch(circle_h4b); plt.gca().add_patch(circle_h4o)\n",
    "    \n",
    "        circle_h5 = plt.Circle((h5x,h5y), node_r, fill=True,facecolor = 'blue',alpha=0.5,edgecolor = 'black',lw=2,zorder=100); plt.annotate(r'$H_5$',(h5x+dx,h5y+dy),zorder=110)\n",
    "        circle_h5o = plt.Circle((h5x,h5y), node_r, fill=False,alpha=1.0,edgecolor = 'black',lw=2,zorder=105) \n",
    "        circle_h5b = plt.Circle((h5x,h5y), node_r*1.5, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10);\n",
    "        plt.gca().add_patch(circle_h5); plt.gca().add_patch(circle_h5b); plt.gca().add_patch(circle_h5o)\n",
    "    \n",
    "        circle_o6 = plt.Circle((o6x,o6y), node_r,fill=True,facecolor = 'orange',alpha=0.5,edgecolor = 'black',lw=2,zorder=100); plt.annotate(r'$O_6$',(o6x+dx,o6y+dy),zorder=110)\n",
    "        circle_o6o = plt.Circle((o6x,o6y), node_r,fill=False,alpha=1.0,edgecolor = 'black',lw=2,zorder=105)\n",
    "        circle_o6b = plt.Circle((o6x,o6y), node_r*1.5, fill=True, facecolor = 'white',edgecolor = None,lw=1,zorder=10);\n",
    "        plt.gca().add_patch(circle_o6); plt.gca().add_patch(circle_o6b); plt.gca().add_patch(circle_o6o)\n",
    "        \n",
    "    plt.plot([i1x-edge,i1x],[i1y,i1y],color='grey',lw=1.0,zorder=1)\n",
    "    plt.plot([i2x-edge,i2x],[i2y,i2y],color='grey',lw=1.0,zorder=1)\n",
    "    plt.plot([i3x-edge,i3x],[i3y,i3y],color='grey',lw=1.0,zorder=1)\n",
    "    \n",
    "    plt.annotate(r'$x_1$ = ' + str(np.round(x1,2)),(i1x-buffer-1.6,i1y-0.05),size=8,zorder=200,color='grey',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=0,ha='left') \n",
    "    plt.annotate(r'$x_2$ = ' + str(np.round(x2,2)),(i2x-buffer-1.6,i2y-0.05),size=8,zorder=200,color='grey',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=0) \n",
    "    plt.annotate(r'$x_3$ = ' + str(np.round(x3,2)),(i3x-buffer-1.6,i3y-0.05),size=8,zorder=200,color='grey',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=0) \n",
    "    \n",
    "    min_wt = np.min(np.abs([w14[iepoch],w24[iepoch],w34[iepoch],w15[iepoch],w25[iepoch],w35[iepoch],w46[iepoch],w56[iepoch]]))\n",
    "    max_wt = np.max(np.abs([w14[iepoch],w24[iepoch],w34[iepoch],w15[iepoch],w25[iepoch],w35[iepoch],w46[iepoch],w56[iepoch]]))\n",
    "  \n",
    "    min_dwt = np.min(np.abs([dw14[iepoch],dw24[iepoch],dw34[iepoch],dw15[iepoch],dw25[iepoch],dw35[iepoch],dw46[iepoch],dw56[iepoch]]))\n",
    "    max_dwt = np.max(np.abs([dw14[iepoch],dw24[iepoch],dw34[iepoch],dw15[iepoch],dw25[iepoch],dw35[iepoch],dw46[iepoch],dw56[iepoch]]))\n",
    " \n",
    "    if (weights == True and forward == False and back == False):\n",
    "        plt.plot([i1x,h4x],[i1y,h4y],color='orangered',alpha = alpha*abs(w14[iepoch]/max_wt)+min_alpha,lw=lw*abs(w14[iepoch]/max_wt)+min_lw,zorder=1)\n",
    "        plt.plot([i2x,h4x],[i2y,h4y],color='red',alpha = alpha*abs(w24[iepoch]/max_wt)+min_alpha,lw=lw*abs(w24[iepoch]/max_wt)+min_lw,zorder=1)\n",
    "        plt.plot([i3x,h4x],[i3y,h4y],color='darkred',alpha = alpha*abs(w34[iepoch]/max_wt)+min_alpha,lw=lw*abs(w34[iepoch]/max_wt)+min_lw,zorder=1)\n",
    "    \n",
    "        plt.plot([i1x,h5x],[i1y,h5y],color='dodgerblue',alpha = alpha*abs(w15[iepoch]/max_wt)+min_alpha,lw=lw*abs(w15[iepoch]/max_wt)+min_lw,zorder=1)\n",
    "        plt.plot([i2x,h5x],[i2y,h5y],color='blue',alpha = alpha*abs(w25[iepoch]/max_wt)+min_alpha,lw=lw*abs(w25[iepoch]/max_wt)+min_lw,zorder=1)\n",
    "        plt.plot([i3x,h5x],[i3y,h5y],color='darkblue',alpha = alpha*abs(w35[iepoch]/max_wt)+min_alpha,lw=lw*abs(w35[iepoch]/max_wt)+min_lw,zorder=1)\n",
    "    \n",
    "        plt.plot([h4x,o6x],[h4y,o6y],color='orange',alpha = alpha*abs(w46[iepoch]/max_wt)+min_alpha,lw=lw*abs(w46[iepoch]/max_wt)+min_lw,zorder=1)\n",
    "        plt.plot([h5x,o6x],[h5y,o6y],color='darkorange',alpha = alpha*abs(w56[iepoch]/max_wt)+min_alpha,lw=lw*abs(w56[iepoch]/max_wt)+min_lw,zorder=1)\n",
    " \n",
    "    if (weights == False and forward == True and back == False):\n",
    "        plt.plot([i1x,h4x],[i1y,h4y],color='orangered',alpha = alpha*abs(x1*w14[iepoch]/max_signal)+min_alpha,lw=lw*abs(x1*w14[iepoch]/max_signal)+min_lw,zorder=1)\n",
    "        plt.plot([i2x,h4x],[i2y,h4y],color='red',alpha = alpha*abs(x2*w24[iepoch]/max_signal)+min_alpha,lw=lw*abs(x2*w24[iepoch]/max_signal)+min_lw,zorder=1)\n",
    "        plt.plot([i3x,h4x],[i3y,h4y],color='darkred',alpha = alpha*abs(x3*w34[iepoch]/max_signal)+min_alpha,lw=lw*abs(x3*w34[iepoch]/max_signal)+min_lw,zorder=1)\n",
    "    \n",
    "        plt.plot([i1x,h5x],[i1y,h5y],color='dodgerblue',alpha = alpha*abs(x1*w15[iepoch]/max_signal)+min_alpha,lw=lw*abs(x1*w15[iepoch]/max_signal)+min_lw,zorder=1)\n",
    "        plt.plot([i2x,h5x],[i2y,h5y],color='blue',alpha = alpha*abs(x2*w25[iepoch]/max_signal)+min_alpha,lw=lw*abs(x2*w25[iepoch]/max_signal)+min_lw,zorder=1)\n",
    "        plt.plot([i3x,h5x],[i3y,h5y],color='darkblue',alpha = alpha*abs(x3*w35[iepoch]/max_signal)+min_alpha,lw=lw*abs(x3*w35[iepoch]/max_signal)+min_lw,zorder=1)\n",
    "    \n",
    "        plt.plot([h4x,o6x],[h4y,o6y],color='orange',alpha = alpha*abs(y4[iepoch]*w46[iepoch]/max_signal)+min_alpha,lw=lw*abs(y4[iepoch]*w46[iepoch]/max_signal)+min_lw,zorder=1)\n",
    "        plt.plot([h5x,o6x],[h5y,o6y],color='darkorange',alpha = alpha*abs(y5[iepoch]*w56[iepoch]/max_signal)+min_alpha,lw=lw*abs(y5[iepoch]*w56[iepoch]/max_signal)+min_lw,zorder=1)\n",
    "\n",
    "    if (weights == False and forward == False and back == True):\n",
    "\n",
    "        plt.plot([i1x,h4x],[i1y,h4y],color='orangered',alpha = alpha*abs(dw14[iepoch]/max_dwt)+min_alpha,lw=lw*abs(dw14[iepoch]/max_dwt)+min_lw,zorder=1)\n",
    "        plt.plot([i2x,h4x],[i2y,h4y],color='red',alpha = alpha*abs(dw24[iepoch]/max_dwt)+min_alpha,lw=lw*abs(dw24[iepoch]/max_dwt)+min_lw,zorder=1)\n",
    "        plt.plot([i3x,h4x],[i3y,h4y],color='darkred',alpha = alpha*abs(dw34[iepoch]/max_dwt)+min_alpha,lw=lw*abs(dw34[iepoch]/max_dwt)+min_lw,zorder=1)\n",
    "    \n",
    "        plt.plot([i1x,h5x],[i1y,h5y],color='dodgerblue',alpha = alpha*abs(dw15[iepoch]/max_dwt)+min_alpha,lw=lw*abs(dw15[iepoch]/max_dwt)+min_lw,zorder=1)\n",
    "        plt.plot([i2x,h5x],[i2y,h5y],color='blue',alpha = alpha*abs(dw25[iepoch]/max_dwt)+min_alpha,lw=lw*abs(dw25[iepoch]/max_dwt)+min_lw,zorder=1)\n",
    "        plt.plot([i3x,h5x],[i3y,h5y],color='darkblue',alpha = alpha*abs(dw35[iepoch]/max_dwt)+min_alpha,lw=lw*abs(dw35[iepoch]/max_dwt)+min_lw,zorder=1)\n",
    "    \n",
    "        plt.plot([h4x,o6x],[h4y,o6y],color='orange',alpha = alpha*abs(dw46[iepoch]/max_dwt)+min_alpha,lw=lw*abs(dw46[iepoch]/max_dwt)+min_lw,zorder=1)\n",
    "        plt.plot([h5x,o6x],[h5y,o6y],color='darkorange',alpha = alpha*abs(dw56[iepoch]/max_dwt)+min_alpha,lw=lw*abs(dw56[iepoch]/max_dwt)+min_lw,zorder=1)\n",
    "    \n",
    "    if (weights == True and forward == True and back == True) or (weights == False and forward == False and back == False):\n",
    "    \n",
    "        plt.plot([i1x,h4x],[i1y,h4y],color='orangered',lw=lw+min_lw,zorder=1)\n",
    "        plt.plot([i2x,h4x],[i2y,h4y],color='red',lw=lw+min_lw,zorder=1)\n",
    "        plt.plot([i3x,h4x],[i3y,h4y],color='darkred',lw=lw+min_lw,zorder=1)\n",
    "    \n",
    "        plt.plot([i1x,h5x],[i1y,h5y],color='dodgerblue',lw=lw+min_lw,zorder=1)\n",
    "        plt.plot([i2x,h5x],[i2y,h5y],color='blue',lw=lw+min_lw,zorder=1)\n",
    "        plt.plot([i3x,h5x],[i3y,h5y],color='darkblue',lw=lw+min_lw,zorder=1)\n",
    "    \n",
    "        plt.plot([h4x,o6x],[h4y,o6y],color='orange',lw=lw+min_lw,zorder=1)\n",
    "        plt.plot([h5x,o6x],[h5y,o6y],color='darkorange',lw=lw+min_lw,zorder=1)\n",
    "    \n",
    "    if forward == True:\n",
    "    \n",
    "        plt.plot(offsetx(i1x,2,-20),offsety(i1y,2,-20)+0.1,color='orangered',lw=1.0,zorder=1)\n",
    "        plt.plot(offset_arrx(i1x,2,-20,0.2),offset_arry(i1y,2,-20,0.2)+0.1,color='orangered',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$I_{1}$ = ' + str(np.round(x1,2)),(lintx(i1x,i1y,h4x,h4y,0.1),linty(i1x,i1y,h4x,h4y,0.1)),size=8,zorder=200,color='orangered',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-20)\n",
    "    \n",
    "        plt.plot(offsetx(i2x,2,12),offsety(i2y,2,12)+0.1,color='red',lw=1.0,zorder=1)\n",
    "        plt.plot(offset_arrx(i2x,2,12,0.2),offset_arry(i2y,2,12,0.2)+0.1,color='red',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$I_{2}$ = ' + str(np.round(x2,2)),(lintx(i2x,i2y,h4x,h4y,0.1),linty(i2x,i2y,h4x,h4y,0.1)+0.24),size=8,zorder=200,color='red',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=12)\n",
    "    \n",
    "        plt.plot(offsetx(i3x,2,38),offsety(i3y,2,38)+0.1,color='darkred',lw=1.0,zorder=1)\n",
    "        plt.plot(offset_arrx(i3x,2,38,0.2),offset_arry(i3y,2,38,0.2)+0.1,color='darkred',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$I_{3}$ = ' + str(np.round(x3,2)),(lintx(i3x,i3y,h4x,h4y,0.08)-0.2,linty(i3x,i3y,h4x,h4y,0.08)+0.2),size=8,zorder=200,color='darkred',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=38)\n",
    "    \n",
    "        plt.plot(offsetx(h4x,2,-12),offsety(h4y,2,-12)+0.1,color='orange',lw=1.0,zorder=1)\n",
    "        plt.plot(offset_arrx(h4x,2,-12,0.2),offset_arry(h4y,2,-12,0.2)+0.1,color='orange',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$H_{4}$ = ' + str(np.round(y4[iepoch],2)),(lintx(h4x,h4y,o6x,o6y,0.08),linty(h4x,h4y,o6x,o6y,0.08)-0.0),size=8,zorder=200,color='orange',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-12)\n",
    "        \n",
    "        plt.annotate(r'$H_{4_{in}}$ = ' + str(np.round(y4in[iepoch],2)),(lintx(h4x,h4y,o6x,o6y,0.08)-0.5,linty(h4x,h4y,o6x,o6y,0.08)+0.5),size=8,zorder=200,color='orange',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-12)\n",
    "    \n",
    "        plt.plot(offsetx(h5x,2,12),offsety(h5y,2,12)+0.1,color='darkorange',lw=1.0,zorder=1)\n",
    "        plt.plot(offset_arrx(h5x,2,12,0.2),offset_arry(h5y,2,12,0.2)+0.1,color='darkorange',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$H_{5}$ = ' + str(np.round(y5[iepoch],2)),(lintx(h5x,h5y,o6x,o6y,0.07),linty(h5x,h5y,o6x,o6y,0.07)+0.25),size=8,zorder=200,color='darkorange',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=12)\n",
    "\n",
    "        plt.annotate(r'$H_{5_{in}}$ = ' + str(np.round(y5in[iepoch],2)),(lintx(h5x,h5y,o6x,o6y,0.08)-0.5,linty(h5x,h5y,o6x,o6y,0.08)+0.5),size=8,zorder=200,color='darkorange',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=12)\n",
    "        \n",
    "        plt.plot([o6x+edge,o6x],[o6y,o6y],color='grey',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$\\hat{y}$ = ' + str(np.round(y6[iepoch],2)),(o6x+buffer+0.7,o6y-0.05),size=8,zorder=300,color='grey',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=0) \n",
    "\n",
    "#        plt.annotate(r'$\\frac{\\partial P}{\\partial \\hat{y}}$ = ' + str(np.round(d6[iepoch],2)),(o6x,o6y-1.2),size=10,\n",
    "#                bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),zorder=200)\n",
    "\n",
    "        plt.annotate(r'$y$ = ' + str(np.round(y,2)),(o6x+buffer+0.7,o6y+0.5),size=8,zorder=300,color='grey',\n",
    "                 bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=0) \n",
    "\n",
    "    plt.plot([h4x,h4x-0.5],[h4y,h4y+1.0],color='red',zorder=5)\n",
    "    plt.annotate(r'$b_{4}$ = ' + str(np.round(b4[iepoch],2)),(h4x-0.7,h4y+1.2),size=8,zorder=200,color='red',\n",
    "        bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=0)  \n",
    "    \n",
    "    plt.plot([h5x,h5x-0.5],[h5y,h5y+1.0],color='blue',zorder=5)\n",
    "    plt.annotate(r'$b_{5}$ = ' + str(np.round(b5[iepoch],2)),(h5x-0.7,h5y+1.2),size=8,zorder=200,color='blue',\n",
    "        bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=0)  \n",
    "    \n",
    "    plt.plot([o6x,o6x-0.5],[o6y,o6y+1.0],color='orange',zorder=5)\n",
    "    plt.annotate(r'$b_{6}$ = ' + str(np.round(b6[iepoch],2)),(o6x-0.7,o6y+1.2),size=8,zorder=200,color='orange',\n",
    "        bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=0)  \n",
    "    \n",
    "    plt.plot([o6x+edge,o6x],[o6y,o6y],color='grey',lw=1.0,zorder=1)\n",
    "    plt.annotate(r'$y$ = ' + str(np.round(y,2)),(o6x+buffer+0.7,o6y-0.05),size=8,zorder=200,color='grey',\n",
    "        bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=0) \n",
    "    \n",
    "    if back == True:\n",
    "    \n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial O_{6_{in}}}$ = ' + str(np.round(d6[iepoch],2)),(o6x-0.5,o6y-1.2),size=10,zorder=100)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial H_{4_{in}}}$ = ' + str(np.round(d4[iepoch],2)),(h4x-0.5,h4y-0.7),size=10,zorder=100)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial H_{5_{in}}}$ = ' + str(np.round(d5[iepoch],2)),(h5x-0.5,h5y-0.7),size=10,zorder=100)\n",
    "    \n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial \\hat{y}}$ = ' + str(np.round(d6[iepoch],2)),(o6x,o6y-1.8),size=10,zorder=100)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial H_{4_{out}}}$ = ' + str(np.round(w46[iepoch]*d6[iepoch],2)),(h4x,h4y-1.2),size=10,zorder=100)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial H_{5_{out}}}$ = ' + str(np.round(w56[iepoch]*d6[iepoch],2)),(h5x,h5y-1.2),size=10,zorder=100)\n",
    "   \n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial X_1}$ = ' + r'${0:s}$'.format(as_si(d1[iepoch],2)),(i1x-2.0,i1y-0.9),size=10,zorder=100)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial X_2}$ = ' + r'${0:s}$'.format(as_si(d2[iepoch],2)),(i2x-2.0,i2y-0.9),size=10,zorder=100)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial X_3}$ = ' + r'${0:s}$'.format(as_si(d3[iepoch],2)),(i3x-2.0,i3y-0.9),size=10,zorder=100)\n",
    "        \n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial I_1}$ = ' + r'${0:s}$'.format(as_si(d1[iepoch],2)),(i1x-1.5,i1y-1.4),size=10,zorder=100)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial I_2}$ = ' + r'${0:s}$'.format(as_si(d2[iepoch],2)),(i2x-1.5,i2y-1.4),size=10,zorder=100)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial I_3}$ = ' + r'${0:s}$'.format(as_si(d3[iepoch],2)),(i3x-1.5,i3y-1.4),size=10,zorder=100)\n",
    "\n",
    "        plt.plot(lint_intx(h4x, h4y, o6x, o6y,0.4,0.6),lint_inty(h4x,h4y,o6x,o6y,0.4,0.6)-0.1,color='orange',lw=1.0,zorder=1)\n",
    "        plt.plot(lint_int_arrx(o6x,o6y,h4x,h4y,0.4,0.6,0.2),lint_int_arry(o6x,o6y,h4x,h4y,0.4,0.6,0.2)-0.1,color='orange',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial \\lambda_{4,6}} =$' + r'${0:s}$'.format(as_si(dw46[iepoch]/lr,2)),(lintx(h4x,h4y,o6x,o6y,0.5)-0.6,linty(h4x,h4y,o6x,o6y,0.5)-0.72),size=7,zorder=200,color='orange',\n",
    "                  bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-11)\n",
    "    \n",
    "        plt.plot(lint_intx(h5x, h5y, o6x, o6y,0.4,0.6),lint_inty(h5x,h5y,o6x,o6y,0.4,0.6)-0.1,color='darkorange',lw=1.0,zorder=1)\n",
    "        plt.plot(lint_int_arrx(o6x,o6y,h5x,h5y,0.4,0.6,0.2),lint_int_arry(o6x,o6y,h5x,h5y,0.4,0.6,0.2)-0.1,color='darkorange',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial \\lambda_{5,6}} =$' + r'${0:s}$'.format(as_si(dw56[iepoch]/lr,2)),(lintx(h5x,h5y,o6x,o6y,0.5)-0.4,linty(h5x,h5y,o6x,o6y,0.5)-0.6),size=7,zorder=200,color='darkorange',\n",
    "                  bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=12)\n",
    "\n",
    "        plt.plot(lint_intx(i1x, i1y, h4x, h4y,0.4,0.6),lint_inty(i1x,i1y,h4x,h4y,0.4,0.6)-0.1,color='orangered',lw=1.0,zorder=1)\n",
    "        plt.plot(lint_int_arrx(h4x,h4y,i1x,i1y,0.4,0.6,0.2),lint_int_arry(h4x,h4y,i1x,i1y,0.4,0.6,0.2)-0.1,color='orangered',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial \\lambda_{1,4}} =$' + r'${0:s}$'.format(as_si(dw14[iepoch]/lr,2)),(lintx(i1x,i1y,h4x,h4y,0.5)-0.6,linty(i1x,i1y,h4x,h4y,0.5)-1.0),size=7,zorder=200,color='orangered',\n",
    "                  bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-20)\n",
    "    \n",
    "        plt.plot(lint_intx(i2x, i2y, h4x, h4y,0.3,0.5),lint_inty(i2x,i2y,h4x,h4y,0.3,0.5)-0.1,color='red',lw=1.0,zorder=1)\n",
    "        plt.plot(lint_int_arrx(h4x,h4y,i2x,i2y,0.5,0.7,0.2),lint_int_arry(h4x,h4y,i2x,i2y,0.5,0.7,0.2)-0.12,color='red',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial \\lambda_{2,4}} =$' + r'${0:s}$'.format(as_si(dw24[iepoch]/lr,2)),(lintx(i2x,i2y,h4x,h4y,0.5)-1.05,linty(i2x,i2y,h4x,h4y,0.5)-0.7),size=7,zorder=200,color='red',\n",
    "                  bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=12)\n",
    "    \n",
    "        plt.plot(lint_intx(i3x, i3y, h4x, h4y,0.2,0.4),lint_inty(i3x,i3y,h4x,h4y,0.2,0.4)-0.1,color='darkred',lw=1.0,zorder=1)\n",
    "        plt.plot(lint_int_arrx(h4x,h4y,i3x,i3y,0.5,0.8,0.2),lint_int_arry(h4x,h4y,i3x,i3y,0.5,0.8,0.2)-0.12,color='darkred',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial \\lambda_{3,4}} =$' + r'${0:s}$'.format(as_si(dw34[iepoch]/lr,2)),(lintx(i3x,i3y,h4x,h4y,0.5)-1.7,linty(i3x,i3y,h4x,h4y,0.5)-1.7),size=7,zorder=200,color='darkred',\n",
    "                  bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=38)\n",
    "    \n",
    "        plt.plot(lint_intx(i3x, i3y, h5x, h5y,0.4,0.6),lint_inty(i3x,i3y,h5x,h5y,0.4,0.6)-0.1,color='darkblue',lw=1.0,zorder=1)\n",
    "        plt.plot(lint_int_arrx(h5x,h5y,i3x,i3y,0.4,0.6,0.2),lint_int_arry(h5x,h5y,i3x,i3y,0.4,0.6,0.2)-0.12,color='darkblue',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial \\lambda_{3,5}} =$' + r'${0:s}$'.format(as_si(dw35[iepoch]/lr,2)),(lintx(i3x,i3y,h5x,h5y,0.5)-0.4,linty(i3x,i3y,h5x,h5y,0.5)-0.6),size=7,zorder=200,color='darkblue',\n",
    "                  bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=20)\n",
    "    \n",
    "        plt.plot(lint_intx(i2x, i2y, h5x, h5y,0.3,0.5),lint_inty(i2x,i2y,h5x,h5y,0.3,0.5)-0.1,color='blue',lw=1.0,zorder=1)\n",
    "        plt.plot(lint_int_arrx(h5x,h5y,i2x,i2y,0.3,0.7,0.2),lint_int_arry(h5x,h5y,i2x,i2y,0.3,0.7,0.2)-0.12,color='blue',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial \\lambda_{2,5}} =$' + r'${0:s}$'.format(as_si(dw25[iepoch]/lr,2)),(lintx(i2x,i2y,h5x,h5y,0.5)-1.2,linty(i2x,i2y,h5x,h5y,0.5)-0.65),size=7,zorder=200,color='blue',\n",
    "                  bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-12)\n",
    "    \n",
    "        plt.plot(lint_intx(i1x, i1y, h5x, h5y,0.2,0.4),lint_inty(i1x,i1y,h5x,h5y,0.2,0.4)-0.1,color='dodgerblue',lw=1.0,zorder=1)\n",
    "        plt.plot(lint_int_arrx(h5x,h5y,i1x,i1y,0.2,0.8,0.2),lint_int_arry(h5x,h5y,i1x,i1y,0.2,0.8,0.2)-0.12,color='dodgerblue',lw=1.0,zorder=1)\n",
    "        plt.annotate(r'$\\frac{\\partial P}{\\partial \\lambda_{1,5}} =$' + r'${0:s}$'.format(as_si(dw15[iepoch]/lr,2)),(lintx(i1x,i1y,h5x,h5y,0.5)-2.2,linty(i1x,i1y,h4x,h4y,0.5)-0.2),size=7,zorder=200,color='dodgerblue',\n",
    "                  bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-36,xycoords = 'data',va=\"top\",ha=\"left\")\n",
    "    \n",
    "    if forward == False:\n",
    "    \n",
    "        plt.annotate(r'$\\lambda_{1,4}$ = ' + str(np.round(w14[iepoch],2)),((i1x+h4x)*0.45,(i1y+h4y)*0.5+0.7),size=8,zorder=200,color='orangered',\n",
    "                     bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-18,xycoords = 'data',va=\"top\",ha=\"left\") \n",
    "        plt.annotate(r'$\\lambda_{2,4}$ = ' + str(np.round(w24[iepoch],2)),((i2x+h4x)*0.45-0.3,(i2y+h4y)*0.5-0.03),size=8,zorder=200,color='red',\n",
    "                     bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=12) \n",
    "        plt.annotate(r'$\\lambda_{3,4}$ = ' + str(np.round(w34[iepoch],2)),((i3x+h4x)*0.45-1.2,(i3y+h4y)*0.5-1.1),size=8,zorder=200,color='darkred',\n",
    "                     bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=38) \n",
    "        \n",
    "        plt.annotate(r'$\\lambda_{1,5}$ = ' + str(np.round(w15[iepoch],2)),((i1x+h5x)*0.55-2.5,(i1y+h5y)*0.5+0.7),size=8,zorder=200,color='dodgerblue',\n",
    "            bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-36) \n",
    "        plt.annotate(r'$\\lambda_{2,5}$ = ' + str(np.round(w25[iepoch],2)),((i2x+h5x)*0.55-1.5,(i2y+h5y)*0.5+0.05),size=8,zorder=200,color='blue',\n",
    "                     bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-12) \n",
    "        plt.annotate(r'$\\lambda_{3,5}$ = ' + str(np.round(w35[iepoch],2)),((i3x+h5x)*0.55-1.0,(i3y+h5y)*0.5+0.1),size=8,zorder=200,color='darkblue',\n",
    "                     bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=18) \n",
    "        \n",
    "        plt.annotate(r'$\\lambda_{4,6}$ = ' + str(np.round(w46[iepoch],2)),((h4x+o6x)*0.47,(h4y+o6y)*0.47+0.39),size=8,zorder=200,color='orange',\n",
    "            bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-12) \n",
    "        plt.annotate(r'$\\lambda_{5,6}$ = ' + str(np.round(w56[iepoch],2)),((h5x+o6x)*0.47,(h5y+o6y)*0.47+0.26),size=8,zorder=200,color='darkorange',\n",
    "            bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=12) \n",
    "        \n",
    "    if forward == True:\n",
    "     \n",
    "        plt.annotate(r'$\\lambda_{1,4} \\times X_1$ = ' + str(np.round(x1*w14[iepoch],2)),((i1x+h4x)*0.45,(i1y+h4y)*0.5+0.70),size=8,zorder=200,color='orangered',\n",
    "                     bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-18,xycoords = 'data',va=\"top\",ha=\"left\") \n",
    "        plt.annotate(r'$\\lambda_{2,4} \\times X_2$ = ' + str(np.round(x2*w24[iepoch],2)),((i2x+h4x)*0.45-0.3,(i2y+h4y)*0.5+0.70),size=8,zorder=200,color='red',\n",
    "                     bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=10,xycoords = 'data',va=\"top\",ha=\"left\") \n",
    "        plt.annotate(r'$\\lambda_{3,4} \\times X_3$ = ' + str(np.round(x3*w34[iepoch],2)),((i3x+h4x)*0.45-1.4,(i3y+h4y)*0.5-1.3),size=8,zorder=200,color='darkred',\n",
    "                     bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=36,xycoords = 'data',va=\"bottom\",ha=\"left\") \n",
    "        \n",
    "        plt.annotate(r'$\\lambda_{1,5} \\times X_1$ = ' + str(np.round(x1*w15[iepoch],2)),((i1x+h5x)*0.55-2.5,(i1y+h5y)*0.5+1.9),size=8,zorder=200,color='dodgerblue',\n",
    "            bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-36,xycoords = 'data',va=\"top\",ha=\"left\") \n",
    "        plt.annotate(r'$\\lambda_{2,5} \\times X_2$ = ' + str(np.round(x2*w25[iepoch],2)),((i2x+h5x)*0.55-1.5,(i2y+h5y)*0.5+0.7),size=8,zorder=200,color='blue',\n",
    "                     bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-12,xycoords = 'data',va=\"top\",ha=\"left\") \n",
    "        plt.annotate(r'$\\lambda_{3,5} \\times X_3$ = ' + str(np.round(x3*w35[iepoch],2)),((i3x+h5x)*0.55-1.0,(i3y+h5y)*0.5+1.1),size=8,zorder=200,color='darkblue',\n",
    "                     bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=18,xycoords = 'data',va=\"top\",ha=\"left\") \n",
    "        \n",
    "        plt.annotate(r'$\\lambda_{4,6} \\times Y_4$ = ' + str(np.round(y4[iepoch]*w46[iepoch],2)),((h4x+o6x)*0.47,(h4y+o6y)*0.47+1.0),size=8,zorder=200,color='orange',\n",
    "            bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=-12,xycoords = 'data',va=\"top\",ha=\"left\") \n",
    "        plt.annotate(r'$\\lambda_{5,6} \\times Y_5$ = ' + str(np.round(y5[iepoch]*w56[iepoch],2)),((h5x+o6x)*0.47,(h5y+o6y)*0.47+1.0),size=8,zorder=200,color='darkorange',\n",
    "            bbox=dict(boxstyle=\"round,pad=0.0\", edgecolor='white', facecolor='white', alpha=1.0),rotation=12,xycoords = 'data',va=\"top\",ha=\"left\") \n",
    "    \n",
    "\n",
    "    \n",
    "    plt.plot([0.5,20,20,0.5,0.5],[-1,-1,10,10,-1],color='black')\n",
    "    #plt.scatter(0,10,color='yellow')\n",
    "    \n",
    "    plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=1.0, wspace=0.2, hspace=0.2); plt.show()\n",
    "       \n",
    "# connect the function to make the samples and plot to the widgets    \n",
    "interactive_plot = widgets.interactive_output(run_plot, {'nepoch':nepoch,'lr':lr,'weights':weights,'forward':forward,'back':back,'seed':seed})\n",
    "interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating   "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "554bb4ff",
   "metadata": {},
   "source": [
    "### Interactive Machine Learning Norms Demonstation \n",
    "\n",
    "#### Michael Pyrcz, Professor, The University of Texas at Austin \n",
    "\n",
    "Interactive artificial neural network dashboard built from scratch to demonstrate, parameter initialization and training. \n",
    "\n",
    "### The Inputs\n",
    "\n",
    "* **$n_{epoch}$** - number of training cycles, **$\\eta$** - learning rate, **$S$** - random number seed\n",
    "* **Weights** - show the network weights, **Forward Pass** - show the contributions and node outputs, **Back Propogation** - show the partial derivatives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "873fbf10",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "acf0334dbd644abe885f2d8c41a160e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Text(value='                                             Machine Learning Simple Artificial Neu…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "198b3e914e0947a7b50b92e7c82666d2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(ui, interactive_plot)                           # display the interactive plot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d3a38fd",
   "metadata": {},
   "source": [
    "Note, I am still making improvements. For now it is best to select weights, forward pass or back propogation and not a combination of two of the choices."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1b27996",
   "metadata": {},
   "source": [
    "#### Comments\n",
    "\n",
    "This was an interactive demonstration of an artifial neural network. Providing students an opportunity to play with machine learning, deep learning for experiential learning.\n",
    "  \n",
    "#### The Author:\n",
    "\n",
    "### Michael Pyrcz, Professor, The University of Texas at Austin \n",
    "*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*\n",
    "\n",
    "With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development. \n",
    "\n",
    "For more about Michael check out these links:\n",
    "\n",
    "#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)\n",
    "\n",
    "#### Want to Work Together?\n",
    "\n",
    "I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.\n",
    "\n",
    "* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you! \n",
    "\n",
    "* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!\n",
    "\n",
    "* I can be reached at mpyrcz@austin.utexas.edu.\n",
    "\n",
    "I'm always happy to discuss,\n",
    "\n",
    "*Michael*\n",
    "\n",
    "Michael Pyrcz, Ph.D., P.Eng. Professor, Cockrell School of Engineering and The Jackson School of Geosciences, The University of Texas at Austin\n",
    "\n",
    "#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)  \n",
    "  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}