week02_classification/homework_part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Salary prediction, episode II: make it actually work (4 points)\n",
    "\n",
    "Your main task is to use some of the tricks you've learned on the network and analyze if you can improve __validation MAE__. Try __at least 3 options__ from the list below for a passing grade. Write a short report about what you have tried. More ideas = more bonus points. \n",
    "\n",
    "__Please be serious:__ \" plot learning curves in MAE/epoch, compare models based on optimal performance, test one change at a time. You know the drill :)\n",
    "\n",
    "You can use either __pytorch__ or __tensorflow__ or any other framework (e.g. pure __keras__). Feel free to adapt the seminar code for your needs. For tensorflow version, consider `seminar_tf2.ipynb` as a starting point.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# < A whole lot of your code > - models, charts, analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A short report\n",
    "\n",
    "Please tell us what you did and how did it work.\n",
    "\n",
    "`<YOUR_TEXT_HERE>`, i guess..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recommended options\n",
    "\n",
    "#### A) CNN architecture\n",
    "\n",
    "All the tricks you know about dense and convolutional neural networks apply here as well.\n",
    "* Dropout. Nuff said.\n",
    "* Batch Norm. This time it's `nn.BatchNorm*`/`L.BatchNormalization`\n",
    "* Parallel convolution layers. The idea is that you apply several nn.Conv1d to the same embeddings and concatenate output channels.\n",
    "* More layers, more neurons, ya know...\n",
    "\n",
    "\n",
    "#### B) Play with pooling\n",
    "\n",
    "There's more than one way to perform pooling:\n",
    "* Max over time (independently for each feature)\n",
    "* Average over time (excluding PAD)\n",
    "* Softmax-pooling:\n",
    "$$ out_{i, t} = \\sum_t {h_{i,t} \\cdot {{e ^ {h_{i, t}}} \\over \\sum_\\tau e ^ {h_{j, \\tau}} } }$$\n",
    "\n",
    "* Attentive pooling\n",
    "$$ out_{i, t} = \\sum_t {h_{i,t} \\cdot Attn(h_t)}$$\n",
    "\n",
    ", where $$ Attn(h_t) = {{e ^ {NN_{attn}(h_t)}} \\over \\sum_\\tau e ^ {NN_{attn}(h_\\tau)}}  $$\n",
    "and $NN_{attn}$ is a dense layer.\n",
    "\n",
    "The optimal score is usually achieved by concatenating several different poolings, including several attentive pooling with different $NN_{attn}$ (aka multi-headed attention).\n",
    "\n",
    "The catch is that keras layers do not inlude those toys. You will have to [write your own keras layer](https://keras.io/layers/writing-your-own-keras-layers/). Or use pure tensorflow, it might even be easier :)\n",
    "\n",
    "#### C) Fun with words\n",
    "\n",
    "It's not always a good idea to train embeddings from scratch. Here's a few tricks:\n",
    "\n",
    "* Use a pre-trained embeddings from `gensim.downloader.load`. See last lecture.\n",
    "* Start with pre-trained embeddings, then fine-tune them with gradient descent. You may or may not download pre-trained embeddings from [here](http://nlp.stanford.edu/data/glove.6B.zip) and follow this [manual](https://keras.io/examples/nlp/pretrained_word_embeddings/) to initialize your Keras embedding layer with downloaded weights.\n",
    "* Use the same embedding matrix in title and desc vectorizer\n",
    "\n",
    "\n",
    "#### D) Going recurrent\n",
    "\n",
    "We've already learned that recurrent networks can do cool stuff in sequence modelling. Turns out, they're not useless for classification as well. With some tricks of course..\n",
    "\n",
    "* Like convolutional layers, LSTM should be pooled into a fixed-size vector with some of the poolings.\n",
    "* Since you know all the text in advance, use bidirectional RNN\n",
    "  * Run one LSTM from left to right\n",
    "  * Run another in parallel from right to left \n",
    "  * Concatenate their output sequences along unit axis (dim=-1)\n",
    "\n",
    "* It might be good idea to mix convolutions and recurrent layers differently for title and description\n",
    "\n",
    "\n",
    "#### E) Optimizing seriously\n",
    "\n",
    "* You don't necessarily need 100 epochs. Use early stopping. If you've never done this before, take a look at [early stopping callback(keras)](https://keras.io/callbacks/#earlystopping) or in [pytorch(lightning)](https://pytorch-lightning.readthedocs.io/en/latest/common/early_stopping.html).\n",
    "  * In short, train until you notice that validation\n",
    "  * Maintain the best-on-validation snapshot via `model.save(file_name)`\n",
    "  * Plotting learning curves is usually a good idea\n",
    "  \n",
    "Good luck! And may the force be with you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}