Studying NLP

Start Date : 2021/11/18

• Materials
  • Coursera NLP
  • HuggingFace
  • FastCampus NLP

1. Review Neural Network and Pytorch

date	conents
2021/11/18 Thu	MNIST with Pytorch	Code
2021/11/19 Fri	Basic Mathematics, Auto Encoder, Manifold Hypothesis	Code1, Code2
2021/11/20 Sat	Data Loader, Pytorch Ignite, CNN, VCG, ResNet, Transfer Learning	Code1, Code2, Code3, Code4
2021/11/21 Sun	RNN, BBTT, LSTM, Gradient Clipping	Code

2. NLP Basic Concepts

date	contents
2021/11/22 Mon	Basic Concept, Regex	Code
2021/11/23 Tue	Corpus Labeling, Tokenization, Subword Segmentation(BPE), Detokenization, TorchText, WordSense, WordNet, TF-IDF	Code1, Code2, Code3, Code4, Code5
2021/11/24 Wed	Word2Vec(Skip Gram), GloVe, FastText	Code
2021/11/25 Thu	Text Classification with RNN	Code
2021/11/26 Fri	Text Classification with CNN	Code
2021/11/27 Sat	Text Classification with FastText	Code

3. Natural Language Generation

date	contents
2021/11/27 Sat	Language Modeling(N-gram), Smoothing and Discounting, Interpolation and Backoff, Perplexity, RNN Language Modeling, Teacher Forcing
2021/11/28 Sun	Data Preparation(AI Hub 'Kor-Eng Paired Data' for building machine translator)	Code
2021/11/29 Mon	Sequence to Sequence, Attention, Masking, Input Feeding, Pack Sequence, Gradient Accumulation	Code, Code
2021/11/30 Tue	AMP, Seq2Seq Implementation	Code
2021/12/1 Wed	Seq2Seq Implementation	Code
2021/12/2 Thu	Trainer.py, Train.py, Data Loader, continue_train.py	Code, Code
2021/12/3 Fri	Inference, Greedy Search, Penalty, Search Function, Translate.py	Code
2021/12/4 Sat	Evaluations(BLEU), Beam Search(Continue)	Code
2021/12/5 Sun	Beam Search, Transformer, Rectified Adam, On Layer Normalization(Xiong et al., 2020)	Code
2021/12/6 Mon	Transformer Implementation/Code Review	Code
2021/12/7 Tue	Transformer Implementation(Search & Beam Search)/Code Review	Code
2021/12/8 Wed	Intro to Advanced topics on NLG(Model Ensemble, Back/Copied Translation)
2021/12/9 Thu	REINFORCE algorithm(with Baseline), Minimum Risk Training for Neural Machine Translation
2021/12/10 Fri	Minimum Risk Training, Reinforcement Learning Trainer	Code
2021/12/11 Sat	Reinforcement Learning Code Implementation	Code, Code
2021/12/12 Sun	Dual Learning
2021/12/13 Mon	Dual Supervised Learning Implementation	Code
2021/12/14 Tue	Dual Supervised Learning Implementation	Code, Code, Code, Code, Code
2021/12/15 Wed	Dual Learning for Machine Translation(Cycle GAN idea)

4.PLM, BERT, GPT, HuggingFace🤗

date	contents
2021/12/16 Thu	Introduction to PLM
2021/12/17 Fri	Transfer Learning, Self-Supervised Learning, Embedding, ELMo
2021/12/20 Mon	Transformer
2021/12/31 Fri	Seq2Seq, Transformer
2021/01/02 Sun	Pre-trained Language Model(GPT, BERT, BART)
2021/01/04 Tue	Huggingface Transformers, BERT Trainer, Pytorch Ignite	Code, Code, Code, Code
2021/01/05 Wed	Huggingface Trainer, ALBERT, Knowledge Distillation, Compressed PLMs via Knowledge Distillation, Tiny BERT	Code
2021/01/06 Thu	Text to Text framework(Google T5), GPT-3(Language Models are Few-Shot Learners), GPT crush

Coursera NLP

1. Natural Language Processing with Classification and Vector Spaces

date	contents
2021/01/07 Fri	Sentiment Analysis with Logistic Regression	Code, Code, Code
2021/01/08 Sat	Week 1 Assignment, Sentiment Analysis with Naïve Bayes, Week 2 Assignment	Code, Code, Code
2021/01/09 Sun	Vector Space Models, Week 3 Assignment	Code, Code, Code, Code
2021/01/10 Mon	Machine Translation, Document Search, Week 4 Assignment	Code, Code, Code

2. Natural Language Processing with Probabilistic Models

date	contents
2021/01/11 Tue	Autocorrect and Minimum edit distance, Week 1 Assignment(Autocorrect)	Code, Code, Code
2021/01/12 Wed	Part of Speech Tagging	Code, Code
2022/01/13 Thu	Part of Speech Tagging
2022/01/14 Fri	Viterbi Algorithm, Week 2 Assignment	Code
2022/01/15 Sat	Autocomplete	Code, Code
2022/01/16 Sun	Autocomplete, Perplexity, Week 3 Assignment	Code, Code
2022/01/17 Mon	Word Embeddings
2022/01/18 Tue	Word Embeddings	Code, Code, Code, Code, Code
2022/01/19 Wed	Week 4 Assignment	Code

3. Natural Language Processing with Sequence Models

date	contents
2022/01/19 Wed	Neural Networks for Sentiment Analysis, Trax	Code, Code, Code
2022/01/21 Fri	Neural Networks for Sentiment Analysis	Code
2022/01/22 Sat	Week 1 Assignment, N-grams vs. Sequence Models	Code, Code, Code, Code, Code
2022/01/23 Sun	Week 2 Assignment, LSTMs and Named Entity Recognition, Gradient Descent Additional, Week 3 Assignment	Code, Code
2022/01/24 Mon	Siamese Networks	Code, Code
2022/01/25 Tue	Siamese Networks, Week 4 Assignment	Code

4. Natural Language Processing with Attention Models

date	contents
2022/01/26 Wed	Neural Machine Translation	Code
2022/01/27 Thu	Neural Machine Translation
2022/01/28 Fri	Neural Machine Translation
2022/01/30 Sun	Neural Machine Translation	Code, Code
2022/01/31 Mon	Week 1 Assignment
2022/02/01 Tue	Week 1 Assignment	Code
2022/02/02 Wed	Text Summarization, T5 Demo, Attention is all you need	Code, Code
2022/02/03 Thu	Week 2 Assignment	Code
2022/02/04 Fri	Question Answering
2022/02/16 Wed	Question Answering, BPE	Code
2022/02/24 Thu	HuggingFace, Week 3 Assignment	Code, Code, Code
2022/02/25 Fri	DSBA review for Reformer