PromptPapers

We have released an open-source prompt-learning toolkit, check out OpenPrompt!

We strongly encourage the researchers that want to promote their fantastic work to the community to make pull request to update their paper's information! (See contributing details)

Contents

Must-read papers on prompt-based tuning for pre-trained language models. The paper list is mainly mantained by Ning Ding and Shengding Hu. Watch this repository for the latest updates!

• PromptPapers
  • Introduction
    • Keywords Convention
  • Papers
    • Overview(Survey)
    • Pilot Work
    • Basics
    • Analysis
    • Improvements
    • Specializations
  • Contribution
    • Other contributors
    • Contributing to this paper list

Introduction

This is a paper list about prompt-based tuning for large-scale pre-trained language models. Different from traditional fine-tuning that uses an explicit classifier, prompt-based tuning directly uses the pre-trained models to conduct the pre-training tasks for classification or regression.

Keywords Convention

The abbreviation of the work.

The key features in terms of prompt learning used in the work.

The main explored task of the work.

The main explored property of prompt learning methods in the work.

Papers

Overview

This section contains the papers that overview the general trends in recent natural language processing with big (pretrained) models.

1. OpenPrompt: An Open-source Framework for Prompt-learning. Preprint.

   Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Hai-Tao Zheng, Maoson Sun [pdf] [project], 2021.11

2. Pre-Trained Models: Past, Present and Future. Preprint.

   Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, Jun Zhu. [pdf], 2021.6

3. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. Preprint.

   Liu, Pengfei, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. [pdf] [project], 2021.7

4. Paradigm Shift in Natural Language Processing. Preprint.

   Tianxiang Sun, Xiangyang Liu, Xipeng Qiu, Xuanjing Huang [pdf], 2021.9

Pilot Work

This section contains the pilot works that might contributes to the prevalence of prompt learning paradigm.

1. Parameter-Efficient Transfer Learning for NLP. ICML 2019.

   Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly. [pdf], [project], 2019.6

2. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. [pdf], [project]. 2019.10.

3. Language Models as Knowledge Bases? EMNLP 2019.

   Fabio Petroni, Tim Rocktaschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, Sebastian Riedel. [pdf], [project] , 2019.9

4. How Can We Know What Language Models Know? TACL 2020.

   Zhengbao Jiang, Frank F. Xu, Jun Araki, Graham Neubig. [pdf], [project], 2019.11

5. Language Models are Few-shot Learners. NeurIPS 2020.

   Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei. [pdf], [website], 2020.5

Basics

This section contains the exploration on the basic aspects of prompt tuning, such as template, verbalizer, training paradigms, etc.

1. Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference. EACL 2021.

   Timo Schick, Hinrich Schütze. [pdf], [project], 2020.1

2. It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. NAACL 2021.

   Timo Schick, Hinrich Schütze. [pdf], [project], 2020.9

3. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. Preprint.

   Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, Sameer Singh. [pdf], [website], 2020.10

4. Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification. COLING 2020.

   Timo Schick, Helmut Schmid, Hinrich Schütze. [pdf], [project], 2020.12

5. Making Pre-trained Language Models Better Few-shot Learners. ACL 2021.

   Tianyu Gao, Adam Fisch, Danqi Chen. [pdf], [project], 2020.12

6. Prefix-tuning: Optimizing continuous prompts for generation. ACL 2021.

   Xiang Lisa Li, Percy Liang. [pdf], [project], 2021.1

7. Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. Preprint.

   Laria Reynolds, Kyle McDonell. [pdf], 2021.2

8. Improving and Simplifying Pattern Exploiting Training. Preprint.

   Derek Tam, Rakesh R Menon, Mohit Bansal, Shashank Srivastava, Colin Raffel. [pdf], 2021.3

9. GPT understands, too. Preprint.

   Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, Jie Tang. [pdf], [project], 2021.3

10. The Power of Scale for Parameter-Efficient Prompt Tuning. Preprint.

    Brian Lester, Rami Al-Rfou, Noah Constant. [pdf], [project], 2021.4

11. Learning How to Ask: Querying LMs with Mixtures of Soft Prompts. NAACL 2021.

    Guanghui Qin, Jason Eisner. [pdf][project], 2021.4

12. Factual Probing Is [MASK]: Learning vs. Learning to Recall. NAACL 2021.

    Zexuan Zhong, Dan Friedman, Danqi Chen. [pdf], [project], 2021.4

13. Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models. Preprint.

    Robert L. Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, Sebastian Riedel. [pdf], 2021.6

14. WARP: Word-level Adversarial ReProgramming. ACL 2021.

    Karen Hambardzumyan, Hrant Khachatrian, Jonathan May. [pdf], [project], 2021.6

15. PTR: Prompt Tuning with Rules for Text Classification. Preprint.

    Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, Maosong Sun. [pdf], 2021.5

16. NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original Pre-training Task —— Next Sentence Prediction

    Yi Sun*, Yu Zheng*, Chao Hao, Hangping Qiu, [pdf], [project], 2021.9

17. Finetuned language models are zero-shot learners.

    ason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, Quoc V. Le. [pdf], 2021.9

18. PPT: Pre-trained Prompt Tuning for Few-shot Learning

    Yuxian Gu*, Xu Han*, Zhiyuan Liu, Minlie Huang. [pdf], 2021.9

19. Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners.

    Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, Huajun Chen. [pdf], 2021.10

20. Multitask Prompted Training Enables Zero-Shot Task Generalization.

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, Alexander M. Rush. [pdf], 2021.10

Analysis

This section contains the analysis of prompt learning methods, including but not limited to why does prompt learning work, various properties of prompt learning methods, limilation of prompt learning methods.

1. What Makes Good In-Context Examples for GPT-3?. Preprint.

   Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, Weizhu Chen. [pdf] 2021.1

2. How Many Data Points is a Prompt Worth? NAACL 2021.

   Teven Le Scao, Alexander M. Rush. [pdf], [project], 2021.3

3. Surface Form Competition-Why the Highest Probability Answer Isn’t Always Right. Preprint. Preprint.

   Ari Holtzman, Peter West, Vered Schwartz, Yejin Choi, Luke Zettlemoyer. [pdf][project], 2021.4

4. Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions. Preprint.

   Swaroop Mishra, Daniel Khashabi, Chitta Baral, Hannaneh Hajishirzi. [pdf], [project], 2021.4

5. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. Preprint.

   Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, Pontus Stenetorp. [pdf] 2021.4

6. Meta-tuning Language Models to Answer Prompts Better. Preprint.

   Ruiqi Zhong, Kristy Lee*, Zheng Zhang*, Dan Klein. [pdf] 2021.4

7. True Few-Shot Learning with Language Models. Preprint.

   Ethan Perez, Douwe Kiela, Kyunghyun Cho. [pdf], [project] 2021.5

8. Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning. Preprint.

   Colin Wei Sang Michael Xie Tengyu Ma [pdf], 2021.6

9. Do Prompt-Based Models Really Understand the Meaning of their Prompts? Preprint.

   Albert Webson, Ellie Pavlick. [pdf], [project] 2021.9

10. Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning. Preprint.

    Prasetya Ajie Utama, Nafise Sadat Moosavi, Victor Sanh, Iryna Gurevych. [pdf], 2021.9

11. Towards a Unified View of Parameter-Efficient Transfer Learning. Preprint.

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig. [pdf], 2021.10

12. Exploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning. Preprint.

    Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Zhiyuan Liu, Juanzi Li, Lei Hou,Peng Li, Maosong Sun, Jie Zhou [pdf]

Improvements

This section contains the improvement of the basic prompt tuning methods, include but not limitedd to using additional resources to improving the performances, making up the shortcomings of previous work or conducting prompt tuning in unsual ways.

1. Calibrate Before Use: Improving Few-Shot Performance of Language Models. Preprint.

   Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, Sameer Singh. [pdf], [project], 2021.2

2. Text Generation with Efficient (Soft) Q-Learning. Preprint.

   Han Guo, Bowen Tan, Zhengzhong Liu, Eric P. Xing, Zhiting Hu. [pdf], 2021.6

3. Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification. Preprint.

   Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juanzi Li, Maosong Sun. [pdf], [project], 2021.8

4. Noisy Channel Language Model Prompting for Few-Shot Text Classification. Preprint.

   Sewon Min, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer. [pdf], 2021.8

5. Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collection.

   Ruiqi Zhong, Kristy Lee* Zheng Zhang*, Dan Klein. [pdf], 2021.9

6. Revisiting Self-Training for Few-Shot Learning of Language Model. Preprint.

   Yiming Chen, Yan Zhang, Chen Zhang, Grandee Lee, Ran Cheng, Haizhou Li. [pdf], 2021.10

Specializations

This section contains the prompt learning methods designed for various NLP task.

1. Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2. Preprint.

   Gregor Betz, Kyle Richardson, Christian Voigt. [pdf] 2021.3

2. GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation. Preprint.

   Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, Woomyeong Park. [pdf] 2021.4

3. Constrained Language Models Yield Few-Shot Semantic Parsers. Preprint.

   Richard Shin, Christopher H. Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, Benjamin Van Durme. [pdf] 2021.4

4. Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction. EMNLP 2021.

   Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, Eneko Agirre. [pdf], 2021.4

5. PADA: A Prompt-based Autoregressive Approach for Adaptation to Unseen Domains Preprint.

   Eyal Ben-David, Nadav Oved, Roi Reichart. [pdf][project] 2021.5

6. Prompt-Learning for Fine-grained Entity Typing. Preprint.

   Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu, Juanzi Li, Hong-Gee Kim [pdf],2021.8

7. KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction. Preprint.

   Xiang Chen, Xin Xie, Ningyu Zhang, Jiahuan Yan, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, Huajun Chen.. [pdf], 2021.9

8. Exploring Prompt-based Few-shot Learning for Grounded Dialog Generation. Preprint.

   Chujie Zheng, Minlie Huang. [pdf], 2021.9

9. SentiPrompt: Sentiment Knowledge Enhanced Prompt-Tuning for Aspect-Based Sentiment Analysis. Preprint.

   Chengxi Li, Feiyu Gao, Jiajun Bu, Lu Xu, Xiang Chen, Yu Gu, Zirui Shao, Qi Zheng, Ningyu Zhang, Yongpan Wang, Zhi Yu. [pdf] 2021.9

10. Template-free Prompt Tuning for Few-shot NER. Preprint.

    Ruotian Ma*, Xin Zhou*, Tao Gui, Yiding Tan, Qi Zhang, Xuanjing Huang. [pdf], 2021.9

11. Learning to Prompt for Vision-Language Models. Preprint.

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. [pdf], 2021.9

12. CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models. Preprint.

    Yuan Yao*, Ao Zhang*, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun. [pdf], 2021.10

13. MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators. Preprint.

    Zhixing Tan, Xiangwen Zhang, Shuo Wang, Yang Liu. [pdf], 2021.10

14. Few-Shot Bot: Prompt-Based Learning for Dialogue Systems. Preprint.

    Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, Pascale Fung [pdf], 2021.10

15. Control Prefixes for Text Generation. Preprint.

    Jordan Clive, Kris Cao, Marek Rei. [pdf], 2021.10

16. The Power of Prompt Tuning for Low-Resource Semantic Parsing. Preprint.

    Nathan Schucher, Siva Reddy, Harm de Vries. [pdf], 2021.10

17. A Good Prompt Is Worth Millions of Parameters? Low-resource Prompt-based Learning for Vision-Language Models.

    Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, Xiang Ren. [pdf]

Contribution

Other contributors

We thank Yujia Qin, Xiachong Feng, Chenglei Si , Tianbao Xie for the paper recommendation.

Contributing to this paper list

• First, think about which category the work should belong to.
• Second, use the same format as the others to discribe the work. Note that there should be an empty line between the title and the authors list, and take care of the indentation.
• Then, add keywords tags. Add the pdf link of the paper. If it is an arxiv publication, we prefer /abs/ format to /pdf/ format.

Don't worry if you put all these wrong, we will fix them for you. Just contribute and promote your awesome work here!

If you recommended a work that wasn't yours, you will be added to the contributor list (be sure to provide your information in other contributors).