Skip to content

Commit

Permalink
Add a paper about BERT
Browse files Browse the repository at this point in the history
  • Loading branch information
@paul356
paul356 committed Jan 12, 2024
1 parent 2c0cbe2 commit 40fe34f
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 5 deletions.
11 changes: 6 additions & 5 deletions _org/2024-01-02-jan-papers.org
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ nav_order: {{ page.date }}
---
#+END_EXPORT

|------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+----------------------------------------------------------------------------------------------------------------|
| Title | Authors | Synthesis | Publisher | Keywords |
|------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+----------------------------------------------------------------------------------------------------------------|
|------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+----------------------------------------------------------------------------------------------------------------|
| Title | Authors | Synthesis | Publisher | Keywords |
|------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+----------------------------------------------------------------------------------------------------------------|
| An End-to-End Learning-based Cost Estimator | Ji Sun, Guoliang Li | This paper shows a cardinality and cost estimation method which takes trees of query plans as inputs and give the estimation of the query cost and cardinalities. The authors show how to encode query operators, predicates, metadata, sample bitmaps into vectors. These vectors go through a embedding method, then through an estimation network to give the estimation of query costs. An abolation study is conducted to show the effectiveness of string embedding choices, the tree structure model and bitmap sampling. | PVLDB Vol 13, November 2019 | Representation Learning, Long Short-Term Memory, String Embedding, Query Cost Estimation, Tree Structure Model |
| Adapative Partitioning and Indexing for In-Situ Query Processing | Matthaios Olma, Anastasia Ailamaki, etc. | This paper presents a online partitioning and indexing framework for in-situ query processing. The framework consists of partition managers, index managers, and a statistics store. The partition managers can generate logical partitions on the fly. The Index Manager will calculate if it is beneficial to build an appropriate index for a partiton. | VLDB Journal 2020 | Online Partitioning, Online Indexing, In-Situ Query Processing |
| Plan-Structured Deep Neural Network Models for Query Performance Prediction | Ryan Marcus, Olga Papaemmanouil | This paper is the first to propose a plan-structured DNN model for query performance prediction. It explains the architecture for the plan-structure DNN. Every operator has a corresponding DNN unit which takes input from its children and gives the performance estimation. The same operators share the network structure and weight. | PVLDB 2019 | Query Performance Estimation, Plan-Structured DNN |
Expand All @@ -21,6 +21,7 @@ nav_order: {{ page.date }}
| How Good Are Query Optimizers, Really? | Viktor Leis, Thomas Neumann, etc. | This paper investigates how query optimizers perform on cardinality estimation, cost estimation, plan enumeration, and it shows how these query optimizers perform on a more realistic dataset. Based on experiments the authors find that cost model errors are dwarfed by cartinality estimation errors, and estimation errors increase along with the number of join relations. Dynamic query plan optimization during query execution can effectively counteract bad query plans. This paper shows the effectiveness of different design considerations and suggests worthwhile research directions. | PVLDB Vol 9, 2015 | Query Optimizer, Cardinality Estimation |
| Attention Is All You Need | Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, LLion Jones, Adian N. Gomez, Lukasz Kaiser, Illia Polosukhin | This is the original paper which propose Transformer the model that forms the network core of many Large Language Models. In order to improve network parallelization the authors suggest to use attention mechanism to learn the relations between sequence poisitions in parallel. But without RNN structure positional encoding is used to inject the positional information. Transformer can outperform many state of art RNN and convolutional models. | NIPS 2017 | Transformer, RNN, Attention |
| Cardinality Estimation: An Experimental Survey | Hazar Harmouch, Felix Naumann | This paper invetigates 12 cardinality estimation algorithms. They are FM, PCSA, AMS, BJKST, LogLog, SuperLogLog, HyperLogLog, HyperLogLog++, MinCount, AKMV, LC, and BF. The authors divide them into four categories - counting trailing 1s, counting leading 0s, kth minimum value and linear synopses. And they compare their accuracy and resource requirements. FM, BJKST, AKMV and BF are the best in their class judging by accuracy. | VLDB 2017 | Cardinality Estimation, LogLog, HyperLogLog, MinCount, Bloom Filter |
| | | | | |
|------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+----------------------------------------------------------------------------------------------------------------|
| BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova | This paper improves the model perform by breaking the limit of a left-to-right Transformer architecture by learning from the whole text context. But in order to prevent the model to directly copy the predicted result. The authors randomly marsk 15% tokens of the input sequence and predict the masked tokens. In this way they can train a LM that can learn the representation from the whole context. | arXiv 2018 | BERT, Language Model, GPT |
| | | | | |
|------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+----------------------------------------------------------------------------------------------------------------|

1 change: 1 addition & 0 deletions _posts/2024-01-02-jan-papers.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,5 @@ nav_order: {{ page.date }}
| How Good Are Query Optimizers, Really? | Viktor Leis, Thomas Neumann, etc. | This paper investigates how query optimizers perform on cardinality estimation, cost estimation, plan enumeration, and it shows how these query optimizers perform on a more realistic dataset. Based on experiments the authors find that cost model errors are dwarfed by cartinality estimation errors, and estimation errors increase along with the number of join relations. Dynamic query plan optimization during query execution can effectively counteract bad query plans. This paper shows the effectiveness of different design considerations and suggests worthwhile research directions. | PVLDB Vol 9, 2015 | Query Optimizer, Cardinality Estimation |
| Attention Is All You Need | Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, LLion Jones, Adian N. Gomez, Lukasz Kaiser, Illia Polosukhin | This is the original paper which propose Transformer the model that forms the network core of many Large Language Models. In order to improve network parallelization the authors suggest to use attention mechanism to learn the relations between sequence poisitions in parallel. But without RNN structure positional encoding is used to inject the positional information. Transformer can outperform many state of art RNN and convolutional models. | NIPS 2017 | Transformer, RNN, Attention |
| Cardinality Estimation: An Experimental Survey | Hazar Harmouch, Felix Naumann | This paper invetigates 12 cardinality estimation algorithms. They are FM, PCSA, AMS, BJKST, LogLog, SuperLogLog, HyperLogLog, HyperLogLog++, MinCount, AKMV, LC, and BF. The authors divide them into four categories - counting trailing 1s, counting leading 0s, kth minimum value and linear synopses. And they compare their accuracy and resource requirements. FM, BJKST, AKMV and BF are the best in their class judging by accuracy. | VLDB 2017 | Cardinality Estimation, LogLog, HyperLogLog, MinCount, Bloom Filter |
| BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova | This paper improves the model perform by breaking the limit of a left-to-right Transformer architecture by learning from the whole text context. But in order to prevent the model to directly copy the predicted result. The authors randomly marsk 15% tokens of the input sequence and predict the masked tokens. In this way they can train a LM that can learn the representation from the whole context. | arXiv 2018 | BERT, Language Model, GPT |
| | | | | |

0 comments on commit 40fe34f

Please sign in to comment.