Computer Science · cs.CL
A Primer in BERTology: What we know about how BERT works
Anna Rogers, Olga Kovaleva, Anna Rumshisky
See on the mapAbstract
Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.
References
The reference list for this paper has not been collected yet.
Cited by 26
26 papers in the index cite this (Semantic Scholar counts 2,001 in total).
The cited-by list is still filling in while reference lists are collected.