arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Computer Science · cs.LG

GLU Variants Improve Transformer

Noam Shazeer

See on the map

Abstract

Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer (arXiv:1706.03762) sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations.

References

The reference list for this paper has not been collected yet.

Cited by 266

266 papers in the index cite this (Semantic Scholar counts 2,294 in total).

The cited-by list is still filling in while reference lists are collected.

More like this