Google has open-sourced Performer, a Transformer deep-learning architecture that scales linearly with input sequence length. This allows Performer to be used for tasks that require long sequences, including pixel-prediction and protein sequence modeling.
A team from Google Research described the model and several experiments in a paper published on arXiv. The Performer uses a generalized attention mechanism called Fast Attention Via positive Orthogonal Random features (FAVOR+) to accurately estimate the standard softmax attention used in the popular Transformer model, reducing the space and time complexity from quadratic to linear.
