The Annotated Transformer

Resource history | v1 (current) | created by janarez

Details

The Annotated Transformer

| created by janarez | Add resource "The Annotated Transformer"
Title
The Annotated Transformer
Type
Interactive
Created
2018-04-03
Description
The Transformer from “Attention is All You Need” has been on a lot of people’s minds over the last year. Besides producing major improvements in translation quality, it provides a new architecture for many other NLP tasks. The paper itself is very clearly written, but the conventional wisdom has been that it is quite difficult to implement correctly. In this post I present an “annotated” version of the paper in the form of a line-by-line implementation. I have reordered and deleted some sections from the original paper and added comments throughout. This document itself is a working notebook, and should be a completely usable implementation. In total there are 400 lines of library code which can process 27,000 tokens per second on 4 GPUs.
Link
https://nlp.seas.harvard.edu/2018/04/03/attention.html
Identifier
no value

authors

This resource has no history of related authors.

topics

discusses Transformer (machine learning model)
v1 | attached by janarez | Add resource "The Annotated Transformer"

resources

This resource has no history of related resources.