We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:

Download:

Current browse context:

cs.CL

Change to browse by:

References & Citations

DBLP - CS Bibliography

Bookmark

(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Computer Science > Computation and Language

Title: New Approaches to Long Document Summarization: Fourier Transform Based Attention in a Transformer Model

Abstract: In this work, we extensively redesign the newly introduced method of token mixing using Fourier Transforms (FNET) to replace the computationally expensive self-attention mechanism in a full transformer implementation on a long document summarization task (> 512 tokens). As a baseline, we also carried out long document summarization using established methods such as Longformer and Big Bird transformer models that are capable of processing over 8000 tokens and are currently the state of the art methods for these type of problems. The original FNET paper implemented this in an encoder only architecture while abstractive summarization requires both an encoder and a decoder. Since such a pretrained transformer model does not currently exist in the public domain, we decided to implement a full transformer based on this Fourier token mixing approach in an encoder/decoder architecture which we trained starting with Glove embeddings for the individual words in the corpus. We investigated a number of different extensions to the original FNET architecture and evaluated them on their Rouge F1-score performance on a summarization task. All modifications showed better performance on the summarization task than when using the original FNET encoder in a transformer architecture.
Comments: 7 pages, 4 figures, 5 tables
Subjects: Computation and Language (cs.CL); Machine Learning (cs.LG)
Cite as: arXiv:2111.15473 [cs.CL]
  (or arXiv:2111.15473v1 [cs.CL] for this version)

Submission history

From: Andrew Kiruluta [view email]
[v1] Thu, 25 Nov 2021 18:03:41 GMT (807kb,D)

Link back to: arXiv, form interface, contact.