Current browse context:
cs.LG
Change to browse by:
References & Citations
Computer Science > Machine Learning
Title: Transformer Embeddings of Irregularly Spaced Events and Their Participants
(Submitted on 31 Dec 2021 (v1), last revised 6 May 2022 (this version, v3))
Abstract: The neural Hawkes process (Mei & Eisner, 2017) is a generative model of irregularly spaced sequences of discrete events. To handle complex domains with many event types, Mei et al. (2020a) further consider a setting in which each event in the sequence updates a deductive database of facts (via domain-specific pattern-matching rules); future events are then conditioned on the database contents. They show how to convert such a symbolic system into a neuro-symbolic continuous-time generative model, in which each database fact and the possible event has a time-varying embedding that is derived from its symbolic provenance.
In this paper, we modify both models, replacing their recurrent LSTM-based architectures with flatter attention-based architectures (Vaswani et al., 2017), which are simpler and more parallelizable. This does not appear to hurt our accuracy, which is comparable to or better than that of the original models as well as (where applicable) previous attention-based methods (Zuo et al., 2020; Zhang et al., 2020a).
Submission history
From: Chenghao Yang [view email][v1] Fri, 31 Dec 2021 20:00:29 GMT (5164kb,D)
[v2] Sun, 20 Mar 2022 00:11:45 GMT (9930kb,D)
[v3] Fri, 6 May 2022 06:03:14 GMT (4723kb,D)
Link back to: arXiv, form interface, contact.