A new model and dataset for long-range memory

Re:A new model and dataset for long-range memory by hooande on Monday February 10, 2020 @04:25PM (#22293539)

I believe the transformer-xl pre-trained model can also be downloaded, to provide a similar long term memory functionality as the compression transformer. I don't have a direct link, but it's available via huggingface https://huggingface.co/transformers/pretrained_models.html

[ Parent | on HN ]

Re:A new model and dataset for long-range memory by gwern on Tuesday February 11, 2020 @01:49PM (#22301323)

Yeah. I didn't mention Transformer-XL because I'm not sure how much of a long-range dependency it actually learns to handle. The only papers I've seen on recurrency indicate that they tend to learn very short-range dependencies, while something like Reformer with direct access to thousands of timesteps seems more likely to actually be making use of them.

[ Parent | on HN ]

Re:A new model and dataset for long-range memory by ColanR on Monday February 10, 2020 @05:00PM (#22293857)

Wow, that's a lot of models. Thanks for pointing that out.

[ Parent | on HN ]