News for hackers. Stuff that matters.
Re:Generate Hacker News Comments from Titles by leod on Sunday April 21, 2019 @05:18PM (#19714644)
Thank you!

The model weighs in at 1.2GB with 100M parameters, which is similar to the smallest GPT-2 model.

I wouldn't be suprised if GPT-2 small (+ finetuning on HN data) performed better than what I have trained. Other than hyperparameters, I think there are two main differences: First, I pretrained the model solely on Wikipedia data, while GPT-2 used more general web data. Second, I used an encoder-decoder model, while GPT-2 is a language model. I'm suspecting that the encoder is not very useful for this task.

There are no comments on this comment yet.