News for hackers. Stuff that matters.
Re:OpenAI releases larger GPT-2 model by minimaxir on Saturday May 04, 2019 @08:40PM (#19830354)
Notably, the 345M model (1.5 GB on disk) is big enough that it's pushing the limits of conventional GPUs, and an alternative method of finetuning the model (https://github.com/nshepperd/gpt-2/commit/47df6da611716b4826...) has to be used to prevent the GPU from going OOM.

I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple (https://github.com/minimaxir/gpt-2-simple) to allow easy finetuning/generation, and am also working on a way to quickly build an API/client for easily deploying GPT-2 to production and generating text at scale and cost effectively. Even the 117M model, managing CPU and RAM performance is tricky.

But given the incredible results of just the 117M model (e.g. Hacker News titles from a retrained 117M model: https://github.com/minimaxir/hacker-news-gpt-2), I'm eager to put the 345M model through its paces.

Re:OpenAI releases larger GPT-2 model by jchw on Saturday May 04, 2019 @09:01PM (#19830427)
>(e.g. Hacker News titles from a retrained 117M model: https://github.com/minimaxir/hacker-news-gpt-2)

Wow, thats great. “The Bullshit Bubble” “Fuck you, Bootstrap” “We should give up on America” - they’re practically comedy, yet very believable too.

Re:OpenAI releases larger GPT-2 model by Judgmentality on Saturday May 04, 2019 @10:48PM (#19830797)
"How to Make a T-Shirt in 10,000 hours" is my favorite.
Re:OpenAI releases larger GPT-2 model by nicwilson on Sunday May 05, 2019 @02:59AM (#19831561)
I think its just edged by "Why I'm ditching ebike hoses for barometers" for me

Though "Hacker makes an infinitely scaleable cup of coffee" gives it a good run for its money.

Re:OpenAI releases larger GPT-2 model by mkl on Sunday May 05, 2019 @12:33AM (#19831186)
I had to do a search to make sure that or something analogous wasn't actually real! It seems incredibly on target.
Re:OpenAI releases larger GPT-2 model by tyre on Sunday May 05, 2019 @12:30AM (#19831173)
This is great. A possible hackathon project for writers. Everyone gets a title and has to write the post
Re:OpenAI releases larger GPT-2 model by qlm on Sunday May 05, 2019 @03:39AM (#19831660)
My favourite is "The secret Nazi medical charity giving out freezers".

Very reminiscent of headlines from The Day Today: https://youtu.be/wdEcO8_2Kl8

Re:OpenAI releases larger GPT-2 model by zrm on Saturday May 04, 2019 @11:36PM (#19830974)
Has anybody tried feeding it comedy to begin with to see what it spits back out?
Re:OpenAI releases larger GPT-2 model by minimaxir on Saturday May 04, 2019 @11:57PM (#19831062)
There was an attempt on Reddit but it didn't turn out well, likely because there wasn't enough input data: https://www.reddit.com/r/MachineLearning/comments/bgvzdu/d_j...

(I have an idea for a more proper approach)

Re:OpenAI releases larger GPT-2 model by CYHSM on Wednesday May 15, 2019 @04:21AM (#19917382)
I tried it on The Office quotes [1] and it seems to generate some more or less humorous examples

[1] https://www.reddit.com/r/MachineLearning/comments/bmn0og/p_l...

Re:OpenAI releases larger GPT-2 model by ddalex on Sunday May 05, 2019 @01:46PM (#19833958)
I can provide more details next week once the article is out, but I've fed it a corpus of one liners and it started to produce weird and sometimes very funny liners :)
Re:OpenAI releases larger GPT-2 model by nprateem on Sunday May 05, 2019 @01:03AM (#19831265)
It'd probably produce a crap sitcom. But then again so does the BBC these days.
Re:OpenAI releases larger GPT-2 model by minimaxir on Sunday May 05, 2019 @01:44AM (#19831366)
Update: 345M model support successfully added to gpt-2-simple and new release pushed to PyPI, plus I've updated the corresponding Colaboratory Notebook accordingly (https://colab.research.google.com/drive/1VLG8e7YSEwypxU-noRN...)

At a glance it's hard to see if the 345M model is "better" at the moment (it's all qualitative), but I'll be doing more testing. Unfortunately, the 345M model might be slightly too resource intensive for the API/client use case I wanted to do, so I'll likely be sticking with the 117M model for now.

Re:OpenAI releases larger GPT-2 model by applecrazy on Saturday May 04, 2019 @09:49PM (#19830594)
One example from your dataset that gave me a laugh:

Hiring technical debt (or "unsortable overtime")

How do you hire technical debt??

Edit: another one-

“In 2009, Africa power creation was switched on for the Google Earth Darth Vader Imperial Warplane Propaganda”

I guess Google is diversifying :P

Re:OpenAI releases larger GPT-2 model by dannyw on Sunday May 05, 2019 @12:07AM (#19831101)
Outsource a project to a contracting agency and you’ve hired technical debt.
Re:OpenAI releases larger GPT-2 model by antpls on Sunday May 05, 2019 @05:18AM (#19831902)
I guess it went through that path :

-Hiring -> Software Developer

-Software Developer -> Code

-Code -> Technical Debt

You only have to hire a Software Developer :)

Re:OpenAI releases larger GPT-2 model by de_watcher on Monday May 06, 2019 @05:28AM (#19837932)
It has found a shortcut.
Re:OpenAI releases larger GPT-2 model by M4v3R on Sunday May 05, 2019 @02:03AM (#19831420)
> Show HN: Blogia, a blogging framework for Ruby

Wow, this sounded so believable I had to check if this exists or not. It seems it doesn't exist, yet :).

Re:OpenAI releases larger GPT-2 model by burtonator on Saturday May 04, 2019 @10:42PM (#19830777)
This thing really does scare me and fascinate me at the same time.

There's so many nefarious things you could do with this...

Generate fake news. spam google. etc.

One valuable use could be to generate comedy and parody.

You could also make it to sabotage others too. You could set it lose on nazi forums and have them argue with bots constantly.

Re:OpenAI releases larger GPT-2 model by lopmotr on Sunday May 05, 2019 @01:50AM (#19831384)
I think people are over-worried about fake material. We've been able to say lies since we became human. We've been able to print lies since printing. We can already edit photos. People routinely make fake screenshots of messages to defame others. Newspapers sometimes print fake stories based on lies given by their sources or which they fail to vet. But provenance still exists to prove things, not how convincing it looks.
Re:OpenAI releases larger GPT-2 model by AJ007 on Sunday May 05, 2019 @10:48AM (#19833060)
The problem is going to be the volume, and how long it takes the reader to figure out what is going on. You can look at a photo and quickly realize something is wrong. With this, you may need to read a few paragraphs or even start reading through what else the "author" has written. That is effectively a denial-of-service attack on human comprehension.

Consider the issue of fake reviews: sometimes fake reviews are really obvious. Sometimes they aren't. Often the best way to pick out the fakes is to analyze all of the other reviews the user has written. That is going to become harder.

For very technical topics, where the reader comes with a strong background knowledge in that topic, picking out the fake material isn't too difficult. I suspect for the hazier things where the writers are more or less stating opinions, like politics, it is going to be incredibly difficult (for a human reader) to separate the bots from real people.

Re:OpenAI releases larger GPT-2 model by ben_w on Sunday May 05, 2019 @04:15AM (#19831757)
If evidence alone was enough, there wouldn’t be any debate about antivaxxers, climate change, the Iraq WMDs, the 9/11 and moon landing conspiracies, Holocaust denial, the correct way to bring peace to the Israel/Palestine conflict, the safety of being white in a Muslim community or being black around an American cop, Brexit, any of the Assange drama, or if Snowden was acting appropriately by leaking what he leaked. And those are just the disagreements I’ve witnessed.

Just because lies and propaganda have always been around, it doesn’t mean they were never a problem.

Re:OpenAI releases larger GPT-2 model by jfoster on Sunday May 05, 2019 @08:24AM (#19832441)
To take an optimistic view, it may eventually teach people to more critically evaluate anything they read.
Re:OpenAI releases larger GPT-2 model by narrator on Sunday May 05, 2019 @01:55AM (#19831402)
If you wanted to run a propaganda department though, you needed to be a government or a large news organization. Now it will be within the reach of 4chan.
Re:OpenAI releases larger GPT-2 model by nprateem on Sunday May 05, 2019 @01:04AM (#19831271)
True. Authors at The Onion should be afraid.
Re:OpenAI releases larger GPT-2 model by leod on Sunday May 05, 2019 @03:27AM (#19831622)
I haven't heard of gradient checkpointing yet, thank you for the link! Do you know how it compares to gradient accumulation? The latter basically reduces the batch size, but takes the sum of multiple gradients before actually performing an update, thereby having the same effect as the original batch size.

The generated titles are great! You can put them into hncynic (https://github.com/leod/hncynic) to get closer to a fully generated HN experience.

Re:OpenAI releases larger GPT-2 model by gwern on Sunday May 05, 2019 @10:45AM (#19833038)
Gradient accumulation and gradient checkpointing are orthogonal. You might want to use them simultaneously.

If I had to compare them, I'd say that accumulation is about working on a minibatch datapoint by datapoint and faking being able to run an entire large minibatch in a single shot, while checkpointing is about working on a model layer by layer and faking being able to run an entire model in a single shot.

The problem with GPT-2-335M and why nshepperd had to mess with gradient checkpointing is that the GPT-2-335M model will literally not fit in your standard 11GB GPU (and from Twitter comments about people trying it on the new 16GB Google Colab instances, it's unclear if 16GB would be enough either!). You can't even run minibatch n=1. It doesn't fit. It OOMs.

The model itself is only a gigabyte or so, the problem is that the self-attention layers, when run, use up a huge amount of memory for their intermediate steps, which must be stored in order to trace everything backwards through each step for the backprop part of training.

(Right now I believe nshepperd's code punts on doing gradient accumulation simultaneous with gradient checkpointing, so we've just been reducing the learning rate, which is sort of similar to faking large minibatches with gradient accumulation.)

Fortunately, because the self-attention layers are so small and cheap to compute, they work well with gradient checkpointing. They're cheap to recompute on the fly, so it's more important to save memory and allow training at all. (This is also how OpenAI is training the Sparse Transformers which are enormous; they haven't said either way, but I assume this is how they trained the larger GPT-2s like the 1.5b parameter version, because I can't imagine what hardware would fit even a single GPT-2 1.5b without tricks.)

Re:OpenAI releases larger GPT-2 model by leod on Sunday May 05, 2019 @11:12AM (#19833187)
Thank you so much for your comprehensive answer, this helps a lot.

If I understand nshepperd's code correctly, it uses a constant and small learning rate. Do you know if this works better than the learning rate schedule that is usually used for Transformer models (https://www.tensorflow.org/alpha/tutorials/text/transformer_...)?

Re:OpenAI releases larger GPT-2 model by gwern on Sunday May 05, 2019 @11:43AM (#19833342)
It's a constant, yes. We haven't tried any other learning rate schedules (for my poetry GPT-2s, I simply drop the LR 10x each day or so). I have no idea if this is optimal for transfer learning or not.
Re:OpenAI releases larger GPT-2 model by indalo on Sunday May 05, 2019 @05:22AM (#19831915)
wow! I just had a blast putting titles into that. the results are amazing. kudos!
Re:OpenAI releases larger GPT-2 model by yeldarb on Saturday May 04, 2019 @08:52PM (#19830399)
Can you elaborate on the expected hardware requirements? (Both for fine tuning and inference)
Re:OpenAI releases larger GPT-2 model by bazza451 on Sunday May 05, 2019 @04:34AM (#19831802)
This is really cool.

Just to pick your brain (I’m an ML noob and it triggered an idea in my head). I was wondering if GPT-2 could be used to generate small paraphrases for an input sentence for search suggestions? e.g “I’m going to the mall today” -> “I will be going to mall today”.

Re:OpenAI releases larger GPT-2 model by czr on Saturday May 04, 2019 @08:59PM (#19830423)
It seems to train fine on Colab without that flag, so that's a decent fallback for anyone running into memory issues on a local machine.
Re:OpenAI releases larger GPT-2 model by minimaxir on Saturday May 04, 2019 @09:04PM (#19830441)
If you did a fresh pull of that fork using a Colab notebook that does so, it will set the flag automatically if you specify the 345M model (I will follow that behavior in my own package).

If it didn't set the flag, it gets messy: https://twitter.com/minimaxir/status/1124742105421631488