Wow, thats great. “The Bullshit Bubble” “Fuck you, Bootstrap” “We should give up on America” - they’re practically comedy, yet very believable too.
Though "Hacker makes an infinitely scaleable cup of coffee" gives it a good run for its money.
Very reminiscent of headlines from The Day Today: https://youtu.be/wdEcO8_2Kl8
(I have an idea for a more proper approach)
[1] https://www.reddit.com/r/MachineLearning/comments/bmn0og/p_l...
At a glance it's hard to see if the 345M model is "better" at the moment (it's all qualitative), but I'll be doing more testing. Unfortunately, the 345M model might be slightly too resource intensive for the API/client use case I wanted to do, so I'll likely be sticking with the 117M model for now.
Hiring technical debt (or "unsortable overtime")
How do you hire technical debt??
Edit: another one-
“In 2009, Africa power creation was switched on for the Google Earth Darth Vader Imperial Warplane Propaganda”
I guess Google is diversifying :P
-Hiring -> Software Developer
-Software Developer -> Code
-Code -> Technical Debt
You only have to hire a Software Developer :)
Wow, this sounded so believable I had to check if this exists or not. It seems it doesn't exist, yet :).
There's so many nefarious things you could do with this...
Generate fake news. spam google. etc.
One valuable use could be to generate comedy and parody.
You could also make it to sabotage others too. You could set it lose on nazi forums and have them argue with bots constantly.
Consider the issue of fake reviews: sometimes fake reviews are really obvious. Sometimes they aren't. Often the best way to pick out the fakes is to analyze all of the other reviews the user has written. That is going to become harder.
For very technical topics, where the reader comes with a strong background knowledge in that topic, picking out the fake material isn't too difficult. I suspect for the hazier things where the writers are more or less stating opinions, like politics, it is going to be incredibly difficult (for a human reader) to separate the bots from real people.
Just because lies and propaganda have always been around, it doesn’t mean they were never a problem.
The generated titles are great! You can put them into hncynic (https://github.com/leod/hncynic) to get closer to a fully generated HN experience.
If I had to compare them, I'd say that accumulation is about working on a minibatch datapoint by datapoint and faking being able to run an entire large minibatch in a single shot, while checkpointing is about working on a model layer by layer and faking being able to run an entire model in a single shot.
The problem with GPT-2-335M and why nshepperd had to mess with gradient checkpointing is that the GPT-2-335M model will literally not fit in your standard 11GB GPU (and from Twitter comments about people trying it on the new 16GB Google Colab instances, it's unclear if 16GB would be enough either!). You can't even run minibatch n=1. It doesn't fit. It OOMs.
The model itself is only a gigabyte or so, the problem is that the self-attention layers, when run, use up a huge amount of memory for their intermediate steps, which must be stored in order to trace everything backwards through each step for the backprop part of training.
(Right now I believe nshepperd's code punts on doing gradient accumulation simultaneous with gradient checkpointing, so we've just been reducing the learning rate, which is sort of similar to faking large minibatches with gradient accumulation.)
Fortunately, because the self-attention layers are so small and cheap to compute, they work well with gradient checkpointing. They're cheap to recompute on the fly, so it's more important to save memory and allow training at all. (This is also how OpenAI is training the Sparse Transformers which are enormous; they haven't said either way, but I assume this is how they trained the larger GPT-2s like the 1.5b parameter version, because I can't imagine what hardware would fit even a single GPT-2 1.5b without tricks.)
If I understand nshepperd's code correctly, it uses a constant and small learning rate. Do you know if this works better than the learning rate schedule that is usually used for Transformer models (https://www.tensorflow.org/alpha/tutorials/text/transformer_...)?
Just to pick your brain (I’m an ML noob and it triggered an idea in my head). I was wondering if GPT-2 could be used to generate small paraphrases for an input sentence for search suggestions? e.g “I’m going to the mall today” -> “I will be going to mall today”.
If it didn't set the flag, it gets messy: https://twitter.com/minimaxir/status/1124742105421631488
I'm working on tools to streamline GPT-2 text generation: I'm currently porting the code above to gpt-2-simple (https://github.com/minimaxir/gpt-2-simple) to allow easy finetuning/generation, and am also working on a way to quickly build an API/client for easily deploying GPT-2 to production and generating text at scale and cost effectively. Even the 117M model, managing CPU and RAM performance is tricky.
But given the incredible results of just the 117M model (e.g. Hacker News titles from a retrained 117M model: https://github.com/minimaxir/hacker-news-gpt-2), I'm eager to put the 345M model through its paces.