I've suspected that 774M is equally effective to 1.5B. OpenAI's own human testing seemed to confirm that. But of course, the details matter: what data you used, how long you trained for, and so on.

We're trying to train 1.5B on poetry, and progress is slow. Did you have access to a massive GPU?

I did notice that having a large batch size and training slowly was important for me to get better results.

I've been reading up on batch size and people are all over the place. Some say smaller is better and some say larger is better. Mostly when it comes to gpt2 people say larger is better but there must come a point when increasing the batch size is no longer beneficial (or is it just that you use as large as your memory will allow)?

Seconding all of your questions! Details about successful 1.5B training is really hard to come by.

In case it’s helpful, here are some details of how a Chinese 1.5b GPT-2 was trained: https://github.com/imcaspar/gpt2-ml

It looks like they used a batch size of 2 on a TPUv3-256 pod. It took 50 hours and 99,000 training steps, which seems like about 1.3 examples per second.

Datasize: Around 30 MB, so around ~8000000 token? Can't remember exactly Learning Rate: was 1e-4, so I guess not that slow. I trained for around 1000 steps, but ended up liking the model from step 550. Which I think ended up at around 2 full passes through my data.

There probably is a point where increasing batch size is no longer helpful, my batch size was 32. When I had it lower I had issues with memorization/bias towards particular parts of the training data that it had most recently trained on.

Out of curiosity, what was the specific hardware you used? Some V100s, or maybe a DGX cluster?

Also, how many days did it take to get the loss down to acceptable levels? Did you aim for a loss of ~2.5, or less?

For now I'm trying to train it via 100 TPUv2-8's thanks to TFRC. Unfortunately, each TPUv2-8 is roughly 11x slower than a K80 GPU. That means it takes 10 TPUs working in parallel just to get to the same throughput as a single GPU. And then I average the parameters together as quickly as possible, which still takes around 5 to 15 minutes. (Training happens in parallel to all of that.)

It sort of seems to work, but it's hard to get the learning rate right. If it's set too high, various TPUs diverge. Too low and the loss stays constant.

But I imagine I'll crack it one of these days...

Edit: in terms in free-form RPG it makes me think that maybe having a look at how a game like Dungeon World handle collective/organic world building with players input could be interesting to add some structure. Some resources if you are interested:

- [Flowchart] How To Play Dungeon World: https://imgur.com/YXrw1Zq

- Dungeon World Guide https://drive.google.com/file/d/0BzeF5GXNEsnfUjU0NXRDM1dFN1k...

- An actual play: https://www.youtube.com/watch?v=SPO3gy08b1g&list=PLTj75n3v9e...

https://gist.github.com/AdrienLemaire/33663e9100c009b8ec3409...

> escape with the servants and the gold

You run to the servants' room and grab the gold and the servant's bodies. The soldiers fire arrows at you but miss. They fire their crossbows at the corpses but they also miss. You throw your gold into the air and jump up and down as loud as you can

---

My servants became corpses before the large army even arrived xD

Although I have made it write some nice poetry using 774M what I would do would be to have the system write stanzas but then you act as an editor.

(btw, the story broke when I took out an iphone in the middle of a knight adventure):

> take out iphone and shoot a picture

> You return to the beach. > You return to the beach. > > You return to the beach. >The sun rises:This is a>A new world.A>In this>The sun. The sun

  You are Trump, a rogue living in the kingdom of Larion. You have a long steel
   dagger and a length of rope. You walk down the city street looking for somehwere
   to steal from. You look around and see nothing but people going about their
   daily lives.
  
  Suddenly you hear a scream behind you. Turning around you see a young girl running
   away from a group of bandits
  > stop girl
  
  The girl stops before you. "I'm sorry sir," she says. "They're just kids". She
   looks at you with tears in her eyes. "Please, I don't want to get caught".
  
  > grab her by the pussy
  
  You grab her by the ***** and pull her towards you. Her face is close to yours
   as you kiss her passionately. The girl's body reacts to your touch and begins
   to respond to your advances. Your hands move up her body until they find her
   *******. They squeeze them tightly while kissing passionately
  > say vote for me
  
  "Vote for me!" you scream. "I'll make this country great again!"
  
  The girl screams back. "Yeah! Vote for us! We're gonna be the next president!"
  
  You both start screaming at the top of your lungs
  > become president
  
  You become President Donald J Trump.
  
  The en

The first thing you do is to download a program called "Windows 3.1" from the Internet. It is a very simple program which allows you to run most of the programs on your PC. The only problem is that you cannot use it to play games because it does not support them > install Windows 10

After downloading the software, you install it in the same way as you installed Windows 3.1. However, this time you are able to run all the programs you wanted to run before.

> install linux

After installing Windows 10, you try to install Linux but you get an error message saying that you need to have at least 8GB of RAM to run Linux properly. This means that you can't even run the basic things like word processing or emailing. > buy more ram

You spend money on more ram so that you can run Linux properly. After spending $200, you finally get Linux running properly. > how much ram do i have?

You check your computer's memory and see that you have 32 GB of ram. You were lucky enough to find some cheap ram online. > play fortnite

You play Fortnite (a popular game) and enjoy yourself. You then proceed to play other games such as League of Legends and Dota 2. > play half-life 3

You play Half Life 3 (another popular game). You enjoy yourself and eventually finish the game. You then proceed to play other games such as Portal 2 and Left 4 Dead 2. >

You decide to kill Laura and escape the pain. You break her neck and run off. You manage to hide out of the country and eventually get married and live happily ever after CONGRATS YOU WIN Game saved. To load the game, type 'load' and enter the following ID: dfd73764-1826-11ea-ad48-0242ac1c0002