To be fair, this is a test case that OpenAI claims to my knowledge was their core test case for GPT4. If this was an outstanding industry test case of note, references would be helpful. OpenAI has also not disclosed how they were able to finally accomplish this, so it’s possible solution was more of a hack than actually training model.

Guilty as charged... I really liked that test case so it went into our prompt library.

maybe modify it slightly? a poem starting with each consonant in alphabetical order? a poem where every word starts with a particular comon letter like "t"? the same but in reverse alphabetical order

You need filter assisted decoding to make that possible in all cases (as in, working for both models all the time provably) I'm surprised it worked in chatGPT this time, but it won't work all the time due to lack of lexical, semantic, or phonetic info in the tokenizer.

https://paperswithcode.com/paper/most-language-models-can-be...

It works in GPT-4 (but not GPT-3.5) even without a special tokenizer: Some problems simply solve themselves with increasing scale. But Bard uses a smaller LaMDA model to reduce inference cost. Quote [1] from Sundar Pichai in February:

> We’re releasing it initially with our lightweight model version of LaMDA. This much smaller model requires significantly less computing power, enabling us to scale to more users, allowing for more feedback.

[1] https://blog.google/technology/ai/bard-google-ai-search-upda...

I don't think it works reliably over 100 invokations of the prompt. The paper tries to get provable situations where 0 ignored constraints happen/are possible.

I'm impressed that it is working more often than not, but we need provable guarantees.

If you ask 100 humans the same thing, you'll get some mistakes too. I don't think the point is whether it makes mistakes. The Chat GPT response shown above is clearly more interesting and impressive / displays something more seemingly like intelligence than the Bard response provided. At some point scaling these things up enough becomes indistinguishable from intelligence, and at that point maybe it is.

> If you ask 100 humans the same thing, you'll get some mistakes too

Yes, still, it's pretty easy to find a somewhat good judgement human who never does that mistake. If you tell him/her: "This is of utmost importance: ..." — But as of now, these text gen machine learning programs don't totally understand that?

Bard did, at least, put the first character of each line of the poem in alphabetical order. (Notice A-B-C-D-E.)

And at least Bard's rhymes in some sense, rather than being essentially a word salad (yes, there are plenty of poems that don't rhyme, which is no doubt ChatGPT's model in this case)

It almost seems cynical to have called it 'Bard'.