IanAWP

CloudFlare ran a benchmark on both services and published their results. The contention is that the benchmark did not represent fastly performance fairly, so fastly reran the benchmark in what they consider to be a fair way, on their service only. They did not retest CloudFlare in their benchmark. The results in the post compare CloudFlare's results for CloudFlare and fastly's results for fastly

This seems unreasonable. Gpt3 is stateless and any given prompt could require a news article, a fiction, a farce, real world logic. Prompt construction absolutely should matter in this situation. Consider that the article you read provided an immense amount of context to you about exactly what kind of tests were being run and how the answers were being evaluated (they want logic not creativity). If they had provided only the prompts and responses with no mention of gpt3 or the purpose of the article or anything else, would you have been able to judge the result? Why should gpt3 be able to do better when it had access to a single prompt at a time.

GPT is both general purpose and stateless. Lacking another mechanism, it's the requirement of the prompt to request a particular genre of response if that's what you want.

There can be no such thing as a bad prompt from gpt3's perspective. A bad prompt is one where the user has a specific purpose which is not expressed. It's bad because you know beforehand that gpt3 can not align with it.

Someone pours grape juice into a bottle and becomes worried that is not safe to drink. GPT3 correctly grasps that there is a hidden context to this weird prompt, however when given no other information it guesses that this hidden context is something known only to the hypothetical character in the prompt. I would probably do this too.

When you give it the correct context (this weird prompt is a logic test) then it gives you the answer you expected.

Blue lines are the 'correct' answers, they are the MVP features of each product.

Next to each line you get a tick or a cross depending on whether your answer matches theirs. For instance if you got a tick next to a line that was not blue, then it means you agreed with them that it was not an MVP feature.

The same author posted a follow-up post explaining why: http://slatestarcodex.com/2018/07/12/did-a-melatonin-patent-...

In summary: "Commenters, including a patent lawyer, have filled in the rest of the story. Because melatonin is a natural hormone and not an invention, patents can only cover specific uses of it. The MIT patent covered the proper way to use it for sleep; a broader patent might not have been granted. The patent probably guided supplement companies, but expired about five years ago. It’s now legal to produce melatonin 0.3 mg pills, but people are so used to higher doses that few people do."