you don't need access to the weights though, unless I missed something? this is akin to the various groups using GPT-3.5/4 to generate data for fine-tuning smaller models

Sure, but that will run you up a hefty bill and add significant latency during training.

The latencies and costs are changing so quickly...

maybe for the model, but if I'm in GCP, then I am reaching out to another cloud, those network latencies and costs are pretty stable

I skim read the paper, the key bits I saw were:

> First, given an LLM and an unlabeled dataset, we prompt the LLM to generate output labels along with rationales to justify the labels

> Second, we leverage these rationales in addition to the task labels to train smaller downstream models.

This doesn't sound like online use of LLMs to me, but rather using them to pre-generate augmented examples.

At the end of the paper is this table:

  Dataset  Train    Validation  Test
  e-SNLI   549,367  9,842       9,824
  ANLI      16,946  1,000       1,000
  CQA        8,766    975       1,221
  SVAMP        720     80         200

Let's do a back-of-envelope calculation:

OpenAI costs "gpt-3.5-turbo: $0.002 / 1K tokens"

let's say each rationale consumes 250 tokens ...that's $275 to augment 550M examples in e-SNLI dataset

Even if that estimate is quite far off in terms of token count, it seems like it'll only cost a few thousand dollars at most? GPT4 costs 10x as much.

It's more than I personally would ever want to pay as a curious bystander/hobbyist, but it doesn't seem like a serious "moat" to competitors.

I think that is about $2 for 1M tokes, so 550M*250\1M * 2 = 550*250/2 = $ 68750, that is far from $275.

But it's not 550M examples, it's 0.55M examples

Edit: just saw I wrote 550M in my original comment, even though I was thinking of it as the correct figure (550k)

Edit: obviously... also have to add the cost of training the T5-large on top of the LLM fees