> First, given an LLM and an unlabeled dataset, we prompt the LLM to generate output labels along with rationales to justify the labels

> Second, we leverage these rationales in addition to the task labels to train smaller downstream models.

This doesn't sound like online use of LLMs to me, but rather using them to pre-generate augmented examples.

At the end of the paper is this table:

  Dataset  Train    Validation  Test
  e-SNLI   549,367  9,842       9,824
  ANLI      16,946  1,000       1,000
  CQA        8,766    975       1,221
  SVAMP        720     80         200

OpenAI costs "gpt-3.5-turbo: $0.002 / 1K tokens"

let's say each rationale consumes 250 tokens ...that's $275 to augment 550M examples in e-SNLI dataset

Even if that estimate is quite far off in terms of token count, it seems like it'll only cost a few thousand dollars at most? GPT4 costs 10x as much.

It's more than I personally would ever want to pay as a curious bystander/hobbyist, but it doesn't seem like a serious "moat" to competitors.