Pretraining on the Test Set Is All You Need

Re:Pretraining on the Test Set Is All You Need by ianand on Saturday October 28, 2023 @07:52PM (#38054576)

Reminds me of “FedEx is all you need” from ICML https://x.com/quantumaephraim/status/1685358421359341568?s=4...

Full disclosure: I have an AI side project that also uses “is all you need” in the title.

[ Parent | on HN ]

Re:Pretraining on the Test Set Is All You Need by gardnr on Sunday October 29, 2023 @12:44AM (#38055852)

This made my day:

The implementation is just enough to run very small workloads:

* Full GPT2 small (124M parameters) model including byte pair encoding, embeddings, multi-headed attention, and multi-layer perceptron stages

* Inference/forward pass only (no training)

* Context is limited to 10 tokens in length

* 10 characters per word limit

* Zero temperature output only

This sheet is very big. Unfortunately, it is not unusual for Excel to lock up while using this spreadsheet. It is highly recommended to use the manual calculation mode in Excel.

from: https://spreadsheets-are-all-you-need.ai/

[ Parent | on HN ]