Databricks Releases 15K Record Training Corpus for Instruction Tuning LLMs

Re:Databricks Releases 15K Record Training Corpus for Instru... by ankitmathur on Thursday April 13, 2023 @02:02AM (#35551632)

Sure! I'll try to briefly summarize though almost certainly will oversimplify. There are a couple of open source language models trained by Eleuther AI - the first one was called GPT-J, and it used some newer model architecture concepts. Subsequently, they released a model architected in the likeness of GPT-3, called GPT-NeoX-20B. Functionally, it was quite similar architecturally to GPT-J, but just with more parameters. Pythia is a model with the same architecture and the same dataset but with different parameter sizes to test scaling laws.

DollyV2 is a Pythia model fine tuned on the Databricks 15K dataset

[ Parent | on HN ]

Re:Databricks Releases 15K Record Training Corpus for Instru... by ankitmathur on Thursday April 13, 2023 @01:59PM (#35559454)

Augmenting the answer to address your followup: parameters are any trainable variable in a model's definition. Model training is a process where you basically tweak the parameters in your model and then re-evaluate the model on a metric judging its quality. A lot of models consist of matrix multiplication, so if you are multiplying matrix A of size 2x2 with matrix B of size 2x2 and both matrices can we tweaked, then you've got 8 parameters, since you've got 8 numbers that can be tweaked

[ Parent | on HN ]