jamiedg
Fully compatible with OpenAI API.
4x lower cost.
Integrated with MongoDB, LangChain, and LlamaIndex for RAG.
New "M2-Bert-Retrieval" models with up to 32k context length
Today we shared progress on training our first model on this dataset, a 7B parameter model using the Pythia architecture. So far we are a bit less than 50% through the training - 440B parameters. We published HELM benchmark results on 16 different scenarios for this checkpoint, showing the model accuracy to be quite high for this stage of training.
RedPajama at 440B tokens higher quality than Pythia and StableLM
Therefore, anyone will be able to fine-tune the RedPajama models using Vicuna or other datasets, given they will be fully open-source.
The RedPajama instruction-tuned models will be fine-tuned only with instruction labels from human labelers and OpenChatKit feedback (). We feel this will keep these models fully "clean" for use in commercial applications without using the output of other commercial models like were used in Alpaca or Vicuna. However, we'll be excited to see all the great fine-tunes created by the open community and are eager to see how close open-source models can get to the quality of leading commercial models over time!!
() OpenChatKit: https://huggingface.co/spaces/togethercomputer/OpenChatKit
You can try it now on Hugging Face! https://huggingface.co/spaces/togethercomputer/OpenChatKit
OpenChatKit includes 4 key components:
1. An instruction-tuned large language model, fine-tuned for chat from EleutherAI’s GPT-NeoX-20B with over 43 million instructions on 100% carbon negative compute available under Apache-2.0 license on Hugging Face.
2. A set of customization recipes to fine-tune the model to achieve high accuracy on your tasks documented and available as open-source under the Apache-2.0 license on Github, along with code to recreate our model results.
3. An extensible retrieval system enabling you to augment bot responses with information from a document repository, API, or other live-updating information source at inference time, with open-source examples for using Wikipedia or a web search API.
4. A moderation model, fine-tuned from GPT-JT-6B, designed to filter which questions the bot responds to, also available under the Apache-2.0 license on Hugging Face.
We collaborated with the tremendous communities at @laion_ai and Ontocord to create the training dataset used for these models, also released as open-source. Read the full details on LAION's blog post!
Can't wait to hear your feedback!
Thanks, -Together