Introducing DBRX: A New State-of-the-Art Open LLM by Databricks

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by K0balt on Wednesday March 27, 2024 @10:32AM (#39839682)

I find that q6 and 5+ are subjectively as good as raw tensor files. 4 bit quality reduction is very detectable though. Of course there must be a loss of information, but perhaps there is a noise floor or something like that.

[ Parent | on HN ]

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by Taek on Wednesday March 27, 2024 @05:21PM (#39844749)

At what parameter count? Its been established that quantization has less of an effect on larger models. By the time you are at 70B quantization to 4 bits basically is negligible

[ Parent | on HN ]

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by K0balt on Wednesday March 27, 2024 @09:10PM (#39846662)

I work mostly with mixtral and mistral 7b these days, but I did work with some 70b models before mistral came out, and I was not impressed with the 4 bit Llama-2 70b.

[ Parent | on HN ]

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by 2099miles on Thursday March 28, 2024 @02:37AM (#39848337)

Source? I’ve seen this anecdotally and heard it, but is there a paper you’re referencing?

[ Parent | on HN ]