Databricks Strikes $1.3B Deal for Generative AI Startup MosaicML

Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by brucethemoose2 on Monday June 26, 2023 @10:52AM (#36480361)

Falcon 7B/40B is good, but hasn't really caught on like LLaMA has. One big reason is that its not well supported by 4-bit inference code (namely llama.cpp and GPTQ).

Mosaic's 7B (and 30B?) models have the same issue, and 7B kinda paled in comparison to LLaMA 7B... But maybe it would be better if finetuned?

To me, its kinda baffling that Mosaic didn't work on adding highly quantized inference to the popular frameworks.

[ Parent | on HN ]

Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by Tostino on Monday June 26, 2023 @11:18AM (#36480803)

Falcon is also slow for inference compared to LLaMA models of similar size. Speed can turn into a quality all its own when you break up problems into small pieces and use the LLM to iterate.

[ Parent | on HN ]

Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by brucethemoose2 on Monday June 26, 2023 @11:52AM (#36481394)

Thats because its unoptimized, right? My impression is that the Falcon GPTQ (and 8 bit bnb inference?) code is just immature.

[ Parent | on HN ]

Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by ml_hardware on Monday June 26, 2023 @01:09PM (#36482614)

Mosaic's MPT models are already supported in GGML: https://github.com/ggerganov/ggml

Here's MPT-30B running in 4-bit precision on CPU :) https://twitter.com/abacaj/status/1673133443339763712?s=20

[ Parent | on HN ]

Re:Databricks Strikes $1.3B Deal for Generative AI Startup M... by brucethemoose2 on Monday June 26, 2023 @01:48PM (#36483171)

Oh I didn't realize this. Everything is moving so fast that I can't even keep up with the features.

[ Parent | on HN ]