News for hackers. Stuff that matters.
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by ldjkfkdsjnv on Tuesday August 29, 2023 @12:01PM (#37309865)
Anyscale's business model was completely disrupted by OpenAI. They are trying to shift to provide hosting/fine tuning for open source LLMs, but the model performance will get crushed by Gpt4/newer open AI models. In theory the alternative to openAI models is nice, in reality anyscale is now competing with Azure/AWS/etc to provide model hosting.

Their original compute platform for running arbitrary ml workloads will become obsolete as the industry consolidates around LLMs.

Re:Llama 2 is about as factually accurate as GPT-4 for summa... by nightski on Tuesday August 29, 2023 @12:16PM (#37310109)
I had never heard of anyscale before but I'm curious why you think the entire industry will consolidate around LLMs. Maybe you mean transformers? Either way there are a lot of problems that LLMs just aren't suited for.
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by ldjkfkdsjnv on Tuesday August 29, 2023 @12:21PM (#37310184)
Transformers sure, but theres evidence that LLMs will continue to outperform models trained for specific tasks. In the past you needed:

1. a model for sentiment analysis

2. a model for summarization

3. ...other NLP tasks

other tasks:

4. a model for object detection in images

5. a model for face recognition in images

Whereas now the LLM does all of the above better than the previous state of the art. This will continue and eat more fields of machine learning. It will happen for images and video. I argue that it will even extend to things like time series analysis

Re:Llama 2 is about as factually accurate as GPT-4 for summa... by msp26 on Tuesday August 29, 2023 @12:38PM (#37310425)
I disagree that LLMs are always better for things like classification. But the barrier to entry is so low with them that it's usually worth trying to use an LLM before training some other model.
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by ldjkfkdsjnv on Tuesday August 29, 2023 @12:42PM (#37310469)
The whole concept of AGI/super intelligence is that one model can do everything. Thats whats happening with LLMS. If you believe in AI, then you believe that specific models will become obsolete
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by spott on Tuesday August 29, 2023 @01:48PM (#37311387)
No, that isn’t the point.

The point is that a model can do everything, not that it is the most cost effective or best way to do everything.

There are tradeoffs with all tech, and there always will be.

Re:Llama 2 is about as factually accurate as GPT-4 for summa... by losteric on Tuesday August 29, 2023 @12:35PM (#37310362)
LLMs come with a higher cost per transaction compared to dedicated models.

imo the industry will consolidate around LLMs for early prototypes and as part of the workflow for building a corpus to train domain-specific models.

Source: team went through this process. LLM cost ~1% unskilled human labelers, then we finetuned BERT to bring the cost down to < 0.001% (savings $120k/y compared to just using the LLM)

Re:Llama 2 is about as factually accurate as GPT-4 for summa... by ldjkfkdsjnv on Tuesday August 29, 2023 @12:45PM (#37310512)
The cost will drop like a rock in the coming years. Its always the same pattern with technology. Theres probably 5-20 Billion in capital working on solving this problem right now.
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by losteric on Tuesday August 29, 2023 @12:56PM (#37310679)
Wouldn't cost savings also translate to simpler models? I anticipate the process of distilling knowledge to simpler models will also become much cheaper and non-technical
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by two_in_one on Tuesday August 29, 2023 @06:44PM (#37315123)
To be fair you need to add the cost of new development and estimate when it pays bask. Without that it's not obvious that custom solution is better. For small tasks it's likely not.
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by adventured on Tuesday August 29, 2023 @12:38PM (#37310423)
It'll happen just as the smartphone ate numerous accessory gadgets.

The tech consolidator works by: if it reasonably can, it will.

PS5? Gaming machine, 4k disc player, streaming box, etc.

Smartphone? Internet, phone, texting, camera/photos, storage, shopping, identity, gaming, music, video, directions. Soon we'll add highly useful AI bots to the list.

Windows? Pretty much non-stop consolidated software from the industry into itself, across decades.

Also known as eating the ecosystem. GPT & Co. will eat their best plugins.

Re:Llama 2 is about as factually accurate as GPT-4 for summa... by mistrial9 on Tuesday August 29, 2023 @01:43PM (#37311309)
"Law of Diminishing Returns" .. your case is weakened by the repetitive confidence of the assertions, whether it is true or not remains to be seen via actual practice
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by ldjkfkdsjnv on Tuesday August 29, 2023 @01:48PM (#37311389)
Yeah definitely is speculative, but I worked in many NLP research teams in FAANG. Worked on large scale ML systems that support systems you have heard of. All of the work we did is now obsolete (except for cost), and the scientists skill sets were completely eroded.
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by two_in_one on Tuesday August 29, 2023 @06:38PM (#37315067)
> It will happen for images and video.

That's for sure. The main problem is how to put them together. There are several ways. First, tokenizing the media and inserting it in the stream. Second, a media controller which accepts verbal instructions generated by LLM. They can be combined. Interesting variation is controller with (immediate) feedback. It can be anything, for example database access.

> I argue that it will even extend to things like time series analysis

AFAIK transformers have been used for low level robotic control. And LLM for high level have been reported by MS and Google.

Re:Llama 2 is about as factually accurate as GPT-4 for summa... by ianbutler on Tuesday August 29, 2023 @12:27PM (#37310273)
OpenAI is listed as a customer on the homepage. One of the names that flits through their "trusted by" carousel.
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by waleedk on Tuesday August 29, 2023 @01:39PM (#37311273)
[Author] OpenAI uses Ray, the open source version of Anyscale Platform, for training their models (video evidence: https://www.youtube.com/watch?v=CqiL5QQnN64).

Ray (https://github.com/ray-project/ray) is available for anyone to use. You can also use Aviary (https://github.com/ray-project/aviary) to serve any of those models yourself.

Re:Llama 2 is about as factually accurate as GPT-4 for summa... by waleedk on Tuesday August 29, 2023 @01:36PM (#37311221)
[Author] I see it differently. I see that Anyscale built some really great infrastructure for scaling AI that could be applied to many domains. What we've seen is the market shift to LLMs, and we've been able to adapt our tech to new domains.

Our prices are competitive (starting at 25c per million tokens) because of the tech we've built that maximizes GPU utilization. That's what we do better than anyone else.

I still think GPT-4 is best LLM out there. It's just very expensive and you don't need all that horsepower you can save bucketloads of money.

Re:Llama 2 is about as factually accurate as GPT-4 for summa... by alfalfasprout on Tuesday August 29, 2023 @02:44PM (#37312089)
Being in the industry it's actually quite the opposite.

OpenAI had a first mover advantage. ChatGPT has a good interface and GPT4 still leads on many benchmarks. But it continues to get worse and worse and others are catching up with significantly lower parameter counts. Their moat is shrinking very quickly.

Llama2 is an open source meta model, btw. Anyscale's ray makes it much easier to leverage the super fast pace of development in the OSS community (or released by large companies as OSS).

Re:Llama 2 is about as factually accurate as GPT-4 for summa... by ldjkfkdsjnv on Tuesday August 29, 2023 @03:23PM (#37312607)
I also work on FAANG NLP research teams. I'm just saying that anyscale is now competing with AWS. Supposedly their moat is shrinking, Google has yet to release a model that competes with GPT-4. Anyone using these models on a daily basis is not impressed with Llama 2, and cannot build useful systems with it, aside from a narrow vertical.
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by Hermitian909 on Tuesday August 29, 2023 @04:10PM (#37313244)
Seeing this right now on the product side. I can't give away untuned GPT3.5, much less sell it. GPT4 is the absolute floor for what I can sell without setting up a pipeline.
Re:Llama 2 is about as factually accurate as GPT-4 for summa... by alfalfasprout on Wednesday August 30, 2023 @06:39PM (#37330083)
> Anyone using these models on a daily basis is not impressed with Llama 2 Being at a FAANG adjacent company working with these models on a daily basis, Llama2 is more than sufficient for a large number of use cases. It ultimately depends what exactly you're looking for. Exposing the LLM directly as a chat interface to users? Then yeah, in many cases it won't feel sufficient. Using it for search, summarization, moderation, etc.? Then it's often more than capable.

FWIW GPT-4 is not one single model as you're aware and it evolves over time. In our experience it has objectively degraded in quality over time.

As for anyscale, ultimately it's more like databricks (and funny enough lots of ex databricks folks there). Anyone can just run hosted spark. Databrick's ecosystem has set them apart and it looks like that's what anyscale is building towards.