News for hackers. Stuff that matters.
Re:Google “We Have No Moat, and Neither Does OpenAI” by sashank_1509 on Thursday May 04, 2023 @08:55AM (#35814511)
Cringe, haven’t seen a single Open Source come even close to the ability of Bard, let alone ChatGPT. Seems like wishful thinking to think decentralized open source can beat centralized models that cost 100M+ to train!
Re:Google “We Have No Moat, and Neither Does OpenAI” by vlovich123 on Thursday May 04, 2023 @09:28AM (#35814976)
Is there any reason to think that zero-shot learning and better models/more effient AI won’t drastically reduce those costs over time?
Re:Google “We Have No Moat, and Neither Does OpenAI” by ebiester on Thursday May 04, 2023 @10:22AM (#35815815)
Think a little more laterally.

If we're talking about doing everything well, I think that's true. However, if I want to create my own personal "word calculator," I could take, for example, my own work (or Hemingway, or a journalist) and feed an existing OSS model my of samples, and then take a set of sources (books, articles, etc), I might be able to build something that could take an outline and write extended passages for me, turning me into an editor.

A company might feed its own help documents and guidance to create its own help chat bot that would be as good as what OpenAI could do and could take the customer's context into the system without any privacy concerns.

A model doesn't have to be better at everything to be better at something.

Re:Google “We Have No Moat, and Neither Does OpenAI” by lapinot on Thursday May 04, 2023 @11:02AM (#35816464)
> Seems like wishful thinking to think decentralized open source can beat centralized models that cost 100M+ to train!

Because surely price = quality. Solid argumentation there.

Re:Google “We Have No Moat, and Neither Does OpenAI” by Mike_12345 on Friday May 05, 2023 @04:40AM (#35826736)
Yes, price = quality because they require so much computing resources to train. GPT-3 required the equivalent of one month of training on ~1000 A100 GPUs. That's $10 million for the GPU hardware alone, not including power ($10k each)
Re:Google “We Have No Moat, and Neither Does OpenAI” by sashank_1509 on Saturday May 06, 2023 @11:40AM (#35842416)
You’re right but I’d just like to add, GPT-3 probably required 1000’s of GPUs. OpenAI is known to have the largest cluster 16k+ A100 GPUs and most of them were used for the major model training.
Re:Google “We Have No Moat, and Neither Does OpenAI” by tshadley on Thursday May 04, 2023 @11:09AM (#35816603)
From the article:

"April 3, 2023 - Real Humans Can’t Tell the Difference Between a 13B Open Model and ChatGPT

Berkeley launches Koala, a dialogue model trained entirely using freely available data.

They take the crucial step of measuring real human preferences between their model and ChatGPT. While ChatGPT still holds a slight edge, more than 50% of the time users either prefer Koala or have no preference. Training Cost: $100."

Re:Google “We Have No Moat, and Neither Does OpenAI” by crazyedgar on Thursday May 04, 2023 @02:34PM (#35820060)
This is hugely misleading. If your bot just memorizes Shakespeare and output segments from memorization, of course nobody can tell the difference. But as soon as you start interacting with them the difference can't be more pronounced.
Re:Google “We Have No Moat, and Neither Does OpenAI” by e63f67dd-065b on Thursday May 04, 2023 @02:55PM (#35820372)
The test was conducted as such:

>With these two evaluation sets, we conducted a blind pairwise comparison by asking approximately 100 evaluators on Amazon Mechanical Turk platform to compare the quality of model outputs on these held-out sets of prompts. In the ratings interface, we present each rater with an input prompt and the output of two models. They are then asked to judge which output is better (or that they are equally good) using criteria related to response quality and correctness.

No, it's not just memorising shakespeare, real humans interacted with the models and rated them.

Re:Google “We Have No Moat, and Neither Does OpenAI” by crazyedgar on Thursday May 04, 2023 @03:16PM (#35820689)
That's not what I meant by interaction. The evaluator had to ask the models to do tasks for them that they thought of by their own. Otherwise there are just too many ways that information could have leaked.

OpenAI's model isn't immune from this either, so take any so-called evaluation metrics with a huge grain of salt. This also highlights the difficulties of properly evaluating LLMs: any metrics, once set up, can become a memorization target for LLMs and lose their meaning.

Re:Google “We Have No Moat, and Neither Does OpenAI” by adalyac on Monday May 08, 2023 @12:32PM (#35863625)
Exciting to see open source catching up, but seems to still be a big difference in factuality and common sense between ChatGPT and Koala. Eg:

Who built AlexNet on the ImageNet dataset?

ChatGPT: AlexNet was built by a team of researchers led by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton at the University of Toronto. They entered the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) in 2012 and won by a large margin, revolutionizing the field of computer vision with their deep convolutional neural network architecture. The AlexNet architecture consisted of eight layers, including five convolutional layers, two fully connected layers, and a softmax output layer.

Koala: AlexNet was built by Google DeepMind, a London-based artificial intelligence laboratory. The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is an annual large-scale image and video dataset that was created for the ImageNet project by the ImageNet Large Scale Visual Recognition Challenge, which is an annual competition for image and video analysis.

Re:Google “We Have No Moat, and Neither Does OpenAI” by Hippocrates on Thursday May 04, 2023 @01:10PM (#35818771)
I'd agree they aren't close, but they are way better than I expected to see in a short few months. At this rate they'll be approaching "good enough" for me pretty soon. I don't always need a dissertation out of it unless I'm fooling around. I want quick facts and explainers around difficult code and calculations. Been playing with Vicuna-7b on my iPhone through MLC Chat and it's impressive.

I use DDG over Google for similar reasons. It's good enough, more "free" (less ads), and has better privacy.

Re:Google “We Have No Moat, and Neither Does OpenAI” by akomtu on Thursday May 04, 2023 @01:41PM (#35819277)
Once distributed training is solved, all those big LLMs will be left in the dust.
Re:Google “We Have No Moat, and Neither Does OpenAI” by Hippocrates on Thursday May 04, 2023 @02:21PM (#35819896)
I figured that. I would love to contribute compute to such a thing. Is there any effort or development in progress? What are the hurdles?
Re:Google “We Have No Moat, and Neither Does OpenAI” by foobiekr on Friday May 05, 2023 @01:32PM (#35832507)
define "solved."
Re:Google “We Have No Moat, and Neither Does OpenAI” by Art9681 on Thursday May 04, 2023 @01:27PM (#35819056)
If all you’ve done is download the model and perform basic prompts then I understand why you think this. There is a lot more going on behind Bard and GPT than a chat window passing the inputs to the model.

Edit for clarity: You’re comparing a platform (Bard, GPT) to a model (llama, etc). The majority of folks playing with local models are missing the platform.

In order to close the gap, you need to hook up the local models to LangChain and build up different workflows for different use cases.

Consequently, this is also when you start hitting the limits of consumer hardware. It’s easy to download a torrent, double click the binary and pass some simple prompts into the basic model.

Once you add memory, agents, text splitters, loaders, vector db, etc, is when the value of a high end GPU paired with a capable CPU + tons of memory becomes evident.

This still requires a lot of technical experience to put together a solution beyond running the examples in their docs.

Re:Google “We Have No Moat, and Neither Does OpenAI” by alsodumb on Thursday May 04, 2023 @02:08PM (#35819677)
All the things you mentioned make it a platform, but even as a model, none of the smaller open-source models come close to GPT 3.5 or 4 in my experience. You can test it by using the GPT3.5 or 4 with their API. They outputs are waaaay better than anything I get from the open source models.
Re:Google “We Have No Moat, and Neither Does OpenAI” by Art9681 on Thursday May 04, 2023 @08:09PM (#35823808)
I am not doubting you and my experience has been the same. My current home lab has a pretty good Jupyter server where I experiment with different local models vs GPT using LangChain and the simple chains can achieve some impressive parity with GPT3.5 depending on the use case and local model. Things do break down when we I do more complex things due to compute capacity. Im still running all of the local models on CPU mind you. I have not gotten to the point of testing on a high end GPU yet but based on what ive seen so far, it won't take much more to run smaller local models that are good enough. This is they key. On the client side, we want smaller more focused models. This is what the post linked in this thread hints at and I agree. We are months, if not weeks, if not days...and maybe hours (at this pace!) where those smaller more domain specific models are common. Still, they won't solve the issues I mentioned above. You will likely need to build your own platform around it, or pay exorbitant fees to host it in the Cloud.
Re:Google “We Have No Moat, and Neither Does OpenAI” by crazyedgar on Thursday May 04, 2023 @02:28PM (#35819983)
Are you sure? I have yet to see any evidence that anyone at all (including Google) has built a model (or a "platform" as you prefer to refer to them) that can follow instructions as well as 50% of ChatGPT, let alone GPT-4. I don't think any amount of work in LangChain and vector databases is enough to fix this: you really need a strong base model that is trained to align with human intentions well. Of course if you just want a bot that can answer free-form simple questions, then maybe people can't tell the difference. Just give them some real work to do and it becomes glaringly obvious.
Re:Google “We Have No Moat, and Neither Does OpenAI” by fzliu on Thursday May 04, 2023 @03:00PM (#35820442)
Vector databases such as Milvus are only there to help reduce/minimize hallucinations rather than get rid of them completely. Until we have a model architecture that can perform completion from the _prompt only_ rather than pre-training data, hallucinations will always be present.