I think it's aimed at other social networks.

TikTok has 1 billion monthly active users for instance

I think TikTok would just use it anyway even if they were denied a license (if they even bothered asking for one). They've never really cared about that kind of stuff.

Anyone who has ever worked in a major social media company knows that this is false - but as another person who has, I will chime in and say this is completely wrong, compliance (especially such obvious compliance) is taken seriously.

I worked at a company that caught a major Chinese Internet company (not ByteDance/TikTok, but one even larger) red-handed engaging in deliberate app install ad fraud (their app would send forged Android INSTALL_REFERRER intents), so it would not surprise me.

I'm curious if you've worked at a Chinese company?

Look at Snapchat: https://techcrunch.com/2023/02/16/snapchat-announces-750-mil...

Just above 700m MAU. So yeah, probably aimed at their direct competitors in social.

I think more Apple. It's not like Google or Microsoft would want to use LLaMA when they have fully capable models themselves. I wouldn't be surprised if Amazon does as well.

Apple is the big laggard in terms of big tech and complex neural network models.

I think Google or Microsoft probably would want to use LLaMa for various purposes like benchmarking and improving their own products. Check out this other condition from the license:

v. You will not use the Llama Materials or any output or results of the Llama Materials to improve any other large language model (excluding Llama 2 or derivative works thereof).

https://github.com/facebookresearch/llama/blob/main/LICENSE

Just like Google scrapes the internet to improve their models, it might make sense to ingest outputs from other models to improve their models. This licensing prevents them from doing that. Using Llama to improve other LLMs is specifically forbidden, but Google will also be forbidden from using Llama to improve any other AI products they might be building.

I can see their business logic but isn't it a bit like do not allow people (or bots) talk to each other, they might all get smarter.

I understand trade-secrets are not free-speech but if the goal is to build better AI to serve the humanity the different bots should learn from each other. They should also criticize each other to find flaws in their thinking and biases.

> if the goal is to build better AI to serve the humanity

It’s not.

>but if the goal is to build better AI to serve the humanity

Whose goal is that?

Google's. Do no evil they say

"Don't be evil" was deprecated from Google's charter around the same time that Apple removed "Computer" from their name.

There are many datasets created by scraping chatGPT and they seem to work out pretty well. In other words, LLM skills are leaky.

The goal is to build better AI to make more money.

That's an ugly position on Meta's part. But Llama models are small, they are not going to be preferred to generate synthetic data, GPT-4 is the darling of synth datasets.

A pointless provision given that the License doesn't cover the output of the model, so I can redistribute outputs to someone else, and then they (since they aren't beholden to the license) can now do as they like.

And they want to be very careful about labeling outputs as derivative works, because the moment they do that then they have no defense against the model being a derivative work of every single input.

Google's model is not as capable as llama-derived models, so I think they would actually benefit from this.

> I wouldn't be surprised if Amazon does as well.

I would - they are not a very major player in this space.

TikTok also meets this definition and probably doesn't have LLM.

Google has far better models than llama based models. They just simply don't put them facing the public.

It is pretty ridiculous that they essentially just set a marketing team with no programming experience to write Bard, but that shouldn't fool anyone into believing they don't have capable models in Google.

If Deepmind were to actually provide what they have in some usable form, it would likely be quite good. Despite being the first to publish on RLHF (just right before OpenAI) and bring the idea to the academic sphere, they mostly work in areas tangential to 'just chatbots' (e.g. how to improve science with novel GNNs, etc). However, they're mostly academics, so they aren't set on making products, doing the janitorial work of fancy UIs and web marketing, and making things easy to use, like much of the rest of the field.

I work in this field. I would love to see what you are basing these assertions off of.

> they mostly work in areas tangential to 'just chatbots' (e.g. how to improve science with novel GNNs, etc)

Yes, Alphabet has poured tons of money into exotic ML research whereas Meta just kept pouring more money into more & deeper NLP research.

Google's LLMs are all vaporware. No one's ever seen them. They're supposedly mind-blowing but when they are released they always sound like lobotomized monkeys.

All the AlphaGo/AlphaFold stuff is very cool, but since no one has seen their LLMs this is about as convincing as my claiming I've donated billions to charity.

I can assure you Google BERT isn't vaporware.

It was probably a challenge to integrate it into search, but they did that.

So your assertion has been refuted based on your use of "all", at the very least.

Haha, that's right. Google has BERT. Their AI stuff isn't all vaporware. There's always BERT.

I’ve been hearing “Google has secret better models” for 7 months now. Maybe some UFOs in the hangers at Moffett Field too?

Would you believe OpenAI has vastly better models that they are not releasing publicly?

No

GPT models were internally available 6-12 months before they've seen public beta, of course OpenAI has more capable internal models.

There's no reason to believe this. The training time and cost is so substantial that they are almost certainly building their next release, but it isn't sitting there rotting.

Much of that training time is RLHF, the absence of which does not make the model less capable of carrying out useful tasks (indeed, in case of GPT-4, it actually made the model slightly less capable).

OpenAI themselves have said they hat GPT-4 internally before they ever released the first version of ChatGPT.

Do you realize that LLaMA-1 is just a very slightly smaller, comparably performing replication of Chinchilla [1], which DeepMind had completed a year prior to LLaMA's release? And has RLHF-ed into a suitable chatbot "Sparrow" [2] months earlier than ChatGPT was launched?

To assume that Google doesn't have anything competitive with Meta is to say that their papers just so happen to contain recipes for Meta's models but they've arrived at those not through training and benchmarking but by divination and bullshitting. This, let us say, does not sound plausible.

Then again, Microsoft uses LLaMA for research, and they should theoretically have some ability to get stuff from OpenAI. Evidently this isn't how any of this works, huh.

1. https://arxiv.org/abs/2203.15556

2. https://en.wikipedia.org/wiki/Sparrow_(bot)

Google _internally_ feels that they are way behind. Forget commenters on HN, literally all of the google employees that I know believe that the company is failing here.

This is not responsive to my arguments. Google can be arbitrarily far behind OpenAI or Anthropic, OP's idea that they feel threatened by LLaMA when they (well, Deepmind) have reached LLaMA level 18-10 months ago is still wrong.

Hard disagree. Google has made it plainly clear that they don't have anything useable in this space. Bard scores below all other commercial model.

Google is getting the asses handed to them, badly. I figured that the code red would whip them into shape but the rot runs deep.

Bard is a 4.5B or so model.

> Google has made it plainly clear that they don't have anything useable in this space.

Google hasn't made their best models public because they're too expensive to run for free.

> Google is getting the asses handed to them, badly.

Bard has 30M active users and isn't even available in large parts of the world. They're in 2nd place - when they were pretty late to the game - that's an odd way to say someone is getting their ass handed to them.

> Google hasn't made their best models public because they're too expensive to run for free.

? It's the same with paid models.

I am paying per each request sent to Google Generative AI and this is what I get: https://i.ibb.co/4KCmz55/bard1.png

...

Why do you think Google had even bothered with Bard?

And then, given that, why is it worse than the competition?

It seems you didn't quite hear the argument. I agree with you that the models Google has released to the public are absolutely worthless. That certainly does not mean they don't have extremely performant models at all however.

If you actually have worked in the area of NLP for about 10 years, you would recognize how the work from Deepmind is much more novel and innovative than other groups. OpenAI certainly has great public facing services, and Meta should be congratulated for releasing these models (although I would still prefer the Galactica training data), but academically Deepmind is one of the best groups around.

On that front, Google's Gemini sounds interesting.

See https://www.tomsguide.com/news/googles-new-gemini-ai-could-b...

It's better to wait and see. Either way, they are scraping everyone and everything. If they can't do it...

Why would they have secret unreleased models?

Surely Google can find another team of code monkeys to whip out a frontend if there is money to be made.

I don't think Google is going to pull back from making some more money.

I think the most likely option is that they have a bunch of talented academics who get paid on time to work on what interest them - but they're the stereotypical large inefficient company and they can't coordinate the effort of productionizing some cool models before the competition.

> but academically Deepmind is one of the best groups around

I think your argument is basically that Google has the potential to create the best models because of superiority in the theory of LLMs, even though we hear of no signs from the board, the ceo, or beta releases or product showcases.

But let’s say you’re right. When do you think we would experience the supremacy of DeepMind in our daily lives?

This reminds me of how any day now their self driving cars are going to work right.

Their self driving cars do work? I rode in one for 30 minutes one-way on Sunday. Used it for my return trip too. No driver. Take at least 2 - 3 rides a week and have been for a few months now.

They work (most of the time) in Phoenix and SF because they've mapped every single inch of the cities by now and there are no adverse conditions. It's not scalable.

Why is that not scalable? Mapping out two large cities for an experimental project in a few years seems scalable, expand to new cities over time with additional resources.

I suppose it is, but not in a Silicon Valley way. They could scale to "large Southwestern city taxi service," but it wouldn't earn back the investment or deliver on the hype. If that becomes the ceiling I bet Google will simply shut Waymo down.

If they work out how to deal with, say, New York weather conditions, there's potential, but they don't seem to be any closer.

I think you’re conflating doable and scalable.

Or perhaps my threshhold for “scalable” takes different parameters and weigh these inputs differentfly from you.

Lol google saying they have better models in private is like that one kid that insists he has an awesome girlfriend, but 'she goes to another school, you wouldn't know her'.

I'm pretty sure if google had something much better, the board and C-suite execs would have at least ensured we saw previews of it by now...

Source?

> Google's model is not as capable as llama-derived models, so I think they would actually benefit from this.

Google's publically available model isn't as capable. But they certainly have models that are far better already in house.

I have no idea how you are so certain of that.

Meta is definitely ahead of Google in terms of NLP expertise and has been for a while. I suspect that Google released their best model at the time with Bard.

We still don't have access to Imagen last I checked, it's still in restricted access. We don't have access to SoundStorm or MusicLM

https://imagen.research.google/

https://google-research.github.io/seanet/soundstorm/examples...

https://google-research.github.io/seanet/musiclm/examples/

Why would it be surprising that they have better models for resarch that they don't want to give out yet?

Because I work in NLP so I have a good sense of the different capabilities of different firms and for the Bard release, it would have made more sense for them to have a more limited release of a better model for PR reasons than what actually happened.

The other things you are describing are just standard for research paper releases.

> Bard release, it would have made more sense for them to have a more limited release of a better model for PR reasons than what actually happened.

Yes I would agree with you if Google wasn't set on to full on panic mode by their investors about releasing something vs Open AI due to Chat GPT's buzz.

Bard was just a "hey we can do this too" thing, it was released half assed, had next to no marketing or hype.

Vertex AI is their real proper offering, and I want to see how PaLM 2 does in comparison.

I can already tell you that PaLM is not anywhere near as good and PaLM-2 is at least not as good before RLHF.

Not going to keep replying, believe what you want about Google's capabilities

@dooraven - I also work in ML (including recently working at Google) and I agree with @whimsicalism.

You seem to be under the mistaken belief that: 1. Google has competent high-level organization that effectively sets and pursues long term goals. 2. There is some advantage to developing a highly capable LLM but not releasing it.

(2) could be the case if Google had built an extremely large model which was too expensive to deploy. Having been privy to what they had been working on up until mid-2022 and knowing how much work, compute and planning goes into extremely large models, this would very much surprise me.

Note: I did not have much visibility into what deepmind was up to. Maybe they had something.

ok now I am confused, as Meta themselves say Palm-2 is better than Llama 2?

> Llama 2 70B results are on par or better than PaLM (540B) (Chowdhery et al., 2022) on almost all benchmarks. There is still a large gap in performance between Llama 2 70B and GPT-4 and PaLM-2-L.

https://scontent.fsyd7-1.fna.fbcdn.net/v/t39.2365-6/10000000...

If Google's publically available model is better Llama 2 already then why is it so inconceivable that they'd have private models that are better than their public ones which are better than LLama already.

Palm-2 isn't better than GPT-4 but the convo was about better than Llama models no?

> I have no idea how you are so certain of that.

Some among us work with it, or have friends or family who work with it. I imagine it is one of those.

Do they? Considering how much was at stack in term of PR when OpenAI released ChatGPT, I would be surprised that Google didn’t put out the best they could.

The other end of the PR stake was safety/alignment. If Google released a well functioning model, but it said some unsavory things or carried out requests that the public doesn't find agreeable, it could make Google look bad.

Comments like this remind me of the old-timers from IBM saying "but wait, we invented the PC! and the cloud! and..."

Gotta put products in the market, or it didn't happen...

It's fine not to give them public credit for in-house only things, but in this subthread we're speculating about whether Llama 2 would be useful to them, which does depend heavily on the quality of their internal models.

bringing back PLOSTFU culture might not actually be a bad thing.

OpenAI seemingly downgraded ChatGPT 4 due to the expense of running it for pro customers (unless you run it through the API).

I just googled "What is the order of object-fields in JavaScript" and the bard-answer said nothing about the differences between ES5 and ES6 and ES2020 how by now the order of object-fields in fact is deterministic.

It seems it is not aware of the notion of historic development, perhaps its world-model is "static"?

Temporal reasoning is interesting , if you google for "news" do you get what was news last year because a website updated last year had a page claiming to contain "Latest News".

REF: https://www.stefanjudis.com/today-i-learned/property-order-i...

Has anyone in this subthread actually read the papers and compared the benchmarks? LLama2 is behind PALM-2 on all major benchmarks, I mean they spell this out in the paper explicitly.

Apple would absolutely not want to use a competitors, or any other, public LLM. They want to own the whole stack, and will want to have their own secret source as part of it. It's not like they don't have the capital to invest in training...

Apple does not have the capability to train a LLM currently.

Apple has shown time and time again that they have the human capital and money to tackle massive projects discretely. It's already fairly well known that Apple's NLP experts from Siri have been reallocated to some secret project. They are more than capable of training an LLM but given their track record in other segments they probably want to wait for the technology to become more "polished" and give less hallucinated answers. They are likely also want the LLM to work locally (at least partially) on their devices using the Neural Engine which adds further engineering complexity to their project. They could even be timing the LLM's launch around a hardware release capable of running the model (M3, M4, etc...).

I have not seen Apple demonstrate ML depth in their talent nor have I seen signs that they are hiring extensively for NLP depth.

They will soon be able to train an LLM because it simply has become commoditized, but they just are not a major player in this space at all.

> I have not seen Apple demonstrate ML depth in their talent

I thought the ml work they do in photos for text selection and facial recognition is pretty neat.

Their approach is different, they build ML tech that runs on-device, so whatever they developed has to be able to run efficiently on iPhone/iPad etc.

I don’t think we will “hear” about Apple using LLMs either way because they will no doubt call it something different like they always have.

Apple only has to slightly open their wallet to become a DL superpower.

Apple is a complete laggard in this space due to years of restrictions on research. They are hiring multiple “AI” roles now and they have the capital and focus to “eventually” catch up — but it is very much a catch-up game.

That said, they seem to prefer catchup waiting till others explore new tech they swoop in an (claim) to perfect it from a usability pov. I have no reason to suspect they won’t do the same here.

I very much doubt that.

If they want to own the whole stack, I don't think they have much to work with. Their highest-end server chip is a duplex laptop SOC, with maxed-out memory that doesn't even match the lowest-end Grace CPU you can buy (nevermind a fully-networked GH200). Their consumer offerings are competitive, but I don't think Apple Silicon or CoreML is ready to seriously compete with Grace and CUDA.

While Apple silicone may not be there for training, I think it's probably there for inference. I expect next years device models to launch with exclusive support for Apples own LLM based Siri.

Sure. Haswell CPUs from 2014 are "there" for inference if they have AVX support and 8gb of RAM. Inferencing isn't the problem though, not on M1 or Macbooks from 2016. Scaling a desirable (and hopefully open) GPGPU programming interface is. This is bottlenecked by both hardware and software decisions Apple has made, making a "home grown" competitive model much more unlikely in my eyes.

I agree that there is an incentive to put AI models on your OS. I just don't think Apple can own the whole stack if they want to play ball right now.

Why not? They have cash and they can rent a bunch of GPUs from Amazon.

It's not unthinkable. Look at a product like Siri - at launch it was basically just a WolframAlpha interface. It wasn't a particularly powerful voice assistant, but it did occupy a league of it's own because of the licensed technology. Sure, Apple could have spent a billion dollars to build a new product division and compete against Wolfram, but why?

While this model in particular isn't likely to land in iOS or Windows, the license definitely suggests Meta is trying to cut off FAANG at the pass.

What makes you think that. Apple is the company that would be most successful at hiding something like this then introduce it as siri ai or something. Not that they are I am just saying Apple keeps everything close to its chest when it comes to products it might introduce in the future.

I work in the field and they just are not hiring the people they need to be hiring.

Interesting. The very early adoption of the neural engines in all Apple products would make you think that they had something brewing. Same with the relatively capable m1/2 GPU's. Various models, stable diffusion runs suprisingly fast on these devices and could be optimised to run much, much faster if Apple actually cared, but they weirdly seem not to.

Considering how much Apple likes to retain control, I’m almost sure they won’t want to use someone else’s model even if it were free in every sense of the word.

> OpenAI's ChatGPT hit 100 million MAUs in January, and has gone down since.

poor reading of the numbers. one guy at a bank pulled up similarweb and guesstimated 100m registered users and it went viral. whisper numbers were closer to 50m. but in the 6 months since they have certainly crossed 100m and probably are north of 500m, and only recently dipped.

You are countering whisper numbers with more whisper numbers.

Fight fire with fire..... ?

How do you find Whisper numbers, it’s open source yea?

It's not open source

He's making a pun referring to OpenAI's open-sourced Whisper voice recognition model:

https://openai.com/research/whisper

Whisper numbers are numbers that are secretly shared among industry insiders, not the usage numbers of OpenAI's Whisper.

AWS is listed as a partner: https://ai.meta.com/llama/#partnerships

now, that is interesting. Alphabet only big co missing in that list?

e: nvm. Apple not there either.

School is out, it will pick up again.

Also, any company with 700 million active users wouldn't have much difficulty reproducing this work.

It's total users, not specifically users of the Llama-2-based product. It's actually quite an elegant way to say "if you're going to produce some super cool new tech with this, let's be friends, unless you're big enough to compete with Facebook in which case rack off."