It does seem like the nicest thing Facebook have ever done by giving so much to the open source LLM scene, I know that it might have been started by a leaker, but they have given so much voluntarily. I mean don’t get me wrong, I don’t like the company but I do really like some of the choices they have made recently.

But I do wonder in the back of my mind why. And I should be suspicious of their angle and I will keep thinking about it. Is it paranoid to think that maybe their angle is putting almost some kind of metadata by style of code being unique to different machines that they can trace generated code to different people? Is that their angle or am I biased in remembering who they have been for the past decade?

Model available, not open source. These models aren't open source because we don't have access to the data sets, nor the full code to train them, so we can't recreate the models even if we had the GPU time available to recreate them.

Everyone using AI in production is using Pytorch by Meta.

Which is open source.

I do not know anybody important in the AI space apart from Google using TensorFlow.

That may be true, but it's largely irrelevant. The ML framework in use has no bearing on whether or not you have the data required to reproduce the model being trained with that framework.

Do you and the GP have 350K GPUs and quality data to reproduce 1:1 whatever Facebook releases in their repos?

Even if you want to reproduce the model and they give you the data, you would need to do this at Facebook scale, so you and the GP are just making moot points all around.

https://about.fb.com/news/2023/05/metas-infrastructure-for-a...

https://www.theregister.com/2024/01/20/metas_ai_plans/

The fact that these models are coming from Meta in the open rather than Google which releases only papers with no model tell's me that Meta's models is open enough for everyone to use.

Besides, everyone using the Pytorch framework benefits Meta in the same way they were originally founded as a company:

Network effects

It's relevant.

There are organisations that are capable of reproduction (e.g. EleutherAI), but yes, you're right, not having the data is largely irrelevant for most users.

The thing that bothers me more is that it's not actually an open-source licence; there are restrictions on what you can do with it, and whatever you do with the model is subject to those restrictions. It's still very useful and I'm not opposed to them releasing it under that licence (they need to recoup the costs somehow), but "open-source" (or even "open") it is not.

If you don’t have a machine powerful enough to compile some heavy AAA game, does that make the game open source?

I'm really curious what their goal is

If you go by what Zuck says, he calls this out in previous earnings reports and interviews[1]. It mainly boils down to 2 things:

1. Similar to other initiatives (mainly opencompute but also PyTorch, React etc), community improvements help them improve their own infra and helps attract talent.

2. Helping people create better content ultimately improves quality of content on their platforms (Both FoA & RL)

Sources:

[1]Interview with verge: https://www.theverge.com/23889057/mark-zuckerberg-meta-ai-el... . Search for "regulatory capture right now with AI"

> Zuck: ... And we believe that it’s generally positive to open-source a lot of our infrastructure for a few reasons. One is that we don’t have a cloud business, right? So it’s not like we’re selling access to the infrastructure, so giving it away is fine. And then, when we do give it away, we generally benefit from innovation from the ecosystem, and when other people adopt the stuff, it increases volume and drives down prices.

> Interviewer: Like PyTorch, for example?

> Zuck: When I was talking about driving down prices, I was thinking about stuff like Open Compute, where we open-sourced our server designs, and now the factories that are making those kinds of servers can generate way more of them because other companies like Amazon and others are ordering the same designs, that drives down the price for everyone, which is good.

Disclaimer: I do not work at Meta, but I work at a large tech company which competes with them. I don't work in AI, although if my VP asks don't tell them I said that or they might lay me off.

Multiple of their major competitors/other large tech companies are trying to monetize LLMs. OpenAI maneuvering an early lead into a dominant position would be another potential major competitor. If releasing these models slows or hurts them that is in and itself a benefit.

Why?

What benefit is there to grabbing market share from your competitors... in a business you don't even want to be in?

By that logic you could justify any bizarre business decision. Should Google launch a social network, to hurt their competitor Facebook? Should Facebook, Amazon and Microsoft each launch a phone?

>Should Facebook, Amazon and Microsoft each launch a phone?

* https://www.lifewire.com/whatever-happened-to-the-facebook-p...

* https://en.wikipedia.org/wiki/Fire_Phone

* https://en.wikipedia.org/wiki/Windows_Phone

> Should Google launch a social network, to hurt their competitor Facebook?

I mean, Google did launch a social network, to hurt their competitor Facebook. It was a whole thing. It was even a really nice system, eventually.

I enjoyed using Google plus more than any other social network, and managed to create new connections and/or have standard, authentic, real conversations with people I didn't know, most of them ordinary people with shared interests that I would probably wouldn't meet otherwise, some of them are people I can't believe I could connect with directly in any other way - newspapers and news sites editors, major SDK developers. And even with Kevin Kelly.

And it turned out that Facebook had quite a moat with network effects. OpenAI doesn’t have such a moat, which may be what Meta is wanting to expose.

Google botched the launch, and they never nurture products after launch anyway. Google+ could have been more successful.

Who says they don't want to be in the market? Facebook has one product. Their income is entirely determined by ads on social media. That's a perilous position subject to being disrupted. Meta desperately wants to diversify its product offerings - that's why they've been throwing so much at VR.

I imagine their goal is to simultaneously show that Meta is still SotA when it comes to AI and at the same time feed a community of people who will work for free to essentially undermine OpenAI's competitive advantage and make life worse for Google since at the very least LLMs tend to be a better search engine for most topics.

There's far more risk if Meta were to try to directly compete with OpenAI and Microsoft on this. They'd have to manage the infra, work to acquire customers, etc, etc on top of building these massive models. If it's not a space they really want to be in, it's a space they can easily disrupt.

Meta's late game realization was that Google owned the web via search and Apple took over a lot of the mobile space with their walled garden. I suspect Meta's view now is that it's much easier to just prevent something like this from happening with AI early on.

Devil's advocate: they have to build it anyway for Meta verse and in general. Management has no interest in going into cloud business. They had Parse long time back but that is done. So why not to release it. They are getting goodwill/mindshare, may set up industry standard and get community benefit. It isn't very different from React, Torch etc.

I think Meta's goal is to subvert Google, MS and OpenAI, after realizing it's not positioned well to compete with them commercially.

Could also be that these smaller models are a loss leader or advertisement for a future product or service... like a big brother to Llama3 that's commercial.

I believe there were rumors they are developing a commercial model: e.g. https://www.ft.com/content/01fd640e-0c6b-4542-b82b-20afb203f...

Commoditizing your complement. If all your competitors need a key technology to get ahead, you make a cheap/free version of it so that they can't use it as a competitive advantage.

The complement being the metaverse. You can’t handcraft the metaverse because it would be infeasible. If LLMs are a commodity that everyone has access to, then it can be done on the cheap.

Put another way - if OpenAI were the only game in town how much would they be charging for their product? They’re competing on price because competitors exist. Now imagine the price if a hypothetical high quality open source model existed that can customers can use for “free”.

That’s the future Meta wants. They weren’t getting rich selling shovels like cloud providers are, they want everyone digging. And everyone digs when the shovels are free.

Zuck just wants another robot to talk to.

Poor guy just wants a friend that won't sell him out to the FTC or some movie producers.

If you want to employ the top ML researchers, you have to give them what they want, which is often the ability to share their discoveries with the world. Making Llama-N open may not be Zuckerberg‘s preference. It’s possible the researchers demanded it.

Commoditize your compliments.

They were going to make most this anyway for Instagram filters, chat stickers, internal coding tools, VR world generation, content moderation, etc. Might as well do a little bit extra work to open source it since it doesn't really compete with anything Meta is selling.

I would guess mindshare in a crowded field, ie discussion threads just like this one that help with recruiting and tech reputation after a bummer ~8 years. (It's best not to overestimate the complexity/# of layers in a bigco's strategic thinking.)

Prevent OpenAI from dominating the market, and at the same time have the research community enhance your models and identify key use cases.

Meta basically got a ton of free R&D that directly applies to their model architecture. Their next generation AIs will always benefit from the techniques/processes developed by the clever researchers and hobbyists out there.

Their goal is to counter the competition. You rarely should pick the exact same strategy as your competitor and count on out gunning them, rather you should counter them. OpenAI is ironically closed, well meta will be open then. If you can't beat them, you should try to degrade down the competitors value case.

Its a smart move IMO

Rule 5: commodify your complement

Content generation is complementary to most of meta's apps and projects

To undermine the momentum of OpenAI.

If Meta were at the forefront, these models would not be openly available.

They are scrambling.

Same as MS, in the game, in the conversation, and ensuring next-gen search margins approximate 0.

Their goals are clear, dominance and stockholder value. What I'm curious about is how they plan to monetize it.

Use it in products, ex. the chatbots

with pytorch (& so many open publications), Meta has had a unimaginably strong contribution to ai for a while

Cambridge Analytica is not a real scandal (did not affect any elections), and FB did not cause a genocide in Myanmar (they were a popular message board during a genocide, which is not the same thing)