Phind did the fine-tuning, not Facebook.

This is actually an excellent demonstration of a good reason to open source something. Another company immediately made a dramatic improvement.

Now anyone from Facebook or any companies they have invested in can benefit from that improvement and information about how they did it. Facebook could even take this exact model and add a coding assistant into Messenger if they wanted (not that they are interested in that). Or apply it to other things.

OpenAI has a big lead. The most effective counter to that is actually to team up.

One possible explanation is to make PyTorch victory over TensorFlow/JAX even more apparent: with Google ceasing to release interesting open-source models, and Facebook / Meta continuing doing so (even if under a less permissive license than usual), it means that there's nothing interesting in the TensorFlow/JAX ecosystem, and a lot in the PyTorch one.

(context: I work for Google DeepMind, opinions are my own)

Another possible reason might be to attract and retain talent? Speculating wildly but I'd guess some top researchers would value having their work be the industry standard rather than internal.

Your speculation is pretty accurate afaik. Mark commented that releasing open source models is what is needed to retain the best so it is what they will do (paraphrasing) on his interview with Lex Friedman.

Hmm, but I feel like Jax/Pytorch interop is only getting better. Tensorflow abstraction has already firmly lost.

> Hmm, but I feel like Jax/Pytorch interop is only getting better

There's definitely some convergence. It's good. It still stands for a fact that most people do/think about PyTorch nowadays, not JAX (speaking as someone who likes JAX).

Facebook has had an AI research lab for the better half of a decade now. Their open release model worked well in the past and it seems to be working well with Llama too. That looks like the extent of it.

We should expect more FAANG-sized companied to do this, frankly. It's surprising to see people antipate a 'gotcha' considering how influental and transparent Pytorch was.

Given that LLMs might be a generation-defining technology it's certainly weird that one of the biggest public companies is giving theirs away for free.

They're actually not giving it away for free. At least not LLAMA v2. Once a product successfully monetizes and gains over a certain number of users, an official license from Meta has to be negotiated otherwise Meta could revoke usage of the model. At least that's roughly what the license for LLAMAv2 implies.

That's not quite right. The license never gets revoked. Once over a certain size, the company will not receive licenses to new versions/releases.

This is how I understand it too, and it's very interesting. In any case, if your company reaches that size, I'm sure you'll have replaced it with something else by that time. I'm confident many fully open-source, high-quality and specialized models will be available in the next years - even thanks to Meta. Whatever their strategy is, the models are on our machines, legally, and this is incredible.

But that clause pretty much only applies to mega companies like FB no?

Yeah IIRC its something like 700m users. ie. meta gets to piggy back off any generational winners

Does FB care about any companies smaller than that? If they're Instagram-competitive-successful, FB can afford to buy them.

Web browsers were also a generation-defining technology.

This would be more like Google giving away their search algorithm in 2000.

That would be it if it was OpenAI releasing GPT4. Facebook releasing Llama is like Google releasing Chrome.

The open research tradition has been open for many defining technology, most companies continued releasing for free even when it became clear this tech would be transformative (I think it was obvious by GPT-2).

If Facebook were to implement a language model inside the platform now it would raise a lot of privacy issues. By publishing free to use models they can rely on others to do the work of training the society into using such tools at mass scale. So by the time they adopt in in all their services there will be much less backlash.

Also, it's cheaper.

Not just for the engineer-hours, but for the endless VP/PM political meetings arguing over this approach or that.

Way cheaper to let the market decide, and then reincorporate the winner into their products. A winner which happens to already be based on a semi-compatible tech stack.

Why wouldn't they? It is their best strategy.

I didn't like them that much before, now I do.

They're in the business of encouraging users to post content to get more eyeballs and therefore more ad revenue. So commoditising a way of generating a lot more content is in their interests.

This. I recall in their earnings report, they gave a similar answer, I.e the more this technology is made accessible, the better the content put on social media becomes. They also pointed out that any improvements made can be incorporated into their stack like what happened with open compute and PyTorch.

It degrades Googles and OpenAis ability to monetize.

That alone is worth gold because tech is not a blue ocean anymore - neither from a market opportunity nor from an investor perspective. Your enemies loss is your benefit. They can’t use money they don’t make to mount attacks to dislodge customers, investors, mindshare from you.

Secondly it commoditizes AI. Zuck believes that his platforms ultimately will benefit if there is more content. Because his platforms sell the ability to show YOUR content over the rest, for coin. Just as with news/mobile games, driving the value of your content lower and lower by fostering the creation/supply of more of it is good for FB and bad for you. You have to advertise to rise above the noise. (And no it doesn’t really “affect users”) because they see the same amount of content (attention is finite), it’s entirely FBs discretion across all their surfaces what you see.

Disclosure: Ex FB but not specific insights into current developments.

> Purely to mess with MS?

Unlikely as Microsoft is the preferred partner for Llama 2 [2]. It's not clear what Meta's end goal is, but large companies can loss lead investments to grab mind share then worry about Business models later, an approach which seems to be working well [2] where Meta went from not being in the leading AI companies conversation to being the company behind the models that most OSS innovation is happening around.

At the very least they've already created a strong brand around Llama whose eyes are watching their every release - which is invaluable reach that other companies could ever hope to dream of, making it easy to promote and gain instant traction for new Llama based products (like this).

[1] https://about.fb.com/news/2023/07/llama-2/

[2] https://news.ycombinator.com/item?id=35813322

My little theory is Microsoft actually learned a lesson from their experience going up against LAMP in the 90s. They got positively wrecked with their "Proprietary, $, closed-source Windows, IIS, ActiveX or whatever" approach.

They release of ton of AI stuff themselves and partner on things like this. They're also big backers/investors in OpenAI and their own proprietary products, of course.

I think they're taking that lesson and hedging their bets this time. It's like how a lot of corporations will donate to both political candidates (in the US) to guarantee they'll have access/influence regardless of how things shake out.

A previous post here on HN [1] discussed the leak of the original LLaMA model and Stable Diffusion. The article points out that the open-source community's effort resulted in more innovation, and an ecosystem built up around these models leading to rapid adoption. That gives more control to the original authors of the model against their competitors. I think Meta has recognized this and wants to keep their lead going.

[1] https://news.ycombinator.com/item?id=35813322

They’re end-users of AI, they’re not vendors selling AI. Giving it away for free is not a loss to them.

They’re also already giving out millions of dollars in man hours in releasing and maintaining open source software, as well as lots of other much less used ML models. So they have an existing open source strategy and it makes sense to double down on this one since it’s clearly more impactful than a lot of the other ones

I think a smart company plans for what they see the future as holding, as opposed to optimizing for what maximizing gains in the short term. And text generation programs seem to be following near to the exact same path as image generation programs did after DALL-E.

So in the future we're probably not going to see any single centralized player, but rather a wide decentralization with many options. It seems in this future, providing a 'decentralization ready' to go package that people can use and deploy for all sorts of things is far smarter than blowing through billions of dollars (and goodwill) desperately trying to become a centralized behemoth when you have no moat, no network effect, and perhaps a temporary technical advantage at best.

And while OpenAI has blown through a bunch of good will going from Open-ForSociety-AI to Closed-ForProfit-AI, Facebook has simultaneously gained a bunch of good will. What this translates to is really difficult to even imagine, but it's certainly desirable. And really I think this intangible of good will is something many modern corporations don't give enough value to, and it's hurting them - catching more flies with honey than vinegar, and all that.

> perplexed

Apt choice of word