Since llama is open source, we're going to see fine tunes and LoRAs though, unlike opus.
Llama is not open source. It's corporate freeware with some generous allowances.
Open source licenses are a well defined thing. Meta marketing saying otherwise doesn't mean they get to usurp the meaning of a well understood and commonly used understanding of the term "open source."
https://opensource.org/license
Nothing about Meta's license is open source. It's a carefully constructed legal agreement intended to prevent any meaningful encroachment by anyone, ever, into any potential Meta profit, and to disavow liability to prevent reputational harm in the case of someone using their freeware for something embarrassing.
If you use it against the license anyway, you'll just have to hope you never get successful enough that it becomes more profitable to sue you and take your product away than it would be annoying to prosecute you under their legal rights. When the threshold between annoying and profitable is crossed, Meta's lawyers will start sniping and acquiring users of their IP.
You seem to be making claims that have little connection to the actual license.
The license states you can't use the model if, at the time Llama 3 was released, you had >700 million customers. It also says you can't use it for illegal/military/etc uses. Other than that, you can use it as you wish.
I call models like this "openly licensed" but not "open source licensed".
Whilst I agree the term isn't ideal, I don't agree with the other comments in the post I originally replied to.
You should be happy that Meta find ways to make money from their models, otherwise it's unlikely that they'd be giving you free access (until your startup reaches 700M+ customers, when the free ride ends).
No it doesn’t. The licence terms talk about that those who on the release date of llama3 had 700M+ customers need an extra licence to use it. It doesn’t say that you loose access to it if in the future you gain that many users.
I don’t care what Zuckerberg says. I care what the licence says. I recommend you to read it. It is shorter and more approachable than the usual rental agreement of a flat.
**
2. Additional Commercial Terms. If, on the Meta Llama 3 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Meta otherwise expressly grants you such rights.
> If, on the Meta Llama 3 version release date, the monthly active users … is greater than 700 million monthly active users in the preceding calendar month …
If that’s not true then the free license applies to you.
OTOH if you are being acquired by Elon Musk, then there may be no due diligence, he will tear up any existing license agreements, spend the next year bickering with Meta on Twitter, then be sued to comply.
I agree too that this is the relevant section.
> It says that if you have 700M+ users then you'll need a new license
It does not say that. It says that if you or your affiliate had 700M+ users on the day of llama3's release date then you need an other licence.
This does not trigger if you just gain 700M+ users. Simply it does not. It does trigger if you become affiliated by someone who in that past date already had 700M+ (for example if google buys you up, or if you become a strategic partner of google).
The key here is "on the Meta Llama 3 version release date" which sets the exact date for when the monthly active users of the products or services should be counted.
> It does not say what the terms of that new license will be
Correct. And I assume the terms would be highly onerous. That I do not dispute.
> or just believe me when I tell you that Zuck said it'll be a commercial license
I believe you on that. That is not what we disagree on. The bit we seem to disagree on is when exactly do you need this extra licence. You state that you need it if your company gains in a future date 700M+ users. That is simply not supported by the very section you quoted above.
There's no reason to suppose that terms of any commercial licensing agreement would be onerous. At this stage at least these models are all pretty fungible and could be swapped out without much effort, so Meta would be competing with other companies for your business, if they want it. If they don't want your business (e.g. maybe you're a FaceBook competitor), then they have reserved right not to license it to you.
In any case, don't argue it with me. In practice this would be your lawyers engaged with Meta and their lawyers, and product licensing team.
No argument here. You can either read it or you can't. :)
I am, this is unambiguously great. Just don't call it open source.
If you get that successful that you cannot use it anymore (have 10% of earth's population as clients) probably you can train your own models already.
Most of these other models, like Llama, are open weight not open source - and open weight is just openwashing, since you’re just getting the final output like a compiled executable. But even with OLMo (and others like Databrick’s DBRX) there are issues with proprietary licenses being used for some things, which prevent truly free use. For some reason in the AI world there is heavy resistance to using OSI-approved licenses like Apache or MIT.
Finally, there is still a lack of openness and transparency on the training data sets even with models that release those data sets. This is because they do a lot of filtering to produce those data sets that happen without any transparency. For example AI2’s OLMo uses a dataset that has been filtered to remove “toxic” content or “hateful” content, with input from “ethics experts” - and this is of course a key input into the overall model that can heavily bias its performance, accuracy, and neutrality.
Unfortunately, there is a lot missing from the current AI landscape as far as openness.
seems like they make everything available.
Models are similar to code you might run on a compiled vm or native operating system. Llama.cpp is to a model as Python is to a python script. The license lays out the rights and responsibilities of the users of the software, or the model, in this case. The training data, process, pipeline to build the model in the first place is a distinct and separate thing from the models themselves. It'd be nice if those were open, too, but when dealing with just the model:
If it uses an OSI recognized open source license, it is an open source model. If it doesn't use an OSI recognized open source license, it's not.
Llama is not open source. It's corporate freeware.
It’s a bit like arguing that Linux is not open source because you don’t have every email Linus and the maintainers ever received. Or that you don’t know what lectures Linus attended or what books he’s read.
The weights “are the thing” in the same sense that the “code is the thing”. You can modify open code and recompile it. You can similarly modify weights with fine tuning or even architectural changes. You don’t need to go “back to the beginning” in the same sense that Linux would continue to be open source even without the Git history and the LKM mailing list.
Linux is open source, because you can actually compile it yourself! You don't need Linus's email for that (and if you needed some secret cryptographic key on Linus' laptop to decrypt and compile the kernel, then it wouldn't make sense to call it open-source either).
A language model isn't a piece of code, it's a huge binary blob that's being executed by a small piece of code that contains little of the added value, everything that matters is in the blob. Sharing only the compiled blob and the code to run makes it unsuitable for an “open source qualifier” (It's kind of the same thing as proprietary Java code: the VM is open-source but the bytecode you run on it isn't).
And yes, you can fine-tune and change things in the model weights themselves the same way you can edit the binary of a proprietary game to disable DRMs, that doesn't make it open-source either. Fine tuning doesn't give you the same level of control over the behavior of the model as the initial training does, like binary hacking doesn't give you the same control as having the source code to edit and rebuild.
In general, software rot is a huge issue, and many projects which may be of future archeological importance are increasingly non-reproducible as dependencies are often not vendored and checked into source, but instead downloaded at compile time from servers which lack strong guarantees about future availability.
Who were the countless unknown contemporaries of Giotto and Cimabue? Of Da Vinci and Michelangelo? Most of what we know about Renaissance art comes from 1 guy - Giorgio Vasari. We have more diverse information about ancient Egypt than the much more recent Italian Renaissance because of, essentially, better preservation techniques.
Compliance, interoperability, and publishing platforms for all this work (HuggingFace, Ollama, GitHub, HN) are our cathedrals and clay tablets. Who knows what works will fill the museums of tomorrow.
So, yeah, unless you own your own world-class datacenter, complete with the nuclear reactor necessary to power the training run, then training is not an option.
I think Zuck's discussion of energy being the limiting factor was one of the more interesting and surprising things to come out of the Dwarkesh interview. We're used to discussion of the $1B, $10B, $100B training runs becoming unsustainable, and chip shortages as an issue, but (to me at least!) it was interesting to see Zuck say that energy usage will be a disruptor before those do (partly because of lead times and regulations in expanding power supply, and bringing it in to new data centers). The sheer magnitude of projected power consumption needed is also interesting.
This is where I have to disagree. Continuing the training of an open model is the same process as the original training run. It's not a fundamentally different operation.
In practice it's not (because LoRA) but that doesn't matter: continuing the training is just a patch on top of the initial training, it doesn't matter if this patch is applied through gradient descent as well, you are completely dependent on how the previous training was done, and your ability to overwrite the model's behavior is limited.
For instance, Meta could backdoor the model with specially crafted group of rare tokens to which the model would respond a pre-determined response (say “This is Llama 3 from Meta” as some kind of watermark), and you'd have no way to figure out and get rid of it during fine-tuning. This kind of things does not happen when you have access to the sources.
That's one of many techniques, and is popular because it's cheap to implement. The training of a full model can be continued with full updates, the same as the original training run.
> completely dependent on how the previous training was done, and your ability to overwrite the model's behavior is limited.
Not necessarily. You can even alter the architecture! There have been many papers about various approaches such as extending token window sizes, or adding additional skip connections, quantization, sparsity, or whatever.
> specially crafted group of rare tokens
The analogy here is that some Linux kernel developer could have left a back door in the Linux kernel source. You're arguing that Linux would only be open source if you could personally go back to the time when it was an empty folder on Linus Torvald's computer and then reproduce every step it took to get to today's tarball of the source, including every Google search done, every book referenced, every email read, etc...
That's not what open source is. The code is open, not the process that it took to get there.
Linux development may have used information from copyrighted textbooks. The source code doesn't contain the text of those textbooks, and in some sense could not be "reproduced" without the copyrighted text.
Similarly, AIs are often trained on copyrighted textbooks but the end result is open source.
You can alter the architecture, but you're still playing with an opaque blob of binary *you don't know what it's made of*.
> The analogy here is that some Linux kernel developer could have left a back door in the Linux kernel source. You're arguing that Linux would only be open source if you could personally go back to the time when it was an empty folder on Linus Torvald's computer and then reproduce every step it took to get to today's tarball of the source, including every Google search done, every book referenced, every email read, etc...
No, it is just a bad analogy. To be sure that there's no backdoor in the Linux kernel, the code itself suffice. That doesn't mean there can be no backdoor since it's complex enough to hide things in it, but it's not the same thing as a backdoor hidden in a binary blob you cannot inspect even if you had a trillion dollar to spend on a million of developers.
> The code is open, not the process that it took to get there.
The code is by definition a part of a process that gets you a piece of software (which is the actually useful binary), and it's the part of the process that contains most of the value. Model weights are binary, and they are akin to the compiled binary of the software (training from data being a compute-intensive like compilation from source code, but orders of magnitude more intensive).
> Similarly, AIs are often trained on copyrighted textbooks but the end result is open source.
Court decisions are pending on the mere legality of such training, and it has nothing to do with being open-source, what's at stake is whether or not these models can be open-weight or if it is copyright infringement to publish the models.
There are places (e.g. in the Linux kernel? AMD drivers?) where lots of generated code is pushed and (apart from the rants of huge unwieldy commits and complaints that it would be better engineering-wise to get their hands on the code generator, it seems no one is saying the AMD drivers aren't GPL compliant or OSI-compliant?
There are probably lots of OSS that is filled with constants and code they probably couldn't rederive easily, and we still call them OSS?
The starting point is the ability to run the LLM as you wish, for any purpose - so if a license prohibits some uses and you have to start any usage with thinking whether it's permitted or not, that's a fail.
Then the freedom where "source" matters is the practical freedom to change the behavior so it does your computing as you wish. And that's a bit tricky - since one interpretation would require having the training data, training code and parameters; but for current LLMs the training hardware and cost of running it is a major practical limitation, so much that one could argue that the ability to change the behavior (which is the core freedom that we'd like) is separate from the ability to recreate the model, and would be more relevant in the context of the "instruction training" which happens after the main training, is the main determiner of behavior (as opposed to capability), and so the main "source would be the data for that (instruct training data, and the model weights before that finetuning) so that you can fine-tune the model on different instructions, which requires much less resources than training it from scratch, and don't have to start with the instructions and values imposed on the LLM by someone else.
I'm curious: given that the model will probably be hosted in a private server, how would meta know or prove that someone is using their model against the license?
You are using all these scary words without saying the obvious, which is that for almost everyone, none of that matters.
Of course random individuals don't care much about the licenses on their personal AI projects. But if you intend to grow something significant, you better read the label from the start.
Most of the qualities that come with open source (which also come with llama 3), matter a lot.
But no, it is not a binary, yes or no thing, where something is either open source and useful or not.
Instead, there is a very wide spectrum is licensing agreements. And even if something does not fit the very specific and exact definition of open source, it can still be "almost" there and therefore be basically as useful.
I am objecting to the idea that any slight deviation from the highly specific definition of open source means that it no longer "counts".
Even though, If something is 99.9% the same as open source, then you get 99.9% of the benefits, and it is dishonest to say that it is significantly different than open source.
I don’t necessarily disagree with your point about there still being value in mostly-open software, but I want to challenge your notion that you still get most of the benefit. I think it being less than 100% open does significantly decay the value, since now you will always feel uneasy adopting these models, especially into an older existing company.
You can imagine a big legacy bank having no problem adopting MIT code in their tech. But something with an esoteric license? Even if it’s probably fine to use? It’s a giant barrier to their adoption, due to the risk to their business.
That’s also not to say I’m taking it for granted. I’m incredibly thankful that this exists, and that I can download it and use it personally without worry. And the huge advancement that we’re getting, and the public is able to benefit from. But it’s still not the same as true 100% open licensing.
Well if the missing piece is a cup holder on the train, yes absolutely! It would absolutely be as good as the binary "contains a cup holder" train design.
So the point stands. For almost everyone, these almost open source licenses are good enough for their usecase and the limitations apply to almost noone.
And you have chosen a wonderful example that exactly proves my point. In your example, the incorrect people are claiming that "99.9%" of a train is dangerous to ride in, while ignoring the fact that the missing .1% is the cup holders.
> You can imagine a big legacy bank
Fortunately, most people aren't running a big legacy bank. So the point stands, once again.
> It’s a giant barrier to their adoption
Only if you are at a big legacy bank, in your example, or similar. If you aren't in that very small percentage of the market, you are fine.
Models aren’t code, some of the concepts of open source code don’t map 1:1 to freely available models.
In spirit I think this is “open source”, and I think that’s how the majority of people think.
Turning everything into some sort of theological debate takes away a lot of credit that Meta deserves. Google isn’t doing this. OpenAI sure as fuck isn’t.
No, it isn't. You do, but, as evidenced by other comments, there's clearly people that don't. Thinking that you're with the majority and it's just a vocal minority is one thing, but it could just as easily be said that the vocal groups objecting to your characterization are representative of the mainstream view.
If we look at these models as the output of a compiler, that we don't have the inputs to, but that we are free (ish) to use and modify and redistribute, it's a nice grant from the copyright holder, but that very much doesn't look like open source. Open source, applied to AI models would mean giving us (a reference to) the dataset and the code used to train the model so we could tweak it to train the model slightly differently. To be less apologetic or something by default, instead of having to give it additional system instructions.
Model Available(MA) is freer than Model unavailable, and it's more generous than model unavailable, but it's very much not in the spirit of open source. I can't train my own model using what Meta has given us here.
And just to note, Google Gemma is the one they are releasing weights for. They are doing this and deserve credit for it.
Meanwhile, there are also good reasons not to water down a well-defined term so it becomes meaningless like “agile” or “open.”
This gets confusing because people want to use “open source” as a sort of marketing term that just means it’s good, so if you say it’s not open source that’s taken to imply it’s bad.
I think at a fundamental level people have to start thinking a little differently about what this is, what open really means, and the like.
Open Source has a specific meaning and this doesn't meet it. It's generous of Meta to give us these models and grant us access to them, and let us modify them, fine tune them, and further redistribute them. It's really great! But we're still in the dark as to how they came about the weights. It's a closed, proprietary process, of which we have some details, which is interesting and all, but that's not the same as having access to the actual mechanism used to generate the model.
So it's a bit silly for anyone to claim a model is open source, but it's not silly to say a model is open. What open means isn't well defined when it comes to a model in the same way that source code is.
Imo if someone reveals the model's architecture and makes the weights available with minimal limitations, it's probably reasonable to call it open. I don't know that that would apply to llama though since I believe there are limitations on how you can use the model.
It's not theological, it's the misuse of a specific legal definition that we all have interest in maintaining. "Freely available models" or "open license" are accurate.
Other companies keeping things for themselves doesn't warp reality, or the existing definitions we use to describe it. Giving them the credit they deserve, especially in comparison to the others, should be enough.
Meta is a corporation, and not subject to rational, good faith human judgment. It's a construct that boils down to an algorithmic implementation of the rules, regulations, internal policies, communication channels, and all those complex interactions that effectively prevent sensible, good faith human intervention at any given stage that would even allow the company to just let people continue to violate their stated license. Like trademarks, if you don't enforce a contract, the inaction dissipates your ability to enforce it later on. They don't pay these lawyers to come up with these licenses and contracts for shits and giggles.
The license is not the outcome of a happy weekend brainstorm session tacked on ad hoc just to maximize the benefit to humanity and blissfully join the wide world of open source.
The license is intended to prevent any serious competitive use of their AI models by third parties. It was crafted deliberately and carefully and expensively. They didn't use existing open source licenses because no license offered them the particular mix of rights and restrictions that fit their overall strategy. It's for PR, the ability to stifle competition, to get free beta testing and market research, and 100% of every part of the license is intentional and an insidious perversion of the idea of "open."
Meta doesn't deserve credit, they deserve condemnation. They could have gone with any number of open source licenses, using GPL or CC licensing with specific provisions to protect their interests and prevent commercial exploitation, or use dual licensing to incentivize different tiers of access. They deliberately and with a high level of effort pursued their own invented license. They are using weasel words and claiming they are open source all over the place in order to foster good will.
The argument "but nobody has been sued" is more than a little silly. There's simply no product known to use their models currently on the market that's both a blatant enough violation and worth enough money to sacrifice the good will they've been fostering. There's no human in organizations that size with the capacity to step in and prevent the lawsuits from happening. It'll be a collective, rules and policies decision completely out of anyone's hands to prevent, even if Zuck himself wanted to intervene. The shareholders' interests reign supreme.
Meta isn't a moral institution. It's a ruthlessly profitable one.
This is interesting. Can you point me to an OSI discussion what would constitute an open source license for LLMs? Obviously they have "source" (network definitions) and "training data" and "weights".
I'm not aware of any such discussion.
https://opensource.org/blog/open-source-ai-definition-weekly...
Here is the latest draft definition:
https://hackmd.io/@opensourceinitiative/osaid-0-0-7
And a discussion about the draft:
https://discuss.opensource.org/t/draft-v-0-0-7-of-the-open-s...
So, no way to reproduce the model. This requirement for 'open source' is absurd. It cannot be reliably done even for small models due to GPU internal randomness. Only the smallest trained on CPU in single thread. Only academia will be interested.
link: https://allenai.org/olmo
At that scale, open weights with permissive license is much more useful than open source.
dunno if english is your mother tongue, but this sounds really good (although a tad aggressive :-) )) !
lmaooo.
I was asking scientifically too. I mean, I had intentions, but I wasnt doing anything outright bad.
It's not quite as bad as Gemini, but in the same class where it's almost not useful because so often it refuses to do anything except lecture. Still very grateful for it, but I suspect the most useful model hasn't happened yet.
"Visible nipples? The defining characteristic of all mammals, which infants necessarily have to put in their mouths to feed? On this website? Your account has been banned!"
Meanwhile in Berlin, topless calendars in shopping malls and spinning-cube billboards for Dildo King all over the place.
So, lets not pretend its something that isnt arousing.
Ankles were arousing in 1800s Britain. They might still be in some places.
Not to worry - uncensored finetunes will be coming shortly.
Sure, it would be nice if we didn't have to do that so that the model could actually spent its full capacity on something useful. But that's a different issue even if the root cause is the same.
Log in to save your conversation history, sync with Messenger, generate images and more.
I've never had a FaceBook account, and really don't trust them regarding privacy
They don't need you to login to get what they need, much like Google
Of course numbers are pretty random, but it's just to give an idea of how these things scale. This is my experience from my company's own internal -deep learning but not LLM- models to train which we had to buy data instead of collecting it. If you can't tap into data "from the wild" -in our case, for legal reason- you can still get enough data (if measured in GB), but it's depressingly more repetitive, and that's not quite the same thing when you want to generalize.
Modern captchas are self driving object labelers; you just need a few to "agree" to know what the right answer is.
That's ominous...
Yeah, almost like comparing a 70b model with a 1.8 trillion parameter model doesn't make any sense when you have a 400b model pending release.
Where is it available? I got this in Norway.
https://about.fb.com/news/2024/04/meta-ai-assistant-built-wi...
Or run a jupyter notebook from Unsloth on Colab
At this point, I think paying Facebook to advertise is a waste of money — the actual spam in my junk email folder is better targeted.
If only we’d be so lucky. I don’t thing they will, but fingers crossed.
[0] https://www.europarl.europa.eu/topics/en/article/20230601STO...
Maybe they mean specific uses of these things in a centralised manner but the way it’s written makes it sound incredibly broad.
The same reason that Threads was launched with a delay in EU. It simply takes a lot of work to comply with EU regulations, and by no surprise will we see these launches happen outside of EU first.
But if you say "It's because of regulations!" I hope you have a source to back that up.
The AI Act will significantly nerf the capabilities you will be allowed to benefit from in the eu.
Regulations in the name of the users are actually just made to solidify the top lobbyists in their positions.
The reasons I hate regulations is not because billionaires have to spend an extra week on some employee's salary, but because it makes it impossible for me tiny business to enter a new business due to the sheer complexity of it (or force me to pay more for someone else to handle it, think Paddle vs Stripe thanks to EU VATMOSS)
I'm completely fine with giving away some usage data to get a free product, it's not like everyone is against it.
I'd also prefer to be tracked without having to close 800 pop-ups a day.
Draconian regulations like the EU ones destroy entire markets and force us to a single business model where we all need to pay with hard cash.
MOSS is a massive reduction in overhead vs registering in each individual country, isn't it? Or are you really just saying you don't like sales tax?
But, in my experience, it is also true that "regulations" is sometimes a convenient excuse for a vendor to not do something, whether or not the regulations actually say that.
Years ago, I worked for a university. We were talking to $MAJOR_VENDOR sales about buying a hosted student email solution from them. This was mid-2000s, so that kind of thing was a lot less mainstream then compared to now. Anyway, suddenly the $MAJOR_VENDOR rep turned around and started claiming they couldn't sell the product to us because "selling it to a .edu.au domain violates the Australian Telecommunications Act". Never been a lawyer, but that legal explanation sounded very nonsensical to me. We ended up talking to Google instead, who were happy to offer us Google Apps for Education, and didn't believe there were any legal obstacles to their doing so.
I was left with the strong suspicion that $MAJOR_VENDOR didn't want to do it for their own internal reasons (product wasn't ready, we weren't a sufficiently valuable customer, whatever) and someone just made up the legal justification because it sounded better than whatever the real reason was
Whether that applies to the specific regulations in question here, I don't know – but even if it doesn't, it may take them some time for their lawyers to research the issue and tell them that.
Similarly, for Serbia, a plausible explanation is they don't actually know what laws and regulations it may have on this topic–they probably don't have any Serbian lawyers in-house, and they may have to contract with a local Serbian law firm to answer that question for them, which will take time to organise. Whereas, for larger economies (US, EU, UK, etc), they probably do have in-house lawyers.
You also said that when Meta delayed the Threads release by a few weeks in the EU. I recommend reading the princess on a pea fairytale since you seem to be quite sheltered, using the term draconian as liberally.
July to December is not "a few weeks"
In my opinion this is a thought stopping cliche that throws the concept of differences of scale out the window, which is a catastrophic choice to make when engaging in comparative assessments of policies in different countries. Again just my opinion here but I believe statements such as these should be understood as a form of anti-intellectualism.
What is anti-intellectual about what I said? If you take a step back you see that your response actually contains no argumentative content.
China's central government also believes it is protecting its citizens.
> while the EU privacy protection laws are put in place to protect citizens
The fact that they CAN exert so much power on information access in the name of "protection" is a bad precedent, and opens the door to future, less-benevolent authoritarian leadership being formed.
(Even if you think they are protecting their citizens now, I actually disagree; blocking access to AI isn't protecting its citizens, it's handicapping them in the face of a rapidly-advancing world economy.)
They don't have any power on information access. They just require their citizen can decide what you do with it. There is no central system where information is stored that can be used in future by authoritarian leadership. But the information stored about American by American companies can be use in such a way if there one day an authoritarian leadership in America.
Anyone who's taking a course in epistemology can tell you that there's more to assessing veracity of a belief than noting its equivalence to other beliefs. There can be symmetry in psychology without symmetry in underlying facts. So noting an equivalence of belief is not enough to establish an equivalence in fact.
I'm not even saying I'm for or against the EU's choices but I think the purpose of analogies to China is kind of rhetorical purpose of warning or a comparison intended to reflect negatively on the EU. I find it hard to imagine one would make a straight faced case that they are in fact equivalent in scope or scale or ambition or equivalent and their idea of the relation of their mission to their values for core liberties.
I think the difference is here are clear enough that reasonable people should be able to make the case against AI regulation without losing grasp of the distinction between European and Chinese regulatory frameworks.
Lol this is the real silly provocative comparison.
China bans sites & apps from the West that violate their laws - the ad tracking, monitoring, censorship & influencer/fake news we have here... the funding schemes and market monopolizing that companies like Facebook do in the West is just not legal there. Can you blame them for not wanting it? You think Facebook is a great company for citizens, yet TikTok threatens freedom? Lol it's like I'm watching Fox News.
Companies that don't violate Chinese laws and approach China with realistic deals are allowed to operate there - you can play WoW in China because unlike Facebook it's not involved in censorship, severe privacy violations etc. and Blizzard actually worked with China (NetEase) to bring their product to market there instead of crying and trying to stoke WW3 in the news like our social media companies are doing. Just because Facebook and Google can do whatever they want unchecked in America and its vassal the EU, doesn't mean other countries have to allow it. I applaud China for upholding their rule of law and their traditions, and think it's healthy for the real unethical actors behind our companies to get told "No" for once in their lives.
US and its puppet EU just want to counter-block Chinese apps like TikTok in retaliation for them upholding their own rule of law. Sounds like you fell for the whole "China is a big scary oppressor" bit when the West is an actual oppressor - we have companies that control the entire market and media narrative over here - our companies and media can control whether or not white people can be hired, or can predict what you'll buy for lunch. Nobody has a more dangerous hold on citizens than western corporations.
As regards stand-alone AI systems, namely high-risk AI systems other than those that are
safety components of products, or that are themselves products, it is appropriate to classify
them as high-risk if, in light of their intended purpose, they pose a high risk of harm to the
health and safety or the fundamental rights of persons, taking into account both the severity
of the possible harm and its probability of occurrence and they are used in a number of
specifically pre-defined areas specified in this Regulation. The identification of those
systems is based on the same methodology and criteria envisaged also for any future
amendments of the list of high-risk AI systems that the Commission should be
empowered to adopt, via delegated acts, to take into account the rapid pace of
technological development, as well as the potential changes in the use of AI systems.
https://ec.europa.eu/commission/presscorner/detail/en/qanda_....
> Our largest models are over 400B parameters and, while these models are still training, our team is excited about how they’re trending. Over the coming months, we’ll release multiple models with new capabilities including multimodality, the ability to converse in multiple languages, a much longer context window, and stronger overall capabilities.
I understand that parameters mean different things for different models, and llama two had 70 b parameters, so I'm wondering if anyone can contribute some guesstimation as to what might be expected with the larger model that they are teasing?
But the fact that open models are beating state of the art from 6 months ago is really telling just how little moat there is around AI.
Hell there are not for profits that can afford that.
I mean anyone can throw out self evident general truisms about how there will always be new models and always new top dogs. It's a good generic assumption but I feel like I can make generic assumptions and general truisms just as well as the next person.
I'm more interested in divining in specific terms who we consider to be at the top currently, tomorrow and the day after tomorrow based on the specific things that have been reported thus far. And interestingly, thus far, the process hasn't been one of a regular rotation of temporary top dogs. It's been one top dog, Open AI's GPT, I would say that it currently is still, and when looking at what the future holds, it appears that it may have a temporary interruption before it once again is the top dog, so to speak.
That's not to say it'll always be the case but it seems like that's what our near future timeline has in store based on reporting, and it's piecing that near future together that I'm most interested in.
Do you support any other editors? If the list is small, just name them. Not everyone uses or likes VS Code.
Your "Double vs. Github Copilot" page is great.
I've signed up for the Jetbrains waitlist.
I also don't understand why I can't bring my own API tokens. I have API keys for OpenAI, Anthropic, and even local LLMs. I guess the "secret" is in the prompting that is being done on the user's behalf.
I appreciate the work that went into this, I just think it's not for me.
I suppose it's perfectly fair for large language models not necessarily to know these things, but as far as manual fine tuning, I think it would be reasonable to build models that are capable of answering questions about which model they are, their training date, their number of training parameters, and how they are different from other models, etc. Seems like it would be helpful for it to know and not have to try to do its best guess and potentially hallucinate. Although in my experience Llama 3 seemed to know what it was, but generally speaking it seems like this is not necessarily always the case.
Losers:
- Nvidia Stock : lid on GPU growth in the coming year or two as "Nation states" use Llama-3/Llama-4 instead spending $$$ on GPU for own models, same goes with big corporations.
- OpenAI & Sam: hard to raise speculated $100 Billion, Given GPT-4/GPT-5 advances are visible now.
- Google : diminished AI superiority posture
Winners:
- AMD, intel: these companies can focus on Chips for AI Inference instead of falling behind Nvidia Training Superior GPUs
- Universities & rest of the world : can work on top of Llama-3
Google's business is largely not predicated on AI the way everyone else is. Sure they hope it's a driver of growth, but if the entire LLM industry disappeared, they'd be fine. Google doesn't need AI "Superiority", they need "good enough" to prevent the masses from product switching.
If the entire world is saturated in AI, then it no longer becomes a differentiator to drive switching. And maybe the arms race will die down, and they can save on costs trying to out-gun everyone else.
That said, Google’s Gemini integration with Google Workplace apps is useful right now, and seems to be getting better. For some strange reason Google does not have Gemini integration with Google Calendar and asking the GMail integration what is on my schedule is only accurate if information is in emails.
I don’t intend to dump on Google, I liked working there and I use their paid for products like GCP, YouTube Plus, etc., but I don’t use their search all that often. I am paying for their $20/month LLM+Google One bundle, and I hope that evolves into a paid for high quality, no ad service.
Interesting how far LLMs can get? Looks like we are close to scale-up limit. It's technically difficult to get bigger models. The way to go probably is to add assisting sub-modules. Examples would be web search, have it already. Database of facts, similar to search. Compilers, image analyzers, etc. With this approach LLM is only responsible for generic decisions and doesn't need to be that big. No need to memorize all data. Even logic can be partially outsourced to sub-module.
Models are pretty much fungible at this point if you’re not trying to do any LoRAs or fine tunes.
I agree scoreboards are not a highly accurate ranking of model capabilities for a variety of reasons.
It's complicated tasks requiring step by step logical reasoning where GPT-4 is clearly still very much in a league of its own.
No real evidence either can pull that off in any meaningful timeline, look how badly they neglected this type of computing the past 15 years.
Better open models lower the barrier to build products and drive the price down, more options at cheaper prices which means bigger demand for GPUs and Cloud. More of what the end customers pay for goes to inference and not IP/training of proprietary models
And announcing a lot of integration across the Meta product suite, https://about.fb.com/news/2024/04/meta-ai-assistant-built-wi...
Neglected to include comparisons against GPT-4-Turbo or Claude Opus, so I guess it's far from being a frontier model. We'll see how it fares in the LLM Arena.