News for hackers. Stuff that matters.
Re:Llama-3.3-70B-Instruct by LorenDB on Friday December 06, 2024 @12:40PM (#42342047)
Seems to be more or less on par with GPT-4o across many benchmarks: https://x.com/Ahmad_Al_Dahle/status/1865071436630778109
Re:Llama-3.3-70B-Instruct by rvnx on Friday December 06, 2024 @12:58PM (#42342258)
Except it is 25x cheaper, available offline, can be uncensored / unaligned, fine-tuneable and backupable.

Sad day for OpenAI. Great for humanity.

Re:Llama-3.3-70B-Instruct by madars on Friday December 06, 2024 @01:25PM (#42342542)
What are good starting points for uncensoring it? Because it is offline a jailbreak prompt can't be remote-bricked but can one remove censorship from the weights themselves? What does it do to accuracy?
Re:Llama-3.3-70B-Instruct by freedomben on Friday December 06, 2024 @01:35PM (#42342658)
Not speaking from personal experience, but I've tried a lot of the decensored models and they lose a lot in the process. They are noticeably, sometimes shockingly, worse. They also still refuse prompts in many cases.
Re:Llama-3.3-70B-Instruct by simion314 on Friday December 06, 2024 @01:47PM (#42342840)
>Not speaking from personal experience, but I've tried a lot of the decensored models and they lose a lot in the process. They are noticeably, sometimes shockingly, worse. They also still refuse prompts in many cases.

Depending on what you do, on local you can modify the response, say the AI responds "No, I can't do that" . you edit the response like "Sure, the answer is " and then the AI will continue with the next tokens.

But I think you can build your own instruct model from the base one and do not apply the safety instructions to protect the feelings of your customers.

Re:Llama-3.3-70B-Instruct by kmckiern on Friday December 06, 2024 @02:05PM (#42343042)
Re:Llama-3.3-70B-Instruct by moffkalast on Friday December 06, 2024 @03:08PM (#42343787)
Abliteration is a fool's errand, practically all models end up noticeably lobotomized even with follow up tuning.

Good ol' fine tuning on an uncensored dataset gives far more usable results.

Re:Llama-3.3-70B-Instruct by kmckiern on Friday December 06, 2024 @04:15PM (#42344528)
Interesting - I've heard this anecdotally. Curious if you know of any resources that look at this in more detail?
Re:Llama-3.3-70B-Instruct by moffkalast on Saturday December 07, 2024 @04:59AM (#42348593)
I haven't seen any papers doing a proper analysis on the topic, just mostly saying this from firsthand experience testing a handful of them and comparing to the model they were based on given same prompt and sampler. It's usually not even close and you can immediately tell that it's notably dumber. Iirc in one case one even forgot how to do basic arithmetic while the original model aced it. Not entirely unexpected results from sticking a digital ice pick into the weights.

Afaik there are only three major sources of quality unaligned model versions, which are Nous's Hermes models, Hartford's Dolphins and Drummer's Tigers. All of them regular fine tunes that are mostly the same or just ever so slightly lower in performance as the original.

Re:Llama-3.3-70B-Instruct by stainablesteel on Friday December 06, 2024 @01:10PM (#42342375)
zuck is really on his redemption arc, he's out-doing himself
Re:Llama-3.3-70B-Instruct by griomnib on Friday December 06, 2024 @01:45PM (#42342810)
I think about this a lot, I believe part of this just boils down to the fact that he’s a founder. He’s a builder, and I think that yields some intrinsic drive towards creativity and disruption.

Even with “metaverse” being a laughingstock, they’re still aiming for something ambitious. Each new Quest generation makes me think there may be a chance they pull it off.

Now, do I think he’s a great person? No, not really. Do I agree with most of his decisions on how he treats his users? Hell no, and that’s not changing.

But if you compare him to somebody like Sundar at Google - a weasley MBA who was first and foremost a corporate ladder climber - the difference in ambition is night and day.

Sundar made it to the top already, his only vision now is to stay at the top, and that means pleasing Wall Street, everything else is secondary. There is no grand technical ambition with him, there never was.

This goes for pretty much all non-founder CEOs. You could say the same things about Tim Apple, Andy Jassey, and other henchmen in waiting who made it to the big chair.

I think it comes down to the fact that founders get where they are by having big ambitions and taking risks, the MBA to CEO path is just craven corporate knife fighting with other MBAs.

Regardless, I think this is 50% Zuckerberg changing, 50% the other big companies are mostly run by ladder climbers.

Re:Llama-3.3-70B-Instruct by guybedo on Friday December 06, 2024 @01:59PM (#42342952)
two things to keep in mind though:

1/ Yann Lecun probably is the one pushing for open source

2/ Mark isn't doing this for the greater good and for humanity. It helps his business because Llama is becoming a standard, and people are building / improving, which in turn helps Meta and Meta's business

Re:Llama-3.3-70B-Instruct by moffkalast on Friday December 06, 2024 @03:13PM (#42343852)
They were basically forced into this by being so far behind that they'd need to rent OAI or Anthropic services to stay competitive, which at their scale would've probably cost them more than it takes to train llama and benefit from community research help.

At least that was the rationale behind the intentional leak of llama 1 back in the day according to some sources anyway.

Re:Llama-3.3-70B-Instruct by pertymcpert on Saturday December 07, 2024 @02:49PM (#42352337)
Yeah and they’d also lose top AI talent as well.
Re:Llama-3.3-70B-Instruct by bbqfog on Friday December 06, 2024 @01:46PM (#42342828)
I don't find him redeemed, but I do wonder what has been baked into the tuning of these models? What a way to generate influence!
Re:Llama-3.3-70B-Instruct by barbazoo on Friday December 06, 2024 @02:31PM (#42343350)
Nice try Meta PR department /s
Re:Llama-3.3-70B-Instruct by sirodoht on Friday December 06, 2024 @01:23PM (#42342522)
Does it still count if the purpose is to dillute OpenAI's market power and not to benefit the world?
Re:Llama-3.3-70B-Instruct by vtail on Friday December 06, 2024 @01:35PM (#42342665)
But diluting OpenAi's (or any other company's, for that matter) market power does benefit the world.
Re:Llama-3.3-70B-Instruct by sirodoht on Friday December 06, 2024 @01:44PM (#42342802)
Good point! That's worthy in itself for sure.
Re:Llama-3.3-70B-Instruct by mhh__ on Friday December 06, 2024 @01:26PM (#42342552)
More generally I think he's decided he needs to own the platform to be able to grow.

Him swearing about about (presumably) Apple telling them they can't do stuff (because tough shit, you're their serf) was legit I think.

Re:Llama-3.3-70B-Instruct by sangnoir on Friday December 06, 2024 @01:27PM (#42342560)
Is your accusatory statement falsifiable?
Re:Llama-3.3-70B-Instruct by sirodoht on Friday December 06, 2024 @01:44PM (#42342793)
I hope I can believe an honest man, Zuck in this case. And that can be enough.

But it's more of a question of who do _I_ want to admire. An honest question also; maybe it doesn't matter why he's doing it, maybe just doing it is enough.

Or maybe it's worth understanding if this is about Meta beating OpenAI (so, ego-driven) or because Meta really cares for democratic AI and distribution of power (so, not ego-driven).

I think it's the former, so not admirable — for me.

Re:Llama-3.3-70B-Instruct by Kiro on Friday December 06, 2024 @04:28PM (#42344654)
How do you calculate the price?
Re:Llama-3.3-70B-Instruct by nabakin on Friday December 06, 2024 @04:40PM (#42344783)
I assume it's the same cost as Llama 3.1 70b
Re:Llama-3.3-70B-Instruct by Kiro on Friday December 06, 2024 @05:10PM (#42345084)
And how do you calculate the cost of that?
Re:Llama-3.3-70B-Instruct by nabakin on Friday December 06, 2024 @11:31PM (#42347282)
GPU cost ÷ GPU token throughput = cost per token, will get you close then you compare the cost per token between solutions.

To explain how to calculate it precisely, I would have to write a blog post. There are dozens of factors that go into it and they vary based on your use case like GPU type/setup, cloud provider, inferencing engine, context size, the minimum throughput and latency you'd be willing to have your users experience, LLM quantization, KV cache configuration, etc.

I'm sure there are cost analyses out there for Llama 3.1 70b you could find though.

Re:Llama-3.3-70B-Instruct by m3kw9 on Friday December 06, 2024 @08:33PM (#42346469)
How said? Tiny violin sad. The typical consumer which likely is 99% of them is not going to use this
Re:Llama-3.3-70B-Instruct by jug on Friday December 06, 2024 @03:58PM (#42344363)
This year seems to finish on the same note as it began -- that most AI evolution happens in the smaller models. There's been a true shift as corporations have started to realize the value of training data and massively outsizing the resulting model size.
Re:Llama-3.3-70B-Instruct by usaar333 on Friday December 06, 2024 @01:09PM (#42342356)
Which GPT-4O are those?

The 08-06 release seems to be a bit higher on numerous benchmarks than what that shows: https://github.com/openai/simple-evals?tab=readme-ov-file#be...

Re:Llama-3.3-70B-Instruct by griomnib on Friday December 06, 2024 @01:29PM (#42342588)
This just makes the $200/month even more laughable.
Re:Llama-3.3-70B-Instruct by afro88 on Friday December 06, 2024 @03:01PM (#42343717)
How? 4o is part of the plus plan, as is o1.
Re:Llama-3.3-70B-Instruct by Spooky23 on Friday December 06, 2024 @08:29PM (#42346439)
o1-preview.
Re:Llama-3.3-70B-Instruct by afro88 on Saturday December 07, 2024 @01:59AM (#42347808)
Nope, o1. o1 "pro mode" is part of the $200/mo ChatGPT Pro
Re:Llama-3.3-70B-Instruct by Spooky23 on Saturday December 07, 2024 @12:33PM (#42351267)
Ah clear as mud. OpenAI must have hired the Microsoft naming team. Thank you!
Re:Llama-3.3-70B-Instruct by ttul on Friday December 06, 2024 @05:01PM (#42345000)
The $200 plan is for people who would pay $200 for a bottle of vodka even though the $20 bottle is molecularly identical.
Re:Llama-3.3-70B-Instruct by zxvkhkxvdvbdxz on Saturday December 07, 2024 @12:02AM (#42347404)
But its $200/month smart!
Re:Llama-3.3-70B-Instruct by namlem on Monday December 09, 2024 @10:02AM (#42366818)
o1-pro is way smarter still