News for hackers. Stuff that matters.
Re:Exponentially Faster Language Modelling by FooBarWidget on Wednesday November 22, 2023 @03:23PM (#38384611)
I find 7B models to be too stupid. They often respond with nonsense or fail to follow instructions.
Re:Exponentially Faster Language Modelling by GaggiX on Wednesday November 22, 2023 @04:34PM (#38385453)
Even like OpenChat-3.5? (Probably the best 7B model out there) Demo: https://openchat.team/

HuggingFace: https://huggingface.co/openchat/openchat_3.5

On the LLM arena (blinded comparisons), it's the third best non-proprietary model: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...

Re:Exponentially Faster Language Modelling by sorokod on Wednesday November 22, 2023 @05:28PM (#38386069)
What is the sum of odd numbers in this set: 4, 7, 12, 1, 3

The sum of odd numbers in the given set is 4 + 7 + 1 = 12. Therefore, the answer is 12.

Re:Exponentially Faster Language Modelling by voiceblue on Wednesday November 22, 2023 @05:58PM (#38386378)
You're using an LLM as if you don't understand how or why it works. Math is not what it's for. Here, look at what ChatGPT 4 does (for a case that hasn't been fed to it by a hundred other users yet):

    What is the sum of odd numbers in this set: 12345654321, 123456543212, 123456543213, 12345654324? Output only the sum, no code.

    The sum of the odd numbers in the set {12345654321, 123456543212, 123456543213, 12345654324} is 246913086434.
What does that prove? Only that I am using an LLM poorly and/or do not understand what it is. Using OpenChat-3.5 for what LLMs are actually good at (e.g. asking it for shell commands to perform certain operations, getting some general information about a topic) seems to work surprisingly well for a 7B model.
Re:Exponentially Faster Language Modelling by sorokod on Thursday November 23, 2023 @02:04AM (#38390200)
Not sure what you mean by 'using'. The prompt is from one of the benchmarks used in scoring LLM models - don't recall which. Here is another from BBH [1]:

>>> (-1 + 2 + 9 * 5) =

... (-1 + 2 + 9 * 5) = 43

[1] https://github.com/suzgunmirac/BIG-Bench-Hard/tree/main

Re:Exponentially Faster Language Modelling by GaggiX on Thursday November 23, 2023 @04:03AM (#38390871)
This is even worse than the other, just straight up use a calculator.
Re:Exponentially Faster Language Modelling by freilanzer on Thursday November 23, 2023 @04:59AM (#38391177)
The use isn't the calculation, but the benchmark for its arithmetic capability.
Re:Exponentially Faster Language Modelling by all2 on Wednesday November 22, 2023 @05:52PM (#38386324)
Technically 3 is even. \s
Re:Exponentially Faster Language Modelling by imatworkyo on Wednesday November 22, 2023 @08:40PM (#38388104)
is this really what you are using AI for ?
Re:Exponentially Faster Language Modelling by moffkalast on Wednesday November 22, 2023 @05:55PM (#38386350)
Are there any comparisons with Mistral-instruct? I've yet to see anything under 30B beat it in any way.
Re:Exponentially Faster Language Modelling by GaggiX on Wednesday November 22, 2023 @06:20PM (#38386635)
Mistral-instruct is in the arena, 1008 ELO vs 1070 ELO (OpenChat-3.5)
Re:Exponentially Faster Language Modelling by cooper_ganglia on Wednesday November 22, 2023 @04:08PM (#38385131)
Even 65B models approach a level of being almost usable, but still fall short in my personal experience.
Re:Exponentially Faster Language Modelling by cloudking on Wednesday November 22, 2023 @04:16PM (#38385239)
This is why I'm not understanding the excitement around open source models, they pale in comparison to GPT-4 quality, so I have no use for them until we have something comparable.
Re:Exponentially Faster Language Modelling by lettergram on Wednesday November 22, 2023 @10:21PM (#38388833)
We utilize a LLama that frankly competes directly with gpt-4, to the point I can't tell which model we are using (we randomly switch between providers to ensure a robust backend).

I think it likely depends on the use case, but many llama models can be returned and there are quite literally thousands of free versions available.

Re:Exponentially Faster Language Modelling by WhrRTheBaboons on Thursday November 23, 2023 @05:36AM (#38391365)
What is your use case?
Re:Exponentially Faster Language Modelling by lettergram on Thursday November 23, 2023 @09:00PM (#38399973)