Re:Exponentially Faster Language Modelling
by FooBarWidget on Wednesday November 22, 2023 @03:23PM
(#38384611)
I find 7B models to be too stupid. They often respond with nonsense or fail to follow instructions.
[ Parent
| on HN ]
Re:Exponentially Faster Language Modelling
by GaggiX on Wednesday November 22, 2023 @04:34PM
(#38385453)
Even like OpenChat-3.5? (Probably the best 7B model out there) Demo: https://openchat.team/
HuggingFace: https://huggingface.co/openchat/openchat_3.5
On the LLM arena (blinded comparisons), it's the third best non-proprietary model: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
Re:Exponentially Faster Language Modelling
by sorokod on Wednesday November 22, 2023 @05:28PM
(#38386069)
What is the sum of odd numbers in this set: 4, 7, 12, 1, 3
The sum of odd numbers in the given set is 4 + 7 + 1 = 12. Therefore, the answer is 12.
Re:Exponentially Faster Language Modelling
by voiceblue on Wednesday November 22, 2023 @05:58PM
(#38386378)
You're using an LLM as if you don't understand how or why it works. Math is not what it's for. Here, look at what ChatGPT 4 does (for a case that hasn't been fed to it by a hundred other users yet):
What is the sum of odd numbers in this set: 12345654321, 123456543212, 123456543213, 12345654324? Output only the sum, no code.
The sum of the odd numbers in the set {12345654321, 123456543212, 123456543213, 12345654324} is 246913086434.
What does that prove? Only that I am using an LLM poorly and/or do not understand what it is. Using OpenChat-3.5 for what LLMs are actually good at (e.g. asking it for shell commands to perform certain operations, getting some general information about a topic) seems to work surprisingly well for a 7B model.
Re:Exponentially Faster Language Modelling
by sorokod on Thursday November 23, 2023 @02:04AM
(#38390200)
Not sure what you mean by 'using'. The prompt is from one of the benchmarks used in scoring LLM models - don't recall which. Here is another from BBH [1]:
>>> (-1 + 2 + 9 * 5) =
... (-1 + 2 + 9 * 5) = 43
Re:Exponentially Faster Language Modelling
by GaggiX on Thursday November 23, 2023 @04:03AM
(#38390871)
This is even worse than the other, just straight up use a calculator.
Re:Exponentially Faster Language Modelling
by freilanzer on Thursday November 23, 2023 @04:59AM
(#38391177)
The use isn't the calculation, but the benchmark for its arithmetic capability.
Re:Exponentially Faster Language Modelling
by all2 on Wednesday November 22, 2023 @05:52PM
(#38386324)
Technically 3 is even. \s
Re:Exponentially Faster Language Modelling
by imatworkyo on Wednesday November 22, 2023 @08:40PM
(#38388104)
is this really what you are using AI for ?
Re:Exponentially Faster Language Modelling
by moffkalast on Wednesday November 22, 2023 @05:55PM
(#38386350)
Are there any comparisons with Mistral-instruct? I've yet to see anything under 30B beat it in any way.
Re:Exponentially Faster Language Modelling
by GaggiX on Wednesday November 22, 2023 @06:20PM
(#38386635)
Mistral-instruct is in the arena, 1008 ELO vs 1070 ELO (OpenChat-3.5)
Re:Exponentially Faster Language Modelling
by cooper_ganglia on Wednesday November 22, 2023 @04:08PM
(#38385131)
Even 65B models approach a level of being almost usable, but still fall short in my personal experience.
Re:Exponentially Faster Language Modelling
by cloudking on Wednesday November 22, 2023 @04:16PM
(#38385239)
This is why I'm not understanding the excitement around open source models, they pale in comparison to GPT-4 quality, so I have no use for them until we have something comparable.
Re:Exponentially Faster Language Modelling
by lettergram on Wednesday November 22, 2023 @10:21PM
(#38388833)
We utilize a LLama that frankly competes directly with gpt-4, to the point I can't tell which model we are using (we randomly switch between providers to ensure a robust backend).
I think it likely depends on the use case, but many llama models can be returned and there are quite literally thousands of free versions available.
Re:Exponentially Faster Language Modelling
by WhrRTheBaboons on Thursday November 23, 2023 @05:36AM
(#38391365)
What is your use case?
Re:Exponentially Faster Language Modelling
by lettergram on Thursday November 23, 2023 @09:00PM
(#38399973)