News for hackers. Stuff that matters.
Re:Ask HN: What's the best self hosted/local alternative to ... by jonathan-adly on Wednesday May 31, 2023 @10:00AM (#36138531)
I don't know the licensing and all that jazz (even if you self-host for your personal use it shouldn't matter). But, this paper[0] released a week ago claims " 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU" (QLORA).

A quick test of the huggingface demo gives reasonable results[1]. The actual model behind the space is here[2], and should be self-hostable with reasonable effort.

0. https://arxiv.org/abs/2305.14314 1. https://huggingface.co/spaces/uwnlp/guanaco-playground-tgi 2. https://huggingface.co/timdettmers/guanaco-33b-merged

Re:Ask HN: What's the best self hosted/local alternative to ... by _laiq on Wednesday May 31, 2023 @10:11AM (#36138639)
Incredible how people are underestimating ChatGPT or overestimating open-source models.

A basic question: How can i join with SQL a column to a string separated with comma

GPT-4: PostgreSQL: SELECT STRING_AGG(columnName, ', ') FROM tableName;

Guanaco: Here is an example of how you could use the CONCAT function in MySQL to concatenate a string to a column value in a single-line query: SELECT CONCAT('The total price for ', product_name, ’ is ', SUM(price)) AS total FROM products; This will result in output like “The total price for Chocolate Bar is 10” or similar depending on your data.

Yeah, no...

Re:Ask HN: What's the best self hosted/local alternative to ... by amluto on Wednesday May 31, 2023 @10:26AM (#36138797)
> How can i join with SQL a column to a string separated with comma

I think I might give this one to Guanaco. I’m moderately familiar with SQL, and I can’t really understand your question. You’re faking the reader out by using “join” to mean something which is not JOIN.

So do you mean to concatenate a string in a column (coming from a single row) to a “string separated with a comma”? If so (and assuming “separated” means starting or ending with), then Guanaco nailed it. If you meant to join (in the Python sense) the values from a given column in _multiple_ rows, then GPT-4 is doing better.

But GPT-4 gave no explanation, and my general experience with it is that it’s happy to write code that does _something_ vaguely related to the prompt. As the prompt gets more complex or unusual, the degree to which the code doesn’t actually do something useful increases. And I have had quite poor results getting ChatGPT to generate code along with some explanation such that the code actually matches the explanation.

Re:Ask HN: What's the best self hosted/local alternative to ... by RC_ITR on Wednesday May 31, 2023 @10:47AM (#36139029)
> You’re faking the reader out by using “join” to mean something which is not JOIN.

Yeah and not having to do precise prompt engineering is a major advantage of GPT-4 vs other models.

Re:Ask HN: What's the best self hosted/local alternative to ... by amluto on Wednesday May 31, 2023 @10:58AM (#36139130)
I don’t think that “prompt engineering” is quite the same thing as clearly specifying the problem. If I’m using a magic AI code-writing companion or asking an actual human to write code for me, I need to define the problem well enough if I want any expectation of getting _correct_ code.

A skilled human will distinguish themselves from a less skilled human by also thinking about the overall problem and helping make sure the specification is correct. This far, I have not seen an AI meet that bar unless the problem is something well-covered already on the web.

I have seen ChatGPT (3.5 and 4) output code that is syntactically correct, executes without errors, and does the wrong things in ways that are not entirely obvious to me until I read the docs myself.

Re:Ask HN: What's the best self hosted/local alternative to ... by hunter2_ on Wednesday May 31, 2023 @03:30PM (#36142953)
> helping make sure the specification is correct. This far, I have not seen an AI meet that bar

I've only used Bard so far, but indeed, it has never replied to my question with a clarifying question. Only with a statement. This seems like an obvious evolution of the chatbot concept...

Re:Ask HN: What's the best self hosted/local alternative to ... by gpderetta on Wednesday May 31, 2023 @10:53AM (#36139076)
To be fair, I had to read the postgresql docs and interpret the answer to figure out what the _laiq was actually asking.

edit: I guess GPT-4 has better reading comprension than me!

Re:Ask HN: What's the best self hosted/local alternative to ... by puntofisso on Thursday June 01, 2023 @08:23AM (#36150148)
I wish there was a model out there that asked me "what do you mean by 'join'" but I guess we're somewhat far from there :)
Re:Ask HN: What's the best self hosted/local alternative to ... by berniedurfee on Thursday June 01, 2023 @09:09AM (#36150563)
I think you’ve just quantified the difference between a text generator (LLM) and AGI.
Re:Ask HN: What's the best self hosted/local alternative to ... by nauful on Wednesday May 31, 2023 @10:34AM (#36138881)
I have a hard time understanding the question as you phrased it. Tested with guanaco-33B and a cleaner question, looks like a much better response:

> How can I join a column with SQL in Postgres to a string separated with a comma?

You can use the `string_agg` function in PostgreSQL to concatenate columns into a string. For example, to concatenate all values from a column named "column1" into a single string separated by commas, you can run the following query:

```sql

SELECT string_agg(column1, ',') FROM table;

```

The `string_agg` function takes two arguments: the first argument is the expression to concatenate (in this case column1), and the second argument is the separator between each value (in this case a comma). The query above will return a single string containing all values from column1 separated by commas.

I generally find WizardLM 1.0, Guanaco and Manticore to be good enough for most GPT cases.

Re:Ask HN: What's the best self hosted/local alternative to ... by wild_egg on Wednesday May 31, 2023 @10:29AM (#36138825)
> join with SQL a column to a string

As a human, I wouldn't know what to give you for this either. Not surprised the models give unexpected results. "Guanaco" gives something more in line with what I understand you to be asking for, but clearly you're asking for something else that I don't understand.

Re:Ask HN: What's the best self hosted/local alternative to ... by yread on Wednesday May 31, 2023 @10:23AM (#36138763)
I also prefer the first but your point would be stronger if the example from Guanaco was actually wrong or internally inconsistent
Re:Ask HN: What's the best self hosted/local alternative to ... by seszett on Wednesday May 31, 2023 @10:24AM (#36138775)
My test for open models is surprisingly simple.

I just ask "What is the capital of France?" and I haven't had a correct answer yet in any model I tried. They often get Paris right at least, but most other details are wrong.

Guanaco says:

> The current capital of France is Paris. It has been so since 1982 when it replaced the previous one which was Vichy.

Re:Ask HN: What's the best self hosted/local alternative to ... by jhbadger on Wednesday May 31, 2023 @10:41AM (#36138953)
What model size? 7B? 13B? 33B? When people say "Guanaco" or "Alpaca" or "Manticore" they are referring to a whole family of models. The really small ones like 7B often are very weird.
Re:Ask HN: What's the best self hosted/local alternative to ... by simonw on Wednesday May 31, 2023 @10:30AM (#36138838)
I just got "The capital of France is Paris." from vicuna-v1-7B running entirely on my iPhone (using the MLC Chat app).
Re:Ask HN: What's the best self hosted/local alternative to ... by edgyquant on Wednesday May 31, 2023 @11:44AM (#36139663)
>They often get Paris right at least, but most other details are wrong.
Re:Ask HN: What's the best self hosted/local alternative to ... by whimsicalism on Wednesday May 31, 2023 @12:02PM (#36139913)
What other detail in that statement is wrong?
Re:Ask HN: What's the best self hosted/local alternative to ... by ac29 on Wednesday May 31, 2023 @12:18PM (#36140105)
Guanaco 7B gives the same exact answer.
Re:Ask HN: What's the best self hosted/local alternative to ... by croes on Wednesday May 31, 2023 @10:32AM (#36138860)
What's the result for: How can I aggregate with SQL a column to a string separated with comma ?
Re:Ask HN: What's the best self hosted/local alternative to ... by dustypotato on Wednesday May 31, 2023 @10:29AM (#36138821)
I have chatGPT with GPT-4 model. It never gives such concise answers. Is it because you used the API?
Re:Ask HN: What's the best self hosted/local alternative to ... by laratied on Wednesday May 31, 2023 @11:42AM (#36139630)
The default is rather wordy. You could instruct chatGPT4 to be less wordy or in the API put instructions to be less wordy in the system prompt or limit answers to a sentence.

I would assume the poster just edited this though.

Re:Ask HN: What's the best self hosted/local alternative to ... by ivanhoe on Wednesday May 31, 2023 @11:14AM (#36139302)
> GPT-4: PostgreSQL: SELECT STRING_AGG(columnName, ', ') FROM tableName;

Well, you asked for sql and ansi sql function is listagg(), so technically this is not the right answer - and definitely a completely useless one for mysql or oracle users.

Re:Ask HN: What's the best self hosted/local alternative to ... by discordance on Wednesday May 31, 2023 @11:58AM (#36139846)
"Thanks ChatGPT. Could you please write that for MySQL and additionally for Oracle?"

MySQL: SELECT GROUP_CONCAT(columnName SEPARATOR ', ') FROM tableName;

Oracle: SELECT LISTAGG(columnName, ', ') WITHIN GROUP (ORDER BY columnName) FROM tableName;

I mean, it's not a big deal.. you just give have to give it a little nudge.

Re:Ask HN: What's the best self hosted/local alternative to ... by api on Wednesday May 31, 2023 @11:51AM (#36139758)
Your question is a bit confusing and ambiguous. I think there are multiple valid answers. Here's a local test of a less ambiguous programming question with "Wizard-Vicuna-30B-Uncensored.ggmlv3.q8_0.bin" on llama.cpp on an M1 Max laptop with 64GiB of RAM.

This model seems roughly on par with GPT-3, maybe GPT-3.5 in some cases.

> Write a function to compute the maximum, minimum, and average of an array of doubles in C.

Here is an example function in C that computes the maximum, minimum, and average of an array of doubles:

```c void compute_statistics(double arr[], int n) { double max = arr[0]; double min = arr[0]; double sum = 0.0; for (int i = 1; i < n; i++) { if (arr[i] > max) { max = arr[i]; } else if (arr[i] < min) { min = arr[i]; } sum += arr[i]; } double avg = sum / n; printf("Maximum: %.2f", max); printf("Minimum: %.2f", min); printf("Average: %.2f", avg); } ```

To use this function, simply call it with the array of doubles and its length as arguments. For example:

```c double arr[] = {1.0, 2.5, 3.0, 4.0, 5.0}; int n = sizeof(arr) / sizeof(arr[0]); compute_statistics(arr, n); ```

This will output:

``` Maximum: 5.00 Minimum: 1.00 Average: 3.00 ```

Here are some simple logic questions:

> Compute the sum of an array of 512 elements with each element having the value 2.

The sum of an array of 512 elements with each element having the value 2 is 1024.

> Compute the sum of an array of 512 elements with each element having the value 3.

The sum of an array of 512 elements with each element having the value 3 is 1536.

Not bad.

Re:Ask HN: What's the best self hosted/local alternative to ... by kgwgk on Wednesday May 31, 2023 @12:42PM (#36140402)
GPT4All-J-v1.3 Groovy [1] gave me the following answer (no idea if this is good or not, but keep in mind that the model comes in a 3.8Gb file and is released under an Apache 2 license, freely available for use and distribution):

    To join a column with SQL in Postgres to a string separated by a comma, you can use the STRING_AGG function. Here's an example query that demonstrates this:
    SELECT STRING_AGG(column_name, ', ') WITHIN GROUP (ORDER BY id) AS joined_string FROM table;
    In this query, we're selecting the joined_string column from a table called table. The STRING_AGG function is used to concatenate all values in the column_name column into a single string separated by commas. We use the WITHIN GROUP (ORDER BY id) clause to specify that we want to group the results by an integer value called id. This ensures that each row of data is treated as a separate entity, even if it has the same column_name.
    The resulting output will be a single column with all values from the joined_string column concatenated into a string separated by commas. You can then use this query in your application to join multiple columns together and store them as a single string value.
[1] https://gpt4all.io/reports/GPT4All_Technical_Report_3.pdf
Re:Ask HN: What's the best self hosted/local alternative to ... by throwupaiethics on Wednesday May 31, 2023 @12:59PM (#36140634)
One of the issues that i've observed with OpenAI, is that things the model got wrong a week ago it suddenly starts getting right but other examples that are not exactly the same question but very similar or another example of the same thing it gets wrong, then a few weeks passes and it gets this right.

My conclusion is that they are using the data from users to improve or some other prompt injection tricks.

So the comparision is not even fair.

Re:Ask HN: What's the best self hosted/local alternative to ... by MacsHeadroom on Wednesday May 31, 2023 @10:29AM (#36138827)
You link to Guanaco-33B, but Guanaco-65B is much more capable.

CPU Version: https://huggingface.co/TheBloke/guanaco-65B-GGML

GPU Version: https://huggingface.co/TheBloke/guanaco-65B-HF

4bit GPU Version: https://huggingface.co/TheBloke/guanaco-65B-GPTQ

Re:Ask HN: What's the best self hosted/local alternative to ... by wing-_-nuts on Wednesday May 31, 2023 @10:42AM (#36138975)
It irritates me to no end that people don't list the system requirements of various models.

How much ram and vram does one need to run 4,13,33,65B models at a reasonable speed?

edit: instead I'll ask this, what's the best model to run on a system with a 24gb 4090 and 64gb of ram?

Re:Ask HN: What's the best self hosted/local alternative to ... by orost on Wednesday May 31, 2023 @10:58AM (#36139126)
You can just barely fit a 33B GPTQ model in 24GB VRAM. It will be in 4-bit mode, and without maximum context size, but it will be quite fast. Or you can run from RAM+VRAM in GGML format with llama.cpp (or a derivative), which will easily fit 65B models even at 5 or 8 bits, but at much lower speed.
Re:Ask HN: What's the best self hosted/local alternative to ... by evanchisholm on Wednesday May 31, 2023 @11:06AM (#36139210)
It's fairly simple to estimate ram requirements based on parameter count: https://blog.eleuther.ai/transformer-math/
Re:Ask HN: What's the best self hosted/local alternative to ... by TheUninformed on Wednesday May 31, 2023 @12:14PM (#36140064)
A 4-bit quantized 33B parameter model will fit on your GPU and you'll be able to use a 2048 token context too. (4-bit quantized larger models are better than smaller 8bit/16bit models)

You can run 4-bit quantized 65B models on your cpu, but it is slow, 1-2 tokens a second instead of 8-15 people typically get with a gpu, but you need two 24gb or an enterprise card with 48gb of ram to load them there.

https://old.reddit.com/r/LocalLLaMA/wiki/models has the information you are irritated about not being listed.

Re:Ask HN: What's the best self hosted/local alternative to ... by ac29 on Wednesday May 31, 2023 @12:20PM (#36140148)
The first link there gives very specific RAM requirements for each of the models.

"Reasonable speed" is subjective, though.

Re:Ask HN: What's the best self hosted/local alternative to ... by londons_explore on Wednesday May 31, 2023 @10:10AM (#36138632)
Table 1 in your link (https://arxiv.org/pdf/2305.14314.pdf) is a good way to compare models.

I'd really like someone to make a big leaderboard/ranking engine which pits all these engines against eachother and publishes the resulting Elo score.

Re:Ask HN: What's the best self hosted/local alternative to ... by the__prestige on Wednesday May 31, 2023 @10:25AM (#36138784)
Re:Ask HN: What's the best self hosted/local alternative to ... by jlund-molfese on Wednesday May 31, 2023 @10:18AM (#36138704)
Re:Ask HN: What's the best self hosted/local alternative to ... by coolspot on Wednesday May 31, 2023 @01:16PM (#36140868)
Guanaco is indeed very capable and can replace GPT 3.5 in almost all scenarios, based on my tests.

Easy way to self-host it is to use text-generation-webui[1] and 33B 4-bit quantized GGML model from TheBloke[2].

[1] https://github.com/oobabooga/text-generation-webui

[2] https://huggingface.co/TheBloke/guanaco-33B-GGML