News for hackers. Stuff that matters.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by thorum on Thursday November 30, 2023 @10:01PM (#38482536)
Current ~70B models like LLAMA 2 70B are on par wih ChatGPT 3.5. The best smaller models can appear on par at first glance, but they hallucinate at a much higher rate and lack knowledge of the world. GPT 4 ‘gets’ things at a deeper level and no open source model is even close.

A year is a good timeframe to evaluate things: the rest of the world seems to lag behind OpenAI by around 12-18 months, at least with LLMs and image generation.

On the other hand open source tech usually has additional features for controlling output that OpenAI never bothers to implement, like llama.cpp’s grammars or ControlNet. So in that sense open source is usually ahead of OpenAI in terms of customizability.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by omeze on Thursday November 30, 2023 @11:02PM (#38482984)
I dont think OpenAI is ever going to ahead in image generation, they were lapped very soon after dall-e and every real workflow Ive seen uses Midjourney or Stable Diffusion. The reverse (GPT 4 vision) is well ahead of open source though
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by vunderba on Thursday November 30, 2023 @11:05PM (#38483001)
The original is leagues behind anything current, but DALL-E version 3 absolutely blows any state of the art generative model out of the water, including mid journey 5.2 and SDXL in terms of pure prompt accuracy and coherence.

Midjourney still has the edge in quality, but it's a moot point if it takes you 1000 v-rolls to get to your original vision.

If all you're generating is anime waifus then MJ/NovelAI/Niji will suffice, but generating prompts particularly featuring relatively complex scenes or actions are amazing on DALL-E 3.

And of course unfortunately, it goes without saying that open AI DALL-E is going to be the most restrictive in terms of censorship.

I generated these from DALL-E 3 instantly. Try to generate them in any other commercial offering. Go ahead. I'll wait...

https://imgur.com/a/2GTRjfK

Descriptions:

A 80s photograph of the Koolaid Man breaking through the Berlin Wall.

Comic illustration set at a festive children's party. The main focus is on the magician who looks uncannily like a well-known fictional wizard. He's trying to say abracadabra but accidentally uses the killing curse.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by mattnewton on Thursday November 30, 2023 @11:13PM (#38483055)
SDXL has controlnet for other kinds of non-text input (like scribbles or just masks). The results are much easier to control in my opinion (a picture is worth thousands of prompt words).

For pure prompt coherence though I think ideogram is not far behind dalle 3.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by vunderba on Friday December 01, 2023 @12:02AM (#38483342)
EDIT: okay, I just tried Ideogram. It's not terrible and seems to do an okay job on text generation but I'd still say its a distant second compared to DALL-E 3. However, having the ability to maintain image continuity to make refinements of your initial image based on corrections like: "Make the building larger", or "He should have a more prominent forehead" is a game changer (e.g. InstructPix2Pix) and DALL-E 3's the only one that's got it.

Ideogram comparisons at bottom:

https://imgur.com/a/2GTRjfK

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by vunderba on Thursday November 30, 2023 @11:18PM (#38483078)
SDXL and even some SD 1.5 checkpoints are great. My current workflow is:

1. Generate initial draft image in DALL-E 3 (iterate as necessary)

It's essentially the ONLY good InstructPix2Pix model.

2. Bring into InvokeAI

Inpaint with stuff that might be considered censored in DALL-E 3.

I'd like to see some proof of Ideogram - it looks... very mobile/instagrammy from the landing page. If you have an account, try out my prompts I'd like to see what you're able to produce.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by FooBarWidget on Friday December 01, 2023 @05:36AM (#38485294)
> Midjourney still has the edge in quality, but it's a moot point if it takes you 1000 v-rolls to get to your original vision.

I can corroborate this. I wanted about 6 images for a presentation. I rolled ~300 MidJourney images. Most of them looked great, but none of them did what I wanted. I rolled ~50 DALL-E 3 images.

In the end, I only picked DALL-E 3 images. They were qualitatively not as good as MidJourney. For example when you zoom in then you see distortions. Or they're a bad fit for 16:9 format. But only DALL-E 3 was able to draw the things I wanted.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by Terretta on Friday December 01, 2023 @08:56AM (#38486669)
> Try to generate them in any other commercial offering. Go ahead. I'll wait...

For interest's sake, this is from the second /imagine on MidJourney (so one of the second set of 4 images):

https://imgur.com/a/YyWHppb

While yours is what you'd want, this arguably looks more like the super cheesy children's TV commercials back in the day and beats the ideogram take.

The Midjourney generations all appear to be referencing Halloween costumes or terrible cosplays, as if there are no trademarked koolaid men in their training set.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by vunderba on Friday December 01, 2023 @11:00AM (#38488213)
yeah, I did some rolls of this image for MJ but that was back in v4 and wasn't very impressed - doesn't look like its made much progress. The original commercials while silly looking are very visually identifiable as the Koolaid man.

I remember hearing that the first versions of MJ used the LAION image set for training data - I'd be curious to see if it has any training data containing the Koolaid man.

I did a search through my MJ history from the past year and added the results to the imgur link to include my attempts at generating the Koolaid man from v3/v4/v5.2.

https://imgur.com/a/2GTRjfK

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by Terretta on Friday December 01, 2023 @11:55AM (#38488951)
If you limit to the "6+" aesthetic set, there are zero for koolaid man, two for koolaid:

https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im...

And only three hundred for berlin wall:

https://laion-aesthetic.datasette.io/laion-aesthetic-6pls/im...

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by zamadatix on Thursday November 30, 2023 @11:34PM (#38483167)
Strong disagree on this from me as well. DALL-E 3 is miles ahead of the latest Midjourney/Stable Diffusion in image generation. The only real area it falls short vs the other options right now is in how nannying it can be.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by kubrickslair on Thursday November 30, 2023 @11:21PM (#38483093)
I have found OpenAI to be the most superior in complex prompts especially where written messages like “Get better, Mom” are expected in the images. The distant second would be ideogram.

I am using these tools to send custom personal messages to close friends and family.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by avereveard on Thursday November 30, 2023 @11:11PM (#38483043)
On the other hand gpt model are converging down. Gpt4 turbo degraded performance so much that now certain 13b produce more consistent results in reasoning. I've a marathon test here for example https://chat.openai.com/share/dfd9b9ae-7214-4dd7-ad20-7ee07a... with purposefully open ended and somewhat ambiguous request to see how models perform and gpt4 turbo chat is just not that good it confuses persons out, didn't pick the right one for abduction, didn't change topic when requested, when recalling persons picked the one from the wrong set, when asked to change language it didn't... It know a lot when asked zero shot questions, but when proving it's self consistency and attention it is nowhere near gpt4.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by BoorishBears on Thursday November 30, 2023 @11:21PM (#38483095)
Nothing kills someone's credibility on LLMs faster than linking to the consumer product that is ChatGPT's web interface in a discussion about models.

You can't even guarantee that you're getting the checkpoint that is known as GPT-4-Turbo via the API, and in fact you can almost guarantee you're not because now they do side-by-side tests between different versions of their models in the web UI.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by avereveard on Thursday November 30, 2023 @11:58PM (#38483313)
The point is exactly that the model people are experiencing is converging down with every subsequent update, and I even mentioned that it's nowhere near the orig gpt4, idk possibly read it again slower instead of jumping to credibility and whatnot.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by BoorishBears on Friday December 01, 2023 @01:49AM (#38483927)
The irony of your comment since it implies you didn't catch half the words in mine: I'm saying you can't compare it to the "original" via the web ui

The "new" model was being used months before they announced GPT-4 Turbo, you could similarly tell by the speed differential in some of the newer tool using modes.

Even the "original" is likely still being used in some responses in the web ui, you can tell because some of the side-by-sides have a drastic speed differential that matches GPT-4 vs GPT-4-Turbo.

-

You're trying to compare models using a tool that doesn't actually tell you what models you're comparing past which family they belong to, and represents more models than you're trying to discuss.

In other words you're talking about a topic you clearly don't understand and the equally clueless mob is glad to roll with it.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by avereveard on Friday December 01, 2023 @02:25AM (#38484115)
> you can't compare it to the "original" via the web ui

Good thing then that I was comparing the chat offering to 7b and 13b open models then, maybe you didn't catch that, try a third reading.

Also, this kind of is a chatgpt thread. I know it's not a perfect comparison, but still. It's a discussion point I'm presenting, not a research paper.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by BoorishBears on Friday December 01, 2023 @03:50AM (#38484632)
Good thing that's orthogonal to what I said?

GPT-4 has an API, when comparing models you use the API otherwise you can't even name the checkpoint of the model that you're testing.

The web ui uses different models, unknown parameters, and even uses RAG... so it's not even an unfair comparision, it's not a comparison.

At the end of the day you're free to convince yourself whatever you want on whatever flawed process you want, but when you're corrected going full angsty sarcasm isn't going to change reality.

-

AI is in such a strange place now that the masses are interested in it: you can say whatever deluded thing you want as long as it aligns with what people want to hear.

I guess while people masturbate to this self-affirming nonsense the rest of us can build useful things on superior tech?

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by kaycebasques on Friday December 01, 2023 @10:56AM (#38488175)
BoorishBear, please stop with the aggressive, condescending tone. You seem to have some productive counterarguments but they're getting lost in your disrespectful language.
[dead]
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by kaycebasques on Friday December 01, 2023 @03:30PM (#38491909)
I don't have a feature to flag your comments and your profile does not point to a way for me to message you privately so this was my only option to call you out

What you call "tone policing" I call "violating numerous HN guidelines":

> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

> When disagreeing, please reply to the argument instead of calling names.

> Please don't sneer, including at the rest of the community.

> Please don't post shallow dismissals, especially of other people's work.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by BoorishBears on Friday December 01, 2023 @05:22PM (#38493338)
It's not your job to call out anyone on HN, and it adds precious little to the conversation.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by kaycebasques on Saturday December 02, 2023 @12:28PM (#38500170)
> It's not your job to call out anyone on HN

You may not personally agree with it, but there's nothing in the HN guidelines or comments from mods saying that I can't do it

> it adds precious little to the conversation

The conversations would have ended a long time ago if you're vitriolic tone was considered acceptable and normal

See dang's response to your comments: https://news.ycombinator.com/item?id=38493663

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by dang on Friday December 01, 2023 @06:02PM (#38493781)
You can flag comments as described here: https://news.ycombinator.com/newsfaq.html#cflag
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by dang on Friday December 01, 2023 @05:51PM (#38493663)
Yikes, you broke the site guidelines really badly in this thread, not just in this comment but in several other places including these:

https://news.ycombinator.com/item?id=38490123

https://news.ycombinator.com/item?id=38484632

We have to ban accounts that post this way. It's not what this site is for, and destroys what it is for. Moreover, we've had warn you about this more than once over the years.

I don't want to ban you, so if you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules from now on, we'd appreciate it. That means no swipes and no personal attacks, among other things.

Edit: you've unfortunately been doing this in other recent threads too:

https://news.ycombinator.com/item?id=38453843

https://news.ycombinator.com/item?id=38453381

https://news.ycombinator.com/item?id=38377068

We really are going to have to ban you if this keeps up, so if you'd please make whatever change is needed to not post like this again, that would be good.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by cooperaustinj on Friday December 01, 2023 @11:43AM (#38488798)
Your credibility is killed by thinking using an API can guarantee which model you're getting. It's entirely black box. If OpenAI wants to lie to you, they can.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by BoorishBears on Friday December 01, 2023 @01:13PM (#38490123)
Your credibility is killed by thinking paying AWS for a c5.metal can guarantee which compute you're getting. It's entirely black box.

If AWS wants to torpedo their business for marginal gain they can. And you won't be able to tell just because your workload falls on its face.

Your comment is the textbook definition of FUD in a comical way.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by infecto on Friday December 01, 2023 @09:19AM (#38486940)
I don't think think using examples derived from ChatGPT are a fair comparison to the underlying models. OpenAI has many optimization tricks on the ChatGPT side that are unrelated to the underlying models being used.

We do know of course that ChatGPT is most likely using 4-Turbo from the decrease in latency and increase in unhelpful answers.

We cannot say that the models are "converging down" though. I don't remember the marketing materials but from the model side we all realize that the Turbo models have some type of quantization/optimization that makes them cheap and fast. 4-Turbo is 3x cheaper than 4, substantially quicker and provides better results than 3.5-Turbo. Amazing progress in my arena.

Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by Workaccount2 on Friday December 01, 2023 @10:47AM (#38488110)
There were many rumors (and it probably was true) that OpenAI was hemorrhaging cash on GPT4 requests. So it makes tons of sense for them to sprint towards a turbo model at the expense of some ability. GPT4-turbo still is ridiculously powerful anyway.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by bugglebeetle on Thursday November 30, 2023 @11:04PM (#38482994)
Function-calling with a JSON schema is about as reliable as llama.cpp’s grammar stuff. I’ve not had any trouble with it.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by infecto on Friday December 01, 2023 @09:22AM (#38486969)
The only thing I would argue is that JSON generation and function calling have noticeable decrease in quality of output in certain uses. I have had a hard time writing tests to measure it but its noticeable for my human eyes when I compare various implementations I have written.
Re:ChatGPT's 1-Year Anniversary: Are Open-Source Large Langu... by ben_w on Friday December 01, 2023 @06:38AM (#38485668)
LLMs perhaps (I'm not sure either way, everything moves too quickly), but SDXL 1.0 (July 26, 2023) was a lot better than DALL•E 2 (6 April, 2022). I think DALL•E 3 (August 10, 2023) is a bit better than SDXL, but other than text generation their quality seems very close to me.

(That said, perhaps I'm Clever Hands-ing myself by only using SDXL for what it's good at. It's terrible at dragons every time I've tried that…)