News for hackers. Stuff that matters.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by hansonkd on Friday December 20, 2024 @01:30PM (#42473481)
It doesn't need to be general intelligence or perfectly map to human intelligence.

All it needs to be is useful. Reading constant comments about LLMs can't be general intelligence or lack reasoning etc, to me seems like people witnessing the airplane and complaining that it isn't "real flying" because it isn't a bird flapping its wings (a large portion of the population held that point of view back then).

It doesn't need to be general intelligence for the rapid advancement of LLM capabilities to be the most societal shifting development in the past decades.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by wruza on Friday December 20, 2024 @02:06PM (#42473858)
And look at the airplanes, they really can’t just land on a mountain slope or a tree without heavy maintenance afterwards. Those people weren’t all stupid, they questioned the promise of flying servicemen delivering mail or milk to their window and flying on a personal aircar to their workplace. Just like todays promises about whatever the CEOs telltales are. Imagining bullshit isn’t unique to this century.

Aerospace is still a highly regulated area that requires training and responsibility. If parallels can be drawn here, they don’t look so cool for a regular guy.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Workaccount2 on Friday December 20, 2024 @02:27PM (#42474104)
What people always leave out is that society will bend to the abilities of the new technology. Planes can't land in your backyard so we built airports. We didn't abandon planes.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by wruza on Friday December 20, 2024 @02:51PM (#42474346)
Yes but the idea was lost in the process. It became a faster transportation system that uses air as a medium, but that’s it. Personal planes are still either big business or an expensive and dangerous personal toy thing. I don’t think it’s the same for LLMs (would be naive). But where are promises like “we’re gonna change travel economics etc”? All headlines scream is “AGI around the corner”. Yeah, now where’s my damn postman flying? I need my mail.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by ben_w on Friday December 20, 2024 @04:58PM (#42475478)
> It became a faster transportation system that uses air as a medium, but that’s it.

On the one hand, yes; on the other, this understates the impact that had.

My uncle moved from the UK to Australia because, I'm told*, he didn't like his mum and travel was so expensive that he assumed they'd never meet again. My first trip abroad… I'm not 100% sure how old I was, but it must have been between age 6 and 10, was my gran (his mum) paying for herself, for both my parents, and for me, to fly to Singapore, then on to various locations in Australia including my uncle, and back via Thailand, on her pension.

That was a gap of around one and a half generations.

* both of them are long-since dead now so I can't ask

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by PaulDavisThe1st on Friday December 20, 2024 @02:45PM (#42474274)
Sure, but that also vindicates the GP's point that the initial claims of the boosters for planes contained more than their fair share of bullshit and lies.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by oblio on Friday December 20, 2024 @03:50PM (#42474892)
We are slowly discovering that many of our wonderful inventions from 60-80-100 years ago have serious side effects.

Plastics, cars, planes, etc.

One could say that a balanced situation, where vested interests are put back in the box (close to impossible since it would mean fighting trillions of dollars), would mean that for example all 3 in the list above are used a lot less than we use them now, for example. And only used where truly appropriate.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by ForHackernews on Friday December 20, 2024 @03:29PM (#42474721)
This is already happening. A few days ago Microsoft turned down a documentation PR because the formatting was better for humans but worse for LLMs: https://github.com/MicrosoftDocs/WSL/pull/2021#issuecomment-...

They changed their mind after a public outcry including here on HN.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by tivert on Friday December 20, 2024 @04:26PM (#42475207)
> What people always leave out is that society will bend to the abilities of the new technology.

Do they really? I don't think they do.

> Planes can't land in your backyard so we built airports. We didn't abandon planes.

But then what do you do with the all the fantasies and hype about the new technology (like planes that land in your backyard and you fly them to work)?

And it's quite possible and fairly common that the new technology actually ends up being mostly hype, and there's actually no "airports" use case in the wings. I mean, how much did society "bend to the abilities of" NFTs?

And then what if the mature "airports" use case is actually something most people do not want?

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by moffkalast on Friday December 20, 2024 @05:05PM (#42475532)
No, we built helicopters.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by skydhash on Friday December 20, 2024 @02:18PM (#42474005)
This pretty much. Everyone knows that LLMs are great for text generation and processing. What people has been questioning is the end goals as promised by its builders, i.e. is it useful? And from most of what I saw, it's very much a toy.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by MVissers on Friday December 20, 2024 @10:10PM (#42477252)
What would you need to see to call it useful?

To give you an example– I've used it for legal work such as an EB2-NIW visa application. Saved me countless of hours. My next visa I'll try to do without a lawyer using just LLMs. I would never try this without having LLMs at my disposal.

As a hobby– And as someone with a scientific background I've been able to build an artificial ecosystem simulation from scratch without programming experience in Rust: https://www.youtube.com/@GenecraftSimulator

I recently moved from fish to plants and believe I've developed some new science at the intersection of CS and Evolutionary Biology that I'm looking to publish.

This tool is extremely useful. For now– You do require a human in the loop for coordination.

My guess is that these will be benchmarks that we see within a few years: How good an AI coordinate multiple other AIs to build, deploy and iterate something that functions in the real world. Basically manager AI.

Because they'll literally be able to solve every single one shot problem so we won't be able to create benchmarks anymore.

But that's also when these models will be able to build functioning companies in a few hours.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by skydhash on Friday December 20, 2024 @11:26PM (#42477551)
> ...me countless of...would never try this without having LLMs...is extremely useful...they'll literally be able to solve...will be able to... in a few hours.

That's marketing language, not scientific or even casual language. So much outstanding claims, without even some basic explanations. Like how did it help you save you these hours? Terms explanations? Outlining processes? Going to the post office for you? You don't need to sell me anything, I just want the how.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by wruza on Saturday December 21, 2024 @12:26PM (#42480922)
My issue with LLMs is that you require a review-competent human in the loop, to fix confabulations.

Yes, I’m using them from time to time for research. But I’m also aware of the topics I research and see through bs. And best LLMs out there, right now, produce bs in just 3-4 paragraphs, in nicely documented areas.

A recent example is my question on how to run N vpn servers on N ips on the same eth with ip binding (in ip = out ip, instead of using a gw with the lowest metric). I had no idea but I know how networks work and the terminology. It started helping, created a namespace, set up lo, set up two interfaces for inner and outer routing and then made a couple of crucial mistakes that couldn’t be detected or fixed by someone even a little clueless (in routing setup for outgoing traffic). I didn’t even argue and just asked what that does wrt my task, and that started the classic “oh wait, sorry, here’s more bs” loop that never ended.

Eventually I distilled the general idea and found an article that AI very likely learned from, cause it was the same code almost verbatim, but without mistakes.

Does that count as helping? Idk, probably yes. But I know that examples like this show that you cannot not only leave an LLM unsupervised for any non-trivial question, but have to leave a competent role in the loop.

I think the programming community is just blinded by LLMs succeeding in writing kilometers of untalented react/jsx/etc crap that has no complexity or competence in it apart from repeating “do like this” patterns and literally millions of examples, so noise cannot hit through that “protection”. Everything else suffers from LLMs adding inevitable noise into what they learned from a couple of sources. The problem here, as I understand it, is that only specific programmer roles and s{c,p}ammers (ironically) write the same crap again and again millions of times, other info usually exists in only a few important sources and blog posts, and only a few of those are full and have good explanations.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by throwaway4aday on Friday December 20, 2024 @03:14PM (#42474573)
Your point is on the verge of nullification with the rapid improvement and adoption of autonomous drones don't you think?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by wruza on Friday December 20, 2024 @06:44PM (#42476265)
Sort of, but doesn’t that sit on a far-fetch horizon? I doubt that drone companies are all the same who sold aircraft retrofuturism to people back then.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by surgical_fire on Friday December 20, 2024 @01:48PM (#42473687)
> to me seems like people witnessing the airplane and complaining that it isn't "real flying" because it isn't a bird flapping its wings

To me it is more like there is someone jumping on a pogo ball while flapping their arms and saying that they are flying whenever they hop off the ground.

Skeptics say that they are not really flying, while adherents say that "with current pogo ball advancements, they will be flying any day now"

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by PaulDavisThe1st on Friday December 20, 2024 @02:46PM (#42474299)
An old quote, quite famous: "... is like saying that an ape who climbs to the top of a tree for the first time is one step closer to landing on the moon".
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by intelVISA on Friday December 20, 2024 @02:07PM (#42473872)
Between skeptics and adherents who is more easily able to extract VC money for vaporware? If you limit yourself to 'the facts' you're leaving tons of $$ on the table...
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by surgical_fire on Friday December 20, 2024 @02:27PM (#42474099)
By all means, if this is the goal, AI is a success.

I understand that in this forum too many people are invested in putting lipstick on this particular pig.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by DonHopkins on Friday December 20, 2024 @03:10PM (#42474541)
Is that what Elon Musk was trying to do on stage?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by zebomon on Friday December 20, 2024 @01:32PM (#42473519)
I agree. If the LLMs we have today never got any smarter, the world would still be transformed over the next ten years.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by handsclean on Friday December 20, 2024 @01:48PM (#42473685)
People aren’t responding to their own assumption that AGI is necessary, they’re responding to OpenAI and the chorus constantly and loudly singing hymns to AGI.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by billyp-rva on Friday December 20, 2024 @01:57PM (#42473771)
> It doesn't need to be general intelligence or perfectly map to human intelligence.

> All it needs to be is useful.

Computers were already useful.

The only definition we have for "intelligence" is human (or, generally, animal) intelligence. If LLMs aren't that, let's call it something else.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by throwup238 on Friday December 20, 2024 @02:03PM (#42473828)
What exactly is human (or animal) intelligence? How do you define that?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by billyp-rva on Friday December 20, 2024 @02:09PM (#42473903)
Does it matter? If LLMs aren't that, whatever it is, then we should use a different word. Finders keepers.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by throwup238 on Friday December 20, 2024 @02:15PM (#42473966)
How do you know that LLMs “aren’t that” if you can’t even define what that is?

“I’ll know it when I see it” isn’t a compelling argument.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by grahamj on Friday December 20, 2024 @02:46PM (#42474305)
they can't do what we do therefore they aren't what we are
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by layer8 on Friday December 20, 2024 @03:09PM (#42474521)
And what is that, in concrete terms? Many humans can’t do what other humans can do. What is the common subset that counts as human intelligence?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by dimitri-vs on Friday December 20, 2024 @11:37PM (#42477580)
Process vision and sounds in parallel for 80+ years, rapidly adapt to changing environments and scenarios, correlate seemingly irrelevant details that happened a week ago or years ago, be able to selectively ignore instructions and know when to disagree
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by jonny_eh on Friday December 20, 2024 @04:23PM (#42475185)
> “I’ll know it when I see it” isn’t a compelling argument.

It feels compelling to me.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Aperocky on Friday December 20, 2024 @04:48PM (#42475395)
I think a successful high level intelligence should quickly accelerate or converge to infinity/physical resource exhaustion because they can now work on improving themselves.

So if above human intelligence does happen, I'd assume we'd know it, quite soon.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by alexalx666 on Friday December 20, 2024 @02:46PM (#42474301)
If I could put it into Tesla style robot and it could do dishes and help me figure out tech stuff, it would be more than enough.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by skywhopper on Friday December 20, 2024 @03:50PM (#42474890)
On the contrary, the pushback is critical because many employers are buying the hype from AI companies that AGI is imminent, that LLMs can replace professional humans, and that computers are about to eliminate all work (except VCs and CEOs apparently).

Every person that believes that LLMs are near sentient or actually do a good job at reasoning is one more person handing over their responsibilities to a zero-accountability highly flawed robot. We've already seen LLMs generate bad legal documents, bad academic papers, and extremely bad code. Similar technology is making bad decisions about who to arrest, who to give loans to, who to hire, who to bomb, and who to refuse heart surgery for. Overconfident humans employing this tech for these purposes have been bamboozled by the lies from OpenAI, Microsoft, Google, et al. It's crucial to call out overstatement and overhype about this tech wherever it crops up.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by noFaceDiscoG668 on Saturday December 21, 2024 @12:38PM (#42480987)
I don’t understand how or why someone with your mind would assume that even barely disclosed semi-public releases would resemble the current state of the art. Except if you do it for the conversations sake, which I have never been capable of.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by AyyEye on Friday December 20, 2024 @01:43PM (#42473637)
> Reading constant comments about LLMs can't be general intelligence or lack reasoning etc, to me seems like people witnessing the airplane and complaining that it isn't "real flying" because it isn't a bird flapping its wings (a large portion of the population held that point of view back then).

That is a natural reaction to the incessant techbro, AIbro, marketing, and corporate lies that "AI" (or worse AGI) is a real thing, and can be directly compared to real humans.

There are people on this very thread saying it's better at reasoning than real humans (LOL) because it scored higher on some benchmark than humans... Yet this technology still can't reliably determine what number is circled, if two lines intersect, or count the letters in a word. (That said behaviour may have been somewhat finetuned out of newer models only reinforces the fact that the technology inherently not capable of understanding anything.)

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by IanCal on Friday December 20, 2024 @02:25PM (#42474076)
I encounter "spicy auto complete" style comments far more often than techbro AI-everything comments and its frankly getting boring.

I've been doing AI things for about 20+ years and llms are wild. We've gone from specialized things being pretty bad as those jobs to general purpose things better at that and everything else. The idea you could make and API call with "is this sarcasm?" and get a better than chance guess is incredible.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by surgical_fire on Friday December 20, 2024 @03:56PM (#42474939)
Eh, I see far more "AI is the second coming of Jesus" type of comments than healthy skepticism. A lot of anxiety from people afraid that their source of income will dry and a lot of excitement of people with an axe to grind that "those entitled expensive peasants will get what they deserve".

I think I count myself among the skeptics nowadays for that reason. And I say this as someone that thinks LLM is an interesting piece of technology, but with somewhat limited use and unclear economics.

If the hype was about "look at this thing that can parse natural language surprisingly well and generate coherent responses", I would be excited too. As someone that had to do natural language processing in the past, that is a damn hard task to solve, and LLMs excel at it.

But that is not the hype is it? We have people beating the drums of how this is just shy of taking the world by storm, and AGI is just around the corner, and it will revolutionize all economy ans society and nothing will ever be the same.

So, yeah, it gets tiresome. I wish the hype would die down a little so this could be appreciated for what it is.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by williamcotton on Friday December 20, 2024 @04:47PM (#42475392)
We have people beating the drums of how this is just shy of taking the world by storm, and AGI is just around the corner, and it will revolutionize all economy and society and nothing will ever be the same.

Where are you seeing this? I pretty much only read HN and football blogs so maybe I’m out of the loop.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by sensanaty on Friday December 20, 2024 @05:22PM (#42475679)
In this very thread there are multiple people espousing their views that the high score here is proof that o3 has achieved AGI.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by AyyEye on Friday December 20, 2024 @03:15PM (#42474587)
Nobody is disputing the coolness factor, only the intelligence factor.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by hansonkd on Friday December 20, 2024 @05:40PM (#42475826)
I'm saying the intelligence factor doesn't matter. Only the utility factor. Today LLMs are incredibly useful and every few months there appears to be bigger and bigger leaps.

Analyzing whether or not LLMs have intelligence is missing the forest from the trees. This technology is emerging in a capitalist society that is hyper optimized to adopt useful things at the expense of almost everything else. If the utility/price point gets hit for a problem, it will replace it regardless of if it is intelligent or not.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Jensson on Friday December 20, 2024 @07:52PM (#42476616)
But if you want to predict the future utility of these models you want to look at their current intelligence, compare that to humans and try to figure out roughly what skills they lack and which of those are likely to get fixed.

For example, a team of humans are extremely reliable, much more reliable than one human, but a team of AI's isn't mean reliable than one AI since an AI is already an ensemble model. That means even if an AI could replace a person, it probably can't replace a team for a long time, meaning you still need the other team members there, meaning the AI didn't really replace a human it just became a tool for huamns to use.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by MVissers on Friday December 20, 2024 @10:17PM (#42477278)
I think this is a fair criticism of capability.

I personally wouldn't be surprised if we start to see benchmarks around this type of cooperation and ability to orchestrate complex systems in the next few years or so.

Most benchmarks really focus on one problem, not on multiple real-time problems while orchestrating 3rd party actors who might or might not be able to succeed at certain tasks.

But I don't think anything is prohibiting these models from not being able to do that.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by shadowerm on Sunday December 22, 2024 @08:10AM (#42486072)
I agree and as a non-software engineer, all that matters to me right now is how much can these models replace software engineering.

If a language model can't solve problems in a programming language then we are just fooling ourselves in less defined domains of "thought".

Software engineering is where the rubber meets the road in terms of intelligence and economics when viewing our society as a complex system. Software engineering salaries are above average exactly because most average people are not going to be software engineers.

From that point of view the progress is not impressive at all. The current models are really not that much better than chatGPT4 in April 2023.

AI art is a better example though. There is zero progress being made now. It is only impressive at the most surface level for someone not involved in art and who can't see how incredibly limited the AI art models are. We have already moved on to video though to make the same half baked, useless models that are only good to make marketing videos for press releases about progress and one off social media posts about how much progress is being made.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by jasondigitized on Friday December 20, 2024 @04:29PM (#42475237)
This a thousand times.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by colordrops on Friday December 20, 2024 @04:39PM (#42475322)
I don't think many informed people doubt the utility of LLMs at this point. The potential of human-like AGI has profound implications far beyond utility models, which is why people are so eager to bring it up. A true human-like AGI basically means that most intellectual/white collar work will not be needed, and probably manual labor before too long as well. Huge huge implications for humanity, e.g. how does an economy and society even work without workers?
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by vouaobrasil on Friday December 20, 2024 @05:36PM (#42475789)
> Huge huge implications for humanity, e.g. how does an economy and society even work without workers?

I don't think those that create AI care about that. They just to come out on top before someone else does.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by colordrops on Tuesday December 24, 2024 @01:46PM (#42503940)
Yes and we should be super worried about that.