News for hackers. Stuff that matters.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by riku_iki on Friday December 20, 2024 @02:13PM (#42473935)
> ~=$3400 per single task

report says it is $17 per task, and $6k for whole dataset of 400 tasks.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by binarymax on Friday December 20, 2024 @02:32PM (#42474152)
"Note: OpenAI has requested that we not publish the high-compute costs. The amount of compute was roughly 172x the low-compute configuration."

The low compute was $17 per task. Speculate 172*$17 for the high compute is $2,924 per task, so I am also confused on the $3400 number.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by bluecoconut on Friday December 20, 2024 @02:33PM (#42474167)
3400 came from counting pixels on the plot.

Also its $20 on for the o3-low via the table for the semi-private, which x172 is 3440, also coming in close to the 3400 number

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by bluecoconut on Friday December 20, 2024 @02:19PM (#42474010)
That's the low-compute mode. In the plot at the top where they score 88%, O3 High (tuned) is ~3.4k
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by ionwake on Friday December 20, 2024 @04:28PM (#42475224)
sorry to be a noob, but can someone tell me doe sths mena o3 will be unaffordable for a typical user? Will only companies with thousands to spend per query be able to use this?

Sorry for being thick Im just confused how they can turn this into an addordable service?

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by JohnnyMarcone on Friday December 20, 2024 @06:06PM (#42476015)
There are likely many efficiency gains that will be made before it's released, and after. Also they showed o3 mini to be better than o1 for less cost in multiple benchmarks, so there're already improvements there at a lower cost than what available.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by ionwake on Friday December 20, 2024 @06:30PM (#42476194)
Great thank you
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by HDThoreaun on Friday December 20, 2024 @08:10PM (#42476704)
The low compute one did as well as the average person though
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by jhrmnn on Friday December 20, 2024 @02:20PM (#42474020)
That’s for the low-compute configuration that doesn’t reach human-level performance (not far though)
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by riku_iki on Friday December 20, 2024 @02:21PM (#42474028)
I referred on high compute mode. They have table with breakdown here: https://arcprize.org/blog/oai-o3-pub-breakthrough
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by junipertea on Friday December 20, 2024 @02:30PM (#42474129)
The table row with 6k figure refers to high efficiency, not high compute mode. From the blog post:

Note: OpenAI has requested that we not publish the high-compute costs. The amount of compute was roughly 172x the low-compute configuration.

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by gbnwl on Friday December 20, 2024 @02:30PM (#42474137)
That's "efficiency" high, which actually means less compute. The 87.5% score using low efficiency (more compute) doesn't have cost listed.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by bluecoconut on Friday December 20, 2024 @02:31PM (#42474141)
they use some poor language.

"High Efficiency" is O3 Low "Low Efficiency" is O3 High

They left the "Low efficiency" (O3 High) values as `-` but you can infer them from the plot at the top.

Note the $20 and $17 per task aligns with the X-axis of the O3-low

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by EVa5I7bHFq9mnYK on Friday December 20, 2024 @02:27PM (#42474105)
That's high EFFICIENCY. High efficiency = low compute.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by xrendan on Friday December 20, 2024 @02:35PM (#42474188)
You're misreading it, there's two different runs, a low and a high compute run.

The number for the high-compute one is ~172x the first one according to the article so ~=$2900

Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by Thorrez on Saturday December 21, 2024 @09:16AM (#42479812)
What's extra confusing is that in the graph the runs are called low compute and high compute. In the table they're called high efficient and low efficiency. So the high and low got swapped.