News for hackers. Stuff that matters.
Re:OpenAI's o1-pro now available via API by jbellis on Wednesday March 19, 2025 @08:53PM (#43418885)
If you're in the habit of breaking down problems to Sonnet-sized pieces you won't see a benefit. The win is that o1pro lets you stop breaking down one level up from what you're used to.

It may also have a larger usable context window, not totally sure about that.

Re:OpenAI's o1-pro now available via API by logankeenan on Wednesday March 19, 2025 @10:54PM (#43419526)
> lets you stop breaking down one level up from what you're used to.

Can you provide an example of what you mean by this? I provide very verbose prompts where I know what needs to be done and just let AI “do” the work. I’m curious how this is different?

Re:OpenAI's o1-pro now available via API by jbellis on Thursday March 20, 2025 @05:29AM (#43421177)
Partly it means you can tell it to do X and it will figure out that implies Y and Z without you having to spell it out

And partly it can actually execute more at the same time without starting to make mistakes

Re:OpenAI's o1-pro now available via API by raylad on Thursday March 20, 2025 @04:29AM (#43420853)
Sonnet 3.7 and O1 Pro both have 200K context windows. But O1 Pro has a 100K output window, and Sonnet 3.7 has a 128K output window. Point for Sonnet.

I routinely put about 100K + of context into Sonnet 3.7 in the form of source code, and in the Extended mode, given the right prompt, it will output perhaps 20 large source files before having to make a "continue" request (for example if it's asked to convert a web app from templates to React).

I'm curious whether O1 Pro actually exceeds Sonnet 3.7 in Extended mode for coding or not. Looking forward to seeing some benchmarks.

Re:OpenAI's o1-pro now available via API by consumer451 on Thursday March 20, 2025 @08:33AM (#43422238)
I am very curious how 3.7 and o1 pro perform in this regard:

> We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%.

https://arxiv.org/abs/2502.05167

Re:OpenAI's o1-pro now available via API by futopy on Sunday March 23, 2025 @06:49PM (#43456523)
Anyone ever tries to restructure a 10K text? For example, structure a 45min - 1hr interview transcript in an organized way without losing any detailed numbers / facts / supporting evidence. I find that none of OpenAI's model is capable of this task. Models are trying to summarize and omitting details. I think such task does not require much intelligence, but surprisingly OpenAI's "large" context model cannot make it.
Re:OpenAI's o1-pro now available via API by qeternity on Thursday March 20, 2025 @08:35AM (#43422254)
"Usable" is the key word here. Not all context is created equal.

Have a look at the RULER benchmark for a bit more detail.