News for hackers. Stuff that matters.
Re:OpenAI's o1-pro now available via API by davidbarker on Wednesday March 19, 2025 @06:37PM (#43417980)
Pricing: $150 / 1M input tokens, $600 / 1M output tokens. (Not a typo.)

Very expensive, but I've been using it with my ChatGPT Pro subscription and it's remarkably capable. I'll give it 100,000 token codebases and it'll find nuanced bugs I completely overlooked.

(Now I almost feel bad considering the API price vs. the price I pay for the subscription.)

Re:OpenAI's o1-pro now available via API by ldjkfkdsjnv on Wednesday March 19, 2025 @07:50PM (#43418502)
As far as I'm concerned, all of the other models are a waste of time to use in comparison. Most people dont know how good this model is
Re:OpenAI's o1-pro now available via API by dinobones on Wednesday March 19, 2025 @08:19PM (#43418687)
Interesting... Most benchmarks show this model as being worse than o3-mini-high and sonnet3.7.

What difference are you seeing from these models that makes it better?

I say this as someone considering shelling out $200 for ChatGPT pro for this.

Re:OpenAI's o1-pro now available via API by jbellis on Wednesday March 19, 2025 @08:53PM (#43418885)
If you're in the habit of breaking down problems to Sonnet-sized pieces you won't see a benefit. The win is that o1pro lets you stop breaking down one level up from what you're used to.

It may also have a larger usable context window, not totally sure about that.

Re:OpenAI's o1-pro now available via API by logankeenan on Wednesday March 19, 2025 @10:54PM (#43419526)
> lets you stop breaking down one level up from what you're used to.

Can you provide an example of what you mean by this? I provide very verbose prompts where I know what needs to be done and just let AI “do” the work. I’m curious how this is different?

Re:OpenAI's o1-pro now available via API by jbellis on Thursday March 20, 2025 @05:29AM (#43421177)
Partly it means you can tell it to do X and it will figure out that implies Y and Z without you having to spell it out

And partly it can actually execute more at the same time without starting to make mistakes

Re:OpenAI's o1-pro now available via API by raylad on Thursday March 20, 2025 @04:29AM (#43420853)
Sonnet 3.7 and O1 Pro both have 200K context windows. But O1 Pro has a 100K output window, and Sonnet 3.7 has a 128K output window. Point for Sonnet.

I routinely put about 100K + of context into Sonnet 3.7 in the form of source code, and in the Extended mode, given the right prompt, it will output perhaps 20 large source files before having to make a "continue" request (for example if it's asked to convert a web app from templates to React).

I'm curious whether O1 Pro actually exceeds Sonnet 3.7 in Extended mode for coding or not. Looking forward to seeing some benchmarks.

Re:OpenAI's o1-pro now available via API by consumer451 on Thursday March 20, 2025 @08:33AM (#43422238)
I am very curious how 3.7 and o1 pro perform in this regard:

> We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%.

https://arxiv.org/abs/2502.05167

Re:OpenAI's o1-pro now available via API by futopy on Sunday March 23, 2025 @06:49PM (#43456523)
Anyone ever tries to restructure a 10K text? For example, structure a 45min - 1hr interview transcript in an organized way without losing any detailed numbers / facts / supporting evidence. I find that none of OpenAI's model is capable of this task. Models are trying to summarize and omitting details. I think such task does not require much intelligence, but surprisingly OpenAI's "large" context model cannot make it.
Re:OpenAI's o1-pro now available via API by qeternity on Thursday March 20, 2025 @08:35AM (#43422254)
"Usable" is the key word here. Not all context is created equal.

Have a look at the RULER benchmark for a bit more detail.

Re:OpenAI's o1-pro now available via API by Tiberium on Wednesday March 19, 2025 @09:02PM (#43418932)
There actually were almost no benchmarks for o1 pro before because it wasn't on the API. o1 pro is a different model from o1 (yes, even o1 with high reasoning).
Re:OpenAI's o1-pro now available via API by ldjkfkdsjnv on Wednesday March 19, 2025 @08:25PM (#43418728)
I regularly push 100k+ tokens into it. So most of my code base/large portions. I use the Repo Prompt product to construct the code prompts. It finds bugs and solutions at a rate that is far better than others. I also speak into the prompt to describe my problem, and find spoken language is interpreted very well.

I also frequently download all the source code of libraries I am debugging, and when running into issues, pass that code in along with my own broken code. Its very good

Re:OpenAI's o1-pro now available via API by Hugsun on Wednesday March 19, 2025 @08:38PM (#43418793)
How long is it's thinking time when compared to o1?

The naming would suggest that o1-pro is just o1 with more time to reason. The API pricing makes that less obvious. Are they charging for the thinking tokens? If so, why is it so much more expensive if there are just more thinking tokens anyways?

Re:OpenAI's o1-pro now available via API by Tiberium on Wednesday March 19, 2025 @09:20PM (#43419045)
I think o1 pro runs multiple instances of o1 in parallel and selects the best answer, or something of the sort. And you do actually always pay for thinking models with all providers, OpenAI included. It's especially interesting if you remember the fact that OpenAI hides the CoT from you, so you're in fact getting billed for "thinking" that you can't even read yourself.
Re:OpenAI's o1-pro now available via API by ldjkfkdsjnv on Wednesday March 19, 2025 @08:50PM (#43418863)
I dont have the answers for you, I just know that if they charged 400$ a month I would pay it. It seems like a different model to me. I never use o3-mini or o3-mini-high. Just gpt4o or o1 pro
Re:OpenAI's o1-pro now available via API by hooloovoo_zoo on Wednesday March 19, 2025 @07:08PM (#43418190)
Is your prompt {$codebase} find bugs?
Re:OpenAI's o1-pro now available via API by davidbarker on Wednesday March 19, 2025 @07:12PM (#43418215)
Typically something like:

  Look carefully through my codebase and identify any bugs/issues, or refactors that could improve it.

  <codebase>
  …
  </codebase>
Doesn't have to be anything overly complicated to get good results. It also does well if you give it a git diff.
Re:OpenAI's o1-pro now available via API by ionwake on Wednesday March 19, 2025 @07:46PM (#43418471)
Sorry if this is a noob question, but are you just pasting file strings inbetween those tags? like the contents of file1.js and file2.js?
Re:OpenAI's o1-pro now available via API by diggan on Wednesday March 19, 2025 @07:58PM (#43418553)
I do something similar, but "raw" markdown instead + filename, so all my prompts end up like this basically:

    Do blah blah blah while taking blah and blah into account. Here is my current code:

    File `file1.js`:

    ```javascript
    console.log('I am number one!')
    ```

    File `file2.js`:

    ```javascript
    console.log("I am number two :(")
    ```
Not sure if I'm imagining, but when I tried with/without the markdown code blocks, it seems to do better when I used markdown code blocks, so wrote a quick CLI that takes a directory path + prompt and creates something like that automatically for me. Often times I send identical prompts to ChatGPT+DeepThink+Claude, compare the approaches and continue with the one that works best for that particular problem, so having something reusable really saved time for this.

Edit: fuck it, in case people are curious how my little CLI works, I threw it up here: https://github.com/victorb/prompta (beware of bugs and whatnot, I've quite literally hacked this together without much thought)

Re:OpenAI's o1-pro now available via API by atxtechbro on Wednesday March 19, 2025 @08:18PM (#43418678)
I actually have a perfect tool called siphon-cli for this. It adds the headers in between files and everything. https://docs.siphon-cli.com/
Re:OpenAI's o1-pro now available via API by diggan on Wednesday March 19, 2025 @08:21PM (#43418701)
Yeah, like a lightweight version of my prompta CLI :)

What I end up with, is one .md file that uses variables like "$SRC", "$TESTS" and "$DOCS" inside of it, that gets replaced when you run `prompta output`, and then there is also a JSON file that defines what those variables actually get replaced with.

Bit off-topic, but curious how your repository ends up having 8023 lines of something for concatenating files, while my own CLI sits on 687 lines (500 of those are Rust) but has a lot more functionality :)

Re:OpenAI's o1-pro now available via API by quartzic on Wednesday March 19, 2025 @09:51PM (#43419210)
Not OP, but practically all of those lines are from a package-lock.json file (6755 lines) and a changelog (541 lines). It looks like the actual source is 179 lines long.
Re:OpenAI's o1-pro now available via API by pridkett on Wednesday March 19, 2025 @07:54PM (#43418530)
Repomix can take care of this for you. I pack it, cat the file to my clipboard with pbcopy, and just paste it into the prompt.

https://github.com/yamadashy/repomix

Re:OpenAI's o1-pro now available via API by diggan on Wednesday March 19, 2025 @08:00PM (#43418570)
I tried the web demo (https://repomix.com/) and it seems to generate unnecessarily complex "packs" for no reason, probably hurts LLM performance too. Why is there "Usage Guidelines" and "File Format" explanations in this, when it's supposed to just be the code "packed"? Better to just have the contents+filename, it'll infer that its directory structure and everything else.
Re:OpenAI's o1-pro now available via API by jakereps on Wednesday March 19, 2025 @08:27PM (#43418736)
While possibly being strange defaults, both of those are options. Remove the file summary and directory structure, both featured on the UI, and on the CLI tool, and voila, it's in your "better" state. There are also additional compression options beyond those two tweaks.
Re:OpenAI's o1-pro now available via API by keizo on Wednesday March 19, 2025 @08:29PM (#43418744)
I wrote a script to do exactly this a while ago (with the help of o1 pro). Makes it way easier https://github.com/keizo/ggrab
Re:OpenAI's o1-pro now available via API by davidbarker on Wednesday March 19, 2025 @08:30PM (#43418751)
That's right. I made a VS Code extension to combine all the files I have open into one long string I can copy & paste.

https://marketplace.visualstudio.com/items?itemName=DVYIO.co...

Re:OpenAI's o1-pro now available via API by crossroadsguy on Thursday March 20, 2025 @02:37AM (#43420316)
Do you get this -

When you say: But is that really a bug?

GPT: That's right. Now that I see it again this is not a bug….and a lot of blah blah.

Re:OpenAI's o1-pro now available via API by jbellis on Wednesday March 19, 2025 @08:57PM (#43418915)
Remarkably capable is a good description.

Shameless plug: One of the reasons I wrote my AI coding assistant is to make it easier to get problems into o1pro. https://github.com/jbellis/brokk

Re:OpenAI's o1-pro now available via API by andrewinardeer on Wednesday March 19, 2025 @09:04PM (#43418945)
I wonder what the input/output tokens will be priced at for AGI.
Re:OpenAI's o1-pro now available via API by stavros on Wednesday March 19, 2025 @09:12PM (#43418996)
They won't. Your use cases won't be something the AI can't do itself, so why would they sell it to you instead of replace you with it?

AGI means the value of a human is the same as an LLM, but the energy requirements of a human are higher than those of an LLM, so humans won't be economical any more.

Re:OpenAI's o1-pro now available via API by dnadler on Wednesday March 19, 2025 @10:14PM (#43419321)
Actually, I think humans require much less energy than LLMs. Even raising a human to adulthood would be cheaper from a calorie perspective than running an AGI algorithm (probably). Its the whole reason why the premise of the Matrix was ridiculous :)

Some quick back of the envelope says that it would take around 35 MWh to get to 40 years old (2000 kcal per day)

Re:OpenAI's o1-pro now available via API by gcanyon on Wednesday March 19, 2025 @10:54PM (#43419527)
I read an article once that claimed an early draft/version that was cut for time or narrative complexity had the human brains being used as raw compute for the machines, with the Matrix being the idle process to keep the minds sane and functional for their ultimate purpose.
Re:OpenAI's o1-pro now available via API by ben_w on Thursday March 20, 2025 @01:48PM (#43426462)
I've read a file that claimed to be that script; it made more sense for the machines to use human brains to control fusion reactors than for humans to be directly used as batteries.

(And way more sense than how the power of love was supposed to be a nearly magical power source in #4. Boo. Some of the ideas in that film were interesting, but that bit was exceptionally cliché.)

Re:OpenAI's o1-pro now available via API by gcanyon on Thursday March 20, 2025 @03:50PM (#43428050)
I'd love to read that file. Of course, we're close (really close?) to being able to just ask an LLM to give us a personalized version of the script to do away with whatever set of flaws bother us the most.
Re:OpenAI's o1-pro now available via API by ben_w on Thursday March 20, 2025 @05:56PM (#43429437)
One of the ways I experiment with LLMs is to get them to write short stories.

Two axies: Quality and length.

They're good quality. Not award winning, but significantly better than e.g. even good Reddit fiction.

But they still struggle with length, despite what the specs say about context length. You might manage the script length needed for a kid's cartoon, but not yet a film.

I'll see if I can find another copy of the script; what I saw was long enough ago my computer had a PPC chip in it.

Re:OpenAI's o1-pro now available via API by gcanyon on Thursday March 20, 2025 @10:36PM (#43431289)
> PPC chip

Pizza box? I loved the 6100.

Re:OpenAI's o1-pro now available via API by ben_w on Friday March 21, 2025 @04:37AM (#43433172)
Beige proto-iMac. I had a 5200 as a teen and upgraded to either a 5300 or a 5400 at university for a few years — the latter broke while at university and I upgraded again to an eMac, but I think this was before then.

Looks like there's many different old scripts, no idea which, if any, was what I read back in the day: https://old.reddit.com/r/matrix/comments/rb4x93/early_draft_...

I miss those days. Even software development back then was more fun with REALbasic than today with SwiftUI.

Re:OpenAI's o1-pro now available via API by gcanyon on Friday March 21, 2025 @07:45AM (#43434484)
HA! I used REALbasic a bit back in the day, then spent my time comparing it to LiveCode, back then called Revolution. Geoff Perlman and I once co-presented at WWDC to compare the two tools.
Re:OpenAI's o1-pro now available via API by 12345ieee on Wednesday March 19, 2025 @10:31PM (#43419399)
You need to consider all the energy spent to bring those calories to you, easily multiplying your budget by 10 or 100.
Re:OpenAI's o1-pro now available via API by kadushka on Wednesday March 19, 2025 @10:47PM (#43419491)
A human runs on ~100W, even when not doing anything useful. It's entirely plausible that 100W will be enough to run a future AGI level model.
Re:OpenAI's o1-pro now available via API by rlt on Wednesday March 19, 2025 @11:26PM (#43419656)
OpenAI doesn’t have the pre-existing business, relationships, domain knowledge, etc to just throw AGI at every possible use case. They will sell AGI for some fraction of what an equivalent human behind a computer screen would cost.

“AGI” is also an under-specified term. It will start (maybe is already there) equivalent to, say, a human in an overseas call center, but over time improve to the equivalent of a Fortune 500 CEO or Nobel prize winner.

“ASI”, on the other hand, will just recreate entire businesses from scratch.

Re:OpenAI's o1-pro now available via API by jasfi on Thursday March 20, 2025 @08:39AM (#43422292)
There's could be something to what you wrote. If AGI were to be achieved by a model, why would they give access to it via an API? Why not just sell what it can do? E.g. business services. That would be far more of a moat.
Re:OpenAI's o1-pro now available via API by foobiekr on Thursday March 20, 2025 @11:34AM (#43424730)
Can you describe this "find a bug" workflow?