What difference are you seeing from these models that makes it better?
I say this as someone considering shelling out $200 for ChatGPT pro for this.
It may also have a larger usable context window, not totally sure about that.
Can you provide an example of what you mean by this? I provide very verbose prompts where I know what needs to be done and just let AI “do” the work. I’m curious how this is different?
And partly it can actually execute more at the same time without starting to make mistakes
I routinely put about 100K + of context into Sonnet 3.7 in the form of source code, and in the Extended mode, given the right prompt, it will output perhaps 20 large source files before having to make a "continue" request (for example if it's asked to convert a web app from templates to React).
I'm curious whether O1 Pro actually exceeds Sonnet 3.7 in Extended mode for coding or not. Looking forward to seeing some benchmarks.
> We evaluate 12 popular LLMs that claim to support contexts of at least 128K tokens. While they perform well in short contexts (<1K), performance degrades significantly as context length increases. At 32K, for instance, 10 models drop below 50% of their strong short-length baselines. Even GPT-4o, one of the top-performing exceptions, experiences a reduction from an almost-perfect baseline of 99.3% to 69.7%.
Have a look at the RULER benchmark for a bit more detail.
I also frequently download all the source code of libraries I am debugging, and when running into issues, pass that code in along with my own broken code. Its very good
The naming would suggest that o1-pro is just o1 with more time to reason. The API pricing makes that less obvious. Are they charging for the thinking tokens? If so, why is it so much more expensive if there are just more thinking tokens anyways?
Look carefully through my codebase and identify any bugs/issues, or refactors that could improve it.
<codebase>
…
</codebase>
Doesn't have to be anything overly complicated to get good results. It also does well if you give it a git diff. Do blah blah blah while taking blah and blah into account. Here is my current code:
File `file1.js`:
```javascript
console.log('I am number one!')
```
File `file2.js`:
```javascript
console.log("I am number two :(")
```
Not sure if I'm imagining, but when I tried with/without the markdown code blocks, it seems to do better when I used markdown code blocks, so wrote a quick CLI that takes a directory path + prompt and creates something like that automatically for me. Often times I send identical prompts to ChatGPT+DeepThink+Claude, compare the approaches and continue with the one that works best for that particular problem, so having something reusable really saved time for this.Edit: fuck it, in case people are curious how my little CLI works, I threw it up here: https://github.com/victorb/prompta (beware of bugs and whatnot, I've quite literally hacked this together without much thought)
What I end up with, is one .md file that uses variables like "$SRC", "$TESTS" and "$DOCS" inside of it, that gets replaced when you run `prompta output`, and then there is also a JSON file that defines what those variables actually get replaced with.
Bit off-topic, but curious how your repository ends up having 8023 lines of something for concatenating files, while my own CLI sits on 687 lines (500 of those are Rust) but has a lot more functionality :)
https://marketplace.visualstudio.com/items?itemName=DVYIO.co...
When you say: But is that really a bug?
GPT: That's right. Now that I see it again this is not a bug….and a lot of blah blah.
Shameless plug: One of the reasons I wrote my AI coding assistant is to make it easier to get problems into o1pro. https://github.com/jbellis/brokk
AGI means the value of a human is the same as an LLM, but the energy requirements of a human are higher than those of an LLM, so humans won't be economical any more.
Some quick back of the envelope says that it would take around 35 MWh to get to 40 years old (2000 kcal per day)
(And way more sense than how the power of love was supposed to be a nearly magical power source in #4. Boo. Some of the ideas in that film were interesting, but that bit was exceptionally cliché.)
Two axies: Quality and length.
They're good quality. Not award winning, but significantly better than e.g. even good Reddit fiction.
But they still struggle with length, despite what the specs say about context length. You might manage the script length needed for a kid's cartoon, but not yet a film.
I'll see if I can find another copy of the script; what I saw was long enough ago my computer had a PPC chip in it.
Pizza box? I loved the 6100.
Looks like there's many different old scripts, no idea which, if any, was what I read back in the day: https://old.reddit.com/r/matrix/comments/rb4x93/early_draft_...
I miss those days. Even software development back then was more fun with REALbasic than today with SwiftUI.
“AGI” is also an under-specified term. It will start (maybe is already there) equivalent to, say, a human in an overseas call center, but over time improve to the equivalent of a Fortune 500 CEO or Nobel prize winner.
“ASI”, on the other hand, will just recreate entire businesses from scratch.
Very expensive, but I've been using it with my ChatGPT Pro subscription and it's remarkably capable. I'll give it 100,000 token codebases and it'll find nuanced bugs I completely overlooked.
(Now I almost feel bad considering the API price vs. the price I pay for the subscription.)