dinp

My guess is, the large score jump for Opus 5 is mainly because of getting the right RL envs for training.

It's becoming harder and more expensive to build and run meaningful benchmarks, it would be interesting to see what they do with arc agi 4, maybe just give it gameboy/steam games and see how they compare vs a human baseline? The latency requirements and very long horizons in games could be an interesting challenge for llms.

https://x.com/lossfunk/status/2034637505916792886

"After the paper was finalized, we ran agentic systems that mimic how humans would learn to solve problems in esoteric languages. We supplied our agents with a custom harness + tools on the same benchmark. They absolutely crushed the benchmark. Stay tuned"

A little harness engineering was enough!

- have bold, strong beliefs about how ai is going to evolve

- implicitly assume it's practically guaranteed

- discussions start with this baseline now

About slow take off, fast take off, agi, job loss, curing cancer... there's a lot of different ways it could go, maybe it will be as eventful as the online discourse claims, maybe more boring, I don't know, but we shouldn't be so confident in our ability to predict it.

While we're here, another thing that's annoying: the token counter. While claude is working, it read some files, makes an edit, let's say token counter is at 2k tokens, I accept the edit, now it starts counting very fast from 0 to 2k and then shows normal inference speed changes to 2.1k, 2.3k etc. So wanted to confirm: is that just some UI decision and not actually using 2k tokens again? If so, it would be nice to have it off, just continue counting where you left off.

Another thing: is it possible to turn off the words like finagling and similar (I can't remember the spelling of any of them) ?

Click on 'Play moves' to watch a replay.

I initially planned to run a chess tournament for LLMs but they are not good: besides obvious mistakes, they output incorrect moves, get stuck in loops by repeating the same moves and the smaller models fail to output valid json frequently. I thought the reasoning models like o3 mini might be good, but they are an incremental improvement in chess.

Feedback and suggestions for other games to explore welcome.

Judging by the quality of comments here and in the linked submissions, it's a good thing.

Also how well do these models work to extract structured output? Eg- perform ocr on some hand written text with math, convert to html and format formulas correctly etc. Single shot prompting doesn't work well with such problems but splitting the steps into consecutive api calls works well.

I know I can access the claude api through 3rd party sites + official partners, but there's no incentive to go through the trouble when llama 3 and gpt-4 apis work great for my use cases.

[0] https://support.anthropic.com/en/articles/8987200-can-i-use-...

Looks like level 3 ddos protection is only available on the enterprise plan, it's not included in the unmetered ddos protection.

I wonder if they use gpt-4 itself to generate the data to keep it upto date.

Slightly OT: everyone feels hiring is broken, can you list some things that are annoying from the employee and employer perspective? Here are some points:

- the process often stretches out over weeks and often months

- job posts often get 100s of applications, a lot are low effort applications, it just muddies the water for both sides

- ATS systems/job boards are annoying with the need to create an account on many sites, some forms have more than 20 questions, often asking what's already there in the resume.

A question to everyone: What would a good application process look like? For me, it should just solve the above mentioned problems. I send an email with my resume, a few sentences about why I might be a good fit for the role/what interests me about the company. The jobs@<company>.com email address could be linked to some Saas product which makes it easy for the employer to go through the applications and further communication about video calls or take home assignments or whatever are all in this email thread. The employer can set the stage of the application such as 2/5 or whatever, they can mark it as rejected or accepted after all rounds to trigger automated emails etc. Is there any Saas like this? (I can build this in a week if it doesn't exist, but no clue how to market it/get users, any pointers in case me or someone else builds this.)

[0] https://openai.com/blog/function-calling-and-other-api-updat...

I've been working on this on and off for a few weeks now. The idea is to let LLMs interact with the outside world using function calling. Eg- I built a simple version of chatgpt web browsing using function calling. There's a django site which is responsible for the auth, managing chats etc and then there's a flask site that's responsible for running functions in a gvisor sandbox. Users will be able to upload their functions as docker images and I'll run them in a sandbox.

There is a lot of self doubt at this point: who is this for? Will people pay for usage? Should I open source the entire thing or part of it? To add to this: I have no online audience, how do I promote this?

I do have some answers to these questions. Eg- should I open source the flask site and get llama 2 to use it? It provides value to the community + hopefully I can get some attention for the project.

I would appreciate any feedback on this project. Please send me an email if you want some openai credits added to your account.

Edit: use testaccount/demouser to login, I've added some credits for testing.

More of a meta comment: this is pretty much why I don't install any extensions in my browser except an ad blocker.

You can use this as an opportunity to teach your friend about security so it doesn't happen again.

Would you be willing to give a few more details about what happened? I'm not interested in the identities of the companies or people, just interested in a high level overview of what happened. We don't hear these stories often.

It's head scratching to users and outside observers, but the incentives are such that there is pressure to grow on a quarterly basis, get those charts looking good and individuals in the management chain are doing what they can to optimize their career growth leading to short term decision making at the cost of users and customers.

The big picture thinking and long term decision making are incredibly hard. Very few companies are able to do this over the long term. Micosoft and apple are doing great currently and it will be interesting to see how stripe and openai navigate this process.

My current opinion is that only small founder owned companies or foss organizations can avoid this trap over the long term and it involves not trying to squeeze out every last bit of value. Both of these require a certain level of financial security + there's the opportunity cost vs just going the vc route.

VC funding is incredibly valuable and it opens up a lot of possibilities that small orgs can never hope for. I guess what I'm saying is: expect enshittification and enjoy the ride while it lasts and then jump ship when trouble starts. Jumping ship becomes incredibly hard with network effects, so that's the challenge we are seeing with social media companies now. Also once companies become too big to fail, it's a drag on society.

Personally I would still go the VC route since I don't have a few million lying around and tell myself this is just the cycle of life (for corporates) to avoid existential questions and going down the rabbit hole of questioning everything around me. Sorry about the disconnected thoughts.

I started around a week ago, another week to go before users can start testing it. If anyone's interested in trying it out, please email [email protected]

I little bit on the technical details, there are 2 parts: a django site that handles the websocket connections, making calls to the gpt apis, user management etc, there's a flask site that will handle running the docker images and returning the output. Initially I'll try to run the docker images using gvisor and maybe later move to firecracker (still exploring my options, not sure).

Here's a screenshot: https://imgur.com/377oPl5 The website is not online yet.