joshka

ha - wh33zie and I have had this discussion before about sans-io on hn at https://news.ycombinator.com/item?id=40872020

https://tangled.org/oppi.li/naut needs migration to the new knot

Start with the person that is instrumental in this - Mitchell Hashimoto - you probably use products built on top of or with his tooling somewhere in your daily life already without knowing it. https://mitchellh.com/

I suspect you're getting down voted on this as it feels like this is a failure to educate yourself on something that feels like it should be relatively easy to do so in less time than it took to type your message.

I've reimplemented ghostty in Rust and it performed on par with ghostty.

I've also given various models access to the tools necessary to measure performance and instructions to make things faster and they do pretty well. Hill climbing on a validatable measure is reasonably easy these days.

My direct experience here directly contradicts pretty much everything you've said can't be done. I.e. the human out of the loop stuff.

I really want to give this a try for a backup to my main email (hosted on fastmail), but a couple of those enterprise features are things I'd want to really try out as a solo hoster (AI Sieve and masked addresses). I can't justify paying $50/year for those though. :|

It's great. Much stronger on the UX side for things that devs need than Claude / Codex. Feels like the issue tracker isn't a black hole. Release notes aren't "Fixes and unicorns", but real links to PRs. I use both Codex and Claude models in it, often interchangably - sub agents from one to the other as needed.

(Bias, I'm more familiar with Codex than Claude as I worked on Codex when I was at OpenAI, but mainly on the TUI not the app - take my better than Claude with a grain of salt perhaps).

> what currently is stopping Anthropic, OpenAI et al. from internally building (and maybe buying) datasets of good and bad architecture, and RL'ing that?

I think to do this well, you have to have all the comms, code for various projects over time and do large scale analysis with tooling to get the various outcomes. A lot of "good and bad architecture" is really post-hoc rationalization of decisions that worked, and it's likely super noisy.

I'd love to see this done, regardless of how difficult it is.

> Corporations can’t use code from GPL projects without contributing to them in some way. Once the license is changed to MIT through a Rust rewrite, the project is free to be used for whatever purpose benefits corporations the most.

Counterpoint: people at corporations who want to contribute to OSS that is GPL licensed often cannot because the legal risks of doing so is too high for many corporations to bear. MIT / Apache licensed software doesn't have this problem. Yes, you might have that software internally forked and used and see that as a net loss, but more often you'll likely find that a company does not want to own their own fork of an open source product and actually contributes back. GPL is friction there, and its supposed benefits seem (at least in my limited experience in 20 something years in the industry) don't seem to outpace this downside. (I put my money where my mouth is on this and release all my OSS as MIT/Apache generally).

Obviously big things like the Linux kernel or other fundamental tooling this doesn't apply, but for anything smaller the GPL is bad.

Love your work on this and the previous site. Thanks!!!

One comment I had (I think I may have posted something like this on a github issue too) is that the site has that feel a bit of the sort of UI from the early 2000s - like helm, or cold fusion dashboards. There's something about it that screams developer generated UI rather than designer influenced. The light mode looks ok, but the greys on the dark mode don't feel particularly scream well suited to a theme website. (Obv. just my purely subjective opinion here).

The swatches preview on the search results pages probably should be much smaller swatches - similar to the copy embed perhaps. And the background of the top part of the card should match the background of the theme.

see https://ratatui.rs/recipes/apps/release-your-app/ for some guidelines

x-posted on their feedback:

https://github.com/orgs/community/discussions/208952#discuss...

Instead or alongside this, how about making notifications good? There's some concrete simple improvements that would help triaging in general.

1. Provide ways to filter notifications of a specific state (merged, shipped release, closed) so these can be handled in bulk easily

2. Filter by type (issue / pr / release)

3. Provide access to the done state as a filter and via the api, so we can use non github website tooling to manage the notifications

4. Filter by author (e.g. let me filter dependabot notifications so I can handle them in one pass)

5. Show and allow filters on PR check status so I can look at PRs that are ready for a quick next step rather than ones that require digging into failures etc.

6. Make this all available properly via the API so I can let my clanker interact with this stuff based on rules that it has to help me triage stuff. Right now there are enough missing fields from the notification APIs that it makes it impossible to work around the shortcomings of the notifications with tooling (without screen scraping /notifications).

All of these things would be better for interacting with work to be done.

A general rule on this is if the information would make it easier to understand what work is to be done, to choose it, to act on it in bulk or in batches of similar actions, then that info belongs in the UI and the API.

---

Also. Please make it possible to subscribe to a discussion comment, but unsubscribe from a discussion. I want to read replies to this comment, but don't care to get a notification on every reply to this entire discussion. Unsubscribing. Tag me if you want me to read a reply here.

> We’d love to hear your thoughts on the new dashboard experience. Drop a comment with any questions or feedback in the Community discussion.

Please someone at github, hear this. Discussions are useless for feedback as you cannot subscribe to just the single thread you posted, only the entire discussion. That means I have a a binary choice of drinking the firehose or ignoring all future comments on some feedback I care about.

Please fix this.

My bugbears are narration - where the page tells me how to use it, or is labelling things in a way that a human would not use. "Choose a ..." where ... would be better as a heading.

I'd put it more generously (albeit biased), that they do take it seriously. But even serious people can be misguided in what things they pay attention to. Security is something that you have to get right 100% of the time and have people whose job it is to say no a lot. Research is the opposite. There's a clash of cultures in those two extremes and OpenAI was born from the wrong side of it. It's worth reminding that ChatGPT was launched as a "low key research preview".

I'd say the narrative that AI agents are a looming danger to the world is probably undersold rather than overhyped. I'm not particularly a doomer on this, but I have an infosec background too, so have a fair idea of what the combination of agentic harnesses + a malicious mindset could do to people/companies/nations/politics/world if wielded incorrectly. I think the good guys will win on this, but there will be plenty of interesting things that happen in that journey.

A good thought process might be to think back to the various large internet worms of the 2000s (Code red, Nimda, SQL Slammer, ...) which were mostly monoculture 0-days (not technically but close enough). Now consider if you no longer have monoculture / single bug as the limitation plus an ability for the hosts to take part not just as attack surface, but also cognition and planning. There's lots of variants of this and they're not particularly far fetched scenarios.

OpenAI's business model would align infra as a cost center rather than infra as a profit center (e.g. Google / AWS). Perhaps there's something there. I'd say also the OpenAI as a grad school that just happens to have a business aspect is also part of this. Bringing a tonne of good process on top of the build fast break things startup stuff would have cramped research speed significantly.

It's likely that OpenAI has gotten as good as it is because it ignored the traditional sysadmin stuff and went scrappy.

I worked there, but this is just my opinion and guesses, not facts.

dupe https://news.ycombinator.com/item?id=49822556

At Astra API prices that's 300B tokens (I saw 130B output tokens claimed elsewhere), large but not unheard of if you consider it across a few people doing random experiments with best-of-n type things. On my personal account, I've done a billion+ token days just on a normal pro 20x subscription. I know many others that wildly outpaced that by orders of magnitude. This was apparently 130B over 89 hours, so about 30x that rate. When things are free and you're expected to token max 30x seems fairly reasonable to me.

If you consider this as a cost to be compared against the question: "What does it take to be able to prove that you have a model that can solve the hardest problems that humans know about?", then spending a some amount of thousands/millions to know the boundaries of that seems not too important in comparison.

You've also got to consider this as compute that's allocated to pushing the frontier of what models can do, so while it's using GPUs that have been paid for etc., it's not like it's a cost that's supposed to be use less of this so that others can have capacity. If you made researchers afraid to use capacity like this, a lot of the things that improve would tend to do so significantly slower. (some may say that's a good thing ;)

A good way to think about this is when tokens are free, you get to choose whether you're optimizing for latency or intelligence rather than having to consider price.

---

Publically, tibo (Codex owner) in Feb this year: https://x.com/thsottiaux/status/2024649339344445825

> OpenAI employees currently get unlimited inference. Usage is now peaking at > XX billion tokens per week for some of them.

Mathew Berman (AI Youtuber) in Jun: https://x.com/MatthewBerman/status/2067270730795134984

> I've used 25 billion tokens in the last 7 days.

I worked at OpenAI previously, but don't know any of the people involved in this.

My guess was it was probably this was more a nerd snipe than any action from OpenAI that was a "massive team" being put on it. Literally someone looking at this and asking "I wonder if our models are good enough yet".

It's easy to assume that having access to massive compute amounts means significant coordination, but this assumes that you're looking at the costs of this sort of thing from an external lens. Internally, tokens are often treated as free and infinite.

I think you're assuming that the output of the page is LLM generated and not the process to produce the page.

Your decompilation threads have all the necessary info in them for this, anyone coming after lacks that foundation and effectively is doing a second inference over the hidden state, assumptions, etc. that your sessions have in them. A simulacrum of a simulacrum in essence is likely to be not particularly good.

If you're using coding agents for this, it may be worth splitting this up into multiple well arranged modules that tell a coherent story and make it easy to browse, and add explanatory docs based on the various things the LLM has found about each function / type.

Astra seems generally available now in codex at least - got any updates on this?

To some extent, and obviously grossly over-simplified here, copyright is the act of taking some already public good and putting private rights on it. My reading of the amicus is that the government is arguing that particular rights that NY Times are asserting that they have are not ones that promote the aims of the original reasons for copyright to exist and so aren't necessarily ones which need to govern the behavior of any other people (in this case OpenAI, but this likely applies to any LLM training no matter the size).

Would be good to see the full details of the tasks / methodology used. https://eebench.org/methodology.html makes this unclear.

> A real capacitor makes the task more interesting. A ceramic part may provide much less than its advertised capacitance once it has voltage across it. Parts have tolerances. Adding more capacitance costs more, takes up space and makes the rail slower to recharge when the power returns. A design that works with nominal values can fail with the parts that arrive.

It sounds like from a reasonable reading of the benchmark post that there's some things that are being tested that are assumed to be criteria that you expect the models to intuitively find those things to be important (i.e. the stuff about working on parts that have tolerances etc.). If that's so, then this really feels like mostly an exploration of whether an LLM has a good understanding of unstated constraints and has an appropriate in distribution set of priors that would be able to form models where it's reasonable to design on those lines.

It's hard to tell whether this is a problem though as the methodology is imprecise.

If you're spending time on evals against your own product, I'd be super curious to see how far you can get to by using a top tier model to produce generalized instructions for lower tier models. E.g. in a loop: "This eval missed X. what's the simplest single instruction that would have helped this session consider that as necessary that can benefit all future runs. Stick that in AGENTS.md and retest."

Reuters: https://news.ycombinator.com/item?id=49538820

NY Times: https://news.ycombinator.com/item?id=49543821

Not sure why the downvotes on this - it's a pretty reasonable take. I suspect that you're right that it's unlikely that the tainted items would be used to update the weights here and are more likely to be something that would be in sessions / memories rather than future model weights.