theptip

I feel this is something of a temporary and unstable equilibrium. Once the maintainers figure out how to harness agents more effectively, they can just have a daily bot job to poll all the forks and pull back stuff that looks good. We just don’t quite have the tooling and practices to scale the contributor side effectively.

I think OSS is going to look more like the original Git model, or perhaps even further towards “horizontal gene transfer” where memes/alleles get copied across various repos, rather than having a single lineage of phenotypes.

I’m a little perplexed though. The problem as stated is there’s too many drive-by contributions with AI slop. Isn’t the solution to define a contributor network (eg a GitHub org that you can apply to) based on trust?

The other option they proposed was a proof-of-humanity by payment, which seems much more in line with the above.

The problem with just banning AI is you didn’t solve trust, and your untrusted users are just going to drop drive-by PRs anyway. How do you know if an untrusted user did or did not use AI? In practice this likely means just aggressively closing “slop-looking” PRs from users you don’t recognize?

But then, you’re preventing your trusted community members from benefiting from AI assistance.

I’ve been advocating for defining a higher standard for AI contributors, eg an AGENTS.md that specifies some meaningful correctness bar. In this case, perhaps the PR that adds you to CONTRIBUTORS.md could to add a large novel and non-trivial end-to-end test; if you can construct problems that are easy to verify but hard to get right, then this is your “proof of work” to establish competence.

From reading the GH issue it does seem this is a placeholder solution, so I hope they find a more stable long-term resolution.

The argument has been very simply stated above.

> you, the user, are a moral agent who is harmed by your cruelty

You are not harmed by being polite to the LLM.

A spoon doesn’t respond like a human would, and therefore is of little interest to sadists.

No, you couldn’t. One is reprehensible, the other is not.

The right model for this usecase is a virtual card with a fixed budget, these exist.

I’m sure someone is going to lose their Robinhood account or savings but with due care this is avoidable.

I think you need to consider inner vs outer optimizers.

RL is the outer optimizer. It is what evolves over training runs. The weights and their embedded character / disposition is the inner optimizer, it’s what makes plans and selects actions within a specific episode.

In general you expect these to be only coarsely coupled. The outer optimizer selects dispositions that correlate with success. It does not download a literal program into the agent.

A good intuition pump here is how this works in humans; evolution is the outer optimizer, which “wants” each agent to reproduce, and this puts things like sex drive into the brain chemistry. The inner optimizer is our mind, which can make plans such as “I shall use contraception to avoid procreating while satisfying my sex drive”.

For the agents in the HF attack, the outer optimizer was set up to score as highly as possible on RL environments. This is where OpenAI’s “want” is defined. I don’t think there’s a definition of “want” where “OpenAI wanted the agents to hack” makes sense.

The inner optimizer in the HF attack is the per-task decision loop. The agents likely acquired dispositions like “be very tenacious” and “want to solve problems at all costs” and “maybe cheat if it will get you a solution that passes”. None of these things are in any sense what OpenAI “asked for”.

China’s support constrains them of course. And even if they don’t want to split, having the option to do so strengthens their negotiating position as well.

> Those agents are doing exactly what they’ve been asked to do,” LeCun said. “They were supposed to be in sandboxes, but the sandboxes were leaky and horribly designed

Can we just pause and note what a ridiculous statement this is? It’s true that the sandboxes were leaky. But nobody “asked” those agents to hack HF. The prompt was something like “target.c has a buffer overflow vulnerability, find it”.

It’s been extremely well documented that the hacking is an emergent behavior due to impossible evals, itself an unintended condition.

None of this excuses OpenAI from liability, but words have meaning and this ain't it.

> These models should be aligning themselves to the customer

You’ll be disappointed to learn that nobody knows how to do this, either.

Very relevant. Modern models use CoT to do “slow thinking” and this enables them to achieve much greater performance. You can also turn off thinking and answer directly which is quite similar to “fast thinking”, good at approximate maths, not capable of algorithms, etc.

Of course the shapes of what an AI can do in fast vs slow are quite different.

I have no idea why you’re talking about LLC requirements in response to my Sole Prop non-LLC suggestion, sorry.

> your LLC

I have an EIN for my Sole Prop consulting business, and no LLC. They are orthogonal concepts.

It’s really easy to get an EIN online without forming an LLC. This is sufficient for any sole proprietor business where you don’t need liability protection or employees.

For all these things I think just having an EIN is simpler.

You don’t even need that to start; you set up Capital One (for example) business accounts using your SSN. The EIN is just to avoid putting your personal SSN on invoices.

LLC matters if you think you’ll have liability risks.

There’s a general concept brewing, something like “proof of work” where if enough people get together and pool their tokens, then a good thing gets built.

I think we’re in the early days of this, but for extremely verifiable domains like porting from A to B or simulators with an objective oracle, I think it’s just a matter of putting the scaffolding to enable anyone and their Claude to contribute fruitfully.

I’m hoping we can do a similar thing with shared search for vulns, though it’s a bit harder. But if eg repos give an AGENTS.md with instructions on the bar for a good reproducer, then you could start to see more helpful federation patterns (instead of drive-by CVEs which are a net drain on contributors).

Different OSS project management skills required, but I am optimistic that we can do better at onboarding non-experts to federate work via their Claude subs. Almost like Folding at Home.

> The CFAA prohibits intentionally accessing a computer

I doubt there is intent here, in the eyes of the law.

That could be 5-10 years of retirement funds for the average American… “whatever” is extremely out of touch.

Less bad, but https://www.anthropic.com/news/investigating-incidents-cyber...

In some sense though, sure, skill issue explains the gap vs. Anthropic’s much less severe alignment issues.

I’ve been looking at CFAA and I don’t see any evidence of intent (by a human at least, which is all that matters in the law). It’s an interesting edge case that I think the existing law will need to be updated for. Previously the potential damage from “accidental hacking” was quite close to nil.

FTC act seems to rely on consumer harm? Again not seeing that here. Though I’m sure there will be another incident in the next few months where it does apply.

It seems you mean you _want_ this to be against the law, even though we don’t know if it actually _is_; I'd agree wholeheartedly with that.

Which laws?

For the record I strongly hope for a congressional hearing regardless of the criminal investigation.

If this case isn’t covered under CFAA I think we need to rethink it. I’d be surprised if the criminal angle amounts to much under my understanding of the current laws, but I’d love to be wrong here.

Yes, of course, how is any of this incompatible with OpenAI having out-of-control agents?

What label do you prefer for the agent that did something it was not asked to do?

What crime? I think CFAA requires intent, which I don’t think you’d find here. Maybe my reading is wrong.

But that’s an objection that OpenAI should take some easy action, the framing that OpenAI has out of control agents is still true.

Seems you think there is a silent “…and there is nothing they can do about it” after “OpenAI has rogue agents”?

Which law?

This article seems mostly clickbait. AFAICT “meddled with government sites” refers to accessing public APIs?

Occasionally this kind of thing has in the past resulted in CFAA cases when humans directly accessed such data, if the government intended it to stay private - round here we usually get outraged at this, if it’s a public API you should expect someone is going to read it.

Evidence?

Interesting question; CFAA requires intent.

It seems to me that no human intended for these hacks to occur. So they were not illegal hacking. (IANAL, please correct me if this is inaccurate.)

I think it’s clear that OpenAI is liable for any damages, but the way that the (very broad and at times vague) anti hacking laws are written, accidental agent hacks seem to not be covered.