Why not read the bill itself? https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml...

It's not that big

The bill only applies to new models which meet these criteria:

(1) The artificial intelligence model was trained using a quantity of computing power greater than 10^26 integer or floating-point operations.

(2) The artificial intelligence model was trained using a quantity of computing power sufficiently large that it could reasonably be expected to have similar or greater performance as an artificial intelligence model trained using a quantity of computing power greater than 10^26 integer or floating-point operations in 2024 as assessed using benchmarks commonly used to quantify the general performance of state-of-the-art foundation models.

…and have the following:

“Hazardous capability” means the capability of a covered model to be used to enable any of the following harms in a way that would be significantly more difficult to cause without access to a covered model:

(A) The creation or use of a chemical, biological, radiological, or nuclear weapon in a manner that results in mass casualties.

(B) At least five hundred million dollars ($500,000,000) of damage through cyberattacks on critical infrastructure via a single incident or multiple related incidents.

(C) At least five hundred million dollars ($500,000,000) of damage by an artificial intelligence model that autonomously engages in conduct that would violate the Penal Code if undertaken by a human.

(D) Other threats to public safety and security that are of comparable severity to the harms described in paragraphs (A) to (C), inclusive.

…In which case the organization creating the model must apply for one of these:

“Limited duty exemption” means an exemption, pursuant to subdivision (a) or (c) of Section 22603, with respect to a covered model that is not a derivative model that a developer can reasonably exclude the possibility that a covered model has a hazardous capability or may come close to possessing a hazardous capability when accounting for a reasonable margin for safety and the possibility of posttraining modifications.

very similar to what the Whitehouse put out [1] in terms of applicability being based on dual use & size. It is hard not to see this as a push for regulatory capture, specifically trying to chill open source development in favor of some well-funded industry closed-source groups which can adhere to these regulations.

A harms-based approach, regardless of the model used, seems more able to be put into practice.

[1] https://www.whitehouse.gov/briefing-room/presidential-action...

Pretty much all models, including today's models, already fall foul of the "Hazardous capability" clause. These models can be used to craft persuasive emails or blog posts, analyse code for security problems, and so forth. Whether such a thing is done as part of a process that leads to lots of damage depends on the context, not on the model.

So in practice, only the flops criteria matters. Which means only giant companies with well-funded legal departments, or large states, can build these models, increasing centralization and control, and making full model access a scarce resource worth fighting over.

Really I feel the opposite way, that none of today's models or anything foreseeable meets the hazardous capability criteria. Some may be able to provide automation but I don't see any concrete examples where there's any actual step change in what's possible due to LLMs. The problem is it's all in the interpretation. I imagine some people will think that because a 7B model can give a bullet point list of how to make a bomb (step 1: research explosives) or write a phishing email that sounds like a person wrote it that it's "dangerous". In reality the bar should be a lot higher, like uniquely making something possible that wouldn't otherwise be, with concrete examples of it working or being reasonably likely to work, not just the spectre of targeted emails.

I've been actually thinking there should be a bounty for a real hazardous use of AI identified. The problem would be defining hazardous (which would hopefully itself spur conversation). On one end I imagine trivial "hazards" like what we test models with today (like asking to build a bomb) and on the other it's easy to see there could be a shifting goalposts thing where we keep finding reasons something that technically meets the hazard criteria isn't reall hazardous.

This sounds entirely reasonable!

640kb should be enough for anyone!

I don’t understand how a law can expect someone to foresee and quantify potential future damage. I understand the impetus to hold companies responsible, but that is simply impossible to know.

Will do, thanks! I must have just missed it if the original page linked to it.

Looks like it's trying to literally regulate speech. This would be struck down per Bernstein v DOJ.

There is freedom of speech regardless if it's written in English or C.

Writing code absolutely does not fall under free speech. Neither does any product development. Ford isn't allowed to ignore seat belt requirements and claim the government is infringing on the designers' freedom of speech/expression.

Haha, confidently incorrect. You made that up didn't you. Absolutely times infinity.

That case was extremely specific and doesn't mean that all code written falls under free speech.

It was also tried in a very different time. Given that we can't even allow free speech on digital platforms today, I'm not sure that many courts would allow for free speech claims to fall under the first amendment.

I dunno. This one seems trickier to me. Since it’s not really the code that’s the key part of the AI—it’s the trained numerical weights. Can you read and write in this language of floating point numbers? I doubt that. So perhaps yes you can write the code that could train a LLM and you could freely distribute that, but does freedom of speech allow you to train the model weights and distribute those?

Yes. Hence freedom of speech. You're allowed to write them in a book and sell the book if you must.

Briefly:

- Developers must assess whether their AI models have hazardous capabilities before training them. They must also be capable of promptly shutting down the model if safety concerns arise.

- Developers must annually certify compliance with safety requirements. They must report any AI safety incidents to a newly created Frontier Model Division within the Department of Technology.

- Cluster Operation Regulation: OOpolicies to assess whether customers intend to use the cluster for deploying AI models. Violations may lead to civil penalties.

- A new division within the Department of Technology will review developer certifications, release summarized findings, and may assess related fees.

- The Department of Technology will establish a public cloud computing cluster named CalCompute, focusing on safe and secure deployment of large-scale AI models and promoting equitable innovation.

I don't work on ML directly so I'm definitely coming at it from a more general CS angle, but I wouldn't feel comfortable anywhere near the first two bullets.

My outsider's understanding is that we really don't know specifically how the models learn what they learn or why they give specific answers. Is it possible that we could even know whether a model could present hazardous capabilities prior to training it? Or after it for that matter?

yeah, my problem with a lot of these people pushing for “regulating“ AI is that they seem to always assume the organizations they are dealing with are big, well-funded corporations. But this will affect individual or open source developers in California just as well. It’s a really dumb proposal, hopefully it doesn’t go through.