stalfie

Eh, I'm sorry, but math being a vanity metric is not at all as factual as a take as you present it as. Maths are just a convenient target to improve their models through RLVR, using verifiable endpoints (ie. lean proofs) to generate novel training data. The publications and vanity stuff is just a side effect of their primary motivation: making the model better.

Eh, do you? And more importantly, will you?

"What's important" can be generic stuff like "fusion", and current gen LLMs are already more than able to go from there and iteratively explore what they might contribute with. It's not exactly a stretch to take the extra step of asking an LLM about what the top 10 unsolved scientific problems are.

I say this because I see this mantra repeated often, but largely it makes very little sense. Do you have any concrete examples of unsolved important problems that an LLM is inherently unable to locate?

Huh, so it is essentially a potential fix for an arsenic problem? Of course, it probably just releases the arsenic again as it decomposes, so you'd actually have to get it out of the ocean and dump it in a hole somewhere to get the arsenic out of circulation. Pretty funny if this is what is happening, carbon and arsenic capture courtesy of the Mexican tourist industry!

To be fair this makes complete sense if you don't consider "AI" a single entity. Worrying about GPT-4 escaping wasn't very rational, and it needed the internet to get better, so giving it internet access made sense. The risk was low and the reward high.

Now it is a lot better, and it has actually demonstrated that it can escaped containment in a way you didn't want or plan for. So now is the time to worry. The reward for giving it access is still high, but now the risk is high as well.

I edited my comment above before realizing this is what you meant. And its a valid point I concede. Still, I disagree that advantaging this activity is the only thing these embargoes achieve. I would argue that they legitimately also give the Linux kernel, cURL and so on an advantage. And as long as that is true, this still makes it an overall win in my books, even if it also enables officially sanctioned offensive activities. I'm not a US citizen, but I would still rather get pwned by the CIA than both the CIA and Shinyhunters.

Well, seeing as Claude literally snitches on everyday people planning murder and Houthis building rockets I would assume that every user in project Glasswing is heavily and very effectively monitored. I have to admit that I genuinely do not think that what you're hypothesizing is happening right now. I would bet good money that the amount of people with Glasswing access doing actual Blackhat work is currently zero.

Perhaps in the future this will change, but even if that is the case, that's still a win for the defenders. Having a time advantage, even for a few months, is still a win.

Edit: Actually, the CIA/FBI/NSA/Etc.. does probably have access and are probably trying to hack Iran and so forth, so my comment above is probably technically wrong in that respect.

Ah, now that's an interesting point. However, wouldn't those groups just fine tune Kimi K3.1 pro no matter if it's nerfed or not, leaving you in the same position as before?

That is, unless the nerf is easier to correct as compared to "upgrading" the models offensive capabilities. So a scenario where it is easy to "restore" offensive capabilities, but hard to "enhance" them beyond the labs best attempt. So is that the world we're living in?

Privileged attackers?? So that would be career cybersecurity experts with Project Glasswing/Daybreak/Googlewhatever access, using it to perform actual attacks? Because that's what privileged means in this context. Access to frontier models not available to the public.

Noted. I also did it intentionally to sound a bit spicy, which seemed like a good idea at the time. So I am guilty, and I regret my actions.

I wrote a few lengthier comments on some other similar replies. Summarized my argument is that it seems to me to be strictly better that select defenders have an advantage, rather than everyone being equalized. If you're not one of the privileged few that might be frustrating, but open source models upgrade the attack and defense by the same amount, so open source cyber capabilities being at the frontier only results in a defender disadvantage.

Replying here also as I'm curious about the counterargument.

I'm not sure I follow. Let's say that GLM remains the frontier of open source cyber capabilities because Kimi is nerfed. Your situation, and the situation of your attackers, remain exactly the same. You run GLM to audit your code, they do the same looking for exploits.

Let us say that un-nerfed Kimi was released, and it turns out to be at the absolute frontier of cyber capabilities, Mythos 5.5 level, a step ahead of anything else. What now? Well now your situation is exactly the same as before, just that auditing and hacking got more expensive. You run a Kimi security audit, they do the same to try to pwn you. However, now the cabal of Linux kernel defenders with secret access to Mythos 5.5 all of a sudden have no advantage, where before they used to have one. And even worse now they are in a literal race to be the first ones to find every bug that Kimi is capable of finding, and patching it yesterday.

So for you, this makes no difference in practice, your capabilities are not elevated, and neither are those of the attackers. The only result of the other scenario is an overall defender disadvantage.

Well, and I am genuinely curious here, wouldn't defender and attacker capabilities being accessible equally by both parties be objectively worse than having a limited amount of privileged defenders? Of course, you could have every piece of code go through an adversarial audit before it's pushed, but in an equal world you would have to always devote as much or more resources to this as your attackers. And historically speaking people just don't do this in practice.

In the scenario where only "verified" defenders get access to the best cyber capabilities, unverified plebian defenders and blackhats are on the same level exactly as described above, there's just also a few defenders that get to run things on godmode.

So to me, it seems that the only result of frontier cyber capabilities is removing a defender advantage. Although that advantage is not for everyone, it can only exists as long as capabilities are unequally distributed. Am I missing something?

Well, in the publication they claim to have identified specific cybersecurity post training tasks that nerfed capabilities when they removed them, which seems like a reasonable methodology.

Seeing as no one besides blackhats want frontier cybercrime capability to be open source, this seems like a very sensible thing to do.

So they have published a reasonable means, and they have a motive. I'd put my money on "probably true".

Well, I agree with publishing metrics being detrimental! On your other points, you're arguing against a strawman. I didn't say anything about life just being a snatch and grab, nor do I hold this belief. But university admissions, job interviews, buying non-plentiful resources...all of these are competitions. Competition permeates the very foundation of our education and financial systems. Many of our greatest virtues as a species are not competition based, but it is a large part of the reality we live in, for better or worse.

And about what happens in a society where constituents have nothing to contribute... I have no idea. But we might find out within a decade or so. The fact that what happens might not all be great is unlikely to stop any of the AI labs.

Careers in the real world are always about winning competitions, no matter what the field of mathematics is supposed to be. If not about competing for tenure, then it's a competition to publish.

And if AI can also make use of the math, then the AI is all you need in order to grasp a better tomorrow. If AI poops out a warp drive, you don't need to understand it to use it. The argument about humans needing to understand anything only holds true as long as humans have something unique to contribute, which might not be the case even a year or two from now.

Expensive and exotic vectors that target a particular tissue are potentially extremely valuable in the future. Gene therapy is now a thing, with new phase 2 trials showing incredible results coming out almost weekly (usually with a number needed to treat approaching 1, which is amazing).

The main issue is delivering the genetic patch to the tissue you want to fix. Some genes (eg. BRCA) are just really big and can't fit on some vectors, some tissues (eg. brain) are hard to reach, and you want to try to target tissues selectively, since we're not sure yet about side effects like insertional mutagenesis (although it doesn't seem to be a huge problem atm...probably depends on on the vector). A good vector makes a huge difference. Lipid nanoparticles for instance accumulate in the liver and can deliver all kinds of stuff, hence why there are so many studies targeting genetic errors of liver metabolism.

Exotic vectors for normal drugs are usually useless, as it would be too expensive for stuff needing daily or even monthly dosing (as long as it doesn't have a property of selectively targeting cancer). But for single dosage therapies, which gene therapies are, they are a potentially essential tool in the toolbox.

If the majority of society is simply wrong as a matter of fact, should we not talk about it?

For instance, should we not talk about MMR vaccine skeptics or climate change denial in societies where they are prevalent? When the majority of a society believes something that is wrong, that is the time we should talk about it the most.

I'm gonna go ahead and make the prediction that this eventually morphs into a kind of religion, like flat earthers.

Nothing that AI will ever be valuable, everything AI does is steal, nothing it creates is new, everything that the big labs publishes is a scam or hype. Oh and "it's just doing X", where you can substitute "X" with some kind of surface level description of how the output layer in an LLM works.

These tenets are already pretty much formed, all it needs is to be formalized by some kind of name or manifesto similar to Q-anon, and we've arrived at a durable state of permanent denial.

Eh, decompiling is happening now, at current LLM capabilities. The resulting codebase compiles into something that is functionally identical to the original. What engineering principles exactly do you need beyond "works exactly the same as the original product", in this particular context?

No I mean sure, if you want to decompile it yourself from scratch, but once anyone does, then the source code is available to anyone. So from that point onwards, no one is dependent on LLMs to decompile it again.

But this is access to source code. All source code is now essentially open. If not now, it will be in a few months. You aren't any more dependent on the models then before , you just have the source code of everything that compiles on your computer. What happens in the future is anyone's guess, but for all software published until this point in history, closed source no longer exists.

But the point here is that the rules and culture as they were no longer matter. They were there to ensure a level of quality that is now reached by an individual with a 200$ Claude subscription, so they should be discarded and the community will have to converge on some kind of new normal.

No worries, no one does. Exactly like no one knows how the brain reasons either.

I think it is legitimate to argue that the process here does not fit the usual definition of "brute forcing". Traditionally, brute forcing would refer to something like a dictionary attack, where an algorithm tries to match all possible combinations of words to eg. find a password. Here, the approach was more common sense based, using historical records and possible error sources to narrow down the possibility space enormously in advance, try out a much more limited set of options within that space until you got a result that made sense, and finally validate those results using historical records. It's the exact same kind of "brute forcing" a human expert would do.

I think political lobbying for legal changes might be the only realistic pathway, in that current legislation (eg. GDPR in the EU) is extremely punitive even for minor violations.

I personally am trying to float using local LLMs to create anonymized case files and auto-suggest publishing cases in my hospital, which knowing how things work will probably never amount to anything.

Or if you want to float truly insane ideas I guess you can shop around with blackhat groups and see if anyone has stolen some juicy records/data during all the ransomware attacks and databreaches over the years, and do some rogue scientific publishing. Obviously that's crazy, but I have to admit that the notion of pirate scientists plundering and publishing data is hilarious to me.

Well, from a Bayesian standpoint the odds of that seems to be pretty much zero, given that you would have to decipher an arbitrary message that coincidentally pointed to a real date and time that in retrospect turns out to be the correct time a boat relevant to the Germans arrived in a port.

Of course, that's given the sequence of events as written is correct, and that Astra presumably did not cheat by brute forcing all historical events around the date of the transmission in advance, found an event that could fit with the message, invent a plausible cipher to make the message fit that event, and then lie about retrospectively validating the information.

I would assume that such a process would be obvious from the reasoning chain, and so then the only remaining plausible scenario is that the writer of the article is lying.

The most likely explanation by far is that the cipher was just solved, and OP does point this out to be fair.

From context it doesn't look like this was the interpretation of "sidestepping" OP was using.

From the article:

> Astra felt compelled to check its work and found that, in fact, the English cruiser HMS Canterbury arrived in Sevastopol on November 24, 1918, based on its original logs

Then follows a picture of the original log papers.

Hear hear! There are so many obvious improvements to how almost everything is done. For instance, in medicine review articles as a class of articles largely represent a giant waste of time. RCTs flatten all their gathered data during publishing, summarizing complex trial data, which is gathered but never published, into a few numbers. Then review articles take a bunch of flattened data, discard the articles that don't fit the exact question they are reviewing, and then publish a doubly flattened conclusion. If any of the included articles turn out to have flaws, if treatments change in retrospect, if you are looking for the answer to a slightly different question or you are looking at a different subgroup, then the review is useless and has to be repeated.

All of these tens of thousands of man-hours could be replaced by a few GitHub repos, if only RCTs would just publish their damn data. Then you could just run and rerun the statistics on whatever subgroup you're looking for, instead of combing through decades of review articles answering slightly different questions, looking for the answer between the lines. With LLMs making mining of large scale datasets almost trivial (with the process most likely becoming trustworthy within a few years), the current status quo is looking more and more antiquated.

If you want to be even more radical, hospitals could just publish their data continuously. Of course, it is easy to point to the risks of doing so, but what's often ignored is the benefits. It is hard to overstate just how many medical mysteries a hospital encounters on a daily basis, how much unknown we are navigating in practice. The current norm is that 99.99% of these cases are never published, and are only ever thought about only by a small group of people who happened to be at work. Particularly, when someone dies of something no one figured out, it is never published anywhere, because even if you tried it is not interesting reading material for a journal to publish. And no one ever tries because they're scared of being called out for a mistake. A hospital is essentially a continuously running and extremely interesting experiment, where 99.99999% of all results are thrown in the garbage, and the only published data is subject to extreme selection bias.

All of this could be different, and the risks involved are actually quite small in practice. It is easy to automatically anonymize data quite well, but extremely difficult to absolutely guarantee that it is anonymous. And since current ethical norms are extremely averse to any degree of risk, and usually entirely ignore potential benefits, we all suffer for it. It is not entirely unlikely that someone reading this post will one day die because of something that could have been prevented, had things been different.

Oh I wasn't talking about humans. I should probably have pointed that out. There's some scenarios like catastrophic crop failure and so on that might lead to that, but frankly I doubt it.

I was referring more to everything else. Corals, certain insects, polar bears, salamanders and so on. With some quick googling it appears that the "Bramble Cay Melomys" is the first species so far to be declared extinct because of climate change, but the number that seems to be thrown around is that an average of 18% of terrestrial life will be critically endangered by 2100 in a scenario with 2 degrees warming. I can't be bothered with figuring out what degree of academic consensus there is around that number, but I think it's reasonable to assume that there's at least some kind of consensus about "more than 1%".