Now the simple knowledge that there might be an exploitable memory issue in some given piece of software is enough for bad actors to point LLMs at it and find the issue in hours. That will only keep getting worse.
At some point, it's going to become a speed game. If you find a vuln, you might as well disclose to everyone immediately. Anyone who runs internet-exposed software will need an agent watching for disclosures around the clock and triaging them automatically (either by upgrading, improving filtering with some protective shell, or taking the whole service offline until humans can help) to keep up with the offensive agents watching around the clock and attacking automatically.
This still doesn't work, since attackers are much more motivated and able to pay attention to such things.
If your goal is finding vulnerabilities / stealing money / whatever, you focus all your energy looking for new exploits.
The rest of the world has other things to do (listen to users, run services, whatever) and by definition cannot spend as much of their time watching every social media in the world for the chance of an announcement.
1. This is also very hard to do correctly.
2. What automated steps would you take without any patches available to apply?
3. Potentially makes you more vulnerable to supply chain attacks
* A bad actor finds a real, theoretically severe but not easy-to-actually-exploit vulnerability in some popular software. E.g. it's an exploit that can get you pwned 100% with RCE, but only if you are using it in configuration X on platform Y with kernel version Z on a Tuesday.
* They design a patch that closes that vulnerability but subtly opens a more dangerous one known only to them. They privately test their patch against LLM reviewers, tuning the approach until the introduced vulnerability typically escapes detection by automated review.
* They "responsibly disclose" the original exploit and helpfully share the patch. In the hurry to fix the original exploit, the patch itself gets only cursory (claudy, codexy) review. By now this has happened so many times that just applying whatever code changes ClodSed were GTG is the status quo.
* When the new vulnerability is eventually discovered, they can plausibly say "oh my agent didn't catch that", which is true. In the meantime, they use the new exploit that only they know about.
> the patch itself gets only cursory (claudy, codexy) review
FWIW Claude is actually quite good at discovering variants of existing bugs.
In any case you don't even need malice --- sincere reporter can vibe up a fix that accidentally introduced a vulnerability, owing to their own lack of domain expertise and model errors... and then this bad fix makes it in because in the thrall of confirmation bias you fail to notice its shortcomings. Reporter also presumably tested it against their own LLM, so that will pass too.
This happened recently in a cryptocurrency project and contributed to a significant exploit.
It is all but certain there are exploitable memory issues in Chrome, Firefox, Node.js, Linux kernel, Nginx, etc.
Can we expect CVEs in a few hours? I'm sorry I ruined everyone's weekend.
I understand several have been accepted since, and he stopped the fuzzers because his backlog of submissions to make was getting out of hand.
Only for projects that are in a maintenance state. New features will produce new bugs, unless everyone pays for the tokens to scan every release with the top-of-the-line agents or otherwise spends more time than many do ATM to ensure that changes and new code don't introduce significant exploit paths. Things will slow down once the low-hanging fruit from the past is mostly detected and pulled, but the level of vigilance will always have to be at least a little higher than it has been until recently.
Time between updates releasing and people installing them has been an increasing problem for some time, we are just going to have to be a lot more disciplined about that. This will be a problem for places that have strict availability requirements as they might have to compromise on the testing time they spend on new patches before updating their installations, and we might have to move to having more flexible maintenance windows (the old “an hour or two once a month after Patch Tuesday” hasn't cut it for some time and certainly won't going forward) to allow faster updates.
The bigger problem, compared to the patch-availability-to-general-use time which can be reduced somewhat via automation (though that might accelerate bad-patch based attacks…), is greater constraints for the time between bug discovery and patch release: as soon as there is a rumour of an issue many will start scanning for it (as that is easier now) so the time to wide exploitation is going to be much shorter. This might lead to buggy patches that add new issues of their own.
Perhaps we'll see a lot more projects having explicit LTS releases that only get security updates or signifiant bug fixes, which will reduce the attack surface area (and fault surface area) most users, letting those with less stability concerns deal with the bleeding edge.
> For people like me, this is just yet one more way LLMs are making my life materially worse.
I'm with you there…
Insufficient. Models available to you are already cyber-gimped today and seems likely that they will be moreso in the future (see the recent K3.1 announcement for example). There will be attackers that lack such handicap.
Yeah I don't buy it either. Look at github right now, it is a total slopfest with people vibing projects they don't know the first thing about. Plenty of LLM use will be delivering code full of new exploits. It will be interesting to see the frequency of these exploits being found correlated to LLM friendly/unfriendly contribution policies once the dust has settled to tell for certain though.
Ask a top agent today to create a to-do app and watch it deploy enterprise grade security: anti CSRF, XSS, CSP, secure cookies, HMACs, rate-limiting of logins, and all kinds of things you never heard of.
I've asked it to implement a simple OAuth and saw it classify inbound IPs for private/public subnets, and put in workarounds for very theoretical DNS re-bindings attacks.
We all know LLMs love to over-engineer stuff, they are having a field day turning your static 3 page blog into crypto-exchange grade security.
1. Simple utilities that a vlogger or a sales guy created to automate routine. Such utilities are created in a hurry, deployed with prodiding private key to an agents, and so on. No attention to security was payed.
2. New generation of enterprise-grade slop that is produced by a relatively large group of people that don't review their work properly due to insane TTM and amounts of code.
Yes, their code might be secure in terms of simple techniques (agents help a lot of course), but in such code bases it's extremely hard to find out what should be fixed, how to fix it without breaking something else, and how to prevent something similar from happening in the future.
The time to exploit is also somewhat hypothetical. We're assuming infinite attacker resources which hasn't really been the assumption.
We see a lot more news and fear but are we seeing more actual "large scale exploitation"?
I'd love to see some numbers in that regard.
It's called job security :)
Like sure it’s true that you’re more necessary now, but when you took the job you weren’t expecting to need to be elbow deep in shit every day, and you’d probably prefer to go back to the days of changing lightbulbs and vacuuming hallways
1. many have underinvested in security for a long time
2. many are checking out of the development process
I generally think we can improve software with Ai tools, but I'm skeptical the current incentives will lead to this.
Or, if you're thinking server-side JavaScript: would the web have been more secure running PHP and Python?
And it's not fully sandboxed. It betrays its user all the time by abusing unrestricted access to the network, or activating browser functionality that most of us would disable if we could.
I'm not saying it's THE bad decision, but it is the product of a generation of thinkers who were not thinking about threats the way we do now. If we were to go back in time and have the opportunity to do it differently, I think we would take that opportunity.
But we don't actually need a time machine. We can just accept some sunk costs and move forward anyhow.
We aren't going back to a page refresh every time you want to update something any time soon
2. I'm not going to feel safe if I go make a web site that works that way. I'm going to feel safe if my bank does.
People will often claim to dislike something, even though they would dislike the alternative even more (and they might even be actively choosing against the alternative). Pay more attention to what people’s actions reveal than what they claim.
(This happens not just in tech.)
Give me a https://motherfuckingwebsite.com/ instead.
And before that, whatever Microsoft "macro" or embedded code variant was popular that week in all those document formats...
But 99% of web applications are forms and displaying stuff. All JS accomplished is making it look prettier and enhanced tracking.
I mean, Facebook, my bank, Amazon… these can all be created with no client side code. It would be a lot of refreshes, but it can be done.
If you want a wildly interactive art experience, just load the appropriate browser for that.
I think it's just a keeping-up-with-the-jonses sort of thing: If the restaurant down the street has a flashy website then I guess we need one too. Having restaurants in competition over the quality if their website is a negative sum game because it distracts them from the food.
HN could too - perhaps the biggest use on HN is not refreshing when up (or down) voting something. YouTube could do something similar - refreshing when 'liking' a video or getting more comments. But would you really find those to be a better experience for you? Even on actions that HN does refresh on, such as 'Next page', or submitting a comment, HN doesn't freeze sorts, so posts and/or comments you didn't see will often invisibly end up before the new location HN sent you to (or posts/comments you already saw earlier in the list now showing up "again" later in the list).
Not saying that's exactly what should be done, but an example that there may be another way than full-on generic code execution wrapping XHR.
As what an actual, safe, efficient implementation, of course not.
The point is there could have been a "dynamic HTML" alternate timeline where you could have had interactive pages for 90% of use cases without executing full blown generic computation arbitrary code on the client device.
But worse is better won and now we get insane trackers and cryptominers at every turn because the web can't degrade to exist without JS massaging DOMs.
AI skynet is killing humans. We know that already though.
The slowest that will ever be.
> second that there is a much shorter window between discovery and large-scale exploitation.
The longest that will ever be.
Aren’t exponentials beautiful?
I wonder if we start to hit a point where the pace that exploits are found and deployed increases faster than they get repaired, because there is a natural higher risk appetite for the bad actors, whereas the good actors still require careful thought and prudence. As a result of this, would the internet/digital economy largely seize up and crash? That would be bad, especially for a lot of people on this site. Maybe this leads to a regression in convenience, but requires more in-person interaction in order to function in society. You can already see this pattern today emerging in higher education. Maybe, in a deeper way, this improves humanity?
Is it? In using it at an opportunity to improve our devops pipelines, software supply chains, etc. Perhaps I'm weird and enjoy strange things? Not to mention that all these security incidents will require plenty of people to fix things = more paid work for us humans going forward.
People are saying the pace will settle down once LLMs find and fix all the bugs. Okay, sure, if you want to believe that sort of thing, but that still leaves the much shorter time window, and that's not going to go away anytime soon.
If you're on the side of the industry where your job or hobby involves breaking things, this is all very exciting. But if you're on the side of the industry where your job is to prevent things from being broken, then it really sucks.
I lost a full day yesterday because some big players discovered a thing, told other big players about it, "oopsed" enough public information to let everyone else know about it, and then left everyone else to figure out what to do with almost no information to act upon.
This cycle has happened several times this year now. If you're on the short list of big hosting providers, you get advance notice of what's coming and early fixes; if you're not, sucks to be you, hope you didn't have any plans today.
For people like me, this is just yet one more way LLMs are making my life materially worse.