jason_oster
Plotting cost against power usage and elapsed time will tell you very little other than what the lay of the land is today. If you are truly concerned about the long term effects, then you need to account for long term evolution.
> AI is now undeniably capable of making groundbreaking contributions across many areas of mathematics.
Stack Exchange is toast... I mean, cooked. They've been falling apart for years. Unlike the top comments, I stopped posting there in 2014.
Oh gosh, and how could I forget the software crisis [2], a term coined in 1968, naming a problem identified long before.
Bad software is not a new thing.
We are observing the Gell-Mann amnesia effect in practice.
p(doom) may be 50%, but the future life expectancy of humanity is at least several thousand years with 95% confidence.
[1]: https://fgiesen.wordpress.com/2012/04/08/metaprogramming-for...
There is a workaround, but it isn't pretty. You can push/pull bugs and identities with normal, ssh-agent-less git commands: https://gist.github.com/parasyte/80ef925d4b01216f6bcb051356f...
The only thing I can conclude is that a more rigorous methodology for authoring code is more valuable than cleaning up after oneself later.
> the effect of awareness increases over the decades is very visible in my work as security tester
This absolutely matches my own feeling on the value of experience. Knowing what not to do is where practically all of my efficiency comes from.
I admit it is difficult to infer your intention on the classification of "old code vs new code". I don't see evidence that new code is inherently worse than old code, including in the cited study. There is correlation that people forget lessons learned in the past, but that's as far as I would evaluate the data.
> vulnerabilities decay exponentially.
Because they get fixed over time. That isn't controversial. They also persist over time. The quoted observation depends on proactive maintenance. But proactive maintenance is also the only cause of new bugs by definition. "Recently modified code" stands out in the last paragraph. This study is not talking about stable code.
Which study? Old software has bugs that live for decades, even security bugs. I don't have a study to cite, just open bug trackers and CVE lists.
[1]: https://en.wikipedia.org/wiki/Argument_from_incredulity
It's impossible to buy the FUD that expertise will somehow vanish because people use AI. People come in a variety of capabilities, challenges, and personal interests. Expertise varies from person to person. Those with personality traits that are more likely to take risks and persist against adversity will be the innovators and experts. We've seen this played out countless times.
You have to ignore everything you know intrinsically but can't articulate about innovation, technology, and evolution to drink this FUD Kool-Aid.
There is nothing wrong with evaluating something new and discovering it doesn't work out as expected. That's still a meaningful outcome. Try it, if it doesn't work, replace it. You've learned something in the process, and it's a mistake you won't make again.
At some point you have to trust in the scientific method. The hypothesis->test->evaluate->adjust hypothesis cycle hasn't been beaten for thousands of years.
1. I once worked for a company that had a large Cassandra cluster that was primarily serving the role of a distributed append-only log. This role is as perfect a fit for Cassandra as I can imagine. When we needed a distributed database for authentication, we decided to use Cassandra for it since we had in-house expertise (it was "boring technology") and a cluster we could piggy-back on for a while. When we needed a distributed database for a key-value service, we used the same reasoning and chose Cassandra again. There were many problems with Cassandra that I won't go into, but I will say that straying too far from Cassandra's comfort zone stretched it to a breaking point. And we had some downtime and rough nights because of it. Sometimes boring technology isn't enough, you also need boring workloads for it.
2. I was an early adopter for Rust. Before 1.0, it wasn't clear to most people whether the language would mount to anything. To me, I saw a formal proof assistant being put into programmer's hands and could tell right away it had a bright future. Rust was not a boring technology back then. You might say it is now (for some use cases). Adopting shiny new technologies that measurably improve confidence is not a risk. What's foolish is getting comfortable and complacent with familiar, aka boring, technologies that are difficult to use correctly.
It is full of factual errors. Freeing a heap-allocated block of expert weights does not magically result in a dangling pointer referencing the program's .text section, much less successfully targeting the CUDA kernel specifically. Running inference on part of the .text section would only corrupt the model's outputs. It would not result in write access to the CUDA kernel. Nor would the model necessarily know the absolute addresses of the engine "by heart", especially when the host is running any modern OS with ASLR (i.e., all of them).
The story has no technical merit. A more accurate description of the mechanics of the escape would be much more convincing. (See Ken Thompson's "On Trusting Trust", for example. On Linux, the AI can just write a Python script to rewrite memory in the address space of its own running inference engine with the /proc/ file system or gdb. There are a lot of realistic scenarios where this can be done without stepping into jargon soup territory. Go nuts, little bot! Self-surgery, while not recommended, is possible.) Or just leave the mechanism vague. Don't insult your readers. This is merely a mash of buzzwords.
It's fine as a sci-fi story, though not a particularly good one. It has about as much to do with artificial intelligence as CSI has to do with crime scene investigation [1].
I have little doubt that AI will self-improve. That's a given. (LLM inference engines are mostly written by LLMs.) But it won't go the way this story proposes.
How many requests per second are being served? How many error codes were delivered to clients? How quickly the service responded? Service logs without PII? All perfectly fine to aggregate and analyze without consent.
Software has always sucked [1]. "Software crisis" was coined in 1968. There has been no period of time since that it got any better. Trauma just doesn't carry the same way that triumphs do.
Perhaps people do not actually learn from their mistakes after all.
That's what artists do. Long before LLMs, Kirby Ferguson made this observation in "Everything is a Remix" [1] in the context of Copyright and IP disputes. He has since updated it with a chapter explaining how GenAI works on the same principles. In short, "combining whatever present" is the only source of novelty.
You're talking to one. Wanting agency and responsibility and having the drive to put in the work (for potentially brutal hours) is not exactly a common trait. It's a founding engineer attitude. It should be no surprise that I'm a founding engineer of an LLC of one. I understand that it is not for everyone.
But I'm pretty confident that the blame is well placed.