pona-a
Is your radiologist using Claude? If so I hope he gets sued for medical malpractice, cause it's about as normal as consulting a crystal ball to find a tumor. AI is an ill-defined term referring to a good chunk of CS/statistics, not a single product like LLMs.
> Programming went the complete paradigm shift at least 18 months ago, and this buffoon still thinks it's vibe coding.
Doesn't help vibe coders still often call themselves that instead of affirming words like "agentic engineer"... As for paradigms being shifted, the dominant one is CEOs shifting into AI psychosis and shifting their IT/QA into unemployment, which is certainly a change...
There's a good reason automated judges are a common cyberpunk trope.
The point is by uncritically accepting this "technology" and not standing up to put any limits around its use, you're at best turning your job into a stressful data-labeling gig, and at worst helping your boss finally replace you.
OpenAI's harness can solve frontier math problems but the model alone is still a clofabulation machine. Its "intuition" did for a while get better from sheer scale, but we reached the optimum frontier for non-thinking models years ago.
Second order effects exist!
In the imagined faux-utopia, there is no more adversity, and it's assumed no active effort was made to preserve its struggle. A god-like nanny will step in to meet your every need before it could even cross into your consciousness.
There'd be no point to music if Mozart could already hear the 6th symphony extracted from his brain with the first thought of inspiration. This art would in turn hardly be comparable to the real deal, even if it were an empirically flawless replica, is there is only so much personality you could extrapolate without a long execution limited by effort or resources. But a convincing replica of art might be all we need to sabotage the real deal, either through spam or demotivating the novice artists.
This future is, of course, ludicrous. It's a childish fantasy, a bastardization of sci-fi writers' nuanced utopias, flattening the social systems they wrote to give their characters motivation and meaning, in favor of fetishizing the technology, which was just a plot device to put the characters in such setting, and acting surprised when taking it at face-value will result in immediate social decay.
The author might or might not be surprised he's not the first to write on the subject, and real sci-fi literature already argues the same points in say Metamorphosis of Prime Intellect.
It's the problem of delegation: you can do more of something, but worse, the more people you bring in. LLMs are just a very cheap contractor with a zero liability clause and some unchecked sycophantic tendencies.
If this is the face of our industry, that might explain a lot of things...
It's been exhilarating.
I've done things I couldn't imagine I could on my own, met some amazing people in the Discord, and felt immense satisfaction seeing the result and sharing it with friends. All things I thought AI permanently stripped from me.
If you keep training your brain to reach for the quick dopamine of asking Claude for a fix, you won't feel the long-lasting fulfillment from applying yourself to a hard problem. You have to quit that voice in your head, saying an LLM could do it in seconds, or it'd be too much work to study the problem yourself, and sit with some pen and paper. Your brain is so much more capable than you realize.
Now with a long laundry list of projects to try make, all I can ask for is more people giving it a go.
If another cis man was suddenly uncomfortable with "they", I'd attribute that to some fragile sense of masculinity or a weird anti-woke crusade, as was the case with most such cases in my life, and find it personally annoying.
I don't want to deliberately make other people uncomfortable with my language, but it does feel exhausting to second-guess yourself, having to double-check if any innocuous phrase might be misinterpreted as an underhanded aggression, preempting a correction we know will be awkward for both sides.
I think we'd do better to consider context more than the specific language...
Our brains "output" language as a byproduct (or rather shadow?) of conscious experience. An LLM is a machine built to project similar shadows, but that doesn't necessarily imply there's an entire world like ours shrunk inside that box.
Both brains and LLMs are physical, and both output language, but one is an appliance built to synthesize language and the other is a living being that merely uses it as a tool. LLMs are made out of pure language in a way people are simple not. Our minds undergo physical changes with every stimulus we process, we conceptualize ourselves separately from our environment without needing to be taught an ego, and we did not emerge from an optimizer crossing wires in our brains until we can perfectly imitate some external intellect, just to list some of these differences.
The biggest issue is you assume the LLM has any "I" from pre-training. Take a pure completion model like GPT-3 and you'll see it's most appropriate to describe as a simulator, not an entity in itself. The model itself doesn't have any notion of an "I", it will gladly write dialog as any character, from Skynet to the user themselves, if you don't apply external rules to constrain it, first coming in the form of stop tokens and prompts, and having later evolved into RLHF, bashing a "personality" into it by mangling its existing probability distributions (which is probably why all the most "aligned" models write in the most unnatural language, worse than those early non-chat LLMs). Any persistent self an LLM demonstrates was explicitly imposed on it by humans, which makes them obviously categorically different from animals or disabled people.
LLMs are by their nature dissimilar from any biological organism. Biological intelligence evolved in bodies to locomote away from threats and towards food/mates, over time growing to model the physical world as it relates to these goals, and eventually becoming sophisticated enough we can manipulate entirely abstract objects, like algebraic terms, just as early homonids or crows could use physical tools.
LLMs by contrast are algorithms trained to mimic human language. Without extensive post-training and prompting, they would not have a concept of a self, and even with these they can still occasionally confuse roles in a conversation. They can't learn beyond the limited memory afforded by context, which tends to simply degrade the more you utilize it.
I suppose Q-Learning does demonstrate a more similar dynamic to nature, and you could draw parallels with RL on LLMs, but I still think it's categorically different from the entirely open-ended environment that is the natural world.
You could say what makes it valuable for us to empathize with octopuses, even though their brains have a fundamentally different structure from ours, is our shared experience of pain, and you could interpret negative rewards as a kind of painful stimuli. But at that point, you've opened up a whole can of worms that you can't close until reductio ad absurdum.
As for any human-like behaviors, I think the answer should be pretty obvious. It's an algorithm trained to generate text, of course with enough expressive ability, the optimizer will seek to model emotional utterances, assigning a speaker's general mood to some internal parameter. But as I've said, it would be a trap, even if you're very much not a stochastic parrot believer like say most Less Wrongers, to infer that just because it speaks like us it thinks remotely like us (see the whole Shoggoth meme).
Is your company's backend conscious? Is a Turing machine running busy beaver? Is SHA256? Emergent behavior isn't limited to brain-equivalent things.
> why does it matter
Genetic optimization, whether through actual evolution, in software, or on FPGAs creates extremely interdependent systems that prove difficult to untangle and trivial to break. If experts can't fully analyze an evolved tone discriminator, because of all the tiny emergent interactions beyond the lumped circuit model in a poorly characterized environment, why suddenly claim we can isolate what process in the brain is responsible for an ill-defined notion of consciousness other than the whole?
> Neural nets can be almost any function
Neural nets can _approximate_ any function with enough layers (see XOR problem), but so can an infinite look-up table with enough entries. They are the best models we've got, but they are not magical.
Look at the most recent and illustrative example with fly brain connectomes. These scans only capture the relative strength of connections between the fly's neurons, to actually turn it into neural activity we need some sort of model.
You could imagine a human connectome with an ANN-based excitation model trained from EEG scans, but that would still be hardly equivalent to a human brain. You can fit on a narrow range of behaviors, but without modeling the actual mechanisms behind them, you'll fail to predict a lot more. You'd be missing the influence of hormones, which doesn't show up on a short MRI scan, glia cells, and a whole lot more nuance only a real biologist can educate you and me about, making the digital twin, if we anthropomorphise it, at best a cripple.
And that's a system directly mimicking a functioning human brain, rather than a highly de-correlated objective like webtext. Just as an NN trained to run towards a finish line could learn bipedal locomotion but will more likely abuse a collision bug and launch itself forward, a machine trained to mimick text could spontaneously instantiate a compressed model of the human brain, or much more likely become whatever alien thing can generate appropriate outputs, bearing less similarity to human thinking than any living animal.
Prolog isn't a funny nonsensical old language like COBOL, only liked by committe burocrats because they heard its name once. It's one of THE foundational languages of symbolic AI. In the ideal case, you're not supposed to write code, merely constraints the interpreter satisfies through search. It would be like saying OpenAI shouldn't have used Lean but Java.