grey-area
Well, in principle yes one system can compensate for another, in practice, errors tend to compound and combine in interesting ways, not go away, particularly in systems like this which are unpredictable and generate slightly different responses each time without human intervention. They are not reliable in the way we're used to considering and don't work well without supervision from careful humans at present.
I don't think there really is a clear distinction between these things though.
Yes your second point is an interesting one - the divisions between these things are arbitrarily imposed on the world, but we do have these distinctions for a reason, even if they are not completely objective.
I do think this is a significant problem specific to LLMs though (that I at least see when using them) - we don't have a way to constrain them in what knowledge they consider. This can be a bonus (they are good at connecting faraway bits of maths in interesting ways), but it can be a significant problem when asking for a summary of a data - confabulations and data from outside the context easily leak into the output of the LLM. That is undesirable behaviour and we have no reliable way to constrain it.
I'm not sure that this particular effort is going to help because it doesn't change or constrain the behaviour of the actual machines and the behaviour is the problem, not the data they are fed.
Perhaps the more important problem for google though is that people really like a query style interface AND response to a list of links. They don't even care if it's sometimes wrong (apparently). Google have added that, but it doesn't really fit with their current design.
The entire premise of google is now under threat.
To me the order of importance would be:
1. The model - if you use a terrible model you'll get bad output
2. The codebase - this definitely forms the model for what claude outputs to some extent.
3. The human operator, what they accept and reject etc.
4. Things like prompts, skills, harnesses etc. The quality of the prompt.
So I think our positions are really just a different ordering of what is important rather than fundamentally opposed.
Re Claude being a mirror, if it is it's a particularly distorting mirror which often changes things in subtle ways and is determined to make people feel smart and addicted to its output - it has been trained to do so after all. I really don't think a mirror is the best analogy, because it's too independent and seductive to be a mirror.
Of course Shakespeare made mistakes or made allusions that contemporary audiences struggle to understand, and he's not the be all and end all. Saying judgement is subjective does not mean judgement is useless or impossible. He's pretty good though and you can tell when you read him, it's gripping, human and tight.
Back to this particular text. From the start, it smells of AI:
> That morning I woke up at 6:58, two minutes before the alarm. I lay there and waited for it to ring. And it rang. We both did our jobs.
Short choppy sentences more suited to advertising than prose, ambiguity without purpose, pointless personification, melodrama and tension over the most mundane details.
It's possible this was written by a human so steeped in the language generated by LLMs they don't know what is human and real any more, but it seems much more likely it's a translation which was translated by LLM and 'improved' by taking its style suggestions, or simply generated/written by the LLM based on a much shorter prompt.
The most important thing though is not who wrote it, but the quality of the writing, and I think the verdict of most people disagrees with you.
I guess it all comes back to the fact that they don't have a clear division between data they have learned about the world, instructions and data under analysis.
You can ask them only to consider the 5 facts in front of them, but they won't do that, and cannot do that.
The internet does not provide anything close to anonymity and never really has. Even a simple IP address is enough to out the majority of users, when combined with fingerprinting, metadata tracing and the use of metadata from other users, there really is no possibility of hiding your identity from a determined attacker.
I would choose the opposite option personally, and would welcome it if this site for example allowed biometric identification and used that to weight comments. While I wouldn't want to force that on anyone I do think it's going to be necessary given the rise of LLMs.
server: nginx/1.14.0 (Ubuntu)
I'm not sure I share your concern about surveillance capitalism - that ship sailed long ago - you are stored in many nation-state and corporate databases under all the aliases, phone numbers, IPs you've ever used. I sincerely doubt your opsec has always been even near good enough to evade this surveillance, it's a little late to start worrying about it now, particularly when there are other concerns.
I'd be more concerned about billionaires spending on bots impersonating humans to sway elections in my local state than a nation state snooping on forum comments and linking them with something else.
Proof of identity is badly needed in order to use the online spaces we've come to know and trust, and time is running out.
I'm afraid your dystopian future where the zone is flooded with shit and nobody can trust anything seems eminently plausible, partly because the internet was built without identity.
I think to solve this you'd have to have strong proofs of identity locked to biometric identifiers. We'll probably need that anyway for other reasons.
It's not a trivial problem. Websites like this could require webauthn tied to a hardware identifier of their choice (like a fingerprint or faceid) - I'm not sure if webauthn provides that level of detail to providers (which method was used), and how secure that attestation is. Of course that's not going to be popular but it might be the lesser of two evils.
Harness, prompt etc etc are all peripheral IMO, though people like to think their super long prompt, custom harness and MakeNoMistakes.md skills are what contributes to their special skill with AIs, but really models are just improving .
You can just trust the AI to generate a perfect answer (and why are humans slowing it down at all if so?), but if you don't expect that yet you have to put in the work.
The number of reads is FAR greater than the number of writes for any given piece of code, and it needs to be comprehensible if it is to be extensible.
So in fact that instruction applies to all software, and I wish more people would take it seriously. It also applies in the AI era, because the AI has to understand the code too, and AI's spend more time reading code than writing it, indeed they base what they write more on what they read than humans do.
If it was all clearly completely incorrect, it wouldn't be such a huge problem, but it is plausible and hard for humans to detect in a short time unless it is long form text.
Even search providers are giving us this new kind of generated result at the top of their results, which most people will be using blindly to verify answers. So it's turtles all the way down.
If you're not a domain expert you have no idea which things are right and which are wrong so it's very hard to determine the truth or trust results.
To top all that, LLMs are being trained on this new untrustworthy data without differentiating it from older data, and trained by other LLM instances in a sort of recursive loop to improve performance on very narrow goals. What could possibly go wrong!
Vouched, no idea why this comment was dead.
Our current version of AI (based on LLMs) is nowhere near ready to do this, in spite of the dreams sold by companies like OpenAI and Anthropic.
We are a long way from AI agents escaping air-gapped computers, let's worry about that when it's an actual problem and when the companies running thousands of agents are acting responsibly.
Why it's urgent: the bug sits in code every Linux server runs when it accepts an IPv4 TCP connection, before any application sees the connection.
It reads like it was LLM generated though so who knows. There is at least a simple mitigation for Golang apps if that is the only concern.