thoughtpeddler

Aren't they running a safety classifier model in parallel to the main user thread to catch and prevent 'banned' outputs?

Same impression here reviewing several hiking trails in California. Some of the most famous ones too, like Yosemite’s Vernal Fall / Nevada Fall apparently have high radiation? What’s going on here? Am I to believe there’s a natural uranium deposit there?

Wow, I'm definitely overthinking and overprompting then. Seems like driving this process can benefit from a more laidback approach. Thanks!

There's another strong benefit to digital minimalism, which is the reduction in personal software footprint. The more software and network dependencies one is using, the greater security risks they take on. We live in an age where the supply chain risk is greater than ever, and it's useful to be mindful of the stacks our digital lives rest on. "How much can I get done with as little software and Internet dependencies as possible?" has always been an important question to be asking ourselves, and it's even truer now. Minimalism is not just a helpful principle to support our own cognitive wellness, but for our digital wellness as well.

Curious about the hardening and optimizing - how much extra prompting is spent to ensure a ‘clean’ software supply chain for this, versus having Claude at its own discretion use third-party packages/libraries/etc (or not)?

Amazon did not kill Barnes and Noble. For a moment, it looked like it did. But B&N is experiencing a resurgence the last several years, because humans don't just value getting books delivered the next day through e.g. Prime. They want to read books in a physical space, and will inconvenience themselves for that tactile experience. There is an important lesson in this when thinking about the AI future.

There's all kinds of telemetry being collected and sent to data brokers, for ad targeting (not directly through Anthropic but third-parties), etc. See this study led by IMDEA Networks in Europe: https://leakylm.github.io/

This article talks about results at a high-level: https://networks.imdea.org/your-conversations-with-ai-may-no...

> The study identifies three main issues: the exposure of conversation permalinks to third-party trackers; the ability to link these interactions to user identities through tracking mechanisms; and the presence of privacy controls and policies that may not accurately reflect actual data flows.

> One of the main findings is the potential transmission of information related to user conversations, such as chat titles, URLs (permalinks), or associated metadata, to third-party trackers such as Meta or Google, along with cookies and other identifiers.

> The study also identifies mechanisms that could enable linking activity in AI systems to real user identities. The combination of identifiers such as cookies (commonly used in tracking services), hashed email addresses, and server-side tracking techniques could facilitate the creation of persistent profiles and user re-identification.

Fascinated by the psychology here. Is it because you view frontier AI labs as beneficial entities that truly care about everyone's well-being, and will do what it takes to ensure, post-replacement, that we are 'taken care of', broadly speaking? Or are you more like "I'm ready to be replaced, and if that results in my death, I was over this reality in the first place, so it just speeds up what I've been hankering for anyway."

How does this follow? Isn't it fair to say that what someone's interacting with at the 'personality surface'-level is a form of emergent complexity / emergent behavior, not readily apparent or discernible at the neuron level (just like how human 'thinking' or 'consciousness' seems to work)? I'm not even arguing these models are conscious, I'm just saying that it doesn't seem like we can take the fact that a single neuron flipping their behavior or triggering 'alter egos' is enough to say they don't "think", etc. We might discover that flipping - if not a single neuron but perhaps really tiny clusters of neurons/synapses - in a human brain, might result in similar effects. We haven't had human connectomics studies to really show this (to my knowledge).

Ambition has to match the problem space. Sometimes I stare at the Bay Bridge here in San Francisco, and I wonder to myself, as an exercise of "how much more ambitious could AI make us?" imagining a 10x version.. would it simply be a bridge 10x bigger? Is that the best I could come up with though? And why is the bridge there at all? ~15,000 years ago you could have walked from SF to Oakland across a dry valley floor (that is now the Bay, true story). Then the glaciers melted, water flooded in, and only then did crossing it become a problem worth solving: first with boats, then with a bridge, and eventually with BART’s Transbay Tube, which isn’t a bigger bridge at all.

Bridges exist to move people and goods across water within a particular envelope of needs and constraints. In this specific case, when the needs grew, the most ambitious answer was not a bigger bridge, but BART's Transbay Tube, which is an entirely different kind of thing.

So when I reflect on Dan's piece, it seems we're in agreement that AI is not provoking us to simply build bigger versions of things we already have.. but what remains unsettled is whether we know what we’re really building for yet.

There's of course the by-now oft-repeated story of electrification of factories, which had a 'first pass' of mostly swapping out the steam engine with one big motor, and otherwise preserving the same layout. Over time people redesigned floor plans around small motors at each machine (with Paul David’s “The Dynamo and the Computer” being the classic telling of this).

A lot of AI-assisted software development right now still feels like the one-big-motor pass at this. I suspect the projects really worth being ambitious about are probably the ones that only make sense once the entire factory floor changes, and I don’t think we're able to see that just yet. What Dan's doing is probably the right answer at the moment, building a lot of odds things and seeing what sticks.

Back to the Bay story, I think this moment is equivalent to our digital glaciers melting, and the water rushing in, than us having to figure out better "cranes", for example. It's changing what the world we're designing for is, what the whole problem space is, instead of whether we build bigger versions of things within the prior paradigm.

We look to our software stack today and we are tempted to imagine 'bigger', 'better', 'faster', 'more efficient', but... why? What are we really building _for_? We shall see!

The US government is not a fan of other sovereign nations (even Western-aligned ones) from gaining "algorithmic sovereignty", as was seen when Australia tried to do so recently and received a stern threat from the US State Department.

I believe that in the fullness of time we will see that by the mid/late-2020s our society was completely information-saturated/overloaded, and hence why turning to these personal AI agents was seen by most as an easy decision (if not downright necessary to gain a feeling of control again over their lives).

Most peers (Millennial knowledge workers) I talk to these days constantly discuss the feeling of "being overwhelmed", and I suspect a lot of it is just the info-sphere.

Neil Postman predicted some of this many decades ago in "Informing Ourselves to Death", where he argued that increasing the speed, volume, and accessibility of information would not necessarily improve society (i.e. information without context/judgment and institutional scaffolding can become socially destructive).

I really feel like we live in that world, and hence the rise of these AI agents as a potential "solution" to that.

Yes, exactly. I remember when the idea of human biological cloning was brought up some decades ago, and society collectively decided "No". Fast-forward to today and digital cloning is suddenly possible and there's absolutely no questioning it - everyone's on board. I'm surprised.

If their security was so poor, why haven't frontier lab weights leaked yet?

Yeah, basically. I suppose as a business owner ideally one tries to walk a tight rope between the 'artistry' and 'fun' of aspects of the business and running it, vs. the cold brutal logic of capitalism, markets, demand, etc. It is interesting that biology is similar: you can as a being of emergent complexity choose to do so many interesting and artistic and fun and social things with your mind, but then you are embodied as trillions of cells, which will eventually succumb to the unfeeling cruelty of natural systems. That's reality it seems!

I'm not saying the 'Claude says this is bad so therefore it's bad' is at the heart of this PHP vuln, but I'm also seeing the same phenomenon in workplace settings with 'meatproxy'-behavior of so much AI use that tends to overfit and hyperfocus on narrow issues, that then punts the work to other people in ways that just waste time (whereby pre-AI, humans would simply not have the capacity to investigate minuscule issues so they'd never be reported as such). But these days, it's just hardcore meatproxy and reverse centaur vibes all over the place (people just acting as the face of AI, and creating so much work that the other people can't really keep up with any of it anymore), and I wouldn't be surprised if that's happening to some extent in infosec more broadly too. (not saying the vulns and exploits aren't real - but there's definitely a long tail of them now where only a few are deserving of a lot of attention, patching, mitigations etc).

I actually rather like 'SOCMED', given that social networks these days can be weaponized in various ways, so it deserves a name akin to 'OPSEC' and other military-like terms

> My north star for Foursquare’s culture (2010s) was always, “We want people to look back and think, ‘That was the best job I ever had because those were the best people I ever worked with’.” And I hear from ex-Foursquare people over and over again that we absolutely nailed this.

Reminds me of what we've been hearing recently from the mathematics community: that if AI completely overtakes their field, it's still about doing math with your fellow humans (just like how people still massively enjoy chess despite AI's dominance over that game).

More and more, technocapital progress reveals the social and communitarian heart of our soul.

But you already have Claude-directed videos that utilize OP's manim project (which he uses himself to do 3b1b) that anyone else can create. Grant open-sourced manim and now anyone can point a model at it to create their own 3b1b-like videos. They're popping up all over the place as Opus 5.5 demo videos just in the last few weeks. So... AGI?

I wonder how much of lagging AI adoption is simply due to people not trusting the existing AI vendors, or the architecture of deployment broadly speaking? If we somehow figured out how to get frontier-class models deployed easily locally (and I don't mean the bumbling demos of today - I mean with the sort of fluidity one experiences when using frontier models from frontier labs), would adoption be way higher for structural reasons?

To be clear, at 0.75 words per token, that's about ~6,100 words (roughly ~12 pages), which I suppose could be useful, but as other commenters have pointed out, you'd need some kind of orchestrated swarm/hive that could smartly fit payload to within that envelope

So refreshing to see actual human dialog at the start of an AI post/paper/explainer, instead of having that visceral "I'm just reading AI slop" feeling. Even if it's totally unrelated, the little bit about playing the Dune board game made me much more engaged and willing to listen to what the rest of the content had to say.

That was more early-mid 1990s, and not so much late-90s and early 2000s, when the NIST/NSA issues came about (from what I understand)

What was so different, culturally?

In light of recent pieces like "The Emperor Pelican Has No Clothes" (https://arcfu.com/post/the-emperor-pelican-has-no-clothes/), I'd be more curious to see how 'PelicanBench' becomes more robust in the time to come -- I feel like all models are now benchmax'd on the standard pelican SVG rendering task and we need something more multi-dimensional. It would be great to have a script that prompts a model to render the pelicans in several different situations (like the linked article) that showcase some kind of broad generalizability or other sort of 'transfer learning' going on with these models that would really then be impressive and worth noting on - otherwise PelicanBench is losing its relevance.

Yes, and that may be due to an influx of bots, unfortunately. Otherwise the core HN crowd has historically been nuanced and thoughtful. Some of the more recent HN data analysis posts also show how just overall submission and comment volume has increased substantially in the current AI moment, so along with that comes these sorts of negative effects.

Multiple things can be true at the same time. We should be able to hold in our minds that large AI labs are not to be trusted, while also understanding that the structural dynamics that support some of their assertions are also true (and are worth thinking about and preparing for).