News for hackers. Stuff that matters.
Re:AI: Markets for Lemons, and the Great Logging Off by anselm on Thursday December 29, 2022 @08:33AM (#34172434)
Authors making this argument (which we see often now) really need to sketch out why cryptographic solutions will fail.

The entire thesis rests on this erroneous sentence in the second point: “This can be done in two ways – just move to invite-only silos where you already know everybody, or big platforms where the owners do the vetting for you.”

There are more options. We employ them when we are forced to; when it becomes cheaper than not.

We can sign posts or sign that somebody is real or to sign that somebody has earned rep, or that somebody has burned their rep… and subjectively score every piece of content that crosses our phone against our social trust graph.

We can rhizomatically scale social networks, can deputize our people in our extended social network to mark content as appropriate for kids or not, or otherwise filter. There’s no specific reason why we cannot grow a kind of social nervous system that has a kind of myelin sheath against noise and spam. It doesn’t have to be specifically only people we’ve shaken hands with or our like 3 closest buddies.

Re:AI: Markets for Lemons, and the Great Logging Off by awestroke on Thursday December 29, 2022 @08:39AM (#34172474)
So, people start signing their AI-generated messages. What have we gained?
Re:AI: Markets for Lemons, and the Great Logging Off by NoGravitas on Thursday December 29, 2022 @08:47AM (#34172529)
Indeed. SPF and DKIM were supposed to reduce spam by ensuring that every sender was verified. Now, we have more spam than ever, and all the senders are verified (on short-lived garbage domains that are not yet on blocklists). The only DKIM failures I ever see are on legitimate mail from badly set-up lists.
Re:AI: Markets for Lemons, and the Great Logging Off by kolinko on Thursday December 29, 2022 @08:51AM (#34172573)
DKIM and SPF are more important for phishing than for spam. Without them, it’s very easy to spoof senders.
Re:AI: Markets for Lemons, and the Great Logging Off by treis on Thursday December 29, 2022 @10:14AM (#34173417)
You can ban them and they'll stay banned. Today they'll come back or switch to one of the other dozen accounts they post crap on. The counter argument to that is that places like Facebook are mostly not pseudonymous and a lot of crap is posted there.
Re:AI: Markets for Lemons, and the Great Logging Off by djmips on Thursday December 29, 2022 @10:45AM (#34173768)
And when you get banned because someone stole your key - that's going to be awesome.
Re:AI: Markets for Lemons, and the Great Logging Off by anselm on Tuesday January 03, 2023 @03:54PM (#34237900)
There are going to be a few cases where keys are lost or stolen. It may be possible to build multi-sig wallets that allow for you to migrate an identity to a new key. What we're looking for is some kind of statistical means of trying to reduce bad actors. Even if there was no recourse for you and you were totally screwed I'm not sure it totally invalidates the concept of having a key or a mechanism for trying to remove bad actors from social discourse. You could trip and die because you wore shoes. You could get locked out of your house... bad stuff does happen - it doesn't mean we shouldn't wear shoes or have keys.
Re:AI: Markets for Lemons, and the Great Logging Off by awestroke on Thursday December 29, 2022 @12:29PM (#34175166)
Ban them on what grounds?
Re:AI: Markets for Lemons, and the Great Logging Off by HDThoreaun on Thursday December 29, 2022 @05:12PM (#34178736)
AI generated content will likely be higher quality than what we have today. AI accounts will spend time building reputation with superior content and then burn some on an ad or spam campaign, but they’ll still have a better reputation than most people.
Re:AI: Markets for Lemons, and the Great Logging Off by anselm on Sunday January 01, 2023 @11:49PM (#34214493)
It is gonna happen for sure. People will leverage powerful tools and claim it is their own voice. Me shrugs.

I more want to at least have that individual emitter be accountable for what they post; to establish continuity. I want to know that that emitter is say 3 hops from me, and is trusted by 12 friends in between, and has a general trust score of say 7/10 overall as an overall rating by my extended trust graph.

It is less that I want people to say good things, or be truthful or whatnot - I just want to know that they are real, that there is a human behind it, that that human has an opinion of some kind. The thing is that there are a ton of sock puppets that are not real - it's more about reducing the noise / spam rather than a perfect solution.

Re:AI: Markets for Lemons, and the Great Logging Off by xg15 on Thursday December 29, 2022 @09:18AM (#34172778)
The same problem we have with other networks of trust: Signing works alright if you already know the people you expect messages from (but in that case, it's also no better than the invite-only social networks the OP talked about).

However, the real problem is getting to know new people or vetting messages from people you don't know. In the future the OP sketched, you can never be sure that an interesting new person isn't actually a bot. Knowing the public key of that person won't solve that problem.

Re:AI: Markets for Lemons, and the Great Logging Off by rocqua on Thursday December 29, 2022 @09:26AM (#34172860)
One of the biggest problems in actually using cryptography in the real world is matching keys to identities.

Using cryptography to solve the fake persona problem only works if the key-identity matching problem is solved. It would be great if the fake persona problem was the impetus that managed a solution to the key-identity matching problem. But I have my doubts.

Notably the key-identity matching problem isn't technical. Its societal. "just do government provided keys" is technically easy, but society (rightfully so) is suspicious of this. Other solutions exist, with other trade-offs. SSL certificates have centralization, revocation, and weakest-link problems. PGP-keys have spoofing, verification, and usability problems (though I liked key-base's approach here). European E-id is an interesting step, one to watch, though I fear the EU bureaucratic system might make a crucial mis-step. I really like SSI based approaches, but SSI is mostly about using crypto when the key-identity matching problem has been solved, and less about solving the actual problem.

Some technical aspects that need solutions that tend to be un-acceptable are handling key-revocation, key-theft, key-loss (as in forgetting), and key-duplication.

Re:AI: Markets for Lemons, and the Great Logging Off by px43 on Thursday December 29, 2022 @10:58AM (#34173936)
How about we let people generate their own keys, then use those keys to make identity claims, when people can generate on their own, or which can be generated by a third party. That gives multiple options to bind identities to arbitrary social media accounts etc without needing some monolithic root of trust.

I also really like the idea of using the keys to hold some amount of value, such that if the keys ever get leaked, there is basically a built in bug bounty to alert the key holder (since the key thief has the option to take all the money). This also gives users the incentive to manage their keys in a sane way.

Social key management schemes are also super interesting, and will likely be a part of future key management schemes. That is, basically, allowing some set of friends and family to re-roll or revoke identities that have been lost or stolen.

Slowly but surely, I think this future is coming. Lots of good people are coming at it from different angles, but basically all converging on the same general concepts.

Re:AI: Markets for Lemons, and the Great Logging Off by throw10920 on Thursday December 29, 2022 @12:47PM (#34175474)
> How about we let people generate their own keys, then use those keys to make identity claims, when people can generate on their own, or which can be generated by a third party. That gives multiple options to bind identities to arbitrary social media accounts etc without needing some monolithic root of trust.

You hit the nail on the head. Matching keys to real people can be done in-person for direct friends, then through a web of trust for indirect friends. For accounts/keys/personas you find on the internet where you don't have a chain of friends, you can either rely on "trusted" third-party attestation ("holder of key 0xdeadbeef earned a degree from this university") - you may never know with complete certainty if that's a real human, a bot, or an alt account for someone you already know, and that's totally fine.

The "problem" of matching keys 1-to-1 with identities for everyone globally (brought up by grandparent post) is a massive red herring that doesn't need to be "solved".

Re:AI: Markets for Lemons, and the Great Logging Off by rocqua on Thursday December 29, 2022 @06:31PM (#34179559)
The problem I brought up in the grandparent post wasn't (meant to be) about a 1-to-1 mapping. It was about asking "who owns this key".

Web of trust sounds nice, but it hasn't caught on. I would say because it has trouble going beyond one hop of trust if you actually consider adversaries. In person physical confirmation works, re-keying is a major hassle though.

I do like the idea of a 'web of trust is good enough for declaring you are a person'. If you get a vouching system with a recursive revocation system, that might work well enough for establishing you are a person (though not well enough for establishing which person you are).

Trusted third parties have problems. They centralize power, either in the state, or in some non-accountable organization.

Re:AI: Markets for Lemons, and the Great Logging Off by peteradio on Thursday December 29, 2022 @10:16AM (#34173442)
You can employ those techniques but real people will get blocked as spam. And the better AI can evade the closer to the bone you will have to cut. Then what? AI is interacting with your algorithm to silence voices.
Re:AI: Markets for Lemons, and the Great Logging Off by anselm on Sunday January 01, 2023 @11:46PM (#34214470)
Even with some defects or imperfections anything is better than what we have now - which is basically nothing.

I think the way I'd think about this is to imagine say a small community, such as a town of say 5000 people or so. While you cannot know each person individually, you can know of people by reputation. People do earn rep over time, and they can burn rep. It is true that some people will be unfairly downscored, or unfairly upscored - but I'm not really trying to argue for those fine grained situations. What I'm trying to argue for is simply distinguishing the very bad actors acting out of pure malice from injecting fake news, media and 'yellow journalism' into human conversations.

True some real people will be downscored (I prefer to think of this as downscoring bad actors rather than 'blocking'). And true an AI can 'sound very human' - but an AI or a bad actor will struggle to build up a reputation over time. An AI can't shake hands with you, it is harder for it to prove it is human... Other bad actors will presumably burn their reputations if they spit out a series of offensive, misleading, false, inflammatory or toxic posts...

Note I am not necessarily advocating for crypto per se as a way to establish social trust graphs (a'la PGP or say Keybase) but I am arguing that there are other options that the OP did not raise. I more want to see a wider discussion around ways to filter malicious media that either "centralized systems" OR "small social clubs". I'm not necessarily saying it has to be a cryptographic solution... but I do think there are more ways to have what we want.

Re:AI: Markets for Lemons, and the Great Logging Off by vlovich123 on Thursday December 29, 2022 @10:44AM (#34173746)
Because using technology to solve human problems rarely/never works. [1] was originally written for spam and bore mostly correct. Explain how replacing spam with “AI-generated spam” changes anything? You can try to fight this stuff but it will look more like AI to detect AI (similar to our current anti-spam tech). No reason to believe cryptography has some kind of magical bullet here as it’s an unrelated problem domain. And to the person claiming you get kicked off and that prevents you from coming back ignores a) we haven’t solved being able to tie disparate online personas for a unique offline one (despite Facebook ostensibly trying really hard) b) there are all sorts of secondary problems that pop up when you try to do that (eg ignores the concept of learning from your mistakes and redemption, key things that happen frequently with the young or anyone else testing boundaries).

[1] https://trog.qgl.org/20081217/the-why-your-anti-spam-idea-wo...

Re:AI: Markets for Lemons, and the Great Logging Off by throw10920 on Thursday December 29, 2022 @12:39PM (#34175354)
> Because using technology to solve human problems rarely/never works.

You're badly misunderstanding the parent post - it is not proposing a technological solution to a human problem, but a technological enforcement of a fundamentally human solution:

> subjectively score every piece of content that crosses our phone against our social trust graph...can deputize our people in our extended social network to mark content as appropriate for kids or not, or otherwise filter

This is a social web of trust, where real people do the ranking and trust assignments - the cryptography and other technology just keeps track of bookkeeping.

Re:AI: Markets for Lemons, and the Great Logging Off by vlovich123 on Thursday December 29, 2022 @05:47PM (#34179143)
Given that GPT is already difficult to distinguish from a person who’s confidently wrong, how does this web of trust system solve the problem?

The belief that anything will “solve” this seems naive when there’s 20+ years of proof of this being an “unsolvable” problem despite repeated technological, social, and legislative attempts. There might be a new normal established with new battlegrounds drawn and we learn to “live” with it, but I’m willing to bet non-trivial domes of money against there being any true “solution” here.

Re:AI: Markets for Lemons, and the Great Logging Off by throw10920 on Thursday December 29, 2022 @11:22PM (#34181748)
You clearly don't actually know anything about webs of trust. I encourage you to read up on them: https://en.wikipedia.org/wiki/Web_of_trust

It's obvious to anyone with a passing familiarity of WoTs that you seed your web with people you know in real life. GPT is not "difficult to distinguish from a person who’s confidently wrong" in real life.

Re:AI: Markets for Lemons, and the Great Logging Off by theamk on Saturday December 31, 2022 @11:04PM (#34203483)
Are you going to be accepting direct confirmations only or indirect as well?

If you are only accepting direct confirmations, this means you are only going to talk to people who you meet in person. This is totally fine and will work, but then you don't need any new tech -- just ask for their email / phone / nickname on your favorite social site. Or make a private forum (or a Signal/Telegram/Whatsapp group) and invite them there.

If you are accepting indirect confirmation, then once the network grows big enough, there will be bots. Maybe some of your friends meet Greg, director of marketing for Widgets Inc., and correctly confirm him as real human, and then Greg will confirm an army of GPT telemarketer bots as "real humans" so they can do the sales and earn Greg a bonus. Or maybe your good friend gets malware on their computer and their key is used to confirm a bunch of keys without their knowledge.

Re:AI: Markets for Lemons, and the Great Logging Off by vlovich123 on Sunday January 01, 2023 @03:55AM (#34204788)
May be a bit wise to point to something other than the Wikipedia page before claiming I don’t knowing something about an entire topic.

You seem to be intentionally or otherwise completely missing what I’m saying. PGP just establishes ownership of a private key. It doesn’t say anything about that person then choosing to sign the output of GPT or giving GPT that private key to do whatever with. And GPT can mimic whatever writing style you give it. Not hard to imagine giving it various writing samples of yours personally to learn from and imitate. So please explain how web of trust solves anything for that. Aside from trying to keep track which person in your personal network is a spam vector, because there are super connectors with thousands or tens of thousands of real-world contacts and logistically that’s not realistic to manage since most people aren’t cryptography nerds.

There’s also a bigger fundamental problem with web of trust in applying it here. Trust is not binary in that way. If I trust person A and they trust person B, trust isn’t a transitive property so in reality there’s nothing we can say about my trust about B. Also, trust isn’t binary. If I trust person A about specific science topics, that doesn’t mean that trust extends to other topics. Trust also isn’t static and sticky whereas it is generally treated as such in computer systems where trust needs to be scores and revoked automatically somehow (and then we’re back to an AI war to detect abuse and better GPT to avoid the abuse). This is also ignoring that trying to model human trust webs with a CS model that doesn’t work anything like it, isn’t a good recipe for success. Also, human trust webs have massive trust and scaling problems) (cough Theranos, WeWork, FTX, Madoff, etc etc). Notice how web of trust always sticks to basic cryptographic primitives which are easy to write papers about and solve academically but not a solved problem by any means in terms of defining what trust actually means or how a web of trust works in terms of AI content. Obviously PGP has been around forever and AI is a bit newer so maybe there will be interesting work coming out of this space at some point. AFAIK today web of trust buys you bupkis in terms of fighting GPT spam. I would recommend reading up any number of articles that discuss why pgp and signing parties failed. It’s not purely a UX issue. The bigger problem is that even in a “trusted” system, fraud arises spontaneously because it’s the prisoner’s dilemma problem - there’s a material advantage to perpetrating the fraud and there’s a better advantage to helping perpetrate fraud (much hard and longer political process to revoke that trust).

As a more concrete practical demonstration of this failure. Consider that certificate authorities, which are assumed to be “trusted core signatories” in a PKI system. A PKI systems is actually the same as a “web of trust”, it’s just that I delegate verification to a third party. As cryptocurrencies should have shown, people still prefer having a virtual account in a bank to manage those funds/ lower fees. Similarly, people will outsource the complexity of validating identities (CAs signing carts for websites). CAs have repeatedly abused this to the point where afaik the security community generally acknowledges that CAs are generally worthless - even the “good ones” struggle to do verification at scale and there’s so many CAs in typical lists that it’s basically guaranteed that there are malicious actors. And we know decentralization doesn’t actually work for end users because it’s too complicated a mental model. People want a named intermediary that they delegate responsibility to. That’s why most people defer their CA validation to browsers and operating systems. PGP would work similarly so now you’ve got people delegating key trust to Apple, Microsoft, Google, Signal, etc etc. and nerds who use open source verified key managers and maintain their own infra to manage these lists. But that’s not a representative sample of what end users will accept at scale. So you’re back to centralized control, which will be better than status quo as OSes and browsers realistically are more resitant to handing out broken signatures. And of course maybe better algorithms and methods will be developed to solve these shortcomings.

But a lot of these issues have existed and been documented for a long long period of time independent of the vague idea of using it as a GPT detector / blocked. I’ve been around the Bay Area for 10+ years and I remember having friends who thought this would take off any day now and worked really hard to make it happen, hosting signing parties and whatnot. It didn’t and I was pretty confident it was a pure nerd activity that wouldn’t have any impact in its current form (regardless of the UX challenges - the problems are much more fundamental and worse). Web of trust is seriously hard even in the simplest possible state which is PGP and that’s failing miserably despite being around for a very long time.

Would you agree that it’s on you to provide some supporting evidence on the claim that

A) I don’t know what I’m talking about

B) something more than hand waving “sprinkle some decentralized cryptography here” and actually explain how you solve the human problems that are so important here + why PGP has largely failed but suddenly it’s going to find a second life in GPT prevention.

Re:AI: Markets for Lemons, and the Great Logging Off by anselm on Tuesday January 03, 2023 @04:07PM (#34238076)
Yes the improving strength of GPT is magnifying the problem but is unrelated to the solution. The solution space steps outside of examining the content of the message for truth. The solution is to sign the messages using a social trust graph; and scoring posts based on the social trust distance.

It's not a very compelling argument for me that a (very short) 20 year history of failing to solve this problem means that it cannot be solved.

I'm willing to bet you $1000.00 that this will be completely solved in 10 years. In 10 years we will know if content we consume is "fake" or "real". It may still be offensive, and harmful to gullible people - but it will all be scored as to how real it is. Real will be defined as the likelihood that the content comes from a real human being type person, as agreed upon by other persons in-between you and that person. Content one hop away from you, from a friend, will have a score of 100% real, and content many hops away will have lower score. You will probably start your day by sorting the content you consume by the likelihood that it is real.

I did actually work on PGP - and I'm willing to concede it didn't succeed. But DNS works and bitcoin works (technically). So we do use various flavors of cryptographic trust to make sure that actions in a network are "real" versus "forged".

And yes - it's true that bots can creep into a social trust graph... so yeah, it make take effort to keep pruning the weeds in the garden in a sense.

Note in some ways I'm not really trying to argue FOR crypto per se - I'm just saying that the OP should at least critique crypto if they want to make the thesis they are making. And I'm arguing that it is a big omission to gloss over the utility of crypto and the argument will be hard to make that crypto is not a significantly powerful modifier to the original thesis of the OP.

Re:AI: Markets for Lemons, and the Great Logging Off by wanderingbit on Thursday December 29, 2022 @12:44PM (#34175428)
Context: I've been working in the cryptocurrency space for a decade.

One of the most interesting technologies the crypto space is working on are the SSI (self-sovereign identities).

https://en.wikipedia.org/wiki/Self-sovereign_identity

Even if the whole idea of cryptocurrencies is found to be a dead-end, the fact that it invigorated the research and development of SSI will change some important things about how we operate online.

There exist prototypes of tech that allows you to prove you are indeed a unique human being online [1], and reveal nothing else about your identity. Most importantly, this tech is not owned or controlled by any FAANG or government, it's an open protocol just like email.

I have listened to a podcast with an expert researcher in AI, and I remember him saying that he predicts some form of cryptographic identity will arise in order to help deal with the bot problem

[1] https://worldcoin.org/ (note, I don't work for them, and have no idea if this will be the tech that finally breaks out, I just think they're the furthest along of any of their competitors)