News for hackers. Stuff that matters.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by londons_explore on Tuesday July 30, 2024 @12:35PM (#41111073)
> AI can already solve captchas, so the arms race for bot protection is pretty much lost.

Require login, then verify the user account is associated with an email address at least 10 yrs old. Pretty much eliminates bots. Eliminates a few real users too, but not many.

Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by _heimdall on Tuesday July 30, 2024 @01:37PM (#41111925)
I must be an outlier here, but I don't keep email addresses that long. After a couple years they're on too many spam lists. I'll wind those addresses down and use them for a couple years only for short interactions that I expect spam from, and ultimately close then down completely the next cycle.

At best any email I have is 4 or 5 years old.

Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by account42 on Wednesday July 31, 2024 @06:29AM (#41117879)
You are definitely an outlier in that you abandon email addresses deliberately. But many people do not have an old address simply because they lost access to their previous ones for one of many possible reasons, the most common one being that it was provided with a business relationship (e.g. ISP contract) that no longer exists.

That's before even getting into how you'd possibly verify email adress age, especially without preventing self-hosting.

Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by londons_explore on Wednesday July 31, 2024 @09:17AM (#41118935)
There are commercial services to verify email address age: https://www.ipqualityscore.com/email-age-checker

They generally look at data leaks and partner with big companies to see when that email address first signed up to any online service.

Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by account42 on Wednesday July 31, 2024 @10:38AM (#41119581)
That doesn't seem remotely compatible with modern privacy laws like the GDPR. And it certainly ads even more false negatives of people locked out because they didn't have their email leaked long enough ago.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by londons_explore on Wednesday July 31, 2024 @06:44PM (#41124315)
GDPR has a massive carve-out for fraud prevention...
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by account42 on Thursday August 01, 2024 @03:33AM (#41126939)
Which doesn't include collecting information about your email and then handing it off to random third parties just because they want to use it for that.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by mcherm on Tuesday July 30, 2024 @01:28PM (#41111814)
This is about OpenStreetMap, so you are proposing that my minor daughter not be allowed to read a map?
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by tempfile on Tuesday July 30, 2024 @12:40PM (#41111134)
> require login

this is not a solution if you want a public internet (and sites that don't care about the public internet already don't have a problem)

Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by londons_explore on Tuesday July 30, 2024 @12:44PM (#41111187)
for read-only content, I just stick it behind a cache and let the bots go wild.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by dartos on Tuesday July 30, 2024 @12:55PM (#41111351)
That’s just passing the buck.

Someone still needs to pay for that traffic. If it gets too much for cloud flare or whoever, you’re gonna get the bill.

Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by account42 on Wednesday July 31, 2024 @06:31AM (#41117893)
Traffic is not all that expensive though if you are not using a cloud provider where that's how they squeeze captured customers.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by jgalt212 on Tuesday July 30, 2024 @01:20PM (#41111714)
You can't cache this stuff for bot consumption. Humans only want to see the popular stuff. Bots download everything. The size of your cache then equals the size of your content database.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by account42 on Wednesday July 31, 2024 @06:33AM (#41117900)
But you can still make sure that you save the data in a form where generating the served webpage takes the least amount of time. For most websites this means saving the HTML - in a giant cache or with a more deliberate pre-generation setup.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by jgalt212 on Wednesday July 31, 2024 @09:24AM (#41118978)
The data structure to html conversion takes milliseconds. That's distinction without a difference.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by account42 on Wednesday July 31, 2024 @10:36AM (#41119552)
Clearly not the case with most websites. And "milliseconds" are already a huge amount of time. Video games simulate huge worlds and render complex 3D graphics within 16ms or even much less with the >60 framerates that are expected these days.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by jgalt212 on Wednesday July 31, 2024 @06:27PM (#41124186)
You've gotten off topic here. The giant cache you speak of approaches the size of the content database when designing for the long tail of bots. A giant cache is non economic and thus not a solution unless you're an AWS salesman.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by account42 on Thursday August 01, 2024 @03:32AM (#41126935)
Not at all. You can either design your website so that pages can be retrieved in sub-millisecond time (and that doesn't have to mean throwing money at cloud providers) or you can cry about bots.
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by londons_explore on Thursday August 01, 2024 @05:58AM (#41127559)
Yes, the cache ends up being bigger than the content database, but for text content that's typically not a problem. The human effort to type some text always hugely exceeds the cost of a few kilobytes of flash to store what they typed in a ready-to-serve form.

The generation process of taking the raw text and assembling the page around it is typically rather expensive for most CMS systems. Sure - it isn't theoretically expensive, but unless you want to engineer a CMS from scratch most people just pick one off the shelf and then end up having to pay the CPU time overhead of wordpress etc.

Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by tempfile on Tuesday July 30, 2024 @12:54PM (#41111343)
I presume OSM has already considered this and ruled it out (probably because the map should be dynamic)
Re:Dear AI Companies, instead of scraping OpenStreetMap, how... by azemetre on Wednesday July 31, 2024 @09:54AM (#41119189)
How does one find the age of an email account?