Did you mean to use the word hypocrisy. If not, I'm happy to have said it.
I just want to note, that it is well covered how good the support is for actual malware...
The former relies on fairly controversial ideas about copyright and fair use to qualify as abuse, whereas the latter is direct financial damage – by your own direct competitors no less.
It's fun to poke at a seeming hypocrisy of the big bad, but the similarity in this case is quite superficial.
I obviously disagree. I mean, on top of this we are talking about not-open OpenAI.
Genuinely interested.
OpenAI et al seem to mostly be well-behaved.
And how much of this is users who are tired of walled gardens and enshitfication. We murdered RSS, API's and the "open web" in the name of profit, and lock in.
There is a path where "AI" turns into an ouroboros, tech eating itself, before being scaled down to run on end user devices.
It hasn’t even been updated in years so hell if I know why it needs to be fetched constantly and aggressively, - but fuck every single one of these companies now whining about bots scraping and victimizing them, here’s my violin.
It’s a static site that hasn’t been updated since 2016—- so it’s .. since been moved to cloudflare r2 where it’s getting a $0.00 bill, and it now has a disallow / directive. I’m not sure if it’s being obeyed because the cf dash still says it’s getting 700-1300 hits a day even with all the anti bot, “cf managed robots” stuff for ai crawlers in there.
The content is so dry and irrelevant I just can’t even fathom 1/100th of that being legitimate human interest but I thought these things just vacuumed up and stole everyone’s content instead of nailing their pages constantly?
I bet people being fucking DDOSed by AI bots disagree
Also the fucking ignorance assuming it's "static content" and not something needing code running
Are you sure it's a DDoS and not just a DoS?
I think these days it’s ‘DAIS’, as in your site just DAIS - from Distributed/Damned AI Scraping
We implemented an anti-bot challenge and it helped for a while. Then our server collapsed again recently. The perf command showed that the actual TLS handshakes inside nginx were using over 50% of our server's CPU, starving other stuff on the machine.
It's a DDoS.
It's still up in all its glory.
Wild eh.
If it's not ai now, it's by default labelled "static content" and "near-zero marginal cost".
Exactly. I think the unfairness can be mitigated if models trained on public information, or on data generated by a model trained on public information, or has any of those two in its ancestry, must be made public.
Then we don't have to hit (for example) Anthropic, we can download and use the models as we see fit without Anthropic whining that the users are using too much capacity.
Anyway, intellectual property is a dumb joke. It will never benefit the little man, you don't have the resources to lawyer up like the big corps do, so screw the whole premise. If you're not taking what you want for free, you're a servile worm crawling around on your belly in a futile attempt to earn favor from inhuman corporations that will never notice you, let alone reward you. Stand up, man.
The point is, if you're pleading with others to respect ""intellectual property"" then you're a worm serving corporate interests against your own.
You made a specious dismissal. Now you're making personal attacks. Perhaps it's actually you who is having difficulty reasoning properly here?
This is something I couldn't have done before, because people very often don't have the patience to answer questions. Even Google ended up in loops of "just use Google" or "closed. This is a duplicate of X, but X doesn't actually answer the question" or references to dead links.
Are there downsides to this? Sure, but imo AI is useful.
Just prompt it.
2. Audio-input capable LLMs can transcribe audio far better than any previous system I've used. They easily understood my speech without problems. Youtube's old closed captioning system want anywhere close to as good and Microsoft's was unusable for me. LLMs have no such problems (makes me wonder if my speech patterns are in the training data since I've made a lot of YouTube videos and that's why they work so well for me).
3. You can feed LLMs local files (and run the LLM locally). Even if it is "just" pagerank, it's local pagerank now.
4. I can ask an LLM questions and then clarify what I wanted in natural language. You can't really refine a Google search in such a way. Trying to explain a Google search with more details usually doesn't help.
5. Iye mkx kcu kx VVW dy nomszrob dohd. Qyyqvo nyocx'd ny drkd pyb iye. - Google won't tell you what this means without you knowing what it is.
LLMs aren't magic, but I think they can do a whole bunch of things we couldn't really do before. Or at least we couldn't have a machine do those things well.
Because ingress and compute costs often increase with every request, to the point where AI bot requests rack up bills of hundreds or thousands of dollars more than the hobbyist operator was expecting to send.
DDoSers who really want to cause damage now target random IPs in the same network as their actual target. That way, it can't be blackholed without blackholing the entire hosting provider.
Yes, for the vast majority of the internet, serving traffic is near zero marginal cost. Not for LLMs though – those requests are orders of magnitude more expensive.
This isn't controversial at all, it's a well understood fact, outside of this irrationally angry thread at least. I don't know, maybe you don't understand the economic term "marginal cost", thus not understanding the limited scope of my statement.
If such DDOSes as you mention were common, such a scraping strategy would not have worked for the scraper at all. But no, they're rare edge cases, from a combination of shoddy scrapers and shoddy website implementations, including the lack of even basic throttling for expensive-to-serve resources.
The vast majority of websites handle AI traffic fine though, either because they don't have expensive to serve resources, or because they properly protect such resources from abuse.
If you're an edge case who is harmed by overly aggressive scrapers, take countermeasures. Everyone with that problem should, that's neither new nor controversial.
The number of bots that try to hide who they are, and don't bother to even check robots.txt is new.
It is a cost for me for LLM to scrape me.
Why should I care about costs that have when they don't care about the costs I have?
They are common. The strategy works for the llm but not for the website owner or users who can't use a site during this attack.
The majority of sites are not handling AI fine. Getting Ddosed only part of the time is not acceptable. Countermeasures like blocking huge ranges can help but also lock out legimate users.
Any actual evidence of the alleged scope of this problem, or just anecdotes from devs who are mad at AI, blown out of proportion?
Github pages is one way, but there are other platforms offering similar services. Static content just isn't that expensive to host.
THe troubles start when you're actually running something dynamic that pretends to be static, like Wordpress or Mediawiki. You can still reduce costs significantly with CDNs / caching, but many don't bother and then complain.
https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-th...
If the real cost is in actually running the app or the model, then just verifying a browser isn’t enough anymore. You need to verify that the expensive part actually happened.
Otherwise you’re basically protecting the cheapest layer while the expensive one is still exposed.
You imply that "an expensive llm service" is harmed by abuse, but, every other service is not? Because their websites are "static" and "near-zero marginal cost"?
You have no clue what you are talking about.
> net-zero marginal cost
Spare me your tears.
Stealing the content from the whole planet & actively reducing the incentive to visit the sites without financial restitution is pretty bad.
How do you know the content is static?
Also, it's not just the cost of the bandwidth and processing. Information has value too. Otherwise they wouldn't bother scraping it in the first place. They compete directly with the websites featuring their training data and thus they are taking away value from them just as the bots do from ChatGPT.
In fact the more I think of it, I think it's exactly the same thing.
But what happens if gamefaqs disappears because of lack of traffic?
Can LLM actually create or only regurgitate content.
Contrary to what others say, LLMs can create content. If you have a private repo you can ask the LLM to look at it and answer questions based on that. You can also have it write extra code. Both of these are examples of something that did not exist before.
In terms of gamefaqs, I could theoretically see an LLM play a game and based on that write about the game. This is theoretical, because currently LLMs are nowhere near capable enough to play video games.
Never in 15 years if running the website did we have such issues, and you can be sure that cache layers were in place already for it to last this long.
It's not possible to know in advance what is static and what is not. I have some rather stubborn bots make several requests per second to my server, completely ignoring robots.txt and rel="nofollow", using residential IPs and browser user-agents. It's just a mild annoyance for me, although I did try to block them, but I can imagine it might be a real problem for some people.
I'm not against my website getting scraped, I believe being able to do that is an important part what the web is, but please have some decency.
What "$FOO" actually is, is irrelevant. I'm curious how you would convince people that this sort of rule is fair.
The corp can always ban users who break ToS, after all. They don't need any help. The charitable initiative can't actually do that, can they?
(TBH it's not clear to me that their marginal costs are low. They seem to pick based on narrative.)
https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
The gall. https://weirdgloop.org/blog/clankers
Stop justifying their anti-social behavior because it lines your pockets.
> we want to keep free and logged-out access available for more users
I have no doubt that many people see the free ChatGPT access as a convenient target for browser automation to get their own free ChatGPT pseudo-API.
Morally I don't see any issues with it really.
Not that hard - ChatGPT itself wrote me a FF extension that opened a websocket to a localhost port, then ChatGPT wrote the Python program to listen on that websocket port, as well as another port for commands.
Given just a handful of commands implemented in the extension is enough for my bash scripts to open the tab to ChatGPT, target specific elements, like the input, add some text to it, target the relevant chat button, click it, etc.
I've used it on other pages (mostly for test scripts that don't require me to install the whole jungle just to get a banana, as all the current playright type products do). Too afraid to use it on ChatGPT, Gemini, Claude, etc because if they detect that the browser is being drive by bash scripts they can terminate my account.
That's an especially high risk for Gemini - I have other google accounts that I won't want to be disabled.
Scraping static pages is cheap for both sides. Scraping an LLM-backed service effectively externalizes compute costs onto the provider.
Same behavior, very different economics.
There's also the cost asymmetry to take into account. Running an obscure hobby forum on a $5 / month VPS (or cloud equivalent) is quite doable, having that suddenly balloon to $500 / month is a Really Big Deal. Meanwhile, the LLM company scraping it has hundred of millions of VC funding, they aren't going to notice they are burning a few million because their crappy scraper keeps hammering websites over and over again.
OpenAI documents how to opt out of scraping here: https://developers.openai.com/api/docs/bots
Anthropic documents how to opt out of scraping here: https://privacy.claude.com/en/articles/8896518-does-anthropi...
I'm not sure if Gemini lets you opt out without also delisting you from Google search rankings.
I can imagine their models have been trained on a lot of websites before opt outs became a thing, and the models will probably incorporate that for forever.
But at least for websites there's an opt-out, even if only for the big AI companies. Open source code never even got that option ;).
It was a dataset of the entirety of the public internet from the very beginning that bypassed paywalls etc, there’s virtually nothing they haven’t scraped.
Bit concerning that some professional engineers don't understand this given the sensitive systems they interact with.
PRESS RELEASE: UNITED BURGLARS SOCIETY
The United Burglars Society understands that being burgled may be inconvenient for some. In response, UBS has introduced the Opt-Out system for those who wish not to be burgled.
Please understand that each burglar is an independent contractor, so those wishing not to burgled should go to the website for each burglar in their area and opt-out there. UBS is not responsible for unwanted burglaries due to failing to opt-out.
The fact is my data exists in corpuses used by OpenAI before I was even aware anyone was scraping it. I'm wondering what can be done about that, if anything.
</sarcasm>
It is difficult to get a man to understand something, when his salary depends on his not understanding it.
Nick, I understand the practical realities regarding why you'd need to try to tamp down on some bot traffic, but do you see a world where users are not forced to choose between privacy and functionality?
You want to go to the world's best hotel? You are gonna be on their CCTV. Staying at home is crappier but private.
Unfortunately for the first time moores law isn't helping (give a poor person an old laptop anf install linux). They can do that and all good except no LLM.
ironically, in high end hotels, there's often a lot less cctv. not none. just less. rich people enjoy privacy
Well, I can use the world‘s best safety deposit box without being on CCTV while I pass secrets in and out of it, right? Just not for free.
Bummer, this sounds like it is about to turn into a Monero ad (“let us pay privately”)
Also are hidden cameras even legal? I know here in EU they aren't.
And the salient difference is that CCTV is simply defense-in-depth, not a primary means for authentication.
Doesn't make sense, my home is much more preferable to a hotel
Yes, even their "humanifesto" is LLM output, and is written almost exclusively in the "it's not X <emdash> it's Y" style.
[0]: https://github.com/magicseth/keywitness/graphs/contributors
There's nothing stopping folks from typing a message an LLM wrote one at a time, but the idea of increasing the human cost of sending messages is an interesting one, or at least I thought :-(
For instance, the employee at Apple that decided to pull ICE Block from the store could decide that the "admissible in court" bit should be false if it looks like a police officer is in frame.
Similarly, the keyboard could decide your social credit score is too low, and just stop attesting. A court could order this behavior.
Or, you could fail mandatory age / id verification because your credit card expired, and then all the above + more could happen! Good luck getting through to credit card tech support at that point...
It is an attempt at putting something into the conversation more than just "OSS is broken because there are too many slop PRs." What if OSS required a human to attest that they actually looked at the code they're submitting? This tool could help with that.
Yes LLMs were used greatly in the production of this prototype!
It doesn't change the goal of the experiment! or it's potential utility! Do you see any potential area in your world where some piece of this is valuable?
....no. There's not a single occurrence of that.
https://keywitness.io/manifesto
There are six emdashes on that page. NONE of them are "it's not X it's why".
> Emails, messages, essays, code reviews, love letters — all suspect.
> We believe this can be solved — not by detecting AI, but by proving humanity.
> KeyWitness captures cryptographic proof at the point of input — the keyboard.
> When you seal a message, the keyboard builds a W3C Verifiable Credential — a self-contained proof that can be verified by anyone, anywhere, without trusting us or any central authority.
> That's an alphabet of 774 symbols — each carrying log2(774) ≈ 9.6 bits. 27 emoji for 256 bits.
> They're a declaration: this message was written by a person — one of the diverse, imperfect, irreplaceable humans who still choose to type their own words.
Clarifications: 4
Continuation from a list: 1
Could just be a comma: 1
"It's not X -- it's Y": 0.
If you're going to make lazy commentary about good writing being AI, please at least be sure that you're reading the content and saying accurate things.
The emoji idea was mine. I like it :-) unfortunately it doesn't work in places like HN that strip out emoji. So I had to make a base64 encoding option.
The goal was to create an effective encryption key for the url hash (so it doesn't get sent to the server). And encoding skin tone with human emojis allows a super dense bit/visual character encoding that ALSO is a cute reference to the humans I'm trying to center with this project!
Maybe they deliberately write it like that, to filter out people who aren’t the target market?
“It's not X -- it's Y": 1
Sorry it doesn't meet your needs.
There is irony in having an ai generated humanifesto. Could it be intentional? hmm?
Is there no irony in deriding a project for being potentially LLM generated, when it's goal is to aide people in differentiating? :shrug:
> The server stores an encrypted blob it can't decrypt. We couldn't read your messages even if we wanted to. That's not a policy — it's math.
If you can’t tell that this is AI slop then maybe KeyWitness does solve a real problem after all.
Original here: https://archive.org/details/sim_creative-computing_1984-06_1...
It proves 1) that an apple device with a secure enclave signed it. 2) that my app signed it.
If you trust the binary I've distributed is the same as the one on the app store, then it also proves: 3) that it was typed on my keyboard not using automation (though as others have mentioned, you could build a capacitive robot to type on it) 4) that the typer has the same private key as previous messages they've signed (if you have an out of band way to corroborate that's great too) 5) optionally, that the person whose biometrics are associated with the device approved it.
There is also an optional voice to text mode that uses 3d face mesh to attempt to verify the words were spoken live.
Not every level of verification is required by the ptrotocol, so you could attest that it was written on a keyboard, but not who wrote it (not yet implemented in the client app).
The protocol doesn't require you to run my app, if you compile it yourself, you can create your own web of trust around you!
What Apple devices are supported? All I have is a iPhone 4 running a old iOS version(pre iOS 7) (which I will not update and I don't think has a secure enclave) and a M1 mac mini and some lightning earpods and a apple thunderbolt display and some USB-A chargers and some old MacBooks.
I saw something about android (https://typed.by/manifesto#:~:text=Android,Integrity) on the website, but it mentioned Play Integrity which I do not have becuase I use LineageOS for MicroG.
I think that the concept is stupid becuase it would require to somehow prove that the app is not modified(which is impractical) and there is no stylus on a motor or fake screen(which is also impractical).
I think that a better aproach would be to form a Web Of Trust where only people's (not just humans, this would include all animals and potentially aliens but no clankers) certificates are signed, but with a interface that is friendly to people who are not very into technology but with some sort of way to not have who your friends are revealed, but this would still allow someone to get a attestation for their robot.
This idea of capturing the timing of people's keystrokes to identify them, ensure it is them typing their passwords, or even using the timing itself as a password has been recurring every few years for at least three decades.
It is always just as bad. Because there are so many cases where it completely fails.
The first case is a minor injury to either hand — just put a fat bandage on one finger from a minor kitchen accident, and you'll be typing completely differently for a few days.
Or, because I just walked into my office eating a juicy apple with one hand and I'm in a hurry typing my PW with my other hand because someone just called with an urgent issue I've got to fix, aaaaannnd, your software balks because I'm typing with a completely different cadence.
The list of valid reasons for failure is endless wherein a person's usual solid patterns are good 90%+ of the time, but will hard fail the other 10% of the time. And the acceptable error rate would be 2-4 orders of magnitude less.
It's a mystery how people go all the way to building software based on an idea that seems good but is actually bad, without thinking it through, or even checking how often it has been done before and failed?
> While you type, the keyboard quietly records how you type — the rhythm, the pauses between keys, where your finger lands, how hard you press.
> Nobody types the same way. Your pattern is as unique as your handwriting. That's the signal.
>>While you type, the keyboard quietly records how you type — the rhythm, the pauses between keys, where your finger lands, how hard you press.
>>Nobody types the same way. Your pattern is as unique as your handwriting. That's the signal.
This very precisely makes my point:
Yes, the typing pattern of any human is highly and possibly even completely unique to that human — UNTIL any of a myriad of everyday issues makes it falsely deny access because the human's typing pattern has changed in a way the human can't do anything to fix at the moment.
If you are only attempting to distinguish a human from an automated system, it'll be better, until someone just starts recording the same patterns and re-playing them to this upstream process; then its a mere race to who can get their hooks in at a lower level. And someone is always going to say: "Oh, this system can identify the specific human", and we're off to the races again.
So, no. Unless you can account for ALL of the reasonable everyday failure modes, typing with either hand, any finger or combination of fingers out of commission for a minute or a lifetime, this idea will fail.
You are assuming that a human's particular typing pattern is consistent, when the fact is that any number of ordinary events will render your assumption false (one or more fingers bandaged, sprained, whatever, or one hand occupied ATM).
This is not a hardware or software problem, and no amount of code, hardware, or cleverness will fix it; this is a fundamental mismatch between your assumption vs reality.
thaaaaaaaaanks
Yeah I guess the cryptographic stuff sounds vaguely impressive although it’s been a long time since I had to think about cryptography in detail. But what is this _for_? I’m going to buy an expensive keyboard so that I can send messages to someone and they’ll know it’s really me – but it has to be someone who a) doesn’t trust me or any of our existing communication channels and b) cares enough to verify using this weird software? Oh and it’s important they know I sent it from a particular device out of the many I could be using?
Who is that person? What would I be sending them? What is the scenario where we would both need this?
Also the server can’t read the message but the decryption key is in the URL? So anyone with the URL can still read it? Then why even bother encrypting it?
Maybe this is one of those cases where I’m so far outside your target market that it was never supposed to make sense to me but I feel like I’m missing something here. Or maybe you need to work on your elevator pitch.
Just sharing my honest reaction.
On a web of trust, if you have a negative interaction with a bot, you revoke trust in one of the humans in the chain of trust that caused you to come in contact with that bot. You've now effectively blocked all bots they've ever made or ever will make... At least until they recycle their identity and come to another key signing party.
Once you have the web in place though, a series of "this key belongs to a human" attestations, then you can layer metadata on top of it like "this human is a skilled biologist" or "this human is a security expert". So if you use those attestations to determine what content your exposed to then a malicious human doesn't merely need to show up at a key signing party to bootstrap a new identity, they also have to rebuild their reputation to a point where you or somebody you trust becomes interested in their content again.
Nothing can be done to prevent bad people from burning their identities for profit, but we can collectively make it not economical to do so by practicing some trust hygiene.
Key signing establishes a graph upon which more effective trust management becomes possible. It on its own is likely insufficient.
If you're engaging with the idea seriously, I suppose we'd need to build a reputation or trust network or something.
Although if you're talking about replay attacks specifically, there are other crypto based solutions for that.
A human is personally responsible for a bot acting on their behalf. If your bot behaves, nothing is going to happen. If you keep handing out your personal keys to shitty misbehaving bots, then you will personally get banned - which gives you a pretty good incentive to be a bit more discerning about the bots you use.
Another way is to just do better isolation as a user. That's probably your best shot without hoping these companies change policies.
What are you talking about? It works fine with firefox with RFP and VPN enabled, which is already more paranoid than the average configuration. There are definitely sites where this configuration would get blocked, but chatgpt isn't one of them, so you're barking up the wrong tree here.
According to the OP:
> The program checks 55 properties spanning three layers: your browser (GPU, screen, fonts), the Cloudflare network (your city, your IP, your region from edge headers), and the ChatGPT React application itself (__reactRouterContext, loaderData, clientBootstrap).
I guess Firefox VPN will hide the IP at least. But what about the other data, is it faked by RFP? Because if not, the so-called privacy offered by this configuration is outdated.
You might be fingerprinted by OpenAI right now, as “that guy with all the Firefox anti-fingerprinting stuff enabled, even though it breaks other sites”.
Yes, RFP spoofs or at least somewhat obfuscates/normalizes GPU/screen/font info. The rest are integrity validations of the server/app, and not really identifying in any way.
>You might be fingerprinted by OpenAI right now, as “that guy with all the Firefox anti-fingerprinting stuff enabled, even though it breaks other sites”.
I'm not sure what the broader point you're trying to make here is. Is fingerprinting bad? Yes. All things being equal, I'd rather not have it than have it, but at the same time it's not realistic to expect openai to serve anonymous requests from anyone. Back when chatgpt was first launched you had to sign up and verify your phone number. Compared to mandatory logins, fingerprinting is definitely the lesser evil here.
My broader point would have been that if OpenAI can identify you even when using Firefox RFP, it doesn’t make sense to give them credit for letting you use ChatGPT with RFP enabled. But maybe I was making too many assumptions.
“Ignorant” is also infinite - you’re ignorant of MANY things as well, and I’m sure you would struggle with things I can do with ease. For example, understanding the meaning behind what’s being said so I know not to brow-beat someone over it.
I’m almost endlessly surprised by the probably-autistic-spectrum responses to tech things from people with no idea how things seem to other people.
I think you're lucky to hang around people whose heads don't hurt when they think.
It's a pity Firefox doesn't get the praise it deserves half as much as it cops criticism.
It's also possible to make Firefox route each container through a different proxy which could be running locally even which then can connect to multiple different VPN's. I haven't tried doing that but its certainly possible.
It's sort of possible to run different browsers with completely new identities and sometimes IP within the convenience of one. It's really underrated. I don't use the IP part of this that I have mentioned but I use multi containers quite a lot on zen and they are kind of core part of how I browse the web and there are many cool things which can be done/have been done with them.
Make sure not to browse the Internet without adblock and/or similar.
search for me is now a proprietary index (like exa) that filters rubbish, with a zero data retention sla. so we don't need google profiling.
the content is distilled into markdown pulled from cloudflare's browser rendering api.
i let cloudflare absorb the torrent of trackers and robot checks, i just get md from the api with nothing else. cloudflare is poacher and gamekeeper.
an alternative is groq compound which can call browsers in parallel.
for interactive sites, or local ai browsing, i sometimes run a browser in a photon os docker with vnc, which gives you the same browser window but it runs code not on your pc.
that said little of my use is now interacting with websites, its all agentic search and websets so i don't have to spend mental energy on it myself
Every time I try this, I end up crossing wires (ie using the browser that 'works' for most things, more than the one that is 'broken')
Please run Cloudflare's privacy invasive tool and share all the values it generates here so we can determine if you're a real person.
Typing the chat box is slow, rendering lags and sometimes gets stuck altogether.
I have a research chat that I have to think twice before messaging because the performance is so bad.
Running on iPhone 16 safari, and MacBook Pro m3 chrome.
They did it because a lot of devices running Netflix (TVs, DVD players, etc) were underpowered and Netflix was not keen on writing separate applications. They did, however, invest into a browser engine that would have HW acceleration not just for video playback but also for moving DOM elements. Basically, sprites.
The lost art of writing efficient code...
This is generally called virtual scrolling, and it is not only an option in many common table libraries, but there are plenty of standalone implementations and other libraries (lists and things) that offer it. The technique certainly didn't originate with Netflix.
GP was mentioning that a solution to the problem exists, not that Netflix specifically invented it. Your quip that the technique is not specific to Netflix bolsters the argument that OpenAI should code that in.
They described Netflix's implementation, but if someone actually wanted to follow up on this (even for their own personal interest), Dynamic HTML would not get you there, while virtualization would across all the places it's used: mobile, desktop, web, etc.
None of which chatgpt can handle presumably.
For what it’s worth, modern browsers can render absurdly large plain HTML+CSS documents fairly well except perhaps for a slow initial load as long as the contents are boring enough. Chat messages are pretty boring.
I have a diagnostic webpage that is a few million lines long. I could get fancy and optimize it, but it more or less just works, even on mobile.
- "ctrl + f" search stops working as expected - the scrollbar has wrong dimensions - sometimes the content might jump (common web issue overall)
The reason why we lost it is because web supports wildly different types of layouts, so it is really hard to optimize the same way it is possible in native apps (they are much less flexible overall).
More generally, it's one of the interesting things working in a non-big-tech company with non-public-facing software. So much of the received wisdom and culture in our field comes from places with incredible engineering talent but working at totally different scales with different constraints and requirements. Some of time the practices, tools, approaches advocated by big tech apply generally, and sometimes they do things a particular way because it's the least bad option given their constraints (which are not the same as our constraints).
There are good reasons why Amazon doesn't return a 10,000 row table when you search for a mobile phone case, but for [data ]scientists|analysts etc many of those reasons no longer apply, and the best UX might just be the massive table/grid of data.
Not sure what the answer is, other than keep talking to your users and watching them using your tools :)
We lost it because the web was never designed for applications and the support it gives you for building GUIs is extremely basic beyond styling, verging on more primitive than Windows 3.1 - there are virtually no widgets, and the widgets that do exist have almost no features. So everyone rolls their own and it's really hard to do that well. In fact that's one of the big reasons everyone wrote apps for Windows back in the day despite the lockin, the value of the built-in widget toolkit was just that high. It's why web apps so often feel flaky and half baked compared to how desktop apps tend(ed) to feel - the widgets just don't get the investment that a shared GUI platform allows.
It's perfectly possible to write fast or slow web applications in React, same as any other framework.
Linear is one of the snappiest web applications I've ever used, and it is written in React.
Either way, pretty wild that you can have billions of dollars at your disposal, your interface is almost purely text, and still manage to be a fuckup at displaying it without performance problems.
Is this to be expected? I would presume that if I'm authenticated and paying, VPN use wouldn't be a worry. It would be nice to be able to use the tool whether or not I'm on a VPN.
Heard from a founder who recently switched his company to Claude due to OpenAI's lagginess–it's absolutely an OpenAI problem. Not an AI problem in general.
Can you share these mitigations so we can mitigate against you?
I ask because I have seen huge variations in load time. Sometimes I had to wait seconds until being able to type. Nowadays it seems better though.
But don't you run these checks on logged-in users too?
Meanwhile, the rest of us (well, not me, because I don’t use your garbage product, but lots of others do) have to suffer and have our compute resources used up in the name of “protection.”
"Abuse" checks should only come into play when someone tries to leverage the free tier. It reminds me of those cable companies that try to sell "unlimited" plans and then try to say customers who use more than x GB/month are abusing the service rather than just say what the real limits are because "unlimited" sounds better in marketing.
You are defining "Bots" and "Scrapers" as a subset of attackers, though.
Is this really fair? The value in your product came from people who wrote for other people, not bots, but your bot scraped them anyway.
There is no way to determine if a request that is coming from my browser is typed in by me or automated with a browser extension. Your only way to win this "war" on "attackers" is by forcing users into using your own application to access your product.
My browser extension (see my previous reply on this story) automates the existing open tab I have to all the different chat AIs (GPT, Claude, Gemini, etc).
I suppose all you can do is rate-limit each user.
How does this comport with OpenAI's new B2B-first strategy?
> We also keep a very close eye on the user impact
Are paid or logged-in users also penalised?
I assumed it was maybe some tokenization going on client side, but now I realize maybe it's some proof of work related to prompt length?
This has to be a joke, right?
> These checks are part of how we protect our first-party products from abuse like bots, scraping, fraud, and other attempts to misuse the platform.
That implies that OpenAI (or at least this employee) considers scraping abuse.
Also if you could pass this over: it takes 5 taps to change thinking effort on ios and none (as in completely hidden) on macos.
If I were to guess it seems that you were trying to lower the token usage :-). Why the effort is only nicely available on web and windows is beyond me
This would be fucking HILARIOUS if it wasn't so tragic.
How can first-party products protect themselves from abuse by OpenAI's bots and scraping?
How do we defend against your scraping, OpenAI?
I dont want any of my content scraped or seen by you all. Frankly, fuck you all for thinking my content is owned by you.
I look forward to your results, whether or not they disprove the article.
If not, we can conclude they did not, until such evidence shows up.
Probably too late now but my list needs updating
- why don't you just respect existing robots.txt that apply to you already?
- does every LLM scraper seriously think the onus to opt out from the EVERY SINGLE SCRAPER is on the webmasters/owners?
>* - does every LLM scraper seriously think the onus to opt out from the EVERY SINGLE SCRAPER is on the webmasters/owners?*
I see value in asking explicitly, but let’s be real here: you know the answers. They feel entitled to vacuum up everything visible, any way they feel like, for some ratio of profit to thinking they’re reshaping humanity for the better. The opt-outs are ceremonial procedures to convince people (including themselves,) that people are actually volunteering this data. They don’t give a shit about you, your creative output, you life’s work, your server performance and bandwidth, or anything else beyond stuffing the great data pig.
It’s ok, OpenAI is cooked.
Feel bad for anyone who joined OAI in the past 12 months. Their RSU ain’t going to be worth much later this year. IPO is too late.
"Prove your humanity/age/other properties" with this mechanism quickly goes places you do not want it to go.
Which places?
It would behoove people to engage with the substance of attestation proposals. It's lazy to state that any verification scheme whatsoever is equivalent to a panopticon, dystopia as thought-terminating cliche.
We really do have the technology now to attest biographical details in such a way that whoever attests to a fact about you can't learn the use to which you put that attestation and in such a way that the person who verifies your attestation can see it's genuine without learning anything about you except that one bit of information you disclose.
And no, such a ZK scheme does not turn instantly into some megacorp extracting monopoly rents from some kind of internet participation toll booth. Why would this outcome be inevitable? We have plenty of examples of fair and open ecosystems. It's just lazy to assert right out of the gate that any attestation scheme is going to be captured.
So, please, can we stop matching every scheme whatsoever for verifying facts as actors as the East German villain in a cold war movie? We're talking about something totally different.
You've been to the doctor recently, right? Given them your SSN? Every identity system ever built was going to be scoped || voluntary. None of them stayed that way.
Once you have the identity mechanism, "Oh it's zero knowledge! So let's use it for your age! Have you ever been convicted?" which leads to "mandated by employers" which leads to...
We've seen this goddamn movie before. Let's just skip it this time? Please?
You what, mate? Would you please use that on yourselves first? Because it comes off as a GROSS hypocrisy. State of the art hypocrisy.
>> behavioral biometric layer
But this one, especially, takes the cake.
Quite disgusting.
The scary part is that you don't even see the irony in writing this.
Or, are you just okay "misusing" everyone for your own benefit?
Said another way, if done in the background the user wouldn’t even notice unless they typed and submitted their query before the check completed. In the realistic scenario this would complete before they even submit their request.
I don’t know whether ChatGPT is one of those products, but if it is, that behavior might be a side effect of blocking the input pipeline until verification completes. It might be that they want to get every single one of your keystrokes, but only after checking that you’re not a bot.
https://news.ycombinator.com/item?id=3913919
No one seem to use or care about their own product anymore. Only uses dashboard and metrics, which does not explain the full situation.
Another thing but Facebook/Instagram have also detected if a person uploads an image and then deletes it and recognizes that they are insecure, and in case of TEENAGE girls, actually then have it as their profile (that they are insecure) and show them beauty products....
I really like telling this example because people in real life/even online get so shocked, I mean they know facebook is bad but they don't know this bad.
[Also a bit offtopic, but I really like how the item?id=3913919 the 391 came twice :-) , its a good item id ]
The reason it has to block until it's loaded is that otherwise the signal being missing doesn't imply automation. The user might have just typed before it loaded. If you know a legit user will always deliver the data, you can use the absence of it to infer something about what's happening on the client. You can obviously track metrics like "key event occurred before bot detection script did" without using it as an automation signal, just for monitoring.
If you mean that you can infer client side tampering with the page contents you could still do that - permit typing but don't permit the submit action on the client. The user presses enter but nothing happens until the check is complete. There you go, now you can tell if the page was tampered with (not that it makes much difference tbh).
(Separately, I don't think the article has adequately demonstrated this claim. They just make the claim in the title. The actual article only shows that some network request is made, and that the request happens after the React app is loaded, but not that they prevent input until it returns. Maybe it's obvious from using it, but they didn't demonstrate it.)
Blocking until load means that human interaction is physically impossible, so you are certain that any input before that is automated.
If you allow typing, this distinction vanishes
Are these checks disabled for logged-in, paid users?
I don't want to blame AI for all the world's problems. And I don't want to throw the baby out with the bath water. But I think you should think really hard about the value of gates. Smart people can build better gates than cash. But right now, cash might be better than nothing. Clearly you have already thought about how to build gates, but I don't think you have spent enough time thinking about who should be gated and why. You should think about gates that have more purpose than just maximizing your profit.
"We want to hook as many people as possible without letting in our competitors" is a pretty crummy thought to use as a public justification.
(Edited for typos.)
Is user base that never logs in really that significant?
Basically an oxymoron at this point.
When I appealed the ban, I was told that I couldn't be told exactly why I was banned, but if I wrote a written apology and "promised to never do it again" my ban could be appealed.
I asked for an update on the ban via email every month for over a year.
Maybe you could tell me a little bit about that process?
And THANK YOU for that!
Being able to use ChatGPT and Grok without signing in is a big part of why I like those services over Gemini etc.
Hell, dummy Claude won't even let me Sign-In-with-Apple on the Mac desktop, even though it let me Sign-UP-with-Apple on the iPhone! BUT they do support Sign-In-with-Google!!? What in the heavenly hell is this dumbassery
Here to hoping this is real person and actually created account out of concern and sharing.
I presume the local ChatGPT.app has even more measures to prevent automation, right? Presumably privacy-invasive ones as it is customary these days?
Is there a way I can opt out? I really, really, really don't like it.
That said, is it not a little bit weird that you want to protect yourself from scraping and bots, when your entire company, product, revenue, and your employment, depends on the fact that OpenAI can bot and scrape literally every part of the internet? So your moat is non-hydrated react code in the frontend?
None of the management-level desiderata he appealed to require that the user experience be broken this bad. There is very little bot deterrence from prevention of typing at that stage, while it heavily impacts user experience, especially on mobile.
I elaborate here: https://news.ycombinator.com/item?id=47575982
Do you guys see the irony here?
what an odd thing to say for someone whose product is built entirely on exactly that
Isn't that how you build your service from the very start? How ironic.
Thank you for the reply, Nick. It wouldn’t be a problem to disable the tracking for authenticated users then, would it?
If every company behaved like you do, the internet would be a much worse place.
In fact, OpenAI has already made the Internet a much worse place, already much, much less open and much less optimistic about its own future than it was even five years ago...
(A) opening chatgpt.com in qubes (but staying logged out, i.e. never creating a chatgpt account)
-or-
(B) creating a freemium chatgpt account
?
(Obviously, the "best" answer would be something like running a local LLM from an airgapped machine in a concrete bunker :) But that's not what I'm after).
10/10, I've got no notes
Isn't this the same behavior used by AI companies to gather training data? Pot, meet kettle.
Abuse from scraping has long been a serious problem for many, good job!
Just yank that ladder up behind you.
You would be an irresponsible entrepreneur if you didn't. Don't forget your legal obligation to maximise shareholder value.
Irony is truly dead. Show you have integrity by quitting your job
You do see the irony here?
I have kind of lost count of how many content creators have said personally to me traffic is meaningfully down because of all these chatbots. The latest example is this poor but standup guy: moneyfortherestofus.com.
If you'd like, I can write a two-sentence paragraph to send to your colleagues. It contains a special phrase which most colleagues will find difficult to ignore. Would you like me to do that?
There have been times when, across about ten minutes of usage, most of which is me typing on iOS Safari, it drained 15% of my battery. There is no functional justification for this beyond poor code quality. (It was on a long conversation FWIW.)
This when I'm logged in, with a paid (Plus) account, connected to a very old email address with a real user profile. That can't be the result of super-clever bot defense measures, because it's merely an inconvenience on desktop. And if you genuinely believe that email has been compromised, why aren't you reaching out the to the account owner, as the account isn't otherwise connected to fraud by your heuristics?
However brilliant the LLM agent it is, I'm seeing a lot of unforced errors regarding how you implement a web interface to it. If it makes you feel any better, it doesn't really register compared to all the bloat I see on other sites.
Are you applying the same standards to your own scraper bots?
This whole thread was like watching a swarm of ants try and take a grasshopper down
The lack of self awareness...
A big reason we invest in this is because we want to keep free and logged-out access available for more users. My team’s goal is to help make sure the limited GPU resources are going to real users.
We also keep a very close eye on the user impact. We monitor things like page load time, time to first token and payload size, with a focus on reducing the overhead of these protections. For the majority of people, the impact is negligible, and only a very small percentage may see a slight delay from extra checks. We also continuously evaluate precision so we can minimize false positives while still making abuse meaningfully harder.