Most of the stuff is implemented in Go because I believe that you've only understood the problem if you can automate it in code, as you can see in my writeups repo [1] :D

[1] https://github.com/cookiengineer/writeups

But in Germany, the section of the copyright law concerning data mining specifically says that scraping websites is legal unless the owner of the website objects in a machine-readable form. robots.txt very clearly fulfils this standard. If any bot owner complains that you labelled them as a bot as outlined above, they would be admitting that they willfully ignored your robots.txt, and that appears to me to make them civilly liable for copyright infringement.

Source: https://www.gesetze-im-internet.de/urhg/__44b.html

I also had a look if these actions would count as self-defense against computer espionage under the criminal code, but the relevant section only outlaws gaining access to data not intended for oneself which is "specifically protected against unauthorized access". I don't think this claim will fly for a public website.

Source: https://www.gesetze-im-internet.de/stgb/__202a.html

https://www.bakerdonelson.com/return-of-the-rocket-docket-ne...

You are going to hit a lot more false positives with this one than actual bots

What in the flying **. Is this a common thing?

For example: https://bright-sdk.com/

> Bright SDK is approved by Apple, Amazon, LG, Huawei, Samsung app stores, and is whitelisted by top Antivirus companies.

humans are a truly horrible species and this kind of thing is a great example of why I believe that.

So is that such a bad thing? If OP is going to use this to provide data about bots, blocking mass amounts of the internet could actually be a terrific example of how many people are at least tangentially connected to bots.

And "a lot" of false positives?? Recall, robots.txt is set to ignore this, so only malicious web scanners will hit it.

Personally I sometimes do a quick request to /wp-admin to check if a site is WordPress, so I guess that has a nonzero chance of affecting me. And when I mirror a website I almost always ignore robots.txt (I'm not a robot and I do it for myself). And when I randomly open robots.txt and see a weird url I often visit it. And these are just my quirks. Not a problem for a fun website, but please don't ban a whole IP - or even whole ISP - because of this.

So that is a balance between a bad actor and even "stop it" blocks, and auto expire means transitory denial.

add a captcha by limiting IP requests or return 429 to rate limit by IP. Using popular solutions like cloudflare could help reduce the load. Restrict by country. Alternatively, put in a login page which only solves the captcha and issues a session.

of course people look at this. it's not an everyday thing for the prototypical web user, but some of us look at those a lot.

I... I do... sometimes. Mostly curiosity when the thought randomly pops on my head. I mean, I know I might be flagged by the website as someone weird/unusual/suspicious, but sometimes I do it anyway.

Btw, do you know if there's any easter egg on Hacker News' own robots.txt? Because there might be.

Also, how do gzip bombs works, does it automatically extract to the 20gb or the bot has to initiate the extraction?

LLMs that are implemented in a manner like this to offer web scraping capabilities usually try to replace web scraper interaction with the website in a programmable manner. There's bunch of different wordings of prompts, of course, depending on the service. But the idea is that you as a being-scraped-to-death server learn to know what people are scraping your website for in regards to the keywords. This way you at least learn something about the reason why you are being scraped, and can manage/adapt accordingly on your website's structure and sitemap.

> how do gzip bombs works, does it automatically extract to the 20gb or the bot has to initiate the extraction?

The point behind it is that it's unlikely that script kiddies wrote their own HTTP parser that detects gzip bombs, and are reusing a tech stack or library that's made for the task at hand, e.g. python's libsoup to parse content, or go's net/http, or php's curl bindings etc.

A nested gzip bomb has the effect that it targets both the client and the proxy in between, whereas the proxy (targeted via Transfer-Encoding) has to unpack around ~2ish GB of memory until it can process the request, and parse the content to serve it to its client. The client (targeted via Content-Encoding) has to unpack ~20GB of gzip into memory before it can process the content, realizing that it's basically only null bytes.

The idea is that a script kiddie's scraper script won't account for this, and in the process DDoS the proxy, which in return will block the client for violations of ToS of that web scraping / residential IP range provider.

The awesome part behind gzip is that the size of the final container / gzip bomb is varying, meaning that the null bytes length can just be increased by say, 10GB + 1 byte, for example, and make it undetectable again. In my case I have just 100 different ~100kB files laying around on the filesystem that I serve in a randomized manner and that I serve directly from filesystem cache to not need CPU time for the generation.

You can actually go further and use Transfer-Encoding: chunked in other languages that allow parallelization via processes, goroutines or threads, and have nested nested nested gzip bombs with various byte sizes so they're undetectable until concated together on the other side :)

The transfer-encoding means that the proxy has to decompress a 200kb request into a 2Gb response to the client, and the client will receive a 2Gb file that will expand to 20Gb.

Small VM gets knocked offline and the proxy gets grumpy with the client for large file transfers.

Yes, correct. A gzip bomb inside a gzip bomb that contains only null bytes, because it's much larger on the client side when unpacked.

A "normal" gzip bomb that would only leverage "Content-Encoding: gzip" or only "Transfer-Encoding: gzip" isn't really good as for compression ratio, because the sent file is in the megabytes range (I think it was around 4MBish when I tried with gzip -9?). I don't wanna send megabytes in response to clients, because that would be a potential DoS.

edit: also note the sibling comment here: https://news.ycombinator.com/item?id=41923635#41936586

But that's a good point; I'd not considered that if you compress the HTTP response it'll almost certainly get automatically extracted which "detonates" the (g)zip bomb.

Python package requests

Whatever is the default these days in C#

Honestly, I have never used a modern HTTP client library that does not automatically decompress.

I guess libCurl might be a case where you need to add an option to force decompress.

Also, the best starter is charmender

Also, the best one is clearly Squirtle.

Venusaur beats Charizard any time in the first edition. Given you use the correct attacks, of course, which are:

Toxin and Leech Seed, Growth and Razorleaf :)

Charizard Fire Blast vs. Venusaur: 236-278 (65 - 76.5%) -- guaranteed 2HKO

Venusaur Razor Leaf vs. Charizard on a critical hit: 31-37 (8.6 - 10.3%) -- possibly the worst move ever

Worth noting that toxic gets downgraded to regular poison after switching out, which does 1/16th of damage every turn, and blocks other status like paralysis or sleep.

Leech seed would probably not do much, and is lost upon switching out.

Growth is ok, but since venusaur is slower, you only reduce damage by 33% on your second hit, not great.

Sleep powder would give you a chance, if it hits (75%) and if zard doesn't crit. But it would also waste your team's sleep if you are playing with sleep clause.

Q.E.D Charizard>Venusaur

That said it's not a transitive superiority, as we know Blastoise> Charizard, and Venusaur>Blastoise.

That said Charizard beats venusaur harder than venusaur beats blastoise, as the turtle has access to ice beam and blizzard, so it can at least deal non-stab super effective damage back before getting wrecked by razor leaf. And as a bonus gets a chance to crit or freeze.

Blastoise Blizzard vs. Venusaur: 158-186 (43.5 - 51.2%) -- 5.4% chance to 2HKO

Venusaur Razor Leaf vs. Blastoise on a critical hit: 256-302 (70.9 - 83.6%) -- guaranteed 2HKO

Finally while Zard doesn't have SE coverage moves against Blastois it is faster, so it can get 2 moves in before dying, either 2 slashes or a swords dance and a hyper beam, which deal the same damage but would leave zard roided against the next mon. (Or a slash +beam if you roll for a crit.)

Blastoise Surf vs. Charizard: 205-242 (57.1 - 67.4%) -- guaranteed 2HKO

+2 (swords dance) Charizard Hyper Beam vs. Blastoise: 194-228 (53.7 - 63.1%) -- guaranteed 2HKO

So while mon superiority by matchups are not transitive, that is, we cannot order starter pokemon from best to worst. We can definitely order their matchups from best to worst:

1) Zard vs Venusaur 2) Blastoise vs Zard 3) Venusaur vs Blastoise

With 2 and 3 being closer than 1 and 2.

The matchups themselves are transitive, and as such, in terms of starter matchups Charizard is the best, and Venusaur is the worst

QED

> Leech seed would probably not do much

No, that is incorrect and exactly my point of the argument (and therefore the reason why you would've failed my website's captcha check).

The first editions we're talking about had a bug which meant that if Toxin is used first, then Leech Seed does an additional damage per round. And it's ever increasing because the multiplier of Toxin was applied to Leech Seed, which no defensive attack can counteract except Rest (and a Relaxo that had enough HP to survive a second attempt of Toxin+Leech Seed stacking).

Note that leech seed will literally in this combination also restore more than 100% HP of Venusaur even when the targeted Pokemon has only 1hp left, each round, making Venusaur almost unbeatable without a Pokemon that has Psybeam and both a high special attack value and a high speed value like Mewtwo, Mew, or Gengar.

Detailed explanation of how the function behind it works:

https://bulbapedia.bulbagarden.net/wiki/Toxic_%28move%29

for sake of reference: (N * max(1, int(0.0625*MaxHP)))

Note that this was fixed specifically in later editions of Pokemon Stadium and second generation editions, wherein Toxic / bad poisoning and leech seed will reset the counter to 0 each round, whereas the counter was ever increasing in the first generation.

- - - - - - - -

It's kinda funny that we have arrived at a generational knowledge transfer problem, in Pokemon. Google's results without specifically searching for "bug" or "glitch" will happily generate bullshit answers about this topic, always referring to newer editions and their online documentation.

Some results that were talking about this bug, just search for "pokemon red leech seed toxic bug":

- https://gamefaqs.gamespot.com/boards/907714-pokemon-blue-ver...

- https://www.elitefourum.com/t/pokemon-red-blue-yellow-glitch...

- https://www.smogon.com/forums/threads/gen-1-glitches-and-how...

- https://glitchcity.wiki/wiki/Leech_Seed_and_Toxic_stacking

So a zip bomb would just decompress up to whatever internal limit and be discarded.

A GET request doesn't have a body. There's nothing to gzip.

For example, as much as I'd like to praise Go's stdlib because of how far it can take you, the defaults for its HTTP server are "the URL+headers of a request can take as much as 1MB" and "no timeouts at all, wait forever if necessary". See the `Server` type and the `ListenAndServe` function (not method) from the `net/http` package for more details.

They're probably trying to be as conservative as possible, but 1MB is huge.

Just like all good c code always checks for overflows.

real pro, man, wow! :))

You're not good at doxxing, Benjamin.

Maybe get outside and touch some grass once in a while? because being a bully online isn't something I would strive for in life.

[1] https://github.com/tholian-network/stealth

Being able to use the web anonymously is a value that should be held up against moral values. Malicious scraping of websites should be, too. Both ideas can simultaneously true, they don't have to be mutually exclusive. I also support initiatives like the web archive which I consider "good behavior" of web scraping.

If a person asks me for the dataset of my website and they don't have a competitive business that they run, I'm even happy to open source the dataset generation part. Contributors get more rights, abusers get less rights. That's how it should be in my opinion.

I don't see them as contradictory.

Would you mind elaborating more on where you draw the line between "good" and "bad" behavior when it comes to scraping?

Is it obeying robots.txt, the monetary purposes (profit / non-profit)?..

I personally have a hard time accepting that a website can allow Google to scrape all their content, but prevent others. In my view, this is anti-competitive behavior favoring a large and evil corporation. Argumenting that Google delivers value back in the form of traffic is unacceptable to me, because it goes against basic principles of net neutrality, which I support.

I'm intetested in reading your thoughts on this.

I wrote you an email, would love to discuss this further.

I would suggest to generate some fake facts like: "{color} {what} {who}", where:

* {what}: [ "is lucky color of", "is loved by", "is known to anger", ... ]

* {who}: [ "democrats", "republicans", "celebrities", "dolphins", ... ]

And just wait until it becomes part of human knowledge.

"You won't believe what celebrities love {color}!!"

nobody cares about "celebrities" :)

> I would suggest to generate some fake facts like: …

Oh, I very much like this.

But forget just LLM ranges, there could be many other unknown groups doing the same thing, or using residential proxy collections to forward their requests. Just add to every page a side-note of a couple of arbitrary sentences like this, with a “What Is This?” link to take confused humans to a small page explaining your little game.

Don't make the text too random, that might be easily detectable (a bot might take two or more snapshots of a page and reject any text that changes every time, to try filter out accidental noise and therefore avoid our intentional noise), perhaps seed the text generator with the filename+timestamp or some other almost-but-not-quite static content/metadata metrics. Also, if the text is too random it'll just be lost in the noise, some repetition would be needed for there to be any detectable effect in the final output.

Anyone complaining that I'm deliberately sabotaging them will be pointed to the robots.txt file that explicitly says no bots⁰, the licence that says no commercial use¹ without payment of daft-but-not-ridiculous fees.

----

[0] Even Google, I don't care about SEO, what little of my stuff that is out there, is out there for my reference and for the people I specifically send links to (and who find it, directly or otherwise, through them)

[1] And states that any project (AI or otherwise) that isn't entirely 100% free and open source and entirely free of ads and other tracking, is considered commercial use.

The text itself without the webfont (which acts like a session, basically) is useless for any kind of machine processing, because it contains the shifted characters as UTF-8. The characters are then shifted back with a custom webfont whose seed is the same as the served HTML, but is different for each client. If you detect a non-bot user, it's currently just setting the seed/shift to 0, and serves the real plaintext, but that's optional as the user doesn't notice a difference (only maybe in the copy/paste function).

For me this was the only kind of web technology I could come up with to find a different way to serve "machine-readable" and "human-readable" content and to be able to differ between them. Anything else that's based on e.g. WebCrypto API or other code would be easily bypassed, because it can run in headless Browser instances.

Though taking screenshots in a headless chrome would kind of work to bypass this, but OCR is luckily currently kinda shitty and the development costs for something like that would explode compared to just adding another rotation mechanism in the webfont :D

You would probably have to keep that for accessibility purposes. Though then however you are detecting bot/not might be easily tricked by a good bot - the CAPCH arms race is currently at a point where such things exclude more human requests than automated ones…

http://www.rexresearch.com

I spent 2 minutes of my life shooting cookies with a laser. I also spent close to a quarter of a minute poking a cookie.

Not possible (unless you're talking double gzip). gzip's max compression ratio is 1032:1[1]. So 100kB can expand to at most ~103MB with single gzip.

Brotli allows much larger compression. Here's[2] a brotli bomb I created that's 81MB compressed and 100TB uncompressed. That's a 1.2M:1 compression ratio.

[1] https://stackoverflow.com/a/16794960

[2] https://github.com/google/google-ctf/blob/main/2019/finals/m...

> Not possible (unless you're talking double gzip). gzip's max compression ratio is 1032:1[1]. So 100kB can expand to at most ~103MB with single gzip.

Not sure if I understand the rest of your argument though. If the critique is that it's not possible and that I'm lying(?) about the unpacked file size on the proxy side, then below you'll find the answer from my pentester's perspective. Note that the goal is not to have a valid gzip archive, the goal is to have an as big as possible gzip archive _while unpacking_ and before the integrity and alignment checks.

In order to understand how gzip's longest match algorithm works, I recommend to read that specific method first (grep for "longest_match(IPos cur_match)" in case it changes in the future): https://git.savannah.gnu.org/cgit/gzip.git/tree/deflate.c#37...

The beauty of C code that "cleverly" uses pointers and scan windows with offsets is that there's always a technique to exploit it.

I'll leave that for the reader to understand how the scan condition for unaligned windows can be modified so that the "good_match" and "MAX_MATCH" conditions are avoided, which, in return, leads to bigger chains than the 258 / 4096 bytes the linked StackOverflow answer was talking about :)

Most people in that ecosystem try to justify that they are somehow better when they can write pointer-magic that nobody else can understand, and feel personally attacked immediately when you mention how much less complex and more maintainable the code would have been if they would have used Go or Rust or another memory safe language.

For me, because I work in cyber, Go is kind of somewhere the middle ground between intentionally crappy C code for offensive purposes and maintainable Go code for defensive purposes. Can't use rust because you can't write exploits in Rust without nullifying the reason you used Rust in the first place :D

Go has a lot of conventions and paradigms I could have an opinion against. The point behind it is that it has opinions, which makes it more maintainable, even when you don't like the enforced opinions.

I'm not saying that there is no reason to use C, there is. But every time you choose C as a language you should be aware that you chose a maintenance burden that increases the risk of the codebase to be abandoned in the future.

Are you saying you can modify your gzip compression code locally to generate a malformed gzip file? That wouldn't be exploiting deflate.c , that would be exploiting the receiver's decompression code, which might be inflate.c or some other implementation of gzip decompression, which might be in some other language. The language used by the compression code doesn't seem relevant to me, rather it's the language used by the decompression code that might have vulnerabilities that can be exploited. If you have a compressed gzip file that expands to more than 1032:1, the file itself is a proof of concept of the vulnerability; it doesn't matter whether the file was generated by C, Rust, Python, or in a hex editor by hand.

If you've found something in gzip code that causes it to use significantly more memory or disk space than it should (either during compression or decompression), I think that's a denial or service vulnerability and should be reported to gzip.

[1] https://git.savannah.gnu.org/cgit/gzip.git/tree/inflate.c

Check out the methods that start with "is_filtered_nmap_" here: https://github.com/tholian-network/firewall/blob/master/kern...

This stuff is just low level automated scanning looking for well known, easy exploits. Default credentials and stuff like that. A lot of it is trying to recruit hosts for illegal VPNs / proxies, DDOS service, and more scanning / propagation. My advice is to block it (maybe with an expiration time on the block), log it, and ignore it. But it can be fun to see what they do with a honeypot.

LLMs don't prompt themselves from training data, they learn to reproduce it. An example of transformer poisoning might be pages and pages of helpful and harmless chatlogs that consistently follow logically flawed courses.

Tell that to twitter propaganda bots and the developers behind it. Don't have to tell me that, you know. Most interactive systems that interact with websites that I've seen are vulnerable to this because of the way they prompt the LLM after the scrape, with the unfiltered or crappily sanitized content.

Basically, this isn't about training, it's about abusing the "let's act like our model wasn't trained in 2019 by adding random Internet data to the chat transcript".

Sounds brutal. A whole ISP typically is a single ASN and any of their subscribers can be running bots while others don't - isn't this so?

Do bots even use QUIC? Either way, holding tcp state instead of udp state shouldn't be a big difference in 2024, unless you're approaching millions of connections.