Nearly 90% of our AI crawler traffic is from ByteDance

Re:Nearly 90% of our AI crawler traffic is from ByteDance by atif089 on Thursday October 31, 2024 @06:14PM (#42012118)

Did you implement this in your web server or within your application? I'd love to see the code if you're willing to share

[ Parent | on HN ]

Re:Nearly 90% of our AI crawler traffic is from ByteDance by mmastrac on Thursday October 31, 2024 @08:22PM (#42012881)

Yep, it's open-source:

Here's where we handle the rate limits:

I actually misremembered my implementation. It's rolling counting bloom filters, not a token bucket. :)

[ Parent | on HN ]