Show HN: Ladder, open source alternative to 12ft.io and 1ft.io

Re:Show HN: Ladder, open source alternative to 12ft.io and 1... by Fnoord on Monday November 06, 2023 @10:42AM (#38164044)

Some possible clues:

> https://github.com/kubero-dev/ladder#environment-variables

> USER_AGENT User agent to emulate Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

> X_FORWARDED_FOR IP forwarder address 66.249.66.1

> RULESET URL to a ruleset file https://raw.githubusercontent.com/kubero-dev/ladder/main/rul... or /path/to/my/rules.yaml

[ Parent | on HN ]

Re:Show HN: Ladder, open source alternative to 12ft.io and 1... by janejeon on Monday November 06, 2023 @10:49AM (#38164156)

Oh wow... I'm surprised that's enough. When I was researching scraping protection bypass, you had to do some real crazy stuff with the browser instance + using residential IPs at a minimum...

[ Parent | on HN ]

Re:Show HN: Ladder, open source alternative to 12ft.io and 1... by 2cpu1container on Monday November 06, 2023 @11:55AM (#38165100)

Thats not the full story. It works on many sites, but some (ft.com as an example) have more severe countermeasures to bypass the paywall. Therefore the ladders modifies the served HTML from origin to remove such.

Those rules still need to be build up. (by me or the OS-community)

[ Parent | on HN ]

Re:Show HN: Ladder, open source alternative to 12ft.io and 1... by ComputerGuru on Monday November 06, 2023 @12:44PM (#38165864)

I don’t know of any off-the-shelf product that respects X_FORWARDED_FOR unless the current request ip originates from a whitelisted (or lan) address.

[ Parent | on HN ]