News for hackers. Stuff that matters.
Re:How to Build a IP Geolocation Database from Scratch? by incolumitas on Thursday September 14, 2023 @10:36AM (#37509599)
Big fan of what articles? On https://incolumitas.com/ or on https://ipapi.is/?

Great idea with latency triangulation, I used latency information for a lot of things, especially VPN and Proxy detection.

But I didn't assume you can obtain that accurate location. I am honestly impressed. But latency triangulation with 600 servers gives some very good approximation. Nice man!

Some questions:

- ICMP traffic is penalised/degraded by some ISP's. How do you deal with that?

- In order to geolocate every IPv4 address, you need to constantly ping billions of IPv4's, how do you do that? You only ping an arbitrary IP of each allocated inetnum/NetRange?

- Most IP addresses do not respond to ICMP packets. Only some servers do. How do you deal with that? Do you find the router in front of the target IP and you geolocate the closest router to the target IP (traceroute)?

Re:How to Build a IP Geolocation Database from Scratch? by withinboredom on Thursday September 14, 2023 @10:50AM (#37509779)
I'm very curious why you'd do VPN/proxy detection...

But at a previous company I worked at that ran a very large chunk of the internet, we did indexing of nearly the entire internet (even large portions of the dark web) approximately every two weeks. There were about 500 servers doing that non-stop. So, I think it is relatively reasonable if you have 600 servers to do that.

Re:How to Build a IP Geolocation Database from Scratch? by meroje on Thursday September 14, 2023 @12:00PM (#37510704)
In the business of media streaming, rightholder will require that you check for vpn and proxies in addition to countries when deciding if a given viewer will be able to stream a given media.
Re:How to Build a IP Geolocation Database from Scratch? by withinboredom on Thursday September 14, 2023 @12:06PM (#37510780)
Does that actually work? That could explain an issue with a particular streaming service I use. There are currently some ongoing routing issues in BGP land and my ISP. When trying to stream, it says I’m using a proxy, so due to the incredible route my packets are taking, that might be it. What’s funny is that the only way to watch this service is to use a vpn right now.
Re:How to Build a IP Geolocation Database from Scratch? by sitzkrieg on Thursday September 14, 2023 @01:31PM (#37512065)
of course it doesnt work but they gotta try clutching pearls and applying whatever pressure they can think of on these fronts
Re:How to Build a IP Geolocation Database from Scratch? by wpietri on Thursday September 14, 2023 @02:32PM (#37512862)
Why is this getting downvoted? It seems to me that a lot of the media-focused anti-piracy tooling is essentially a performance of toughness to make rightsholder execs comfortable. Everybody accepts you can't stop piracy entirely, and nobody's willing to say, "Fuck it, we'll compete on convenience and strong consumer relationships," so we all put up with this weird middle ground of performative DRM and the like. With only the rare occasional bit of honesty, as from Weird Al: https://sfba.social/@williampietri/110906012997848549
Re:How to Build a IP Geolocation Database from Scratch? by at_a_remove on Thursday September 14, 2023 @03:40PM (#37513830)
[delayed]
Re:How to Build a IP Geolocation Database from Scratch? by fomine3 on Thursday September 14, 2023 @09:53PM (#37517252)
I love that HLS+AES "encryption" is still in use, thanks to Apple.
Re:How to Build a IP Geolocation Database from Scratch? by vGPU on Thursday September 14, 2023 @03:39PM (#37513804)
They probably just keep a list of known VPN server IP’s.
Re:How to Build a IP Geolocation Database from Scratch? by meroje on Friday September 15, 2023 @04:34AM (#37520138)
Routing should not impact the detection, it's usually based on maxmind's anonymous/datacenter database using your IP. Accuracy won't be 100% of course but you have to show compliance.
Re:How to Build a IP Geolocation Database from Scratch? by withinboredom on Friday September 15, 2023 @07:54AM (#37521630)
I doubt it. According to that database my ip is in a totally different country but I'm served the correct content. Despite my efforts to fix this for years...
Re:How to Build a IP Geolocation Database from Scratch? by reincoder on Thursday September 14, 2023 @10:52AM (#37509822)
https://incolumitas.com/

This is my all-time favorite article: https://incolumitas.com/2021/11/03/so-you-want-to-scrape-lik...

I used to do freelance web scraping, and that article felt like some kind of forbidden knowledge. After reading the article, I went down the rabbit hole and actually found a Discord server that provided carrier-grade traffic relay from a van which contained dozens of phones.

For the questions..... we have to kinda wait a bit, someone from our engineering team might come here and reply.

By the way, as I have you here have you considered converting the CSV files to MMDB format? I was planning to do that with our mmdbctl tool later today.

https://github.com/ipinfo/mmdbctl

Re:How to Build a IP Geolocation Database from Scratch? by sambazi on Thursday September 14, 2023 @12:51PM (#37511432)
> I used to do freelance web scraping

"don't sell warez"

Re:How to Build a IP Geolocation Database from Scratch? by carlhjerpe on Thursday September 14, 2023 @05:48PM (#37515291)
You can guess pretty well how IP's are related by BGP announcements, so as long as a few per block and if small, ASN. You can use that logic.