News for hackers. Stuff that matters.
Re:How to Build a IP Geolocation Database from Scratch? by TZubiri on Thursday September 14, 2023 @08:26AM (#37507991)
"how to scrape an ip geolocation database"

You know you can just run a whois query per ip you want to analyze, no point in scraping the whole ipvN space.

Re:How to Build a IP Geolocation Database from Scratch? by incolumitas on Thursday September 14, 2023 @08:38AM (#37508111)
I have to scrape the whole IP address space since I offer location information as part of my API.

Also I only need to scrape as many WHOIS records as there are different networks out there. So for example for the IPv4 address space, there are much less networks as there are IPv4 addresses (2^32).

Also, most RIR's provide their WHOIS databases for download.

Therefore, "scraping" is not really the correct word, it's an hybrid approach, but mostly based on publicly available data from the five RIR's.

Re:How to Build a IP Geolocation Database from Scratch? by notlukesky on Thursday September 14, 2023 @09:04AM (#37508442)
What was the easiest and the most frustrating part?
Re:How to Build a IP Geolocation Database from Scratch? by incolumitas on Thursday September 14, 2023 @07:43PM (#37516298)
LACNIC is a fucking pain in the ass since they block after like 27 records

ARIN is awesome

Re:How to Build a IP Geolocation Database from Scratch? by djbusby on Thursday September 14, 2023 @09:06AM (#37508465)
The whois data for IP is not accurate.
Re:How to Build a IP Geolocation Database from Scratch? by gsich on Thursday September 14, 2023 @06:09PM (#37515487)
whois has no sane format.
Re:How to Build a IP Geolocation Database from Scratch? by ggm on Thursday September 14, 2023 @08:36PM (#37516691)
RDAP is run by all the RIR and is Json and has all the whois data except IRR.

And it does 302 redirect to best source.