incolumitas
Partially new headless Chrome detection signals
Comparing the Accuracy of Various IP Geolocation Providers
In this study I wanted to find out how good various different geolocation providers rank when being compared with real GPS location data obtained via the JS Location API.
Seems like IP Geolocation is 93% correct to the country level and like 70% correct to the city level.
This is way better than a "random guess" and therefore the utility is definitely given.
it's not hard to infer location data from IP addresses, albeit it's not necessarily an exact science
Live Proxy and VPN Detection
I am the creator of https://ipapi.is/ and already some time in the niche. I also have my own IP Geolocation Database as described here: https://ipapi.is/geolocation.html
Do you have your first customers?
You can derive an enormous amount of geolocation data by looking at WHOIS data: https://ipapi.is/geolocation.html
But I picked just one faulty classification of ipinfo.io, that's not fair, I know. I only wanted to point out that what you are doing is exactly the same as https://ipapi.is/ is doing and that we both make mistakes
----
You are using the 700 measuring servers to interpolate geolocations of IP addresses, right?
That works sometimes, but more than often it does not. It does not scale either.
Active latency triangulation of every IPv4 address (let's not even speak about IPv6) is simply not possible. The reasons are manyfold:
- Most hosts don't reply to ICMP
- Many routers block ICMP traffic, or they throttle / downgrade it, thus skewing measurements
- Traffic from your probing servers is probably not handled in the same way as is normal residential ISP traffic
- You have to constantly measure all IPv4, since IPs are constantly reassigned, which is simply not possible with only 700 servers
Latency triangulation works in theory, but in practice it is just not applicable to the full IP space.
Having said that, active geolocation with probing servers is still better than not doing it :D
Latency triangulation works much better in a passive way, meaning that a client is visiting a server that is under your control and you triangulate the client with JS for example (web sockets).
But I doubt that ipinfo.io has a significant share of the Internet's traffic...
Maybe I am missing something?
Hosting detection is a finite process, meaning that there is a finite amount of hosting providers out there to detect after all. Challenge lies in staying up to date.
So maybe we can make a compromise and state that ipinfo.io is likely a bit better than ipapi.is in hosting detection.
But ipapi.is for sure detects some hosting provider's that you don't. Examples:
https://api.ipapi.is/?q=185.45.13.144
https://api.ipapi.is/?q=185.50.248.0
https://api.ipapi.is/?q=91.102.88.0
Summary: Your product is maybe a bit better in regards to hosting detection than https://ipapi.is/, but it is much much much much more expensive ;)
https://ipapi.is/ is not as good in regards to geolocation, but our hosting detection is more advanced and https://ipapi.is/ is much cheaper...
The Economics of Making 3.7 Billion DNS Reverse Lookups
An Algorithm to Detect Hosting Providers and Their IP Ranges
If you hate their pricing, then you should join https://ipapi.is/
I have more competitive pricing and all my pricing plans include the full API output.
See for yourself: https://ipapi.is/pricing.html
If you sign up and mail me the code: "HN-IPAPI.IS-2023" I will provide you with the large API plan for 3 months for free.
ARIN is awesome
Great idea with latency triangulation, I used latency information for a lot of things, especially VPN and Proxy detection.
But I didn't assume you can obtain that accurate location. I am honestly impressed. But latency triangulation with 600 servers gives some very good approximation. Nice man!
Some questions:
- ICMP traffic is penalised/degraded by some ISP's. How do you deal with that?
- In order to geolocate every IPv4 address, you need to constantly ping billions of IPv4's, how do you do that? You only ping an arbitrary IP of each allocated inetnum/NetRange?
- Most IP addresses do not respond to ICMP packets. Only some servers do. How do you deal with that? Do you find the router in front of the target IP and you geolocate the closest router to the target IP (traceroute)?
Also I only need to scrape as many WHOIS records as there are different networks out there. So for example for the IPv4 address space, there are much less networks as there are IPv4 addresses (2^32).
Also, most RIR's provide their WHOIS databases for download.
Therefore, "scraping" is not really the correct word, it's an hybrid approach, but mostly based on publicly available data from the five RIR's.
How to Build a IP Geolocation Database from Scratch?
On High-Precision JavaScript Timers
With only 10 dongles and 10 dataplans, you can have a lot of IP addresses that are extremely hard to block. It's an one time investment, paying proxy providers is a fixed cost.
https://github.com/NikolaiT/Crawling-Infrastructure
And here I am writing about it (but its quite old): https://incolumitas.com/2019/08/31/web-scraping-puppeteer-aw...