Great idea with latency triangulation, I used latency information for a lot of things, especially VPN and Proxy detection.
But I didn't assume you can obtain that accurate location. I am honestly impressed. But latency triangulation with 600 servers gives some very good approximation. Nice man!
Some questions:
- ICMP traffic is penalised/degraded by some ISP's. How do you deal with that?
- In order to geolocate every IPv4 address, you need to constantly ping billions of IPv4's, how do you do that? You only ping an arbitrary IP of each allocated inetnum/NetRange?
- Most IP addresses do not respond to ICMP packets. Only some servers do. How do you deal with that? Do you find the router in front of the target IP and you geolocate the closest router to the target IP (traceroute)?
But at a previous company I worked at that ran a very large chunk of the internet, we did indexing of nearly the entire internet (even large portions of the dark web) approximately every two weeks. There were about 500 servers doing that non-stop. So, I think it is relatively reasonable if you have 600 servers to do that.
This is my all-time favorite article: https://incolumitas.com/2021/11/03/so-you-want-to-scrape-lik...
I used to do freelance web scraping, and that article felt like some kind of forbidden knowledge. After reading the article, I went down the rabbit hole and actually found a Discord server that provided carrier-grade traffic relay from a van which contained dozens of phones.
For the questions..... we have to kinda wait a bit, someone from our engineering team might come here and reply.
By the way, as I have you here have you considered converting the CSV files to MMDB format? I was planning to do that with our mmdbctl tool later today.
"don't sell warez"
We have a simplified explanation of our probe network here: https://ipinfo.io/blog/probe-network-how-we-make-sure-our-da...
The only update is the number of servers is like 600+ now. The probe network is growing extremely rapidly.
Our IP geolocation process is quite complicated, and we have a team of data engineers, infrastructure engineers, and data scientists working on various aspects of it. Therefore, our approach is users can ask us questions, and we will try our best to answer them.
I love your company and service, but I hate your pricing. I work with a lot of small clients/apps that paying for usage would be a no-brainer, but the defined monthly price buckets don't make any economical sense at their scale. If you added a "pay as you go" tier that a small app could reasonably start by using dollars worth of API calls per month and grow from there, I'd be spreading your seed all over the place. I'm not saying this to rag on you, just trying to provide some constructive feedback as a thank you for your info sharing!
# Check out the free IP databases
https://ipinfo.io/products/free-ip-database
The free databases come with commercial usage permission, and because they are databases, you can make unlimited lookups from them. The databases provide full accuracy and are updated daily. They are just a subset of our IP geolocation database that only provides IP to Country information.
# Complement the database with the API service
If you only want city-level information, switch to the API service. Use the database to look up IP-to-country information as many times as you want. However, use the API service only when necessary.
Additionally, if you include a credit link to us, we will double your API limit to 100k/month. Visit https://ipinfo.io/contact/creditlink.
# Cache data
All of our API libraries have native caching support. We strongly recommend that users reduce their number of requests by caching the response. I highly recommend you check out our libraries: https://github.com/ipinfo
---
The only challenge with the free IP databases is that you need to host the database somewhere to lookup the IP to Country information. Having an API service with nearly unlimited lookups for IP to Country information will be fantastic.
If you know someone who has an IP to Country as API service please, let me know. We only require an attribution for using our database. If you have a similar service that is popular but don't want to maintain it let us know as well, we can takeover the site and host it ourselves with the IP to Country data.
If you hate their pricing, then you should join https://ipapi.is/
I have more competitive pricing and all my pricing plans include the full API output.
See for yourself: https://ipapi.is/pricing.html
If you sign up and mail me the code: "HN-IPAPI.IS-2023" I will provide you with the large API plan for 3 months for free.
iptables -A INPUT -p icmp -j DROP(But the guy running the probes is making a good counter argument)
I disabled ICMP reply on my home router.
it's a bit like greeting-back ppl on the street.
not doing it will not make you invisible. it will break somebody's assumption of decency, but most ppl don't care either way.
Doesn't actually help at all because the BGP announced prefix of your IP can still be tracerouted. You won't be physically far from it.
Say if your ISP announces 125.15.18.0/17 and you're in 125.15.29.145, a traceroute will still yield a pretty good approximation of where you're at. The last hop ping is really quite immaterial here.
Edit: they provided a method: https://news.ycombinator.com/item?id=37510063
The data is not derived from the IP address itself, but rather from the process itself. And it's just a ping. Moreover, the majority of the IP addresses are not pingable. So, we rely on other in house statistical and scientific models to estimate the location. The probe infrastructure is extremely complicated and there are billions and billions of IP addresses, which is why we do not have a robust range filter mechanism.
You can implement a dynamic ping blocking mechanism or use our data to find hosting ASNs and block ranges of those ASNs. You can download the database for free: https://ipinfo.io/developers/ip-to-country-asn-database
I wonder if you meant `concession`.
Also, it's a false dichotomy. One can use VPN or proxies, to limit exposure or to encapsulate it. Of course, you can't get perfect location privacy.
IP geolocation is mainly used in cybersecurity and marketing analytics. There are many ways to geolocate someone. I once came across a project that could estimate the country a user is from based on their writing style and grammar mistakes. For example, American people sometimes use "should of" instead of "should have". Knowing the geolocation of an IP address isn't super creepy. It's just how things work on the internet.
Just because it's "how things work on the internet" doesn't make its mass collection right.
I apologize if I was rude in any way by saying the word "meme". I saw a sister comment and thought you were being sarcastic. There is a popular meme about "I have your IP address", so I thought you were referencing that. I have had conversations with many young people who were concerned about their IP address being leaked through a game server. Therefore, I try to use humor to alleviate their stress. However, I now realize that this situation was different, and I am sorry for not understanding that.
We provide a service that helps users keep their internet-connected services secure by providing IP metadata information. Are you being attacked by malicious actors? Use our free IP database to identify the location and ASN to block them. Do you want to restrict access to your service to certain regions? Do that for free with our services.
We have the most accurate data available, and yet we offer the most generous free tier. We provide a full accuracy IP database for free, without any range aggregation, and with daily updates and a commercially permissible license. We have built a community forum solely dedicated to answering users' questions. We invest in website tools and open-source tools, all with the goal of helping users maintain the security and functionality of their services.
We do have premium tier services, but if you use our free data as a foundation, you can always replicate those premium features to a reliable degree.
Our IP metadata information is being used in marketing and sales intelligence. It is the same data that you use to protect your internet connected devices, used by our customers to sell you something.
IP metadata information that we provide is a cornerstone of keeping the internet safe and accessible for everyone. That is how things just are. The deepweb is immune to IP meta data information, and that is why it is such a messy and chaotic place.
That is just truth of the internet. We are essential and we prefer to be open about our process and listen to our stakeholders (users + customers + non-users).
It's good that you are using a VPN. I advocate for the usage of VPNs, and many VPN companies actually use our data to verify their server locations. In the VPN industry, VPN companies get their VPN servers from specialized hosting services that cater to dozens of VPN companies. You can check out the ASNs of the VPN IP addresses to find them.
VPN companies use our IP geolocation data to confirm the actual location of their servers. Let me tell you a fun story. One VPN company claimed to have a server in the Bahamas, but upon investigation, we discovered that the server was actually located in New York. It was a surprising find. Getting a server in the Bahamas is more challenging than getting one in NY. Just imagine users thinking their internet activity is immune to US jurisdiction because they are using a VPN service based in Bahamas but in fact it is actually located in NY. So, we might not be essential, but we are certainly very useful!
Thank you for the great conversation, dude. Appreciate it.
When people work in advertising, they mostly forget that the core of their business is for-profit manipulation of people with little or no regard for truth or the people concerned. But I personally think that's kinda creepy, and only getting more so as it goes from broad manipulation of millions via mass media down to thousands, hundreds, or single individuals.
In the context of a butcher and a vegan we are not either of them, we would be someone who sell salt. Salt as an ingredient is used by vegans, vegetarians, and meat eaters.
Salt is useful, and it is a complimentary item that makes the food taste better. But it is not a main feature of any dish. Salt plays an important and somewhat universally required item in any dish, but this is not the main feature of the dish you eat.
We are a data company. This IP metadata we sell is important to run a service that is connected to the internet. We are focused on serving cybersecurity as we are focused on other industries which includes adtech and sales intelligence.
IP metadata is used in threat intelligence, building firewalls, and attack surface management. On the other hand, when you are running a business at scale, you need to invest into understanding your target demographics. That is where largely adtech companies use our data. They need data and context to understand who their users are.
IP metadata helps to prevent cyberattacks to happen at a very early stage through firewall blocking. IP metadata, used through threat intelligence, can further strengthen firewalls to prevent future attacks. And IP metadata, can also be used in website personalization, adtech and sales intelligence etc.
We believe that every internet connected services for profit and non-profit alike, should have access to some security resilience, that is why we offer so many good quality free data.
If you progressively remove the layers of metadata associated with internet-connected devices one by one from GPS data (user-permitted action) → IP geolocation (IPinfo.io)→ IP address (ISP), and come to a system that is completely anonymous, you have the dark web.
The dark web represents an ecosystem that is completely devoid of ad tech because there is no identity, and the users are purely homogenous. The system itself is not widely adopted because internet participants need to feel secure from cyberattacks and harmful words. Traditional internet services do not exist there because they are getting bombarded with anonymous attacks, and they cannot provide any value to the customer because they don't even know who/what the user is.
IP metadata helps with running a business and serves as a way to protect yourself. There are organizations and users who actively utilize our services to check email headers, find the IP address, and lookup the IP metadata of those IP addresses. The data is utilized for recognizing spam and phishing emails. Conversely, our service is also used in adtech-driven cold emails and newsletter services.
IP metadata is required for a functioning and widely adopted internet.
..any? It makes me nervous that you actually mean that.
Can IP geolocation be used for positive things? Yes. We agree on that.
But it can also be used for things that can reasonably be seen as creepy and/or bad. Hopefully you can agree on that.
If you're going to sell your services to all comers, then you are going to be supporting and profiting from both good and bad things. And rather than spewing paragraphs about the good bits when somebody points out the bad bits, you should own what you're doing.
Think of it like selling guns. Are there legitimate defensive uses for them? Sure, ask anybody in Ukraine. But if a gun deal just focuses on how they "enjoy talking with people and helping them with the technical [...] aspects" while being "happy to assist "any business, organization, or person" to buy a gun, then they end up morally responsible for all the other uses guns have, too.
It feels like it couldn't be abused by 'freeloaders', because i'd guess their use-case is viewing other peoples.
Wait - how does this work for cell IPs? A lot of cellphone v4 IPs are now shared between hundreds or thousands of devices, right?
It probably has something to do with important routers. What tags do we show when you visit the IP data page? The IP data page can be accessed by visiting ipinfo.io/<IP_address>.
We use the generic term "data experts," but it actually consists of about 2 dozen engineers, including data engineers, data scientists, infrastructure engineers, backend engineers, and a great technical CEO working on all that. All those folks have gone on a boating trip off the coast of Spain for a retreat.....except for me.
I will ask them and try to circle back with some answers.
You can enter IP addresses on the right side to look up information here: https://ipinfo.io/what-is-my-ip
Additionally, we offer some enjoyable tools that you can use here: https://ipinfo.io/tools
The CLI tool is particularly entertaining.
You can also use our API service without signing up, with a limit of 1000 requests per day.
If you do choose to sign up for a free account, you will receive 50,000 requests per month, free IP databases, a bulk lookup feature, and more.
I don't know if that provider terminates long running calls, but the calls would stay up too regardless of tower.
The challenge of being a data provider is that you can use our data in a million ways, and we don't have coverage of all. So, when you come up with questions or ideas, we can help you better.
As you mentioned, audit logs. I highly recommend you look into the ASN field.
The ASN identifies an organization that owns a block of IP addresses. In my experience, I have found that the combination of ASN+Country is the most valuable information you can use in spam and fraud detection. You can fake the IP geolocation information with a VPN. However, it is not as easy to fake the ASN information of the IP address. So, when you use a combination of country + ASN, you can have a robust cybersecurity system.
{
"city": "Mumbai",
"connection": {
"asn": 24560,
"isp": "Bharti Airtel Ltd."
},
"continent_code": "AS",
"continent_name": "Asia",
"country_code": "IN",
"country_name": "India",
"currency": {
"code": "INR",
"name": "Indian Rupee",
"plural": "Indian rupees",
"symbol": "Rs",
"symbol_native": "\u099f\u0995\u09be"
},
"ip": "2401:4900:1f38:7402:5569:2e45:3bb:9c0d",
"latitude": 19.076000213623047,
"location": {
"calling_code": "91",
"capital": "New Delhi",
"country_flag": "https://assets.ipstack.com/flags/in.svg",
"country_flag_emoji": "\ud83c\uddee\ud83c\uddf3",
"country_flag_emoji_unicode": "U+1F1EE U+1F1F3",
"geoname_id": 1275339,
"is_eu": false,
"languages": [
{
"code": "hi",
"name": "Hindi",
"native": "\u0939\u093f\u0928\u094d\u0926\u0940"
},
{
"code": "en",
"name": "English",
"native": "English"
}
]
},
"longitude": 72.87770080566406,
"region_code": "MH",
"region_name": "Maharashtra",
"time_zone": {
"code": "IST",
"current_time": "2023-09-15T10:52:42+05:30",
"gmt_offset": 19800,
"id": "Asia/Kolkata",
"is_daylight_saving": false
},
"type": "ipv6",
"zip": "400203"
}[0] https://ipinfo.io/2401:4900:1f38:7402:5569:2e45:3bb:9c0d
I know it can be done with CSV but it's not as smooth.
We usually just send users the documentation of ingesting the data in CSV or NDJSON format (Newline Delimited JSON). We don't actually get many requests for data downloads in Parquet format. I think we have a few customers where we deliver the data in parquet format directly to their cloud storage bucket.
But keep an eye out for our emails if we announce the parquet data downloads. I will talk with the folks about this.
BUT, there are some good news.
At least for the free database, we deliver the data directly to data warehouse platforms. Not even storage buckets. And we supply a good amount of documentation.
We have the free database in Snowflake, GCP, Kaggle, and Splitgraph, and we are working on a few more deals. For the free database, atleast, we are working on better things than parquet. Like literally one-click solution to bring the IP data to your data warehouse.
Kaggle: https://www.kaggle.com/code/ipinfo/ipinfo-ip-to-country-asn-...
Snowflake: https://app.snowflake.com/marketplace/listing/GZSTZSHKQ4QY/i...
If you want to use our free IP database on Google Cloud or BigQuery, please send us an email ([email protected]) and mention that the DevRel sent you from HN. I can easily set you up with the free IP database in GCP/BQ.
A time series IP database requires a substantial amount of storage and computational cost to query, as I imagine. The city level geolocation data we have is ~1.5 gb in size. IP range data is complicated to query efficiently as you need to understand data platform settings and good amount of computer network math and computer science stuff. Adding a layer of time series complexities on top of that, makes this process quite difficult.
To give you some context of how IP metadata lookups work, you can check out this article
https://ipinfo.io/blog/ip-address-data-in-snowflake/
Even if you keep all your database in a binary format, the computational cost is still non-negligible.
But I encountered 2 things using ipinfo: Hetzner Server that are in Germany in a fixed location that never moved are sometimes located in another country, for me it was once s Server placed into Moscow and once in South America.
How does this happen?
I guess it is because of IPv4 trading or IP address shuffling.
As far as I know, Hertzer, like many hosting companies, is buying IPv4 addresses around the world. Here is an article on the IPv4 trades:
https://tech.marksblogg.com/ipinfo-free-ip-address-location-...
When a company buys an IP address block or relocates an IP block from one of its data centers to another, the location of those IP addresses changes.
If your IP address is static, but we have made an error in geolocation, I would love to take a closer look. You can email our support ([email protected]) and send a link to the comment. We can discuss it further from there.
A behavior pattern could be that your IP address is being shuffled around random locations that go beyond the normal location shuffling of an ISP connection.
Also, if your IP range is listed in some public datasets that belong to a VPN service, we could recognize your IP as a VPN.
Please reach out to our support and let us know about this. Thanks
Our geolocation methodology expands on the methodology you described. We utilize some of the publicly available datasets that you are using. However, the core geolocation data comes from our ping-based operation.
We ping an IP address from multiple servers across the world and identify the location of the IP address through a process called multilateration. Pinging an IP address from one server gives us one dimension of location information meaning that based on certain parameters the IP address could be in any place within a certain radius on the globe. Then as we ping that IP from our other servers, the location information becomes more precise. After enough pings, we have a very precise IP location information that almost reaches zip code level precision with a high degree of accuracy. Currently, we have more than 600 probe servers across the world and it is expanding.
The publicly available information that you are referring to is sometimes not very reliable in providing IP location data as:
- They are often stale and not frequently updated.
- They are not precise enough to be generally useful.
- They provide location context at an large IP range level or even at organization level scale.
And last but not least, there is no verification process with these public datasets. With IPv4 trade and VPN services being more and more popular we have seen evidence that in some instances inaccurate information is being injected in these datasets. We are happy and grateful to anyone who submits IP location corrections to us but we do verify these correction submissions for that reason.
From my experience with our probe network, I can definitely say that it is far easier and cheaper to buy a server in New York than in any country in the middle of Africa. Location of an IP address greatly influences the value it can provide.
We have a free IP to Country ASN database that you can use in your project if you like.
https://ipinfo.io/developers/ip-to-country-asn-database