Same here. I would like to integrate a simple blacklist into our system. Does anybody know of a good IP lists that contains known bots, data centers, and so on?

Also from a performance perspective, I can't do hundreds of millions of requests a month over the wire. That just a waste.

IPDetective collects data data from bot nets, datacenters, tor nodes, proxies and vpns. Is that what you are looking for?

Also IPDetective can be used just as a detection solution rather than a prevention solution.

Sometimes I use it against my nginx access logs and see how much of my traffic could be from bots.

I think your service would be close to what I'm looking for. But can you handle 120+ million requests per month? I would rather integrate an open-source list into our open-source tool than using a proprietary service.

I feel you. Maybe we could work something out.

Thank you very much for your input. I will look into having the privacy be more verbose around the system logs and for how long I store them.

> I keep look for something I can just add lists to my server and check against that, but not going to spend thousands on it.

What do you mean by this? Like having the ability to create your own deny/black list of ip addresses and then you can validate against it?

>> I keep look for something I can just add lists to my server and check against that, but not going to spend thousands on it.

> What do you mean by this? Like having the ability to create your own deny/black list of ip addresses and then you can validate against it?

They want to download your database and run queries on it locally, rather than call your API. This way they don't share data with you and don't need to worry about your data practices.

Ahhh makes sense. I could definitely offer that it's pretty large being 250 million IP addresses. I wonder what the best format would be? Csv? SQLite?

Currently to make things a little smaller I am putting many ip's in with cidr notation..

whichever format you would offer I could likely find ways to convert if needed.

I am still researching what is the fastest / lightest on server resources for blocking ips (mainly from data centers lately been the main problems) -

ip-block-file that is ready via cpanel's WHM is what I am using on one system, and iptables/uffirewall / fail2ban on another server..

I'm using wordpress firewall plugin to block some on another site - but I think it's lighter on server resources to block before the requests hit wordpress.. I wonder if it'd be lighter to pull the file from a separate server and return yay or nay - or if it's fine loading up 250 million ips via one of the firewall options already in place.

How large of a file it 250 mill ips? (non-compressed) - I suppose available RAM could also play into these things.

All of the IP are stored in memory. This takes up about 3.5 GB. I have a text file that is 445K lines that contains all of the specific IP addresses and the IP prefixes (CIDR) that make up that 250 million ips.

Uncompressed I do not know. Are you interested in the ranges and ip addresses or a list of all the ip addresses that make up all the ranges as well?

Let me know what you need and I could definitely try and accommodate for you.