It's from a DHT/BitTorrent crawler that runs around downloading all the torrent metadata it can find I've been running for ages.

Do you make visualizations of the traffic?

Some different views. For instance, to give some sense of a scale for the 250 or so v2 torrents seen every day is, here's a view of the total number of torrents it discovers each day [1]. (Or, more properly, this is the number of torrents it gathered complete metadata and validated against the infohash each day).

One of these days I should find somewhere to put all these bits of metadata I've been downloading.

[1] https://imgur.com/a/8pWhjxD

Consider creating a collection at the Internet Archive for these data sets.

Sure:

https://archive.org/details/torrent_metadata_archive_sample

I'll start uploading monthly archives.

I genuinely appreciate that you took the time to do this. Thank you.

For anyone looking to download the data: Click on my name in archive.org to see the monthly archives. I'm slowly working my way through the months, though there may be breaks in the work as other work takes priority.

The first few months will be the biggest, then it should settle down to something like 15-30gb a month.

Neat! I don’t have it handy, but years ago I scrapped the states traffic page and then made time lapses from them. One of the road segments had a bad sensor that would toggle into a stuck state.

Keep it up!

cool. Did you write it yourself or are you using an open source project of some kind? Would be interested to run one of these myself.

I stumbled across a DHT crawler, and started running it. Over time I added random things to it, like IPv6 and the ability to download metadata. I was mostly interested in using it to learn how DHT worked both in theory and in practice, and for that, it did it's job.

The page for the crawler I started with is long gone, but this appears to be a reasonable copy on github:

https://github.com/zlzlovezl/simDHT-1

I'm sure you can find other copies, with features added. Looking around, it seems like quite a few people have taken this and made it their own over time.

It's far from perfect in this state, but a fun toy to play with. I will say, from a point of painful experience: The way this crawler works tends to make some of the various people that track such things think you're sharing pretty much all the things. If that's an issue where you live, don't run it from your home connection.