Track HN: Survival Rate of 120,396 Show HN Stories (June 2023)

Re:Track HN: Survival Rate of 120,396 Show HN Stories (June ... by hawski on Tuesday June 13, 2023 @02:58PM (#36315114)

Regarding database hosting, if you would consider giving the data away, I would suggest converting it to an SQLite database and sharing it over Torrent.

[ Parent | on HN ]

Re:Track HN: Survival Rate of 120,396 Show HN Stories (June ... by xnx on Tuesday June 13, 2023 @03:14PM (#36315409)

I second this. You've done a great service to collect this data. I'm guessing the file must be much smaller than 20GB when compressed.

[ Parent | on HN ]

Re:Track HN: Survival Rate of 120,396 Show HN Stories (June ... by zX41ZdbW on Tuesday June 13, 2023 @05:56PM (#36318123)

It is only around 5 GB in ClickHouse. Details: https://github.com/ClickHouse/ClickHouse/issues/29693

[ Parent | on HN ]

Re:Track HN: Survival Rate of 120,396 Show HN Stories (June ... by zX41ZdbW on Tuesday June 13, 2023 @05:58PM (#36318145)

I've also did an experiment by generating and searching embeddings for all the comments on HN. Here is the walkthrough: https://www.youtube.com/watch?v=hGRNcftpqAk

[ Parent | on HN ]

Re:Track HN: Survival Rate of 120,396 Show HN Stories (June ... by rahimnathwani on Tuesday June 13, 2023 @09:53PM (#36320372)

I'm guessing OP wants to share a database that's always up to date.

A torrent containing a single sqlite file would be good for a snapshot in time, but each update would require a new torrent, even if it only contains the updates since the base or last release.

IIRC IPFS can be used to distribute files that change over time, with only the changes being transferred, although of course there would need to be a place where OP publishes the hash of the most recent file.

In either case, someone would need to seed the file to guarantee it's always available.

[ Parent | on HN ]