Show HN: Search over 100M archived posts and comments scraped from Parler

Re:Show HN: Search over 100M archived posts and comments scr... by throwaway888abc on Friday March 12, 2021 @02:02AM (#26432926)

Kudos to op / author for putting this out.

>Please consider supporting this project so I can be more relaxed with server/bandwidth costs.

Why not bittorrent (instead of expensive AWS S3) as any other datadump ?

[ Parent | on HN ]

Re:Show HN: Search over 100M archived posts and comments scr... by formalsystems on Friday March 12, 2021 @01:39PM (#26439203)

Thank you! Agree 100% on bittorrent - in all honesty I am new to these kinds of projects, and the wonderful people of the Internet like yourself have been really helpful in showing me the standard way to go about things. Should be a torrent link on there by tomorrow!

[ Parent | on HN ]

Re:Show HN: Search over 100M archived posts and comments scr... by nagyf on Saturday March 13, 2021 @11:32AM (#26447631)

S3 actually supports Torrent: https://docs.aws.amazon.com/AmazonS3/latest/userguide/S3Torr...

It might be the easiest solution for you

[ Parent | on HN ]

Re:Show HN: Search over 100M archived posts and comments scr... by cosmodisk on Saturday March 13, 2021 @06:05PM (#26450999)

I'm impressed that AWS supports torrent, however that won't reduce S3 hosting costs, as each new download would only increase it.

[ Parent | on HN ]