Show HN: I scraped 25M Shopify products to build a search engine

Re:Show HN: I scraped 25M Shopify products to build a search... by mfrye0 on Wednesday December 13, 2023 @09:28PM (#38637076)

If you're already on AWS, I recommend switching to postgres for now. For context, I have 3 RDS instances, each multi zone, with the biggest instance storing several billion records. My total bill for all 3 last month was $661.

Postgres has full text search, vector search, and jsonb. With jsonb you can store and index json documents like you would in Mongo.

- https://www.postgresql.org/docs/current/textsearch.html - https://aws.amazon.com/about-aws/whats-new/2023/05/amazon-rd...

[ Parent | on HN ]

Re:Show HN: I scraped 25M Shopify products to build a search... by neeleshs on Thursday December 14, 2023 @01:53AM (#38638564)

how big is the disk for the biggest instance?

[ Parent | on HN ]

Re:Show HN: I scraped 25M Shopify products to build a search... by mfrye0 on Thursday December 14, 2023 @06:01PM (#38648603)

Pretty small still at 500gb. It only stores hot data right now and a subset of what's important. Most of our data is in S3.

[ Parent | on HN ]

Re:Show HN: I scraped 25M Shopify products to build a search... by LunaSea on Thursday December 14, 2023 @03:11AM (#38638991)

I have troubles seeing how this is possible.

$220 dollars per instance gets you 8Gb of RAM which is way, way, below the index size if you are indexing billions of vectors.

[ Parent | on HN ]

Re:Show HN: I scraped 25M Shopify products to build a search... by philippemnoel on Thursday December 14, 2023 @12:55PM (#38644546)

You can even do Elastic-level full text search in Postgres with pg_bm25 (disclaimer: I am one of the makers of pg_bm25). Postgres truly rules, agree on the rec :)

[ Parent | on HN ]