Show HN: I scraped 200M Shopify products to build a search engine

Re:Show HN: I scraped 200M Shopify products to build a searc... by LuigiElsa on Thursday February 22, 2024 @04:48AM (#39465093)

Well done! A lot of progress since last time. Have you guys considered using AI to categorise products (ie; create labels using product images), instead of using the text to match the search? I say this cause I sometimes see some irrelevant products and I can tell you guys are basing the search on text

[ Parent | on HN ]

Re:Show HN: I scraped 200M Shopify products to build a searc... by pencildiver on Thursday February 22, 2024 @05:07AM (#39465197)

Thanks! Definitely a work in progress but getting better by the day.

We have run tests with image-detection to try to categorize products. We currently do search based on the name, description, price, store, and brand.

The problem with image-detection is cost. Given the size of our data set, it's very costly to run 800m - 1b images through a model (i.e. most products have 4 - 5 images). We've considered only doing the first 'hero' image to start though. Open to any cost-effective ideas though.

For example, if you search for "wooden chair", it would be nice to select a filter for 'category' to narrow down if I want to see "office furniture", "dining room", or "art".

https://www.searchagora.com/search?query=wooden+chair&count=...

[ Parent | on HN ]

Re:Show HN: I scraped 200M Shopify products to build a searc... by LuigiElsa on Thursday February 22, 2024 @05:47AM (#39465375)

I found some things on Github you could use, I'm not a dev myself and I'm not sure how scalable these are, but have a look, maybe there's something useful. https://github.com/jhc13/taggui

The category filtering is what I wanted to get at, I think the search would improve a lot.

[ Parent | on HN ]

Re:Show HN: I scraped 200M Shopify products to build a searc... by pencildiver on Thursday February 22, 2024 @06:20AM (#39465561)

Super cool, thanks! Will check it out.

[ Parent | on HN ]