Show HN: I scraped 25M Shopify products to build a search engine

Re:Show HN: I scraped 25M Shopify products to build a search... by muratsu on Wednesday December 13, 2023 @06:44PM (#38635854)

Agora also doesn't return red shoes for the search query "red shoes". Seems like you haven't fully solved the problem yet :)

From a technical perspective, crawling 25M products is impressive but the search itself doesn't provide much value to me. I already use large e-commerce sites (amazon, wallmart, ...) and targeted ones (Nordstrom, SSENSE, ...). Sure I may not be searching through all the shopify, wix stores but I need to know why that's valuable to me to begin with. Perhaps understanding the value prop of SMBs and educating me about it would be a better positioning for Agora than simply being a search engine.

[ Parent | on HN ]

Re:Show HN: I scraped 25M Shopify products to build a search... by pencildiver on Wednesday December 13, 2023 @06:58PM (#38635988)

Definitely have not solved the problem yet! The search algorithm prioritizes the brand called "Red Wing Shoe" so still figuring out ways to show real 'red shoes'. Have been thinking about passing the images through a detection tool and tag them to enhance the search experience.

Re: Value Proposition. Absolutely, I think focusing on the SMB-angle and 'local shopping' will help direct users better. I'll definitely take this into account.

[ Parent | on HN ]

Re:Show HN: I scraped 25M Shopify products to build a search... by paulddraper on Wednesday December 13, 2023 @07:21PM (#38636196)

Best of luck on your marriage

[ Parent | on HN ]

Re:Show HN: I scraped 25M Shopify products to build a search... by charliejuggler on Monday December 18, 2023 @09:10AM (#38682744)

You could try the method we used for our vector search demo for e-commerce (all open source, natch) - use CLIP to get vector embeddings for product pictures and then use these for boosting or matching. https://opensourceconnections.com/blog/2023/03/22/building-v... Our demo works pretty well for searches like 'blue network cable' when the colour isn't always explicitly mentioned in the product data.

[ Parent | on HN ]