News for hackers. Stuff that matters.
Re:Show HN: Data on Shopify Stores by rajool on Thursday February 04, 2021 @10:13AM (#26025961)
Hi Everyone! I’m Ali, co-founder of Shopgram. We recently launched Shopgram (https://shopgram.io) which gathers, analyses and publishes data about Shopify stores to help merchants having a successful business.

My team and I released a Shopify app to help merchants boost their sales. Upon launch, we realized we need more data about Shopify stores. We crawled more than 1 million shops and assumed it might be useful for merchants too. With online shopping growing massively in the last few years and the additional acceleration due to the pandemic, more users have headed towards online shopping. As a result, Shopify, a platform for building online stores expedited its growth. There are many merchants who would benefit from the data about trending Shopify stores and products, so we published the gathered data for free and it was well received by Product Hunters. Learning more about Shopify stores made us wonder what challenges shop owners deal with and what their major problems are. Throughout dealing with the challenges, we took a step back and put ourselves in a new merchant or to-be merchant position, searching for the most popular niche markets or products to sell. Price, margin, quality, delivery time, etc., can affect the product merchants are trying to sell on their website.

For this matter, first we identified Shopify-made stores, gathered their data then cleaned them. Same products with different names and vast merchandise made the process hard. We solved this by applying AI tools and Machine Learning to classify the products. e crawled and categorized stores based on products they sell so that one can explore stores or products in a category. At the end we also added a search ability on the processed data. You can see the results at https://shopgram.io

Shopgram has three main parts for addressing the challenges merchants face. Stores - https://shopgram.io/stores Best Shopify stores created by Shopify per product categories based on geographic information. Products: https://shopgram.io/products Popular products sold in Shopify are shown here. We’ve also added features to provide more metadata on them, how to procure such products and finally what are their similar products. Insights: https://insights.shopgram.io Statistics which helps merchants choose the niche market to increase their sales. We’ll be pleased if you can take a look at Shopgram and share your thoughts on how we can improve the site and make it more useful.

Re:Show HN: Data on Shopify Stores by madelyn on Thursday February 04, 2021 @06:44PM (#26031155)
Hi Ali! I'm a software engineering lead working in the Shopify ecosystem as well.

This is an awesome resource, I just shared it around at my company. It's a lot of data to crawl and no joke!

Thank you so much!

Re:Show HN: Data on Shopify Stores by rajool on Friday February 05, 2021 @09:51AM (#26036863)
Hi Madelyn, Thanks for your comment I would be gald if we could have a talk. My email is myname at shopgram dot io
Re:Show HN: Data on Shopify Stores by moretta on Thursday February 04, 2021 @12:39PM (#26027701)
Seems interesting. So I suppose your focus long term is on the products side of thing only, to allow merchants to work out which products they should sell? Do you plan on adding any other insights for data such as shipping etc?
Re:Show HN: Data on Shopify Stores by rajool on Thursday February 04, 2021 @01:00PM (#26027951)
That's right. We are going to help them with choosing best vendors and channel, to supply high quality products. That's an interesting idea to add insights about shipping too. Thanks for your comment.
Re:Show HN: Data on Shopify Stores by type0 on Thursday February 04, 2021 @03:29PM (#26029417)
It's a nitpick but to me the title on the page reads like shoporam, "g" looks more like "o".
Re:Show HN: Data on Shopify Stores by rajool on Friday February 05, 2021 @09:53AM (#26036888)
Ooh I see :) Thanks for sharing it with me.
Re:Show HN: Data on Shopify Stores by mritchie712 on Thursday February 04, 2021 @11:18AM (#26026723)
how'd you get the list of shopify merchants to start with?
Re:Show HN: Data on Shopify Stores by rajool on Thursday February 04, 2021 @11:42AM (#26026998)
We've extracted a list of IP addresses for shopify address (by checking whois on a relatively small number of stores we already know they're on Shopify) Then we've used some reverse whois service (like https://myip.ms) to find other domains that refer to the same IP. Those stores are definately on Shopify as their domain points to an IP address that belongs to Shopify.
Re:Show HN: Data on Shopify Stores by abusaidm on Thursday February 04, 2021 @03:03PM (#26029163)
This is really interesting. Thank you for the explanation.

I'd love to hear about how you crawl and store the data. Doesn't shopify block you as you are hitting there servers soo much for all the site?

Re:Show HN: Data on Shopify Stores by rajool on Saturday February 06, 2021 @01:27AM (#26044594)
Till now, No :)
Re:Show HN: Data on Shopify Stores by armenarmen on Thursday February 04, 2021 @11:34AM (#26026912)
Crawling all subdomain.myshopify.com I’d expect
Re:Show HN: Data on Shopify Stores by bilekas on Thursday February 04, 2021 @11:43AM (#26027009)
Isn't this limited to the guestimated subdomain ?
Re:Show HN: Data on Shopify Stores by rajool on Thursday February 04, 2021 @11:43AM (#26027007)
Reverse IP Lookup
Re:Show HN: Data on Shopify Stores by mtgentry on Thursday February 04, 2021 @12:38PM (#26027691)
Would love to see a list of all the stores in a particular category. Would pay for that. I know there are lists you can buy but your data seems more in-depth.
Re:Show HN: Data on Shopify Stores by rajool on Thursday February 04, 2021 @12:58PM (#26027924)
Glad to see you liked it. We'll add more info to it in near future too.
Re:Show HN: Data on Shopify Stores by ultrasounder on Friday February 05, 2021 @11:46AM (#26038243)
I thought you could search by store category already!
Re:Show HN: Data on Shopify Stores by rajool on Saturday February 06, 2021 @01:25AM (#26044585)
+1
Re:Show HN: Data on Shopify Stores by errantmind on Thursday February 04, 2021 @07:20PM (#26031438)
Cool tool, I'd be interested to see more categories, specifically for vintage and antique items (please note the distinction as well). These sellers are mostly on shopify these days and the community around them is growing fast as it bridges costumers, collectors, cosplayers, and many others
Re:Show HN: Data on Shopify Stores by rajool on Saturday February 06, 2021 @01:24AM (#26044577)
Glad to see you liked it.

You could search for any categories already. Wasn't it useful?

Re:Show HN: Data on Shopify Stores by glaucon on Thursday February 04, 2021 @06:42PM (#26031142)
Very interesting thanks. For future developments enhancing the search to allow exclusions would be good. I'm currently searching for "cotton sun hat" but I get masses of caps (which I don't want) so being able to do this "cotton sun hat brim -cap" would be great.
Re:Show HN: Data on Shopify Stores by rajool on Saturday February 06, 2021 @01:18AM (#26044563)
That's great.

Thanks for your feedback.

Re:Show HN: Data on Shopify Stores by bilekas on Thursday February 04, 2021 @11:42AM (#26027004)
I'm curious how you're establishing the location of the stores.. There seem to be a lot from JP that are surprising.

Are the locations simply the DNS location?

Also, I guess what's the aim ?

Re:Show HN: Data on Shopify Stores by rajool on Thursday February 04, 2021 @11:57AM (#26027192)
We are using the data crawled from the homepage together with currency information and estimate the location based on them.
Re:Show HN: Data on Shopify Stores by edoceo on Thursday February 04, 2021 @07:49PM (#26031718)
Could be you looked like JP to the store because your crawler was in JP and shopify is locale aware.
Re:Show HN: Data on Shopify Stores by rajool on Saturday February 06, 2021 @01:25AM (#26044583)
:-?

I'll check.

Re:Show HN: Data on Shopify Stores by ibdf on Thursday February 04, 2021 @12:15PM (#26027418)
this is actually a cool way to find online stores. I would recommend a link to the store right from the list, instead of having to go to the details page.
Re:Show HN: Data on Shopify Stores by rajool on Thursday February 04, 2021 @12:22PM (#26027489)
Thanks for your recommendation, got it.
Re:Show HN: Data on Shopify Stores by qnsi on Saturday February 06, 2021 @11:14AM (#26047470)
how do you learn what popular products are?
Re:Show HN: Data on Shopify Stores by qnsi on Saturday February 06, 2021 @11:15AM (#26047479)
also what happens after sign in? Is there any need to sign in?