Hey HN,

Im an indiehacker who loves to solve for niche pain points

I realized I spend a lot of time just scrolling through subreddits looking for user complaints, insights, etc

I decided to have ChatGPT do the heavy lifting for me, and consolidate a more curated list of posts with insights for me to read

Stats:

Posts Processed: ~15k Insights Found: ~1K API Costs: ~$150

I hope a few of you may find this useful, would appreciate any feedback

~Justin

~1K API Costs: ~$150

$150 for 1000 API requests? Isn't that way too high?

ah sorry poor formatting. insights found was 1k. api costs were $150 for the 15k posts processed

Still, isn't that high? With $150 you can rent a powerful server for a month and have a big LLM like Llama 2 running on it.

yea definitely not cheap, to break it down further:

I first ran each post title through gpt-3 to classify as either likely or unlikely to be insightful to founders. Then, if it is likely to be insightful I pass the entire post content including up to 100 comments into gpt 4.

This comes down to around 13-14 million tokens used for the 15k posts, which for the latest gpt-4 model is ~$150.

Im sure this could be done cheaper with something like Llama 2 but tbh I am fine with this for now since I was aiming for speed with the initial list to gauge interest :)

I think the feedback you'll get is that ideas are fairly worthless. They are not validated with a market, and it is all about implementation anyway.

So this is the kind of thing I'd absolutely browse with interest. But there is zero chance I'd pay for it.

thanks for the insight! - yes absolutely this is not a list of validated ideas by any means, more so a feed of inspiration and hints towards market interest (if you find 5-6 reddit posts complaining about a similar thing it has higher likely hood of validation then random idea you think of in the shower). I'm working on a free plan which gives 10 a day to scroll through so currently looking at ways of increasing the value here beyond just the static list

I look at your screenshot list and those are huge problems with huge companies trying to solve them. This seems more like a big opportunities startup list than something for side projects. That may not be representative of what is in the database, but those are huge pain points, not niche problems.

yea great point, I chose those in the screenshot since they caught my eye when scrolling through. But in general yes I think i need to work on the database more to only contain the super niche opportunities

I didn't realize the first round that it was actually just a nice and neat list of scraped information, which I am quite happy to spend time going through. Just purchased a membership.

happy to hear! hoping to 10x the value of the product over the next few months so stay tuned :)

What sub-reddits were analyzed in this dataset?

the current list is ["startups, SaaS, bigdata, analytics, content_marketing, web_design, UXDesign, growthhacking, telecommuting, remotework, BusinessIntelligence, indiebiz, advertising, agency, PRConsulting, sales, AskMarketing, eventplanners, EventProduction, EventProfessionals, CustomerSuccess, userexperience, Marketresearch, VirtualAssistants, OfficeManagers, devops, sysadmin, webdev, LawFirm, legaltech, LegalAdvice, sustainability, Green, enviroaction, business, travelhacks, humanresources, InstructionalDesign, socialmedia, consulting, AskManagement, DataScience, networking, B2BForHire, retail, warehouse, QualityAssurance, AskHR, projectmanagement, Construction, ITManagers, SmallBusiness, CustomerService, Automate, cybersecurity, AskNetsec, finance, FinancialPlanning, Emailmarketing, marketing, TechRecruitment, BusinessProcess, techsupport, ITdept, copywriting, TeamBuilding, RemoteWork, smallbusiness, dataisbeautiful, visualization, ecommerce, EntrepreneurRideAlong, LeadGeneration"] and adding a bunch more in the coming weeks :)

"Hey ChatGPT, give me a list of 30 subreddits where I can find startup ideas"

(...)

(Beep. Boop.)

["r/startups", "r/leadership", "r/GrowMyBusiness", "r/venturecapital", "r/AdvancedEntrepreneur", "r/marketing", "r/LadyBusiness", "r/productivity", "r/socialmedia", "r/startups_promotion", "r/Entrepreneur", "r/EntrepreneurRideAlong", "r/Entrepreneurship", "r/advancedentrepreneur", "r/startup", "r/crowdspark", "r/sideproject", "r/InternetIsBeautiful", "r/Plugyourproduct", "r/Imadethis", "r/growthmemes"]