To put it somewhat in context, the two types of scrapers currently are traditional http client based or headless browser based. The headless browsers being for more advanced sites, SPAs where there isn't any server side rendering.
However headless browser scraping is in the order of 10-100x more time consuming and resource intensive, even with careful blocking of unneeded resources (images, css). Wherever possible you want to avoid headless scraping. LLMs are going to be even slower than that.
Fortunately most sites that were client side rendering only are moving back towards have a server renderer, and they often even have a JSON blob of template context in the html for hydration. Makes your job much easier!
I'm fast with Python scraping but for scraping one page ChatGPT was way, way faster. The biggest difference is it was quickly able to get the right links by context. The suit wasn't part of the link but was the header. In code I'd have to find that context and make it explicit.
It's a super simple html site, but I'm not exactly sure which direction that tips the balances.
Sure, it may be more resource intensive, but it's not slow by any means. Our users process hundreds of rows in seconds.
Indeed... and they could periodically do an expensive LLM-powered scrape like this one and compare the results. That way they could figure out by themselves if any updates to the traditional scraper they've written are required.
Notion does not have a good moat. The increase of AI usage isn't going to strengthen their moat, it's going to weaken it unless they introduce major changes and make it harder for people to transition content away from Notion.
There are a lot of middle men who are going to be shocked to find out how little people care about their layer when openAI can replace it entirely. You know that classic article about how everyone's biggest competitor is a spreadsheet? That spreadsheet just got a little bit smarter.
* Ensuring data accuracy (avoiding hallucination, adapting to website changes, etc.)
* Handling large data volumes
* Managing proxy infrastructure
* Elements of RPA to automate scraping tasks like pagination, login, and form-filling
At https://kadoa.com, we are spending a lot of effort solving each of these points with custom engineering and fine-tuned LLM steps.
Extracting a few data records from a single page with GPT is quite easy. Reliably extracting 100k records from 10 different websites on a daily basis is a whole different beast :)
The landing page does not provide nearly enough information on how it works in practice. Is it automated or is custom code written for each site?
Had a conversation last week with a customer that did exactly that - spent 15 minutes in chatGPT generating working Scrapy code. Neat to see people solve their own problem so easily but it doesn't yet erode our value.
I run https://simplescraper.io and a lot of value is integrations, scale, proxies, scheduling, UI, not-having-to-maintain-code etc.
More important than that though is time-saved. For many people, 15 minutes wrangling with ChatGPT will always remain less preferable than paying a few dollars and having everything Just Work.
AI is still a little too unreliable at extracting structured data from HTML, but excellent at auxiliary tasks like identifying randomized CSS selectors etc
This will change of course so the opportunity right now is one of arbitrage - use AI to improve your offering before it has a chance to subsume it.
I've been using Duckling [0] for extracting fuzzy dates and times from text. It does a good job but I needed a custom build with extra rules to make that into a great job. And that's just for dates, 1 of 13 dimensions supported. Being able to use an AI that handles them with better accuracy will be fantastic.
Does a specialised model trained to extract times and dates already exist? It's entity tagging but a specialised form (especially when dealing with historical documents where you may need Gregorian and Julian calendars).
The problem with many indie hackers is that they just build products to have fun and try to make a quick buck.
They take a basic idea and run with it, adding one more competitor to an already jamed market. No serious research or vision. So they get some buzz in the community at launch, then it dies off and they move on to the next idea. Rinse and repeat.
Rarely do they take the time to, for example, interview customers to figure out a defensible MOAT that unlocks the next stage of growth.
Those that do though usually manage to build awesome businesses. For example the guy who built browserbear also runs bannerbear which is one of the top tools in his category.
They key is to not stop at « code a fun project in a weekend » and actually learn the other boring parts required to grow a legit business overtime.
Source: I’m an indie hacker
A month or two ago, there was some drama (which I'm sure you've seen as well) about an IHer who found a copycat. I looked into it and it didn't seem like a copy at all, yet this person was complaining quite heavily about it. But I mean, it's the fundamental law of business, compete or die. If you can't compete, you're not fit to run your business, and others who can, will.
Yeah I think some people confuse copycats with competitors:
- Copycats who just flat out copy your design / messaging / landing page: that's something to complain about
- Someone doing a product that solves a similar problem but build their own solution and design: that's perfectly normal and acceptable
More generally, I wonder how a lot of smaller startups will fare once OpenAI subsumes their product. Those who are running a product that's a thin wrapper on top of ChatGPT or the GPT API will find themselves at a loss once OpenAI opens up the capability to everyone. Perhaps SaaS with minor changes from the competition really were a zero-interest-rate phenomenon.
This is why it's important to have a moat. For example, I'm building a product that has some AI features (open source email (IMAP and OAuth2) / calendar API), but it would work just fine even without any of the AI parts, because the fundamental benefit is still useful for the end user. It's similar to Notion, people will still use Notion to organize their thoughts and documents even without their Notion AI feature.
Build products, not features. If you think you are the one selling pickaxes during the AI gold rush, you're mistaken; it's OpenAI who's selling the pickaxes (their API) to you who are actually the ones panning for gold (finding AI products to sell) instead.