cpeffer
Re:2markdown – Transform Websites into Markdown
on Wednesday May 01, 2024 @06:39PM (#40230518)
If you’re looking for an open core version of this check out firecrawl.dev
Show HN: From SERP to LLM-Ready Data Markdown in Seconds with Firesearch
Show HN: API to turn entire websites into Markdown
While building mendable - we found that feeding LLMs well-structured markdown improved accuracy. We also found it surprisingly hard.We found some great tools online, but none reliably handled the entire process. We wanted an API that took a URL, crawled the pages in the URL, and gave us an easy-to-use, up-to-date markdown we could feed into our index.So, we released an open-source repo and an...
Re:Show HN: I made a tool to clean and convert any webpage to Markdown
on Sunday April 14, 2024 @06:10PM (#40034921)
Very cool. We posted about a similar tool we built yesterday
It also crawls (although you can scrape single pages as well)
Re:Turn entire websites into LLM-ready data
on Saturday April 13, 2024 @08:27PM (#40027557)
It wasn't in our initial version (we didn't plan on launching today), but we are pushing an update to do so now.
Re:Turn entire websites into LLM-ready data
on Saturday April 13, 2024 @07:55PM (#40027352)
It crawls webpages (finds subdirectories), handles JS blocking with fallbacks to headless browsers, and does this all concurrently.
If only that script worked for every website. But, alas, it does not.
Re:Turn entire websites into LLM-ready data
on Saturday April 13, 2024 @07:44PM (#40027261)
* Creator here - Thats the goal!