Ask HN: What are you working on? (March 2025)

Re:Ask HN: What are you working on? (March 2025) by andrethegiant on Sunday March 30, 2025 @06:26PM (#43528339)

I'm working on pure.md[1], which lets your scripts, APIs, apps, agents, etc reliably access web content in markdown format. Simply prefix any URL with `pure.md/` and you get the unblocked markdown content of that webpage. It avoids bot detection and renders JavaScript-heavy websites, and can convert HTML, PDFs, images, and more into pure markdown.

pure.md acts as a global caching layer between LLMs and web content. I like to think of it like a CDN for LLMs, similar to how Cloudinary is a CDN for images.

[1] https://pure.md

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by wild_egg on Sunday March 30, 2025 @08:10PM (#43529178)

Thanks for sharing. I was planning on building something like this in April after hitting too many issues with Jina and Tavily but it looks like you've already done the hard work!

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by andrethegiant on Sunday March 30, 2025 @09:44PM (#43529828)

Thanks! Still a work in progress :-)

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by wanderingbit on Sunday March 30, 2025 @10:43PM (#43530285)

What a great idea, I will soon be a paying customer. This solves a problem of an app I'm using that I was hesitant to try to develop myself.

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by andrethegiant on Sunday March 30, 2025 @11:33PM (#43530618)

Much appreciated!

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by hardlyfun on Monday March 31, 2025 @12:17AM (#43530942)

Very nice, how did you manage to bypass sites with cloudflare turnstile setup?

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by udev4096 on Monday March 31, 2025 @01:22AM (#43531386)

Flaresolverr most probably

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by udev4096 on Monday March 31, 2025 @01:24AM (#43531399)

Good job on enshittifying the web even more. You must be proud. Also, from the looks of it, there is no difference between pure.md and any generic scraper. It's plain old tactics, designed for bloating the web with more slop

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by NationOfJoe on Monday March 31, 2025 @02:42AM (#43531880)

i have no skin in the game and honestly i am wondering how this idea contributes to enshittifying the web more?

this idea just seems like it provides the same content as visiting the site in a different view, like reader mode?

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by elric on Monday March 31, 2025 @08:00AM (#43533947)

I think the parent is referring to the goal of making the web more "LLM friendly".

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by hbsbsbsndk on Monday March 31, 2025 @08:53AM (#43534455)

The service seems designed to bypass anti-scraping measures. If site owners don't want their content scraped by AI this is subverting their will.

It also obfuscates responsibility between the AI vendor and the scraping service. One can imagine unethical AI providers using a series of ephemeral "gateways" to access content while avoiding any legal or reputational harm.

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by shoebham on Monday March 31, 2025 @02:07AM (#43531648)

Love the recursion redirect at pure.md/pure.md

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by andrethegiant on Monday March 31, 2025 @12:58PM (#43537184)

You found the easter egg!

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by m0rde on Monday March 31, 2025 @06:20AM (#43533233)

Is there an example we can see?

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by 27theo on Monday March 31, 2025 @06:26AM (#43533272)

https://pure.md/https://news.ycombinator.com/item?id=4353323...

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by WillAdams on Monday March 31, 2025 @09:04AM (#43534591)

It seems to miss URLs?

At: https://willadams.gitbook.io/design-into-3d/2d-drawing the links for:

https://mathcs.clarku.edu/~djoyce/java/elements/elements.htm...

https://mathcs.clarku.edu/~djoyce/java/elements/bookI/bookI....

https://mathcs.clarku.edu/~djoyce/java/elements/bookI/defI1....

are rendered as:

_Elements_ _:_ _Book I_ _:_ _Definition 1_

Maybe detect when a page is on gitbook or some other site where there is .md source on github or some other site and grab the original instead?

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by andrethegiant on Monday March 31, 2025 @12:01PM (#43536509)

By default, href values of <a> tags are removed, because they add significant token length without adding more context. Coming soon, you can specify a request header to set whether or not you want links removed from the response. Those underscores you mentioned are from the italics.

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by erekp on Monday March 31, 2025 @09:47AM (#43535014)

how do you exactly fallback to common crawl? isn't the cost to even hold and query common crawl insane?

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by andrethegiant on Monday March 31, 2025 @11:57AM (#43536466)

With AWS Athena, you can query the contents of someone else’s public S3 bucket. You pay per read, but if you craft your query the right way then it’s very inexpensive. Each query I run only scans about 1MB of data.

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by wfn on Tuesday April 01, 2025 @12:46PM (#43548910)

Since I was just looking at this accidentally, here are some examples of how to query at a ~cent-per-query cost level (just examples but quite illustrative): https://commoncrawl.org/blog/index-to-warc-files-and-urls-in...

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by metadat on Monday March 31, 2025 @09:06PM (#43541785)

Cool project!

Recently discussed, too: https://news.ycombinator.com/item?id=43462894 (10 comments)

[ Parent | on HN ]

Re:Ask HN: What are you working on? (March 2025) by sharpshadow on Tuesday April 01, 2025 @08:47AM (#43546152)

Works great on mobile thanks, helpful tool to bypass flaky websites, js and even some paywalls.

[ Parent | on HN ]