andrew_zhong

Show HN: Resurf – realistic, reproducible test framework for AI browser agents

Posted by andrew_zhong on Thursday May 07, 2026 @04:41PM
from the show-hn dept.

Systematic testing of browser agents today is not easy: testing on real websites is flaky, rate-limited and potentially expensive (e.g. using proxies or bypassing Captcha), while static-HTML benchmarks lack state and dynamic behavior.Resurf gives your browser agent a realistic, stateful, instrumented framework — built on synthetic websites with failure-mode injection:- Realistic, dynamic,...

( Read More... | no comments | 4 points | on HN )

Show HN: HTML to Markdown with CSS selector & XPath annotations for LLM Scraper

Posted by andrew_zhong on Sunday April 05, 2026 @11:58PM
from the show-hn dept.

https://github.com/lightfeed/scrapedown

HTML-to-Markdown converters produce clean, readable content for both humans and LLMs — but the DOM structure is lost along the way. You can always feed Markdown to an LLM to extract structured information, but that costs tokens on every page, every time.What if the LLM could also see where each piece of content lives in the DOM? Then it can generate robust scraping code — stable selectors and...

( Read More... | no comments | on HN )

Re: on Saturday March 28, 2026 @12:15PM (#47555926)

You can use a browser automation library for rendering JS + interaction (like click collapse button) and then extract the HTML after interaction. Here is an example of using a AI browser automation library with prompt to interact (but you can also use playwright if you know the exact element to interact)

https://github.com/lightfeed/extractor?tab=readme-ov-file#us...

Re: on Thursday March 26, 2026 @02:55PM (#47534258)

[Update]] I will replace the stealth browser with plain playwright and remove anti-bot as a feature.

Re: on Thursday March 26, 2026 @02:54PM (#47534243)

I hear you loud and clear - will replace the stealth browser with plain playwright and remove anti-bot as a feature.

Re: on Thursday March 26, 2026 @02:53PM (#47534228)

[Update] I will replace the stealth browser with plain playwright and remove anti-bot as a feature.

Re: on Thursday March 26, 2026 @11:26AM (#47531689)

I agree that When pages have similar structure, for one time extraction as it is (not reasoning from context), scraping with selectors is the way to go.

This library also supports HTML as input so running a browser is not required.

Re: on Thursday March 26, 2026 @11:22AM (#47531638)

What kind of LLMs are you using? In structured output mode?

In this library we recover nullable and optional fields, invalid elements in nested array, bad urls, repair incomplete JSONs. If these issues are what you see, yes it should work for your case.

Re: on Thursday March 26, 2026 @06:06AM (#47528542)

Put things to perspective - Gemini 2.5 flash is 0.3/1M tokens - assuming each page is 700 tokens and output is not much you are looking at $210 for 1M pages

Re: on Thursday March 26, 2026 @05:57AM (#47528488)

Agreed - in this project I did a one path sanitation to recover invalid optional / nullable fields or discard invalid objects in nested array.

I know multi path LLM approaches exist: e.g. generating JSON patches

https://github.com/hinthornw/trustcall

Re: on Thursday March 26, 2026 @05:37AM (#47528375)

We do see fewer invalid JSONs on latest bigger LLMs but still can happen on smaller and cheaper models. There is also case when input is truncated or a required field not found, which are inherently difficult.

On XML vs JSON, I think the goal here is to generate typed output where JSON with zod shines - for example the result can type check and be inserted to database typed columns later

Re: on Thursday March 26, 2026 @04:45AM (#47528028)

HTML -> markdown -> LLM is standard practice. We strip elements like aside, embed, head , iframe etc. the criteria is conservatively set to avoid removing too many elements (especially in extractMain mode)

https://github.com/lightfeed/extractor/blob/main/src/convert...

I have used gemma 3 and had good results.

Once Gemini 3 flash drops the preview suffix, will update the examples. Thank you for the pointer.

Re: on Thursday March 26, 2026 @04:01AM (#47527781)

In context of e-commerce web extraction, invalid JSON can occur especially in edge cases, for example:

price: z.number().optional() -> price: “n/a”

url: z.string().url().nullable() -> url: “not found”

It can also be one invalid object (e.g. missing required field, truncated input) in an array causing the entire output to fail.

The unique contribution here is we can recover invalid nullable or optional field, and also remove invalid nested objects in an array.

Re: on Thursday March 26, 2026 @03:31AM (#47527617)

I will add a PR to enforce robots.txt before the actual scraping.

Re: on Thursday March 26, 2026 @03:27AM (#47527589)

We do respect robots.txt production - also scraping browser providers like BrightData enforces that.

I will add a PR to enforce robots.txt before the actual scraping.

Re: on Thursday March 26, 2026 @02:21AM (#47527180)

The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — fixing CDP leaks, removing automation flags, etc. For sites behind Cloudflare or Datadome, that alone usually isn't enough — you'll need residential proxies and proper browser fingerprints on top. The library supports connecting to remote scraping browsers via WebSocket and proxy configuration for those cases.

Re: on Thursday March 26, 2026 @02:03AM (#47527090)

Yeah that's a good observation. XML's closing tags give the model structural anchors during generation — it knows where it is in the nesting. JSON doesn't have that, so the deeper the nesting the more likely the model loses track of brackets.

We see this especially with arrays of objects where each object has optional nested fields. For complex nested objects, the model can get all items well formatted but one with an invalid field of wrong type. That's why we put effort into the repair/recovery/sanitization layer — validate field-by-field and keep what's valid rather than throwing everything out.

Re: on Thursday March 26, 2026 @02:02AM (#47527075)

Good point. The anti-bot patches here (via Patchright) are about preventing the browser from being detected as automated — things like CDP leak fixes so Cloudflare doesn't block you mid-session. It's not about bypassing access restrictions.

Our main use case is retail price monitoring — comparing publicly listed product prices across e-commerce sites, which is pretty standard in the industry. But fair point, we should make that clearer in the README.

Re: on Thursday March 26, 2026 @02:01AM (#47527073)

Show HN: Robust LLM Extractor for HTML/Markdown in TypeScript

Posted by andrew_zhong on Wednesday May 14, 2025 @11:29AM
from the show-hn dept.

https://github.com/lightfeed/lightfeed-extract

While working with LLMs for structured web data extraction, we saw issues with invalid JSON and broken links in the output. This led me to build a library focused on robust extraction and enrichment:- Clean HTML conversion: transforms HTML into LLM-friendly markdown with an option to extract just the main content - LLM structured output: Uses Gemini 2.5 flash or GPT-4o mini to balance accuracy...

( Read More... | no comments | 2 points | on HN )

Re:Show HN: LLM-powered News Hub, e.g "Games, science or open source" ... on Friday May 17, 2024 @05:19PM (#40394311)

Thank you for the feedback. Our intention is to let users try for 10 days without commitment (no credit card required), so that after 10 days they can choose to subscribe or not.

I see your confusion on pricing. It will be a paid service given the storage and LLM cost. I put "10 day free trial" under the main action button. Hope it can clarify a bit.

Show HN: LLM-powered News Hub, e.g "Games, science or open source" on HN

Posted by andrew_zhong on Friday May 17, 2024 @04:53PM
from the show-hn dept.

https://www.lightfeed.ai

We created LightFeed to transform any websites into lightweight and focused news feed. Use LLM to sort and summarize posts using user's custom prompt. For example, you can get anything related to "Games, science or open source" on HN.The news feed is automated daily and you can receive it in browser, email or RSS.We will also open source the LLM web parser/sort library soon o...

( Read More... | 3 comments | 3 points | on HN )

Re:Jina Reader: Convert any URL to an LLM-friendly input on Sunday April 21, 2024 @03:03PM (#40108256)

Have you used them? They offers it for free and I don't find their website mentioning API limit or whether they support dynamic JS

Re:Show HN: LLM Scraper – turn any webpage into structured data on Sunday April 21, 2024 @02:29PM (#40108005)

Great work! I’ve worked on the same problem and used LLM to extract feed into structured data (in my case have to use a more affordable model like GPT3.5 for a Saas app, looking at llama3 now)

Have you thought about automatically extract schema?

Re:Show HN: LLM Scraper – turn any webpage into structured data on Sunday April 21, 2024 @02:20PM (#40107935)

I’ve worked on this exact problem when extracting feeds from news websites. Yes calling LLM each time is costly so I use LLM for the first time to extract robust css selectors and the following times just relying on those instead of incurring further LLM cost.

Re:Ask HN: Is there a way to customize the stories on Hacker News? on Saturday April 20, 2024 @04:33PM (#40100784)

If you are looking for broader less heuristic based filter, you can try LightFeed (full disclosure, I am the creator of LightFeed)

It filters and summarizes stories using LLM and your prompt on any site including HN!

Ask HN: Best practice to use Llama 3 8B on production server

Posted by andrew_zhong on Friday April 19, 2024 @02:48PM
from the ask-hn dept.

The new Llama 3 8b in on par with 22b models, better but could be 10x cheaper than GPT3.5AI builders, if you are using Llama 3 in backend, where do you host it or what API do you use? (For production usecases with good speed and rate limits close to ChatGPT or Claude)-AWS sagemaker-Self host on cloud GPUs-Replicate API (just found them, 0.05/1m token, legit?)-AWS bedrock (seems pricy)-Others -...

( Read More... | 1 comment | 3 points | on HN )

Re:Ask HN: Where to put AI summary and filter in custom RSS feed on Thursday April 18, 2024 @10:02PM (#40082764)

I got reply from someone in Reddit who mentioned that feed readers will just ignore the custom namespaces. SO I think the best option (but hacky) will be to put the summary into the item description HTML...

Hope there are cleaner ways

Ask HN: Where to put AI summary and filter in custom RSS feed

Posted by andrew_zhong on Thursday April 18, 2024 @08:13PM
from the ask-hn dept.

Hi folks,I am working on an AI newsfeed app. It uses LLM to summarize and filter stories on any website or RSS. When exporting this custom AI feed to custom RSS (so that users on RSS readers can subscribe them), I am wondering which RSS field should I use for AI summary, the summary prompt or AI filter. Namely:1. AI summary per top feed links (AI will summarize the page pointed by the link...

( Read More... | 1 comment | 2 points | on HN )