Re:Show HN: I built an API for agents visiting my personal w...
by dormento on Tuesday May 05, 2026 @01:31PM
(#48025729)
> Why your website is invisible to the agents visiting it.
I only _wish_ I could make a website invisible to agents. Instead now we're hostages to Cloudflare /shrug
Edit: so this isn't as "driveby-ish": the tldr; seems to be, if you want your site crawl-able, don't go overboard on the JS. This was good wisdom back then just as it is now in the current situation.
Re:Show HN: I built an API for agents visiting my personal w...
by selvmvde on Wednesday May 06, 2026 @11:48PM
(#48045187)
Fair point on llms.txt, but I think we're talking about two different layers here;
llms.txt helps agents find content -> API helps agents collaborate with a human
Re:Show HN: I built an API for agents visiting my personal w...
by joao_sobhie on Thursday May 07, 2026 @09:19AM
(#48049087)
Got you. You made me realize something. I've been developing a scraping api and after reading your post I tried to do the same as you did. First I've asked claude about the company, the services and prices. Than I've used the api to test if the response is the same and It's not. Basiclly using the api I got thinks that just asking claude, didn't show. The difference is that api turn the website into markdown and don't see the llms.txt. Using api looks like the AI understand the company as a whole.
Thanks for sharing that experiment, you gave me a good ideia on creating a brain for my scenario. I'm thinking of use Obsidian as the brain, and use my scraping api to colect the data from my company website, linkedin and instagram. With all the knoledge, use paperclip to orchestrate marketing, projects, fixes.
I will build that.
I only _wish_ I could make a website invisible to agents. Instead now we're hostages to Cloudflare /shrug
Edit: so this isn't as "driveby-ish": the tldr; seems to be, if you want your site crawl-able, don't go overboard on the JS. This was good wisdom back then just as it is now in the current situation.