It's not that hard to roll your own web search MCP.

I made one for Crush a while ago.

https://anduil.neocities.org/blog/?page=mcp

I'm not sure about the issues with reddit though? Do they block Claude's web fetch tool? I think Codex runs it thru some kind of cache proxy.

Rolling your own is not the solution for the common case where you’re asking an LLM a question that may or may not be supported or supplemented by a web search. ChatGPT decides by itself when and how to consult the web, and then links the relevant sources in its result. You don’t get that functionality from Claude chat, you’d have to completely build your own chat harness and apps.

Sites like Reddit are blocking AI providers, they have to have some contract with them for access. OpenAI does seem to have that.

That's so frustrating with Claude. If I need to widely search the web or if I need it to read a specific URL I pasted, I always turn to ChatGPT. Claude seems to hit a lot more roadblocks while trying to navigate the web.

They're playing too nice. It's time to roll out the residential proxies.

The issue is Reddit though. They're the ones blocking. They're very aggressive.

When sites are working in one chatbot and not another, there's a good chance that the latter is respecting the website rules. As an example with Reddit, you're probably blocked when using a VPN like Mullvad

That’s a feature not a bug.

Why not just write a skill and script that calls crawl4ai or similar and do this using Claude code?

You can store the page as markdown for future sessions, mash the data w other context, you name it.

The web Claude is incredibly limited both in capability and workflow integration. Doesn’t matter if you’re dealing with bids from arbor contractors or researching solutions for a DB problem.

Want this w/o killing the free open web.

Maybe I run an old PC adjacent to the scraper to manually visit the scraped pages without an adblocker, & buy something I need from an ad periodically (while a cohesive response is being generated in the meantime)

Ya sounds dumb, wishing for a middle ground that lets us be effective but also good netizens. Maybe that Cloudflare plan to charge the bots…

See https://news.ycombinator.com/item?id=47208741.