The original designers of the web envisioned a world where every user could have their own chosen "agent" to navigate the web for them. This is where the User-Agent HTTP header comes from. There would be search agents, data mining agents, and interactive browsing agents.

We missed out on like 90% of what the information revolution was supposed to bring to users, and instead the data giants have captured all the value.

We expect everything on the web, including our user agent, to be free as in no cost. The only way to pay for the development cost is to sell to third parties aka advertisers. If people actually paid money to develop more advanced user agents there would be a market for them.

Why do people prefer free over paid? I think people underestimate the influence of advertisers and underestimate the value of their own time spent researching and dealing with crappy products. It’s almost like a psychological bias.

I like the idea of a better user agent but would I pay for one? Judging by my refusal to pay for quality journalism I’m guessing it’d be a hard sell. That’s another thing I should be willing to pay for but don’t for some weird reason.

The guy who posted the story of splicing a 500kW cable wasn't looking for money. He wanted to amuse people and thanks to him we got amused endlessly. The costs of putting things online are $5 per month at Digital Ocean.

> The costs of putting things online are $5 per month at Digital Ocean.

Plus the knowledge you can, and non-trivial technical expertise like: registering a domain, DNS configuration, setting up web software (even if "one click"), and maintaining it over time.

> the story of splicing a 500kW cable

Do you have a link to this? I've tried googling "splicing a 500kW cable" (and variants) and haven't found much relevant.

https://www.jwz.org/blog/2002/11/engineering-pornography/ is the version I've found.

Warning (NSFW?): if you're coming from HN, the link redirects to this somewhat entertaining image https://imgur.com/32R3qLv

> We expect everything on the web, including our user agent, to be free as in no cost.

I don't think this was an accident. Back in the 90's I paid for Netscape Navigator and for email. People used to even pay money to indexed by search engines (like paying to be in the Yellow Pages). Companies started "giving these things away" in a calculated move that I guess the nascent web population was just not cynical enough to reject.

Web browsers used to cost money. I vaguely remember getting a boxed copy of IE from an MS rep that came to my school as a kid. ISPs started bundling Netscape if I remember right, then MS started giving IE away for free with the OS. In the end we got Firefox, so I'm glad for that, but we can blame the browser wars in part for the unwillingness to pay for agent software.

If you use a Mac, DevonThink and its associated products are great examples of paid user agents that seek out and organize information for you. Their free trials are fairly generous.

I don't use a mac. Is there any equivalent for other platforms?

The closest I’ve ever found is https://www.zootsoftware.com/ - famously championed by James Fallows of The Atlantic. I don’t believe its web crawling tools are as sophisticated as Devon Technologies’, but it can do a lot and has a lot of power tools to process what it ingests. I haven’t used it since 2010, so my information may be out of date.

> I would pay a massive amount of money for a search engine with a “no commercial results” switch.

Yeah but what does that even mean. How do you define commercial? Who is going to go through all of the pages and categorizing them into commercial and non-commercial? And how do you keep shills and “influencers” from littering your results with seemingly innocent but actually commercial content?

Agreed that it’s semantically tricky but if you give me a bag of heuristics to work with it’d be worlds better than what we have now.

Keep in mind there is incentive to have a commercial meta tag so that when I’m actually looking for a good or service I’d be able to easily find it (or see the lack of it). The other 99% of the time ads and products are just an annoying distraction and a waste of money to serve.

TBH google kind of sucks for looking for stuff, amazon + reddit recommendations are my main source of surfacing things I want to buy.

Here's an idea : do the 0.1 version by crawling and indexing the web in a perfectly standard way, apply the usual, run-of-the-mill search algorithms, then diff it against the Bing/Google/etc first results which will be biased towards commercial stuff. The higher a page is among their results, the higher the probability that it's commercial in some sense.

So...you want to build a Google except each query starts on the last page of results?

No, it's just a simple heuristic to get started.