News for hackers. Stuff that matters.
Re:The internet is an SEO landfill by nitrogen on Sunday June 23, 2019 @01:59PM (#20257420)
The original designers of the web envisioned a world where every user could have their own chosen "agent" to navigate the web for them. This is where the User-Agent HTTP header comes from. There would be search agents, data mining agents, and interactive browsing agents.

We missed out on like 90% of what the information revolution was supposed to bring to users, and instead the data giants have captured all the value.

Re:The internet is an SEO landfill by mostlyjason on Sunday June 23, 2019 @02:21PM (#20257623)
We expect everything on the web, including our user agent, to be free as in no cost. The only way to pay for the development cost is to sell to third parties aka advertisers. If people actually paid money to develop more advanced user agents there would be a market for them.

Why do people prefer free over paid? I think people underestimate the influence of advertisers and underestimate the value of their own time spent researching and dealing with crappy products. It’s almost like a psychological bias.

I like the idea of a better user agent but would I pay for one? Judging by my refusal to pay for quality journalism I’m guessing it’d be a hard sell. That’s another thing I should be willing to pay for but don’t for some weird reason.

Re:The internet is an SEO landfill by wbl on Sunday June 23, 2019 @03:12PM (#20258021)
The guy who posted the story of splicing a 500kW cable wasn't looking for money. He wanted to amuse people and thanks to him we got amused endlessly. The costs of putting things online are $5 per month at Digital Ocean.
Re:The internet is an SEO landfill by paulryanrogers on Sunday June 23, 2019 @03:38PM (#20258184)
> The costs of putting things online are $5 per month at Digital Ocean.

Plus the knowledge you can, and non-trivial technical expertise like: registering a domain, DNS configuration, setting up web software (even if "one click"), and maintaining it over time.

Re:The internet is an SEO landfill by stordoff on Sunday June 23, 2019 @05:05PM (#20258653)
> the story of splicing a 500kW cable

Do you have a link to this? I've tried googling "splicing a 500kW cable" (and variants) and haven't found much relevant.

Re:The internet is an SEO landfill by wbl on Sunday June 23, 2019 @07:00PM (#20259283)
Re:The internet is an SEO landfill by dasv on Monday June 24, 2019 @04:15AM (#20261454)
Warning (NSFW?): if you're coming from HN, the link redirects to this somewhat entertaining image https://imgur.com/32R3qLv
Re:The internet is an SEO landfill by c22 on Sunday June 23, 2019 @07:55PM (#20259499)
> We expect everything on the web, including our user agent, to be free as in no cost.

I don't think this was an accident. Back in the 90's I paid for Netscape Navigator and for email. People used to even pay money to indexed by search engines (like paying to be in the Yellow Pages). Companies started "giving these things away" in a calculated move that I guess the nascent web population was just not cynical enough to reject.

Re:The internet is an SEO landfill by nitrogen on Sunday June 23, 2019 @06:19PM (#20259047)
Web browsers used to cost money. I vaguely remember getting a boxed copy of IE from an MS rep that came to my school as a kid. ISPs started bundling Netscape if I remember right, then MS started giving IE away for free with the OS. In the end we got Firefox, so I'm glad for that, but we can blame the browser wars in part for the unwillingness to pay for agent software.
Re:The internet is an SEO landfill by 1123581321 on Sunday June 23, 2019 @02:46PM (#20257797)
If you use a Mac, DevonThink and its associated products are great examples of paid user agents that seek out and organize information for you. Their free trials are fairly generous.
Re:The internet is an SEO landfill by Filligree on Sunday June 23, 2019 @03:13PM (#20258025)
I don't use a mac. Is there any equivalent for other platforms?
Re:The internet is an SEO landfill by 1123581321 on Sunday June 23, 2019 @08:03PM (#20259526)
The closest I’ve ever found is https://www.zootsoftware.com/ - famously championed by James Fallows of The Atlantic. I don’t believe its web crawling tools are as sophisticated as Devon Technologies’, but it can do a lot and has a lot of power tools to process what it ingests. I haven’t used it since 2010, so my information may be out of date.