News for hackers. Stuff that matters.
Re:Why Is the Web So Monotonous? Google by marginalia_nu on Thursday August 04, 2022 @06:10PM (#32348948)
> What I dream of is Google circa 2006. A time where a search engine searched what you asked for. A time before aggressive SEO. A time before social media, when the only people on the internet had a reason to be there. A time before sticky headers and full-screen modal pop-ups asking you to subscribe to a newsletter before reading the article. A time before click-bait and subscription-only websites which tease you with a paragraph before blurring out the rest of the content.

Well, I built sort of exactly this: https://search.marginalia.nu/

It's not great, but it sure has its moments.

Should be noted, regarding Koh Lanta, that travel is one of the most aggressively SEO-spammed topics, along with pharma and online casinos. It's extremely difficult to cut through the noise and reach any sort of signal.

Re:Why Is the Web So Monotonous? Google by varispeed on Thursday August 04, 2022 @06:44PM (#32349268)
Social media existed way before 2006, it just wasn't so popular.

But this was indeed a great time. Remember being on a social network ca 1999 and people used it as a diary where others could comment, give advice and meet others with similar interests. There was a timeline and posts from people you "followed". I think it didn't have likes or ratings though. Most people were super friendly and we were doing meet ups, so very much everyone knew each other. At peak time there was about 20k people and at that point owners couldn't cope with it. With that amount of people you'll find some nasty ones that will ruin it for everyone else and that's what happened. Owners couldn't afford to host it, so they were doing crowdfunding to cover the server costs and maintenance. Some bitter people didn't like that and started making unfounded accusations of theft or that they will report everything to the tax man and the police. So owners eventually closed it.

Re:Why Is the Web So Monotonous? Google by taftster on Thursday August 04, 2022 @06:14PM (#32348988)
[Quoting your quote] - A time before aggressive SEO? No, just that Google was able to defeat it with Page Rank (one of their possibly few innovative ideas to the search arena). Mathematically, Page Rank did very well to weed out the SEO'd websites of the time.

Point is, I don't think SEO at the time was any more or less "aggressive", per se. Just different. And clearly, the game has since escalated altogether.

Re:Why Is the Web So Monotonous? Google by onlyrealcuzzo on Thursday August 04, 2022 @06:14PM (#32348994)
> Well, I built sort of exactly this: https://search.marginalia.nu/

What's the elevator pitch? What do you actually do that is special?

From the about section:

> This is an independent DIY search engine that focuses on non-commercial content, and attempts to show you sites you perhaps weren't aware of in favor of the sort of sites you probably already knew existed.

> The software for this search engine is all custom-built, and all crawling and indexing is done in-house. The project is open source. Feel free to poke about in the source code or contribute to the development!

But that doesn't really give me much info.

Re:Why Is the Web So Monotonous? Google by marginalia_nu on Thursday August 04, 2022 @06:25PM (#32349094)
The point is to surface human websites and discriminate against content mills, click funnels and e-commerce.

For one I have a sort of budget for how much javascript I will tolerate. Some is fine, like your standard wordpress config probably will fly, but not much more. I also do a personalized pagerank biased toward the blogosphere. The likelihood your website shows up in the results is directly determined by whether real humans link to the website.

It's not a Google replacement by any measure, it's meant as a complement, but if you are looking for more in-depth content on a topic than you can find elsewhere, it's often a good starting point.

Re:Why Is the Web So Monotonous? Google by spacemadness on Thursday August 04, 2022 @08:40PM (#32350143)
I like the idea of a search engine that acts as a sort of filter for known dark patterns along with returning useful results. Limiting on javascript quantity seems like a sledgehammer approach, but I like the spirit of it all the same.
Re:Why Is the Web So Monotonous? Google by droidist2 on Thursday August 04, 2022 @09:03PM (#32350262)
> For one I have a sort of budget for how much javascript I will tolerate. Some is fine, like your standard wordpress config probably will fly, but not much more.

What about sites built with React, etc? Like would you be filtering out cool things like Ableton's "Learning Synths"[1] or do you mostly mean third party scripts?

[1] https://learningsynths.ableton.com

Re:Why Is the Web So Monotonous? Google by marginalia_nu on Thursday August 04, 2022 @10:10PM (#32350689)
Yeah, those sited go out the window entirely. It's a search engine for documents, not applications.
Re:Why Is the Web So Monotonous? Google by iakh on Thursday August 04, 2022 @10:35PM (#32350878)
Why should it matter if a website is marked up on the server side or client side to a search engine?
Re:Why Is the Web So Monotonous? Google by marginalia_nu on Friday August 05, 2022 @06:13AM (#32354012)
I don't know why it matters, but enforcing a JS budget turns out to rather effectively reduce the amount of problem websites down to nearly none.

Do babies go out with the bathwater? Almost certainly. But when you're running a small scale search engine, you're never going to index everything anyway, so that's entirely fine.

Re:Why Is the Web So Monotonous? Google by spc476 on Friday August 05, 2022 @03:59AM (#32353054)
Why should I download 50M of Javascript implementing a buggy, bespoke browsing engine to render perhaps 1k of text when I'm using a browser that is already capable of rendering 1k of text? For example, I used to enjoy browsing the C2 wiki, but recently they decided they needed to experiment with their own Javascript based browser, thus rendering their site unreadable unless you are using the very latest web browser (that needs to be updated every 20 minutes mind you).
Re:Why Is the Web So Monotonous? Google by joshlemer on Friday August 05, 2022 @12:58AM (#32351894)
Thanks for the link! I’ve tried it on a few queries and I really really like the results so far! Cheers!
Re:Why Is the Web So Monotonous? Google by tigerlily on Thursday August 04, 2022 @06:26PM (#32349106)
I know it’s of low value to ask, but who really gives a shit?
Re:Why Is the Web So Monotonous? Google by tigerlily on Thursday August 04, 2022 @11:19PM (#32351142)
Sigh, I guess I should’ve clarified I meant “who cares whether or not there’s an elevator pitch, this is just something that someone thought we all needed and would be cool to build”.

FWIW I think it is cool and much needed.

Re:Why Is the Web So Monotonous? Google by interroboink on Thursday August 04, 2022 @07:01PM (#32349397)
I do!

(also low value, perhaps :)

Re:Why Is the Web So Monotonous? Google by joshka on Friday August 05, 2022 @06:24AM (#32354082)
Neat, I wonder if applying something like GPT-3 to understand what each page gives might help with that a little. Kind of like reversing the SEO spam. I wonder if that's something that ML tools could detect given enough input?
Re:Why Is the Web So Monotonous? Google by marginalia_nu on Friday August 05, 2022 @06:30AM (#32354113)
My guess is doing exactly that is most the source of Google's problems right no, meanwhile the spammers do the exact same thing except trying to teach a ML model to successfully pose as authentic content, and what you get is the gloriosu mess we are in. (Even though it's just the last battle in a long war against spam)
Re:Why Is the Web So Monotonous? Google by joshka on Friday August 05, 2022 @06:41AM (#32354190)
Too much / too little, both are problematic I guess. This ends up becoming a war of algorithmic generation vs understanding, and devolves to effectively answering the age old question "is this spam"?

I was going to suggest that perhaps adding info about whether the site had ads, etc. might help the usability. Then I tried marginalia ... :) Perhaps we need to invest in similar detection for other antipatterns (listicles, blogspam etc.) - this is mostly what I was talking about. Less the understanding for indexing, and more the classification for user safety and authenticity of results.

But again, this is very cool.

Re:Why Is the Web So Monotonous? Google by marginalia_nu on Friday August 05, 2022 @06:51AM (#32354275)
Yeah maybe.

I don't know how useful these "hobo signs" actually are, but they did feel like a natural response to a lot of web sites justifying their use of trackers and cookies and affiliate links and whatnot with "everyone is doing this!"

Re:Why Is the Web So Monotonous? Google by carschno on Friday August 05, 2022 @03:40AM (#32352929)
Impressive! I'm genuinely curious: Where do you host the index and the DB, and how do you finance it? Hosting an index of that size on AWS is not cheap, and maintaining the server infrastructure yourself seems quite some effort.
Re:Why Is the Web So Monotonous? Google by marginalia_nu on Friday August 05, 2022 @06:11AM (#32353994)
It's literally hosted off a beefy PC in my living room. I could definitely not afford to host this on AWS or rented servers.

About ~5k USD total investment including a small UPS, operational costs are $40/mo. Doesn't actually require all that much maintenance, at least not compared to the work I've put into building a search engine (including crawler, index) from scratch.

Re:Why Is the Web So Monotonous? Google by Eratosthenes on Thursday August 04, 2022 @08:48PM (#32350186)
Brilliant! This is fantastic. I'm in love with it already :)