In my experience, the forum experience is far from dead, but it's effectively impossible to surface in a search engine - any search engine - unless you know the name of the forum.

Oh, and the content must also be "fresh". If the content isn't "fresh" (which most of the best forum/blog posts are not), nobody shows it anymore. I can search for a specific blog post using a verbatim quote, but the result (if it exists) is buried under 10+ pages of "fresher" content, no matter how disconnected it may be from the search.

The forum experience is dying. I spent about 4 years of my time in-between Google stints working on a searchable feed for forum sites. Finally gave it up when I realize the extent to which the forum scene had died and moved to Reddit & Facebook while I was working on the project.

The root problem is that attention has gone from abundant to scarce, and people already have their habits. That makes it really hard to build a new forum site and attract an audience that's willing to type your URL in every day (and if they don't visit daily, forget about building a viable community). Forum hosts like Facebook and Reddit don't have this problem - you can view your Buy Nothing Group and Moms of Springfield posts interspersed with your feed of friends, or your r/factorio content interspersed with a steady stream of r/AskReddit.

There's also emerging technological barriers. If you don't sign up for CloudFlare, as a new website, you're going to get hosed - but at the same time, CloudFlare makes it basically impossible for any new search engine other than Google to spider the site. Ditto security patches, and keeping software up-to-date. Most people don't want to deal with sysadmin stuff at all, particularly if they're trying to build a community as a hobby. So that pushes people further toward hosted solutions with a turn-key secure software stack, which is Facebook and Reddit.

> The root problem is that attention has gone from abundant to scarce

I don't think that's necessarily true.

I think the root problem is that running & using a forum is too difficult. That is why centralized forums (like you mentioned, reddit and facebook) that handle it for you won out against decentralized forums run by forum members.

Even before facebook/reddit/etc forums tended to live or die by individual effort of one passionate system admin dealing with all the hosting, updates, accounts, and spam until they get fed up and the forum closes because they can't find someone else to take the keys.

One of my favorite niche forums is https://archboston.com. It has years of deep-dive discussion from passionate users about the history and progress of Boston area infrastructure and real estate development projects.

For a while there the site was up but not allowing new accounts to be created -- someone was paying the hosting bills but didn't have time to do any admin tasks. Thankfully, someone else stepped up and people post new stuff every day (albeit with banner ads at the top of each page now, which is honestly not too bad)

I'm happy, but it could have gone poof so easily.

It's not so much that running a forum is difficult.

But anyone who launches a forum today is competing with the large, metaforum platforms like Reddit and Facebook and Discord.

It's just too difficult to assemble a userbase.

> The forum experience is dying.

Perhaps the experience is dying, but the wealth of curated information in forums is still there, is still incredibly valuable, and in some cases is still being added to. Here's one example I used extensively recently; it was sent to me by a colleague, since I never could have found it via a search engine.

https://gearspace.com/board/studio-building-acoustics/610173...

> but at the same time, CloudFlare makes it basically impossible for any new search engine other than Google to spider the site

I hadn't heard about this, can anyone supply a link for more context?

Is there anything a Cloudflare customer can do to "opt in" to being scraped by other search engine bots?

Aside from prioritizing Ads, I actually think the root of the problem is sort of the opposite: information has simply gone from scarce to too abundant. Finding information you need is like trying to find a needle in a haystack. To solve this problem, search engines like google search came into being. Initially, having multiple search engines caused a problem in itself: if you have 700 search engines, which search engine do you use? So the industry naturally ended up coalecesing into a near monopoly, as having 1 (or a few) engine(s) to use is simpler than having 700. However, the root of the problem is still growing. Now it's not so much as "which search engine to use?" but more: "what's the precise combination of text to feed the search engine needed to find my needle?" And as more information gets created, the problem just gets worse: your information gets drowned in the sea of other information and thus gets harder to search for...

Amusingly, appending "reddit" to your search is like a pseudo search engine in itself, instructing the search engine to act as a search engine of only a specific domain of information. Almost like we're back to having multiple search engines, and no one knows which one to use...

> information has simply gone from scarce to too abundant

no, the signal to noise ratio has gone from sufficient to far too low

> hard to build a new forum site and attract an audience that's willing to type your URL in every day

This is exactly the problem that RSS has been solving since it was created over 20 years ago.

If you have this problem, it means you do not have granular-enough RSS feeds. Per-discussion-thread at minimum.

There's a few niche ones out there that are even growing a bit! For example for medium format cameras the largest groups are not on reddit.

> the forum experience is far from dead

If you find a forum for a given subject, it is almost always an authoritative source filled with experts. This is especially true in engineering disciplines.

It's unfortunate that Reddit and social media took over and led to their decline, because it's suboptimal setup in so many ways.

- Reddit in the large is a high noise, low signal monetization chamber. Some subreddits have good moderation, but that doesn't stop the spill over and drama.

- You can't assume much about any given Reddior, and you won't typically form relationships or associations with them. It's pretty much pseudonymous.

- Reddit doesn't focus on authorship. It doesn't allow inclusion of images, media, or carefully formatted responses in threads.

- Reddit corporate is the authority and owner of all content. They can change the rules at any time, and that's a fragile and authoritarian setup for human discourse.

- Reddit corporate is constantly changing the UI and engaging in dark patterns to earn more money. This flies in the face of usability.

Forums should make a comeback. It would be better if each community had real owners and stakeholders that had skin in the game rather than a generic social media overlord that is optimizing for higher order criteria that sometimes conflict with that of the community.

But forums have problems too. They should be easier to host, frictionless to join, easy to discover, and longer lived.

Another way to think of this: every major subreddit is a community (or startup) of its own and could potentially be peeled off and grown. You'd have to overcome the lack of built-in community membership and discovery, but if you can meet needs better (better tools for organizing recipes, community events, engineering photoblogs, etc.), then you might be able to beat them. Reddit can't build everything, just like Facebook couldn't.

This is depressing. Good information is useful for far longer than a carton of milk in your fridge! And a lot of that new "milk" is apparently made of chalk and bilge-water.

Are people still drinking cow milk? Oat milk all the way.

Yes, of course they do, the alternative milk market is growing fast but most people still go for cow milk if they want milk. That's why grocery stores still devote a ton of space to cow milk.

I guess it depends on where you are. Here in Sweden it’s about 50/50 (if you take into account the non-refrigerated milks).

I'm relatively well informed but I always just assumed oat "milk" was an inferior substitute marketed to the actively or wannabe lactose intolerant. [EDIT: and vegans of course.] (No idea if "wannabe lactose intolerant" is really a thing, but i'm thinking of the way gluten sensitivity became a faddish self-diagnosis for a while.)

I still don't drink the stuff, but it's only dawned on me in the last year that there are other reasons, such as environmental concerns or ... actually, I'm not sure. Opening two tabs to Google "oat milk why" and "oat milk why site:reddit.com" now, which conveniently makes this relevant to TFA :)

I use cow milk in my coffee and almond milk in my bland cereal of choice. I’m not lactose intolerant, but too much milk definitely feels “heavy”. The almond milk plus some sultanas substitutes nicely and is very cheap. Mostly its not for the taste, it just makes breakfast easy and efficient so I can focus on fancy stuff later in the day.

Sweden was the 4th highest consumer of milk per capita in 2013. Unfortunately that seems to be the most recent data.

https://en.wikipedia.org/wiki/List_of_countries_by_milk_cons...

Here (mid-tier city in the US) it's more like 80/20 or 90/10 at any normal grocery store, and I suspect there's higher product turn-over for dairy so the actual sales figures favors dairy more than that suggests.

Non refrigerated milk can still be from cows.

Of course it depends on where you in Sweden you are. The super market in the the more affluent part of a large city where I live is about 50/50. Out in the 'sticks' where my parents live it's much closer 80/20

The entire information ecosystem has internalized a bias toward "freshness." It's even really strong in software. Evidently code is more valid and correct if it has recent GitHub commits.

Almost no software just works if left unattended for years. If its a library it means it will likely not work with the latest versions of everything else. Your bug reports will go unattended.

People also have a lot more tolerance for missing features or issues if they see it improving regularly. While getting something unsatisfactory as the last and final version is not nearly as acceptable.

I have built Unix/Linux stuff from the 1990s with no code changes. Programs that do well defined things generally have a long shelf life. Even X stuff often works, though it can look bad on modern displays.

Math kernels, codecs, and so forth can more or less live forever.

Software is the most toxic environment possible for freshness bias. Every month it seems like there's a new framework/ecosystem/whatever and everyone's migrating to it and you're behind the times if you're not using it.

This is only if you bother chasing the absolute freshest trend. Things like React, Ruby on Rails, Node.JS, etc have been the standard and most popular tools for close to 10 years now and aren't going anywhere.

I guess this is why a lot of sites now removed dates from their articles.

Google allows for searches within ranges of dates.

Anyone know the origin of the fresh rule, and the purpose? It makes sense in some niches but in others it is so obviously bad I wonder why Google added it

There was a big news event once (forgot what) and Google was only showing aged pages at the time. So they started prioritizing freshness.

Google had button to search only forums. Monopoly shows.

So frustrating to find background Information to a big current event. Google will aggressively show the same news articles over and over.

I feel this would be solved if the search engine weighted results based on whether users trust the domains.

That only works if you validate users, otherwise the trust database becomes 99% the result of the actions of SEO firms.

Doesn't cloudfare do this? (extremely extensively imo - just about every single click is a captcha of some type sometimes) As much as HN despises blockchains because of crypto, (no, that's not the only use of that tech) it does seem like verification of websites could be a decentralized consensus - along with DNS as well -and user verification could be a portion of the consensus of website quality. This could even help with making self hosted search engines more accessable

Hmm, or you outsource the trust problem to your users. Let them select which domains to trust or who to trust to weight trustworthiness of domains for you.

Imagine that user A can create a list of domains with trustworthiness score. User B can then use that list by going to user-a-awesome-curation.koogle.com?q=nice+shoes

It might create filter bubbles but it would be transparent filter bubbles. You could even wikify/open source the curation.

A lot of this could be solved if we could signal intent or context before searching. But that would require that you know how to use the tool which is a gargantuan user barrier from Google's point of view. Meanwhile, we have to hack about trying to signal context.

pretty much what we're doing at https://breezethat.com, a topic search engine

- we currently curate domains / pages internally based on trust - we'll be adding an open low-code way for others to vote / moderate

the forum experience has effective been totally replaced by either discord or subreddits, or any other kind of self-moderated social media group you can think of.

Its a plus in minus in a lot of ways but the biggest con is that its just straight impossible to search a discord log effectively.

> Can people suggest good alternatives

I've been on the Kagi beta test for a few weeks now and, for the kind of searches I mostly do, it seems to be a massive improvement on Google. Strongly recommended.

https://kagi.com/

I've also been using Kagi for ~1 month and god can I testify for how fantastic it's been. You have to TRY to find blogspam and the allowance of blacklisting domains plus some other handy search customization features make it an absolute joy to use.

It may lack "instant answer" widgets or other fancy search engine features but it gets the actual "search" part of the equation so right that I find it astonishing how I ever used DDG/Google in the past.

You can enable Instant Answers in the settings, it is not as complete as other search engines yet, but it is improving :)

Yeah I've flicked it on a couple times but rarely notice much change, though honestly it's not a search engine feature I've come to desire, as even if you get a widget the majority of the time it may parse incorrect data or not even show what you wanted (which happens a lot with google's widgets IIRC).

Same. Over the years I've trialed most search engines out there, but always find my way going back there after at most 2 days of trying them, because I end up adding "@google" before every query anyways because the results are bad.

With Kagi most of the results are what I'm looking for. If they are not, I'll still try "@google", but so far with very few queries Google's results were actually better. The biggest drawback is worse "smart cards" results, but I hope they keep those optional/unobtrusive anyway.

The strange thing is that the feeling Kagi gives me, isn't even unknown. It just feels like Google circa 2010.

I think the hardest thing with trying a new search engine is the constant question "i wonder if the old one would have done this better"

I wish there was an extension that would search simultaneously on different engines and present them in columns side by side.

Like this? https://chrome.google.com/webstore/detail/double-shot-search...

I've signed up for the beta, but it's hard to shake the feeling that signing in to a search engine is a mistake. "We respect your privacy", "we'll never sell your data", I've heard these claims before and they've almost always been lies. They can tell me that they don't maintain an eternal history of all my queries, but how can I ever verify that?

Kagi ultimately will be a paid service. As I've noted in the context of other services (email), by providing revenue in the form of paid services (and only paid, no freemium tier) the service doesn't have to implement ads at all, and thus can skip the pressure to deliver more and more data in exhange for better rates.

I guess it comes down to trusting that they'll sit on that big stack of user data, exactly the same stuff Google used to build a trillion-dollar company, and continue to decide day after day not to sell it. Will they continue to hold out if VCs get involved?

Yes, I'm aware that all the same arguments apply to DuckDuckGo, and that Google & Bing already explicitly do sell this stuff, but the stronger the promises, the more I demand to see proof.

I think part of it is also an implementation pressure. If you put your dev work into building a system around managing subscriptions and processing user payments, its less easy to flip a switch and start siphoning data to 3rd-parties when they come calling.

In theory, this would be visible to end-users as a halt in feature roll-outs, because the dev team has to pivot to building ad-tech.

I hope to god VCs don't get involved; if they do I'd be the first to bail. I'm hoping that the revenue model makes VC money allergic to them in general. Bootstrapping is preferred for this type of service (see also: Pinboard, sr.ht)

Neeva has a similar ad-free model. The experience is getting better all the time.

Umm... Google and Bing track the hell out of you whether you sign in or not. DuckDuckGo claims not to but the sites you hit through it certainly employ tons of fingerprinting.

Unfortunately with the advanced state of browser fingerprinting you are always "signed in" when you search on any search engine.

> how can I verify that?

Write and ask them. GDPR requires that users can retrieve all stored information associated to their profile/person. Most large internet companies have this functionality.

I love this search engine, it gives me the same feeling that Google did when it became a thing. Their business model after beta will be that users pay to use it, and it has no ads. This is a very encouraging sign, and personally I'll be willing to pay for quality search without ads. I hope enough other people feel the same to make Kagi profitable and functioning for years to come.

Looks interesting, but am I crazy for thinking that $10/month is an insane price to pay for a general purpose search engine? Surely Google wouldn't making anywhere near $10/month off of me even if I disabled adblock.

Those two sentences have no connection. You can gain a huge amount of consumer surplus, even as the seller reaps almost none of it. Google's gain from ads bears little relation to your gain from Google. (This is why people are so much better off in markets: it's actually very hard for a business to get more than a small fraction of consumer surplus.) As it happens, when people try to estimate your gains from a general purpose search engine, it usually comes in at like $100+/month (search 'willingness to pay search engine' and think about how long it would've taken you to find that in a physical library, and how you wouldn't've bothered in the first place because such a search would be impossibly expensive). So, $10/month would be a steal compared to not using a search engine at $0/month.

> So, $10/month would be a steal compared to not using a search engine at $0/month.

But that's a fantasy dichotomy as long as free search engines exist, and there's plenty of them. If you could somehow change the world and wipe them all out in an instant, I guarantee that people would scramble to provide alternatives. We will never live in a world where your only option is $10/month or no search. (Free & open source search engines already exist and you could host one at home or on a cheap VPS; there are also P2P search engines)

Of course people would scramble, but it has nothing to do with how much value you can screw users out of using advertising, because even if ads were worth $0 revenue, search is so valuable you could just plain charge users. This is why there is essentially no relationship between the value of search and the ad revenue. The value of the ad revenue could be $0, and the value to the user would still be $100+/month.

And because the consumer surplus is literally an order of magnitude or two more than the subscription fee quoted, that is prima facie a case that a subscription search engine could have a marginal benefit of >$10 compared to the free ad-supported engine. It, or the subset of searches you opt to use it for, only needs to deliver a little more value to be worth it. Quite aside from the problem of Google Search being increasingly jammed full of ads, wasting your time, or any distortion of ranking, people just plain dislike and avoid ads (https://www.gwern.net/Ads).

The value proposition is in surfacing a result that you wouldn't already have from the free search engine. Your personal calculus will of course vary but lets do a basic business case with the following assumptions

Free search engines work 95% of the time for your employees searches

Kagi can get a result in half of the remaining 5% (this is definetly the biggest assumption and I haven't had enough experience with kagi to say if this is realistic)

Your employee does 1 search a day and 30 days in a month (so kagi gets you 0.75 more completed searches a month).

It takes an employee 15 minutes to search manually through documentation or come up with a solved algorithm from first principals when the search fails.

In that situation your employees time needs to be worth less than $53.33 dollars an hour for the $10 dollar plan not to break even.

So play with the numbers how you want to make up your own mind but it does seem reasonable to argue there's a market for it at that price. Personal use where missing a result could have no cost is ofcourse another question.

edit: 0.75 not 1.5 searches a month extra

Sure, I'm not contesting that there's a market for it. Going along that line of reasoning, there's also a market for a group of experts you can phone and get an answer from at $100 / hour, and so on. But let's not push the goalposts too much :) My personal calculus says it's not worth $10 for me (I don't rely on search much for my work).

Their pricing is closer to $30 a month if you are searching more than a handful of times per day.

At $10 a month you have to pay per search if you search over 20 times per day or something.

The price is crazy only because you're used to not seeing the price you're paying (ads). I spend $10 on things way less valuable very often. A good search engine is at least as much value as intellij to me, and my company pays 5 times more than that per month for intellij.

The problem is that ads price discriminate. Google may not get much money off of you, but there are other users that are very valuable (think of a manager in a billion dollar enterprise searching for a subscription product to buy). Would you be okay with it costing 0.05% of your income?

Of course it will be hard to compete with ads as business model if the alternative doesn't allow for price discrimination.

You'd be surprised. Google makes more than that from showing ads to an average US user. Same for Facebook iirc.

Looks interesting.

In the FAQ they mention potentially charging around $10/month.

Not sure if I'm being entitled or anything, but I was expecting something more like the original WhatsApp model of a few dollars a year.

Perhaps I'm under-estimating how computationally heavy search is.

> Perhaps I'm under-estimating how computationally heavy search is.

It's because they aren't rolling their own search, they pay Google and Bing to do the search for them (via the Google and Bing Search API's which are charged), mix in a few results from their own crawler, and then reorder the results.

So they will always have a higher cost base than both Bing and Google, because they are paying for 3 different search indexes (including their own), plus Bing and Google's margins on the API, plus their own infra costs.

(Now if this is a sustainable model or not is another question...)

Kagi has a "consumption" section where they show how much your searches cost to perform.

I've done 20 searches this month (I haven't switched it to default) and it says I have incurred $0.25 (between $0.01 and $0.02 per search) so it would seem that it's very expensive for them to provide results at the moment. It would absolutely be unsustainable at a few dollars a year given these consumption numbers.

I'm not sure how they are counting the searches, mine is indicating more than 100 searches per day and I certainly don't access their website that much (maybe search suggestions count too?). Their results and overall product are much better than Google but I won't pay 30 usd per month for that.

Every time you make a 'search action' it charges you, including changing categories on the site or changing the filters on the search.

Lets say you search for "Photo Of A Mother" * Then you click "images" * Then you click "Sort by Recent" * Then you click "Licence -> Public" * Then you click "Size -> Large" * Then you click "Size -> Extra Large" *

Every time I have put a star in the above is a time where you would be charged a search (so the above would be charged as 6 searches). It's the same thing with switching to news, applying filters, blocking a site or boosting a site etc - I've validated this on my Kagi account by clicking actions and seeing what it does to the billing, and just by using the search engine and using the lenses feature for example you can quickly rack up loads of searches.

Now let's say you search for an error while you program, visit the first site, it's not got what you want so you click the back button and then visit the second site, that's not got the right answer so you click back and visit the third site... That's currently counted as 3 different searches rather than 1 search. If you open them in different tabs it's counted as 1 search though.

And with all this then you are suddenly over 25% through your daily allowance on the $10 plan. Even choosing to 'block' a site in the search charges you to block it. They are talking about charging $0.015 per search if you go over-quota (which is something like 20 searches on the $10 plan), but as far as I can tell if you are using it moderately heavily you will blast part the 20 searches and could end up with an eye-watering bill.

I think the team are great, and when I was on discord they were really receptive, but I ended up giving up on Kagi after trying to use it as a daily driver as I figured they wouldn't be able to find a good enough monetisation strategy for my level of usage (after hearing discussions in the pricing channel). The product is good, it's just that they can't offer it at a price I can accept (and I suspect they can't offer it at the moment at a price that the market can accept, unless they can reach a deal on API licencing or roll their own search).

That pricing model is insane. I think it will remain a niche product only used by the wealthy unless they can get it down to somewhere like $5/month for unlimited searches.

Yeah, it's not that hard to get results as good as Google when you are paying Google and Bing to give you search results.

It IS hard to get results as good as Google at a price where people are willing to pay for it.

IMO I think the issue with Kagi is that their current architecture has made it easier for them to get great results (as they are just taking their results from Google + Bing and adding their own special sauce) but it makes it much more difficult in the long run to be cost-competitive.

Their nearest competitor (Neeva) does offer $5 a month search, has a free tier, and rolls their own engine, so I don't really see how Kagi's current model is sustainable unfortunately (but maybe they will be tremendously successful and I will look back at this post and feel silly!).

The premise is that for people willing to pay for a search engine, quality of results trumps everything.

I've been using Kagi for a while now too. I like their stance on privacy, but I don't really like that I have to create an account to use it.

https://kagi.com/privacy

Any reasonable justification for requiring sign-up/a user account?

Presumably because, as they move out of beta, they want people to start paying to use their search engine. Thats the only way they can afford to be ad-free.

I'm 100% happy to pay for an ad-free search engine that doesn't sell my data. I don't really like having all my search terms being linked to me via an account, but I suppose that's already happening anyway. I guess I picked the right day to stop looking at clown porn.

I guess how would you implement monetization without an account in some form ?

You could just allow a given IP address ? But that's just as trackable and has tons of downsides with using from other locations.

They could do the SaaS model and give you an API key or unique URL allowing up to 10000 searches per month which you could share with friends/your company

That's probably a worse business model, but it would be really interesting to hear what other monetization ideas Kagi considered or is considering.

Works at an enterprise level (e.g. exchange rate providers) but would be a bad fit for a B2C product.

Managing customer expectations on api key usage (esp. if that key is publicly visible e.g. URL parameter vs. HTTP header) is not worth it unless you have higher-priced products.

Also api keys would mean you might have to prevent re-selling, etc. Furthermore, they could still analyze api key usage to get the same historical data on you as if you logged in.

Would still link searches to your API key.

That's missing the point. If the same API key is used by 1000 people at a company, you don't have the expected 1:1 mapping of search terms to individuals with an account.

An individual account which isn't permitted to be shared is different.

You could sell tokens that get used up after each search or expire after certain amount of time since first use. Browser extension could store tokens and provide them to website as needed in random order. Tokens could be resold so no tracking by payment processor

To buy a token you still need payment details. How would this be different to just buying a subscription?

If you buy from reseller they have no way to correlate payment details to search queries. With subscription, however, service provider have your payment details and knows everything about your queries so it can build a profile on you.

yeah but they've been in "beta" for four years now.

As a bit of a weird hobby, I like to read up on right wing conspiracy theories. That means I do a fair number of searches for specific terms and people mentioned in fake-news facebook/forum posts.

Google seems to slowly oscillate between thinking that I am a right wing loon, and thinking I am Joe Public who must not be shown misinformation. That is, sometimes google is perfectly willing to vomit forth results from the propaganda mills, even when I'm not specifically looking for it, and other times I can't get conspiratorial-minded results even when I am making an effort to find them.

This most frequently manifests itself when I am looking for sources for claims that I know exist. Like if I remember reading an earlier conspiracy that has just been invalidated, or someone posts some a video of someone reading a blog post. If google has decided I am an innocent bystander not to be shown conspiracies it can be nearly impossible to track down the original blog or posts about the conspiracy.

Recency bias is another huge problem with google results. Older content gets heavily de-prioritized, even when it is clearly what you want. Google is willing to give up on terms in your search before it is willing to show you old stuff. For example, if you tried to research early Ukrainian political corruption during Trump's impeachment, your results would be nearly entirely Trump-related content even if you tried to use google's date-filters and exclude terms like -Trump.

I noticed this recently when trying to find primary sources for flat earth claims. They don’t exist on Google, for me at least. You can still find them on duck duck go if you search for something like “flat earth ice wall” but Google just returns generic debunk articles.

This sounds like filter-bubbling. From what I can tell, Google doesn't have user specific filter bubble but user-category filter bubbles, and it's constantly updating the category of users it thinks you're in.

Limiting search results by date is essential if you're trying to find something with a popular name of something else that just came out.

And it's infuriating if that's not what you're trying to do.

Which is why reasonable search engines should have a "sort by {relevance,date,etc}" dropdown.

`site:reddit.com` has been worked poorly for me recently, although I've used it many times in the past. He's my most recent search (I was traveling and trying to watch Netflix, but geo-block was preventing some shows from appearing):

https://www.google.com/search?hl=en&q=site%3Areddit.com%20ne...

The entire first page is for NetflixViaVPN subreddit (not linking to avoid SEO). They have a stickied post that seems to shill two VPN providers I haven't heard of. This is plausible, as maybe Netflix hasn't either... The stickied post has comments disabled, so it's hard to tell. Then if you click other posts, a bot auto-links the stickied post, but everyone is making different suggestions that may imply the stickied post is wrong.

Interestingly, the same search on DuckDuckGo only has three posts from that subreddit. This better matches what I wanted! The first one I'm seeing is:

https://www.reddit.com/r/VPN_Guide/comments/rgh2xn/best_netf...

This seems much more plausible. All those comments suggest a provider I've heard of and that I've heard other people mention IRL. Google seems to rely too much on the URL or the page header, so it's stuck in a single subreddit.

imo, google is still king, but you have to be a bit of a power user. You're already using `site:` which is good if you know exactly where you're looking. If not you can use `related:` in the same way. I find using `-something` to remove terms the most useful. I'll search for something (usually an error message) then add `-react` (and mumble "ffs not everything is react"). Then if I still see things I DON'T want add more `-` to the string.

It's not GREAT that you have to do that, but it's pretty functional and certainly better than going past page 1 of search results.

Anyways, here are some other things you can do for reference: https://support.google.com/websearch/answer/2466433

The major problem with this that I've experienced is even if I use operands like + and - to specify or remove terms--more often remove--Google ends up using a synonym in place of that word that means the same thing.

So if for instance I'm looking up info about ADHD meds as an adult, I might get tons of articles about childhood ADHD since that's where all the research is. I search Adult ADHD meds, I still get articles about childhood ADHD. So then I:

-child -childhood -adolescent -teen -kid -momgroup -mother -parent -teenager -children -kids -school -offspring -smallhumans -minor -underage +adult +work -rehab -addiction

and I still get crap blog spam that's probably related to teaching or raising children or some other bullshit like warning about the dangers of addiction or something, and never information about my ADHD or the meds for it.

It's not GREAT is the understatement of the decade.

I've had some luck inserting "forum" into search terms to find real human content. Mostly when trying to find technical info about cars, but may apply to other fields.

Yep. <search_term> site:reddit.com OR forum OR thread

>What I'm having a heck of a time finding is technical content; long-form programming tutorials, deep dives into academic concepts

github search is good for that. search for 'list of awesome anytopic'/'curated list of anytopic'/'list of anytopic' and you might find a repository with a curated lists of links on anytopic. (search box on the main page of github)

You even have the 'The definitive list of lists (of lists)' https://github.com/jnv/lists

You might also want to check my side project: I have a search tool / catalog of duckduckgo !bang operators, i am hoping that it allows for better discoverability of specialized search engines.

https://mosermichael.github.io/duckduckbang/html/main.html - (best viewed on a PC)

here is the project page on github: https://github.com/mosermichael/duckduckbang

Since there's only a handful of sites you target your searches at, it would be nice if you could just have your own search engine that focuses on those few sites, and perhaps crawls a little deeper.

I've sometimes thought the death of Google will be the self hosted search engine.

Hard to find a business model for a self hosted search engine though.

They already exist.

I've found my jobs' internal (social) message boards/mailing lists/Slack channels/etc to be great resources as the only contributors are those who work/worked at the company. Your (ex)coworkers presumably met/meet a certain competency bar and are less likely to spam. At larger companies there are message boards/mailing lists/Slack channels/etc for nearly every topic.

For local information, I've found forums for local sport teams to be great resources during the off season. Posters are often happy to engage in any sort of chat during the off season. Even if you haven't gotten to know the frequent posters during the sport's season you can use the (usually highly visible w/o any additional clicks) account age/# of posts/"karma" as a proxy of posters' trustworthiness. note: If you don't normally contribute on-topic (i.e., about the team and sport) posts, I would only search the forums for your questions and not post off-topic questions as that'll get you quickly banned.

It's particularly awful on mobile where you get Google's "smart" cards which can be ads, followed by ads then the actual results which are mostly SEO trash. Trying to find support for Google Fiber routers was nearly impossible because Google just tried to interpret what I wanted as signing up for Google Fiber and just overwhelmingly suggests that. It gets even worse on Youtube where after like 10 results for what you typed in they just show you "things you might like".

For technical content I had great results from using safari books online in the past. Having most tech literature an easy web search away was super convenient, because typically the best treatment of any subject is in book form. The downside is that it is expensive, so when I switched employers I lost access and I wasn’t willing to pay for it myself.

Change your search strategy. Most forums require a membership to view them, and most long form posts are on personal websites. Google can't or won't serve those. You have to navigate like it's the old web. Find a good place to make landfall, read old posts, ask around, and follow all your leads.

> Google with site:news.ycombinator.com if I want to find "forum-like" discussion on topics I'm interested in.

Use https://hn.algolia.com/ instead. You can see your results as you type your query and even sort it based on time.

Kagi search (currently in beta) is actually pretty good.

I have noticed the low effort posting. This may extend beyond just Reddit though. Everyone wants the biggest reward for the least amount of effort.

I've replaced "site:news.ycombinator.com" with hn.algolia.com and couldn't be happier.

If I'm interested in anything technical, I'll search hn.algolia, it's a brilliant tool.

youtube has some great technical content