It's much more difficult now to build a competitive search engine, but saying it's impossible and discussions should be banned is toxic. (And already basically proven wrong with existing competing search engines.)
I was looking for this and it really annoys me how it thinks it knows what I want.
I wouldn't be surprised if there's some limit to prevent abuse, but it hardly gets in the way of my normal usage.
Apparently all of this is to offset the costs from paying Google and Bing to run searches for them (the site basically takes results from Google and Bing and then combines them together with their own special sauce) but with a pricing model like this it seems like Kagi will remain a niche tool solely used by the wealthy.
The main paradigm shift necessary will be to pay at all for something that was "free" and is provided "free" to this day - even if the quality is worse.
They need to support their efforts somehow. Instead of showing ads or selling your data, they are going to charge a fee to use it.
I'm logged into Google all the time. I do this to get sharing working across Chrome. Others doe the same with Firefox. Some don't, because privacy.
I hear you think logins are 1000x more hostile than paying with your eyeballs. How would you suggest resolving this clearly conflicted view with logging in vs. how the revenue is made on a given site?
But that is changing. Internet marketing companies are now teaming up with websites so the websites to their dirty work for them and you end up getting tracked anyway -- but with much higher certainty as to who you are. That means that now, yes, logins are more hostile.
The web continues to get smaller and smaller.
As long as it remains niche, it might actually stay that way indefinitely. I’m just worried about Google taking inspiration from them. I bet whatever “tricks” they use now would get a lot less effective if they did.
To my knowledge besides the old experiments with client side search, Google has never allowed (even paid) api access to their search. I thought they did not do this, because it would have allowed somebody to jump start a search engine that might eventually become a competitor.
Kagi seems to be open about their Google relationship, so I assume they have agreement in place.
This even feels like some experiment from Google to create a premium, paid, search product (like what they did with YouTube).
* It blocks all the GitHub and Stack Overflow copycat sites out of the box.
* I can add additional sites to the block list with two clicks.
* It up-ranks (semi-)official documentation like the Python docs or MDN over blog spam sites.
* I can manually up-rank sites (and even pin them to the top of the results) if I'd like to see them more often.
Kagi is awesome because where Google thinks it can guess what you want, Kagi gives you an easy way to tell it what you want.
1) New system comes out that indexes/controls/regulates a naively created dataset
2) Data consumers adopt that system and experience benefits
3) Data suppliers learn rules of system and take advantage of it to improve the positioning of their data, thus breaking the intent of a system built on the assumption of naive creation
4) Users complain about the broken system
5) New entrants realize that the original system actually solved the core problem really well, and there are no easy ways to solve the 'gaming the system' problem
6) Flawed system remains the best available option indefinitely
It's like entropy, there's just no fighting it.
EDIT: And to extend this beyond Google - do you see a lot of long text blocks in 7 second tik tok videos? That's because the creator found a way to game the algorithm.
For bot detection, when was the golden age? I'd say that, rather than follow the above description, it started as non-existent and there was basically a smooth descent downward to where we are now.
Sites with people hired to optimize the robots.txt file are ranked higher than those without. If we ban all websites that hire people to optimize their robots.txt, then a lot of the web isn't indexed and the search results suffer.
Or if we ban every creator on tik tok who posts a lot of text in a short video, then a lot of creators are going to be de-platformed and moved elsewhere.
The point of what I wrote is that 'humans are really god at following rules, so much so that they often are able to manipulate rules to their advantage, without breaking said rules.
That's why this is such a hard problem to counter.
Their only job is to stack order the internet. Down ranking a site to the 1,000,000th result is the same as banning them. They necessarily do this all the time (like for every search with more than 10 potential hits).
Perhaps there is no border to be defined since nearly all SEO is abuse?!
The only acceptable SEO should be "provide website content that is more relevant/interesting for the viewer".
And where is the line between "provide website content that is more relevant/interesting for the viewer" and SEO? If I realise that (to take an example from something found on Google) I could take the sentence "Identify the best customers and convert more" and rewrite it as "Marketing automation helps you identify the best customers and convert more", is that SEO because I'm intentionally adding a keyword, or is that making the content more relevant to the user by being more explicit in what I'm saying?
This does of course not imply that these things can be proven, which implies that no action can be taken. Not everything that is immoral is punishable.
It's true building a good search engine is an gigantic undertaking but if Google (or someone else entirely) is aware of the current issues, they may have ideas to tackle those problems and then we can all share a better internet.
Same for the argument with democracy and any other similar arguments, pointing out problems within our current institutions doesn't remotely mean that we want to abolish everything and start from scratch.
https://neeva.com/ better than pretty much any solution (Qwant, DDG, etc) I've personally tested. It also indexes specific websites like StackOverflow, GitHub, and GMail.
Edit:
Neeva does require an account to create because eventually the product is going to require a subscription.
"To continue searching and access all of Neeva's features, create your free account. Already a Neeva member? Sign in"
Yeah, no.
But essentially we all have de facto accounts on Google anyway.
They track and monitor usage and then map you to a user account when you finally login somewhere.
I'm totally fine with that being upfront and made clear from the get go!
1. I have more than one GitHub account (one is for work, one is for personal stuff). I would like to somehow keep them separate, but at least being able to add multiple GitHub accounts for indexing would be cool.
2. Could Neeva offer something for segmenting the parts of my online presence? For instance, there's personal me and then there's work me. I'd like to turn a different profile on when I'm doing different things.
3. I don't know if it's possible to partner with ProtonMail, but I'd love to index that and my Proton Calendar as well.
Also didn't know the free tier is permanent, but I think that's probably a good move. Thanks for what you do!
If you trust these people are doing what they say, it's a pretty good deal for securing your internet spaces and trying to get away from google, imo.
I can't find shit with DDG and the experience is like using Google from 20 years ago when it didn't have any of its bells and whistles.
I’m glad it works for you, it just reminds me how different are everyone’s expectations of good search results.
"pizza near me" - Google suggested a well rated place within 5 miles. DuckDuckGo suggested a pizza place 91 miles away??
"busted kids lip" - Google says gauze and a cold pack. DuckDuckGo says salt water??
"best monitor for mac" - Google says Dell Ultrasharp with 4.5 stars and 738 ratings on amazon. DuckDuckGo says BenQ 4 stars with 174 ratings on amazon?
how is this better?
No, they are underestimating the difficulty of funding a good search engine.
I liked the runnaroo search very much as did several of my friends. The guy who ran it couldn't fund it. He shut it down.
Altavista (Yeah, that far back) had a nice feature where it would draw a cluster graph of your search results. So, if you searched for "python", it would show your results but would also draw a little graph and you could see that "Hey, there are two clusters here--programming and reptiles." You could then click on the "programming" node and the "reptiles" cluster would go away. It allowed you to drill through irrelevant stuff really quickly.
Note how that feature doesn't exist today--in spite of orders of magnitude more programmers being thrown at search, graph algorithms, and nifty Javascript web UIs. I wonder why ...
(/sarcasm in case you missed it. I don't wonder why. Such a feature would let you drill through irrelevant Ad and SEO garbage too quickly and would impact Google's revenue.)
we started out with topics and then moved to web, and are now folding topics into the web search experience, it's really hard stuff to get right
our first main filter was blogs, which is getting renamed to posts for a mix of reasons, adding forums shortly, along with other more specific topics
re: https://breezethat.com/ -- & better on laptop / desktop atm, premium version will be ad-free
However, I think Google has severely degenerated from just two years ago, when most of the problems were fully in effect.
Google is a bit of a product of the situation of scams being the easiest way anyone makes online.
Has anyone considered the possibility that all this Machine learning and AI models is what made Google and YouTube searches worse?
The Amazon recommendations engine, the poster child of ML in 2014 isn't any good at recommending products that I actually want to buy either.
There’s no comparing how things were in the 90s and early 00s to now.
Every social media company before Facebook faltered or began losing a good chunk of their user base fairly quickly. Until that stopped happening with Facebook and IG for over a decade now.
Reddit’s position has also been here for a decade now.
Humans are incredibly good at solving engineering problems they can see from a mile away, although it takes time to solve.
Honestly I think people are completely underestimating
the difficulty of a good search engine
Boutique search engines pop up all the time here on HN, but they can't compete fairly against Google, without the resources to crawl a billion webpages day after day.
At the time you would get tons of results from a myriad of small blogs, forums, niche websites. Nowadays it's Pinterest, blogspam and more SEO optimised algorithmic crap. If you're lucky you get a forum result that might actually be relevant.
Given the joke that if you can't Google it it doesn't exist, it follows that the Internet has considerably shrunk in the past two decades.
https://pdfs.semanticscholar.org/fc32/72302461b74217662085a8...
The original article's description of how Google search signals work doesn't even scratch the surface of all the signals it's using.
There will be people that do not have favorite hobbies. There will be people with no ambition who benefit from being told what they can or should do to contribute. Those people, in order to have their needs met could choose to take on a Necessary Task, even if it isn't their favorite thing. Maybe they learn to love it and excel at it.
Don't want to do anything to contribute to society despite being physically and mentally able? No needs met. Meet them yourself, you magnificent lone wolf. Want your needs met but don't want to do Necessary Task that you hate? Find a purpose and pursue it.
Have the talent and desire to be a heart surgeon? Go for it, and you'll be rewarded. Want to make life sized statues of the Beatles out of used printer cartridges? Eh, fine but maybe also get a day job.
Sorry but Capitalism is a broken system that is not meeting mine or million's of others needs, much less enabling us to pursue what we're actually passionate about. I at least know that I am passionate about building and supporting technology but I still struggle to pursue it professionally because I had the poor luck to be born without the resources to get a piece of paper that says I know how.
It seems like everyone just thinks it's how you describe the contemporary world
No, what they're describing isn't capitalism. Not even close
These posts should almost be blocked from hacker news. ITs a fantasy. Its like saying that democracy has failed so lets replace it, replace with what? Its the best we can get given the alternatives, and its flaws will always be exploited.