News for hackers. Stuff that matters.
Re:Kagi Small Web by freediver on Thursday September 07, 2023 @11:45AM (#37420695)
Vlad from Kagi here, thanks for posting. The RSS feed unexpectedly broke (edit: the feed is back up! [3]) just as we published the blog post, in the true spirt of "small web" :) Should be up in 30 minutes which will enable the site to function too (it uses the same feed).

This has been a personal pet project of mine and I spent considerable time getting my hands dirty with the code, as the team was busy with other initiatives. When I said the "feed broke" for the launch I meant I broke it. Software is messy especially for an old school dev. I learned in the process I am not a very good coder anymore (if I ever was one?), constantly going back and fixing stuff I previously thought was solid. Check it out in the linked repo [1].

I found the site replace the need for discovery for me, and getting to know various different humans and their writing felt good. There is a glimpse of hope in the concept and I hope you get to see it too (as soon as the feed is back up!)

You can check the list of included websites here [2]. And all the recent posts already surface in Kagi results (for relevant queries).

[1] https://github.com/kagisearch/smallweb

[2] https://github.com/kagisearch/smallweb/blob/main/smallyt.txt

[3] https://kagi.com/smallweb

Re:Kagi Small Web by BbzzbB on Thursday September 07, 2023 @12:19PM (#37421297)
Isn't this more likely to have been inspired by marginalia* than a personal blogs thread? Doesn't seem to have the same results from the Apple Watch example, but it's what immediately came to mind for me when I read this post's title.

*https://search.marginalia.nu, commonly mentioned on HN https://hn.algolia.com/?query=marginalia

Re:Kagi Small Web by marginalia_nu on Thursday September 07, 2023 @12:21PM (#37421340)
Vlad was dabbling with similar things to Marginalia with Teclis around the same time Marginalia first made headlines.

And to be fair, I was simultaneously inspired by the blog thread to build a curated blog filter for my search engine, which led to a series of changes that overall tends to promote more of this types of results in general.

The fact that we're several to have similar ideas sort of validates the ideas I think.

Re:Kagi Small Web by BbzzbB on Thursday September 07, 2023 @12:36PM (#37421600)
Interesting lore, thanks.

I'm not complaining, glad there's more tools for searching outside the scope of SEO spam or content mills, and competition can only make the niche better (presumably, unless it gets big enough to incentivize disguising commercial content as niche blogs). Just felt like it might be a bit disingenuous to quote a blog list as inspiration for a search engine focused on "the non-commercial part of the web" which is exactly your engine's aim and wording, didn't know they've been working on small web initiatives since a similar timeline.

Hopefully you both find success and with it uplift hobbyist websites.

Re:Kagi Small Web by flexagoon on Friday September 08, 2023 @02:20AM (#37430097)
Kagi actually uses Marginalia as one of its result sources, especially if you enable the "non-commercial" lens!
Re:Kagi Small Web by endorphine on Friday September 08, 2023 @06:16AM (#37431770)
It could be more likely yet not actually true, at the same time.
Re:Kagi Small Web by raybb on Thursday September 07, 2023 @12:10PM (#37421123)
Why do you let YouTube channels have advertisements/sponsors but require sites to be ad-free?

I understand the spirit of it and don't have any counter examples but seems like a bummer if someone has a nice indie blog but can't be added because they have a few ads or a sponsored post.

Re:Kagi Small Web by freediver on Thursday September 07, 2023 @12:16PM (#37421257)
Most of the focus was on the website curation. The YT channels were curated from the initial list of sources [1] and then automatically filtered by recency and subscriber count criteria.

That is to say that very little other effort was made to curate the YT channels and we expect the user community will contribute to edit the list.

Also I was under impression that all YouTube channels had ads, so that is why this was not considered as a seperate criterion.

[1] https://github.com/kagisearch/smallweb#youtube-channels

Re:Kagi Small Web by elashri on Thursday September 07, 2023 @12:34PM (#37421564)
I think we are here talking about organic ads (sponsorship, …etc.) not YouTube as platform ads.
Re:Kagi Small Web by freediver on Thursday September 07, 2023 @12:38PM (#37421626)
Thanks for clarifying. Yeah we should establish a criterion around that. Curating YT is much more work it feels like. Let's see how "Small YouTube" [1] feels like to begin with and we can take it from there.

[1] https://kagi.com/smallweb/?yt

Re:Kagi Small Web by flexagoon on Friday September 08, 2023 @02:27AM (#37430143)
By the way, you can perhaps check https://nebula.tv for a list of good YouTube channels (not all of them are great IMO, but a lot of people who make content on Nebula fit the "small web" criteria)

Now that I think of it, perhaps you can call Nebula "the Kagi of video streaming". Maybe you guys could collaborate somehow in the future ;D

Re:Kagi Small Web by mrcgnc on Thursday September 07, 2023 @12:00PM (#37420957)
I'm a paying customer and this is yet another nice addition. It's something only you can do. Please keep doing this kind of stuff!
Re:Kagi Small Web by manuelmoreale on Thursday September 07, 2023 @12:17PM (#37421268)
This is so neat!

So happy so see the small, independent and more humane web being highlighted.

I’m trying to do my part[0] but I have no doubt that a search engine—even if still a niche one— can have a much bigger impact.

Really well done Vlad!

[0] https://peopleandblogs.com

Re:Kagi Small Web by marginalia_nu on Thursday September 07, 2023 @12:25PM (#37421407)
I kinda wish there was a sort of proper blog-DMOZ. Would make this sort of stuff a lot easier. Right now everyone and their grandma is curating their own blog list. Lots of wasted effort that could be saved by collaborating somehow.
Re:Kagi Small Web by manuelmoreale on Thursday September 07, 2023 @12:32PM (#37421538)
I agree but it’s incredibly hard to collaborate on this kind of projects because everyone has their own take on what should and shouldn’t be included.

Curation is still hard. Also, I personally don’t mind having different curated directories because those will reflect the taste and interests of the curator.

Re:Kagi Small Web by marginalia_nu on Friday September 08, 2023 @03:36AM (#37430651)
I figure collaborate on answering the easy questions, is the website online, is it chock full of spam, is it a blog about horses or about keyboards, etc. Some of that could be automated (like onlineness, large change detection, etc.), but some of it needs manual supervision.

Right now everyone who is running any sort of curated discovery service needs to answer these same questions about roughly the same site.

Re:Kagi Small Web by mgreg on Thursday September 07, 2023 @12:27PM (#37421441)
That is a very good idea.
Re:Kagi Small Web by smallerfish on Thursday September 07, 2023 @01:41PM (#37422685)
Because curation to keep spam out would be work. And how do you prevent trojan horses, where an initially promising blog is turned over to SEO once included in the official dmoz?

Could be done, but it'd take time and effort.

Re:Kagi Small Web by manuelmoreale on Thursday September 07, 2023 @02:10PM (#37423150)
There’s also the issue of quality. Do you just include everything that is a blog, no matter how low quality it is?

And if the answer to that is no, then where do you draw the line when it comes to quality?

Curating anything is complicated and different people will have different opinions which is why you end up with different lists curated by different groups of people with different ideas about what should and shouldn’t be included.

Re:Kagi Small Web by fabianholzer on Thursday September 07, 2023 @03:19PM (#37424207)
ooh.directory gained some traction a few month ago [0] - and they are steadily adding blogs although they do it slowly. Manually reviewing and trying to have some diversity is no easy task.

[0] https://news.ycombinator.com/item?id=33719983

Re:Kagi Small Web by Zetobal on Thursday September 07, 2023 @06:09PM (#37426285)
I always thought bookmarking services like delicious or springboard should have done it. Seemed like a natural evolution.
Re:Kagi Small Web by aftergibson on Thursday September 07, 2023 @11:51AM (#37420798)
No other search engine would do this and it's awesome! Always feel like my subscription is well worth it. Thanks Vlad!
Re:Kagi Small Web by epilys on Thursday September 07, 2023 @12:24PM (#37421386)
Discord really? why not IRC? I don't think having to use proprietary communication software that require accounts did with small web and open source.
Re:Kagi Small Web by mikae1 on Thursday September 07, 2023 @04:01PM (#37424784)
> why not IRC?

why not Matrix? I think that's a more realistic alternative. :)

Re:Kagi Small Web by coding123 on Thursday September 07, 2023 @12:19PM (#37421310)
Honestly the reality is a good coder is one that keeps chugging and doesn't give up.
Re:Kagi Small Web by mhitza on Thursday September 07, 2023 @12:54PM (#37421891)
I really enjoy that this feature got released. Adding more visibility to the indie/small web is a great thing. I've complained in the past on HN that any useful personal blog is buried under spammy content in search engines.

It would be nice to also be able to just search within the small web, maybe using a lens in the future?

Regarding the topic of self promotion, I would disagree with the current rules and I would ask you to allow people to self promote. As long they have an old enough blog, maybe even cut that down to a year, would be helpful. Most users on the small/indie web lack visibility and this would do them service. My blog is already within the index because I think it might have been picked up during the "HN share your blog" post that happened a while ago, but others might not have been that lucky.

Re:Kagi Small Web by skinkestek on Thursday September 07, 2023 @05:47PM (#37426058)
There is already a lens called the non commercial web or something that I suspect overlaps to some degree,but I don't think they are equal and I don't know if I am right.
Re:Kagi Small Web by digitalsin on Thursday September 07, 2023 @01:01PM (#37422014)
This is absolutely fantastic, thank you!
Re:Kagi Small Web by sergiosgc on Thursday September 07, 2023 @01:29PM (#37422489)
Congratulations on the initiative. The results seem cool, and integration in the main search results may give a glimpse outside the big web.

I'm a Kagi customer, and a very happy one. The search engine is amazingly good. This only makes me happier with my search engine choice.

Re:Kagi Small Web by mbb70 on Thursday September 07, 2023 @01:40PM (#37422648)
FYI your "Why pay for search" and "FAQ" links on https://kagi.com/pricing are broken, perhaps another refactoring victim?
Re:Kagi Small Web by freediver on Thursday September 07, 2023 @01:54PM (#37422882)
Thanks, will be sorted out.
Re:Kagi Small Web by eitland on Thursday September 07, 2023 @05:56PM (#37426145)
And this is the second major reason why I use Kagi:

If I notice something is wrong I can report it and expect to have a developer look into it within reasonable time.

They are grateful for reports from users and if there is an actual problem (and not a misunderstanding) it is also fixed, again with a reasonable.

(The first major reason for me is premium search quality, no tracking probably comes at a distant third or something.)

Re:Kagi Small Web by dharmab on Thursday September 07, 2023 @08:45PM (#37427790)
Vlad helped me troubleshoot why my Orion browser wasn't syncing bookmarks (turned out to be a config issue on my device) and we got the fix added into the official documentation.

Imagine getting that level of support for any other web browser!

Re:Kagi Small Web by spiritplumber on Thursday September 07, 2023 @02:31PM (#37423481)
Thanks for doing this! Our fiction and rpg stuff is at https://www.emlia.org/
Re:Kagi Small Web by palmer_fox on Thursday September 07, 2023 @03:25PM (#37424295)
The more I uses Kagi (paid) the more I am feeling sad for the possibility that it might go away if it doesn't get more traction.

For me you represent an incredible accomplishment: the first search engine that gives better results than Google, respects privacy, offers customization and so much more.

Thank you.

Re:Kagi Small Web by sshine on Thursday September 07, 2023 @04:54PM (#37425429)
Kagi is the best.
Re:Kagi Small Web by Phemist on Thursday September 07, 2023 @05:22PM (#37425789)
Sometimes I have to use somebody else's laptop, for some reason or other I end up searching for something. I think, why are these results so crap?

I notice google/bing and quickly kick in my muscle memory built up over the years to type duckduckgo.com and do another search.

After this I still am confused why the results are so crap. Then I notice the duck laughing at me and think... shouldnt this be a dog?

And then I start feeling bad for the original user of the device and tell them to do the search themselves and let me know when they get a result

Re:Kagi Small Web by Phemist on Thursday September 07, 2023 @05:27PM (#37425846)
Just to say: the search engine frog has long been boilt and it is amazing what crap people have to put up with when searching for anything. How jarring the experience of seeing TV commercials is after years of Kodi, is how the experience is of moving back to an ad-supported search engine after now months of Kagi.
Re:Kagi Small Web by skydhash on Thursday September 07, 2023 @10:06PM (#37428385)
Youtube doesn’t show ads in my country. Anytime I use a vpn or I travel, I can’t believe how bad the free tier is. My twitch consumption has gone way down because of the ads. Watching the last NBA season was so annoying. I felt more than half of the broadcast was ads.
Re:Kagi Small Web by reacharavindh on Friday September 08, 2023 @01:25AM (#37429698)
Which country is it that YouTube would not shows ads?!
Re:Kagi Small Web by tacon on Thursday September 07, 2023 @04:22PM (#37425025)
Your reference [2] is to the YouTube channel list. I think you meant to link to

https://github.com/kagisearch/smallweb/blob/main/smallweb.tx...

Re:Kagi Small Web by TradingPlaces on Thursday September 07, 2023 @06:22PM (#37426427)
Thanks Vlad. Another great feature.
Re:Kagi Small Web by BoorishBears on Thursday September 07, 2023 @07:37PM (#37427190)
Semi-unrelated, but is there any reason you don't make your search API as turn key as the other APIs?

It's expensive enough that I can't imagine anyone repackaging it profitably (2x Bing search prices for me) but having to email someone adds just enough friction to discourage a lot of tinkerers from even trying it.

Re:Kagi Small Web by freediver on Thursday September 07, 2023 @08:39PM (#37427752)
This is planned, just can get to the top of priorities given everything else. But expect it by the end of the year.
Re:Kagi Small Web by matheusmoreira on Friday September 08, 2023 @12:21AM (#37429278)
I have been developing a "small web" static site, forked and customized a templating engine into a static site generator just for my site. Even wrote a new post recently, last week or so. How can I add it to the list? The GitHub says to make a pull request, is that really all there is to it?
Re:Kagi Small Web by freediver on Friday September 08, 2023 @03:12PM (#37438037)
I checked your site and do not see an RSS feed there? Feel free to submit it once implemented :)
Re:Kagi Small Web by matheusmoreira on Friday September 08, 2023 @11:15PM (#37441918)
I'll work on it!
Re:Kagi Small Web by VapidLinus on Friday September 08, 2023 @02:41AM (#37430268)
Are English "Small Web" results included even if I use the "Sweden" search region?

I noticed the example result for useyourloaf wasn't included if I switched it to "Sweden" and not sure if this is just an oddity or if the entire feature is nerfed because I just leave my locale on all the time.

Re:Kagi Small Web by freediver on Friday September 08, 2023 @11:13AM (#37434590)
Can you post this on kagifeedback.org so that we can have a look in more depth?