This is great, and is something I've wanted for a while. I use pinboard which is supposed to have similar capabilities (click 'search full text', 'search mine' after turning on and paying for 'archiving'), but I've never been totally confident in it (pages would change, and the cached version was updated to a 404 page), and ended up letting my archiving subscription lapse.

I think google used to offer something that did this as well as search all your local files, but I think that went the way of all gThings.

There's also https://historio.us/ - it's essentially same as what OP is doing but it's as easy as bookmarking

That does look pretty cool, and unusually for a SaaS has chosen a pricing I think is reasonable for the service (not everything should be $9 a month!).

Do you know if it does pdfs? That's a key thing I want in this kind of service.

I use Zotero for this and it shines at pdfs. It does webpage snapshots too. Discussed below: https://news.ycombinator.com/item?id=31849987

All of Stavros' stuff is reasonably priced and sexy.

That's a good one, but I was thinking of https://en.wikipedia.org/wiki/Google_Desktop

> Google Desktop was a computer program with desktop search capabilities, created by Google for Linux, Apple Mac OS X, and Microsoft Windows systems. It allowed text searches of a user's email messages, computer files, music, photos, chats, Web pages viewed, and the ability to display "Google Gadgets" on the user's desktop in a Sidebar.

Discontinued in September 2011

Google desktop was so great. It’s also weird how bad windows search and Mac search sucks.

Google desktop was able to find files anywhere in milliseconds. Still today if I search for a file on windows it takes many seconds and frequently misses the file. And there’s an indexing service that seems to peg my cpu every once in a while.

You want Everything. I've been using it for probably close to 10 years now, it makes searching on Windows not just bearable, but capable.

I'd love something similar to automatically crawl and index every site I visit. I'm forever losing stuff. I know I saw it but I can't remember where.

This is Mac only and I have no affiliation other than I like this developer but your request reminded me that he just launched this app: https://andadinosaur.com/launch-history-book

That’s a really genius idea. I also like the author’s pricing mode. I was fearing some stupid “$10 is just the price of coffee. This is worth 24 coffees a year for the rest of your life” and have a reasonable $7 purchase price.

that's some expensive ass coffee considering i buy a bag for less than that and use it for a week or two, making tens of cups

You are not Starbucks’ market.

who the fuck is starbucks' market if a daily coffee costs over $2000 each year (assuming you do a 5-day workweek and throwing two months off the year to account for days when you don't get morning coffee from starbucks)

you have to be swimming in money

that makes me doubt that anyone would pay more than $2-4 and even that's overpriced as hell

I use BrowserParrot for this. Works really well.

https://www.browserparrot.com/

> Right now we only support MacOS

loving :) thanks

ArchiveBox documents how to automatically archive links from your browser history:

https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-l...

Archive box can do authentication too. Haven’t checked if yacy can. Pinboard can’t. Pocket can.

https://heyday.xyz comes to mind

Use the yacy-proxy funktion.

I had trouble with that, I ended up using this userscript https://github.com/JeremyRand/YaCyIndexerGreasemonkey

I use Google for this. It's really annoyingly good at finding previously visited pages.

Vortimo

A bit different, but I've been building something similar that runs locally: https://github.com/a5huynh/spyglass

You create some rules for topics you want to index and it'll go out and crawl them. Searching through it is a global hotkey away.

Spyglass looks amazing! Thank you for this.

Thank you! Feel free to ping me on Github/Discord if you run into any issues, always looking to make it better :)

Nice

I have wished for a while that browser would store the entire page of any bookmark you save automatically, and put a decent search engine on it. I wrote a script once to do it for my bookmarks, and it didn't even take that much space on my hard drive.

Your system could be a Firefox addon, kinda like what scrapbook used to be, but automatic. Even with a note system, and storing metadata, Zotero style, but without the need for the dual setup.

How mozzilla is not pursuing that kind of innovation in Firefox itself is beyond me. Instead they continously try to ape the Chrome UI and collect telemetry without permission so that they can give you "exciting" "new" features like colorways.

What a good idea. A search engine like Kagi could support importing your existing bookmarks as a custom lense.

Nice! How about getting it to automatically index your whole search history?

Not what you're going for -- you don't have a list of specifically opted-in 'bookmarks' to browse.

but I have often wanted "wait, what was that site involving X I was looking at maybe last week?"

It would also be nice to be able to search through my aggregated browsing history on every device I use.

Maybe I should open a feature request to Google/Fracebook to provide an API hook for that, since they probably already have all that information anyway.

This. I wonder if there is a way to direct my searches first to the domains I have ever visited. Oftentimes I will search for something that I am sure I’ve hit before but can vaguely remember which result set it was that scored my search.

My chrome history appears to aggregate my browsing across devices Anyways, so it should just be a matter of exposing that info

Firefox probably does too, if you're logged in.

Yes works, you just need to use the proxy function of yacy, and everything get s indexed.

Facebook doesn't share data out in APIs!

This is insane! Thanks for recommendation! Was able to set it up relatively quickly and now have a personal search of all the things that I find interesting, this is insane!

How does the process of bookmarking goes then? meaning, how hard is it to add something to crawl with depth0 on a day to day. Can it be done with a bookmarklet?

Ive never even considered something like this before, but its genius!

The offline caching sounds awesome.

Thanks for sharing

This is how I always imagined the search engines of the future to work. All the data is local first and the user is in control.

me too, when I first used delicious I was hoping that individual curators could provide a more in-depth "meta search" engine! Pocket is not bad, it provides some interesting links to new stuff on the web.

I really liked this setup. The only point of friction for me was adding the links to the index via the crawler everytime. So I created a Firefox Extenstion to do it directly from the address-bar.

If someone is interested, you can download it from here: https://addons.mozilla.org/en-US/firefox/addon/yacy-it/

Limitation: It currently supports only YaCy running on localhost and unprotected. You might have to configure CORS as outlined here -> https://github.com/tecoholic/yacy-it#configuring-yacy

Thanks for this recommendation! Do you only index things with a depth of 0? I just set Yacy up a couple days after seeing this post, and didn't realize what that meant at first.

Now I have 10s of 1000s of pages indexed after importing my bookmarks/history, and I'm wondering if it'd be more useful to _only_ index the pages I've visited/bookmarked, or if it'd also be good to crawl those sites further.

I guess one distinction would be whether or not I thought I could use Yacy as a full-time replacement for google/ddg. It'd be nice if I could index "everything", but then have a toggle to search only my bookmarks/history or something similar to that.

Thanks this look neat. Can you easily share index accros clients?

Edit : looks like the docker config allow to mount a arbitrary folder , that folder can be shared. I don’t need it to be concurrent proof.

Again, thanks this look nice.

If anybody is interested in this, there is also a service which offers very similar thing: https://historio.us/

That's mine! Feel free to drop a line if you need anything.

I have 3 tech related subscriptions; BorgBase, JetBrains, and historio.us. I self-host everything, but I've never found anything that replaces historio.us. It indexes just enough to always get me a complete copy of the data I want cached and the search results are just right. I often have about 1 page of search results when I'm looking for old info and I can pick out the page I'm looking for instantly.

I use it a lot when I'm learning something new. As I'm looking for beginner info I'll often find more advanced stuff that I'd like to try or learn at some point, but I don't have a good enough understanding to know if it's truly useful info. I historify those sites and move on knowing that I can find it in my historio.us search at a later date.

I also reference cached pages in a lot of my personal docs. I recently started using ArchiveBox for that, but the search doesn't make it a good replacement for the above use case.

I've been using historio.us since 2011 (!) and have never found anything to replace it. Great job!

Thanks, I'm glad you like it! I really should give it some love, but I'd need to do a fairly sizable rewrite for most stuff...

"i like historious" as the only review at the bottom gave me a laugh for some reason

product looks very promising!

Some UI feedback: I went to check out https://historio.us/ and on my macbook air I saw the top of the green button "see our plans". Took me a couple of clicks until I realized I had to scroll down to click the real "plans and pricing" button that was off my screen.

Ahh, thanks! The same link is at the top, but I'll see if I can reposition.

That sounds really cool. How difficult is this to set up?

How confident are you that this will keep your bookmarks safe? Losing indexed data might not be such a big concern for a search engine that is intended to re-index everything continuously.

I defnitely felt betrayed when I found out that Firefox automatically deletes old history :/ I guess they think that most users need to trash their profile to fix some obscure bug before that anyway...

Can it cache indexed pages? If not how do you deal with disappearance/changes online?

Yes, it does automatically keep a cache from when it indexed the site. I have it set to not automatically recrawl sites, so the cache is from when I added the site.

I'd love to hear more about how you set all this up, particularly, do you have some sort of extension or bookmarklet that submits the current page to yacy's crawler?

Nice! I’ve been working on and off on a similar idea (searchable index of link contents) as a cli app eventually web web frontend. It’s on Python so packaging has been an issue.

Didn't know about yacy – interesting! Thank you

This is fascinating! I've been meaning to set something up on a spare rpi for this and I hadn't heard of yacy before. Thanks!

That's an awesome idea. I'll have to look into this. Been looking for something like this.

That's amazing! I wanted to make something similar. Thank you!

This is an incredible idea.

How does yacy handle paywalls? Does it use the cookies from your browser instance or can it use bypass services like 12ft and co?