News for hackers. Stuff that matters.
Re:An Introduction to the WARC File by beeboobaa on Monday January 29, 2024 @09:19PM (#39185639)
Anyone archiving every website they visit? I often want to search for something I read recently, which is awkward to do if you don't remember where you read it.

I'd like to store everything I visit with my browser once the page is loaded, but haven't found any nice solutions for this.

Re:An Introduction to the WARC File by autocole on Monday January 29, 2024 @11:31PM (#39186412)
I'm using the Forethink web extension for that. It keeps a local index of every page I visit for searching

https://forethink.ai

Re:An Introduction to the WARC File by daniel31x13 on Monday January 29, 2024 @10:12PM (#39185985)
Created Linkwarden exactly for this:

https://linkwarden.app

Re:An Introduction to the WARC File by beeboobaa on Monday January 29, 2024 @10:54PM (#39186219)
Nice! This is very close to what I'm hoping to find. My main issue is that it seems the pages are fetched from the linkwarden server, instead of preserved by my browser extension. For example if I use the browser extension to save a HN page, the saved version is an unauthenticated view of the page instead of showing my username.

If the pages were preserved directly from my browser now additional requests to the server would be required, and it would be realistic to archive everything automatically, instead of only when I press the button.

Re:An Introduction to the WARC File by 112233 on Tuesday January 30, 2024 @03:34AM (#39187705)
was that not literally a built-in feature of Opera 12?
Re:An Introduction to the WARC File by bandie91 on Tuesday January 30, 2024 @04:12AM (#39187928)
YaCY (self-hosted search engine + index) can do this when using it as a proxy.

https://yacy.net/

Re:An Introduction to the WARC File by jeroenhd on Tuesday January 30, 2024 @06:38AM (#39188909)
Interesting! How well does this proxy deal with bot detection (Cloudflare and such)? Most proxies I've used, usually without archiving capabilities, have me fill out CAPTCHAs for every other site.
Re:An Introduction to the WARC File by robjwells on Tuesday January 30, 2024 @06:50AM (#39189016)
On MacOS HistoryHound gives you a searchable index of webpage content, though it doesn't store the full contents of pages in a way that can be viewed offline.

https://www.stclairsoft.com/HistoryHound/

Re:An Introduction to the WARC File by iansinnott on Saturday February 03, 2024 @10:40PM (#39247332)
A bit of a late response, but yes I've been storing full text of every website I visit and it's excellent for finding stuff again.

The idea is to index pages as you visit them using a browser extension, thus avoiding all the pitfalls of being treated like a bot.

Here's the project: https://github.com/iansinnott/full-text-tabs-forever