I totally feel your pain. I have been slowly building a local search engine for navigating the ebooks, PDFs, and other documents I've got stored on my NAS device.

I ended up replacing my NetApp StorVault (6TB (4TB usable), RAID6) with a FreeNAS box from iXSystems (24TB (20TB usable), RAID6) which cost me a bit more than $2500. (it is also much quieter than the StorVault was :-)). And on that 20TB I've stored a bit more than 200 book volumes that I had digitized at 1DollarScan, probably close to 1000 magazines (scanned generally manually with my ScanSnap 1500), and perhaps as many PDF documents that were never printed to begin with (like data sheets). I also have my music collection, and I have recently looked at putting movies on there as well since the streaming services are letting me down here as well.

At the end of the day it is the 21st century version of one's personal library.

I have had a FreeNAS mini for 5 years now. Best 2K on tech I have ever spent. So much better then rolling my own (which I used to do). 4x4TB (2 striped 4TB mirrors). Thinking about replacing the drives with 8TB ones. I had one drive fail in 5 years.

Currently I can only really do metadata search, but I'd really love to set up fulltext at some point. I have this dream where someday I can do things when the Internet is down.

The other thing I'm interested in, and wonder if you have thoughts on: When you pass, your personal library is inherited. Will people know what to do with your digital library? Will it be useful? Will people want it?

My goal is exactly that, usability without me present :-).

The tools for processing PDFs into searchable text have a lot of warts. For a while IBM was offering a free Watson service to do this (now its part of Watson Discovery) which has some warts. I did manage a set of perl scripts that would post process the statements that I downloaded from the bank into CSV files, but I would still like to pull tabular data out of PDF book scans to make the data they provide more useful.

I have a simple frontend based on the perl Mojolicious module which Blekko had developed as part of another project but my indexing tools are still quite primitive. Simple bi-gram and tri-grams, and a growing synonym index. I don't give it enough queries to use my own traffic for ranking feedback. So basically everything is nearly equal rank. Basically I am about to the AltaVista level of search capability :-).

The vision is it just runs as a server and anyone on the same network can access it like a web service an pull up documents (and in the future media) of interest.

For my eBooks, I use a java app called DocFetcher. It's indexing and word search capabilities are pretty nice, and I've had great luck with it, especially as I can set it up portably.

Did you get very far with other enterprise stuff like Sharepoint?

I didn't try Sharepoint, although I looked at Elastic Search on AWS briefly. The goal though was to have all of this stuff on premises both for latency reasons and to maintain a credible defense should someone come after me for copyright infringement.

A bit of a tangent, but if you use a Mac, I've found that Dash is really great for getting things done without internet access. It's main purpose is to pull in local copies of docsets and search those, but it can also pull in sections of stack overflow (such as all Python questions, or all Pandas questions), and it integrates with things like Alfred, so you can do scoped searches without any GUI interaction.

So do you have ~19TB still free? It doesn't seem like your items would take too much space.

I do still have a lot free, when I replaced the StorVault it was about 75% full (3TB out of 4TB), I'm up to about 5TB now. But there are more things on there as well. The Postgres data for my local gitlab instance for example, and time machine backups of my Macbook pro. So it isn't all library stuff.

There's actually great tools to help you build and organize that library... assuming you pirate everything.

I'm just hoping the movie/TV industry gets the picture soon and goes the way the music industry has with streaming.

I don't pirate everything. In fact, I don't pirate anything. For instance, with regards to eBooks, I hit Humble Book Bundles hard. They're bloody incredible for the discerning legitimate DRM-free digital content purchaser.

I was more just making the point that it's actually easier to do library organization for movies and TV as a pirate than it is as a legit owner of Blu-Rays for example.

Sad state for the industry, really.

Meh, it isn't that difficult with ripped blu-rays either. Plex can do it quite well.

I've found Plex great for movies, my ripped television shows on the other hand tend to be weird. Since the order of DVDs, BRs don't always match up to the airing order there can be a lot of manual work in cleaning that up. Plex is also not super great with music yet. It's getting better but it's definitely having problems with a lot of my music since I've got a lot of local/indie bands.

You have backups.

You don’t have to throw away the disk after you rip it. The disk itself is a great backup.

Personally, I prefer everything to be in my Plex server, but I’m in no rush and just rip as I go. I’ll probably never actually rip every one. By the sounds of it, I don’t even have anywhere close to the number of disks to rip as you, so I get it no being appealing.

Backblaze is $5 a month for unlimited storage. If your hard drive fails, you can order a hard drive with your data.

You better store encrypted if you're pirating movies/music.

It's rather easy to get TOS'ed even if you match filenames of common pirate name signatures.

Backblaze insists that your data is encrypted and that they can't see it. It's supposedly not like Box/Dropbox/Google Drive.

Do you have any examples where someone's Backblaze account was terminated for backing up pirated media?

I can comment on this - it's flawed. They encrypt locally and only send encrypted data to their servers, true, but they deduplicate that data between customers meaning they know file hashes or something similar and could remove pirated content if they so desired. Second, in order to get your data back their solution is to go ahead and type your "private" encryption passphrase into their site so their servers can then decrypt it and send you a zip or hard drive full of decrypted data.

I still suggest it for friends and family because it's cheap and damn sure that encryption is better than nothing, but if you have a lot of sensitive data I wouldn't recommend it. If you demand privacy, use Restic and Backblaze's B2 service, you're paying per GB then though.

> I still suggest it for friends and family because it's cheap and damn sure that encryption is better than nothing, but if you have a lot of sensitive data I wouldn't recommend it. If you demand privacy, use Restic and Backblaze's B2 service, you're paying per GB then though.

First, though, find out if your friends or family have an Office365 subscription. A lot of people get one because they want the Office apps, and don't make heavy use of the 1 TB of OneDrive storage that it includes. Restic vis rclone should be able to use that for backups.

Backblaze is cheap, but using something you have already paid for is even cheaper. :-)

How many of your friends are going to know how to use rclone?

You'd only need rclone if you wanted to use OneDrive with restic, because restic doesn't have direct OneDrive support.

Several other cloud backup programs do have OneDrive support built in, including Arq, duplicity, duplicati, duplicacy, and GoodSync. There are probably others.

Luckily passive commercial scanners don't require much; tossing it into an encrypted zip with everything set to fastest still runs over everything passive that's not nationstate

It makes more sense to encrypt the entire data store and just sync the entire data store. This is proof against anyone who isn't willing to hack your computer or break into your house while its open.