You're not alone. I have about 75 ebooks organized by a few directories depending on what type of book it is.

I just move around my file system like normal and then open them in light weight readers like Zathura. Browsing is the same as any other directory, the files open immediately, it works offline and it's easy to sync between devices.

I apply this same strategy to MP3s too.