What's your solution for a new tag being introduced late, which could/should be retroactively applied to already tagged media?
Re-tag the entire inventory by hand? Re-tag automatically with some ML/AI tool after you've tagged enough media to train a discriminator? Or just live with old inventory not having that tag?
So for example I have a top level `Documents/docs` directory. In this everything is recorded in format `yyyy-mm-dd Title.ext` format. I have a saved search which says "give me everything untagged". I will select each thing that is untagged and then tag it!
I have my version of a zettelkasten system, where I put a tag/ list of tags at the top of the note file (timestamp being the note name). Combining this with something like ripgrep allows me to search the note file contents by those tags, and keywords.
A tag I used, didn't seem like the right one for the content as it matured and grew over time.
This meant creating a tiny script which would take the name of the tag and the note name, in which it is being used and look at backlinks and forward links and rename the tag based on usage.
A never changing library, is a dead library.
In the end maintaining order always requires effort, if you have a hierarchical folder structure and you decide to add another category you have the same problem. And realistically folders are "good enough" for most situations, you can choose for each file how fine-grained the categorization should be and the mental overhead for finding a file is often not that bad.
I hope she managed to keep all her files and didn't lose anything.
I literally have no idea what to do with it :(
Note: I specifically remember these speculations in the lead-up to the release of Windows 98. They don't seem to be easy to google at this point.
At least one I tried ages ago was one you could use to browse your music collection, it automatically generated "directories" based on mp3tags.
> TMSU is a tool for tagging your files. It provides a simple command-line tool for applying tags and a virtual filesystem so that you can get a tag-based view of your files from within any other program.
> TMSU does not alter your files in any way: they remain unchanged on disk, or on the network, wherever you put them. TMSU maintains its own database and you simply gain an additional view, which you can mount, based upon the tags you set up. The only commitment required is your time and there's absolutely no lock-in.
> TMSU not automatically detect file moves and renames
I dunno if we can call it a tagged filesystem with this limitation. (Why not store tags, or at least a GUID, in file metadata?)
But presumably there is some API or ABI to monitor, or some signal emitted, when files are moved, added, updated, or deleted. However iwaitnotify works, could that be incorporated (even theoretically) into TMSU?
There are a couple very barebones wrappers around mv and rm, though they could be better (pass through arguments, etc.).
https://github.com/oniony/TMSU/wiki/Tricks-and-Tips#filesyst...
Those searches might be based on title, author/creator/contributor, publication/creation dates, assigned identifiers, full-text search, relations between documents, subjects or topics, document type / format, or other aspects.
How that is specified precisely ... I haven't settled on entirely, though short mnemonics, two- or three-letter where possible (ti -> title, au -> author, date, isbn, oclc, doi, etc.) might work.
If the filesystem lives under /docfs, then, say, /docfs/au:barrie/ti:peter+pan would turn up J.M. Barrie's Peter Pan. Specific document identifiers such as ISBNs, OCLCs, DOIs, or LCCSs could pinpoint specific documents, looser searches such as for Peter Pan might generate a list rather than a unitary response but those individual documents would be further distinguished by other properties.
The filesystem would be virtual rather than static-on-disk. You're effectively exploring the namespace. Creating this as a filesystem rather than, say, as a database query or application-specific data store means that standard filesystem tools would be available to interact with the contents, though modifications of works might require some additional work.
The system could also provide access to works not directly in the stacks, including remote resources (e.g., accessed via URIs and network calls), or applications (through some API). The underlying data store could be in any of several formats, and again the filesystem-based access could abstract between several independent stores.
Now all I need to do is build it ...
There are some similarities, and I'm thinking that TMSU might serve as a component of this system.
Definitely thinking about categorization differently (folders which are similar to this, tags, organic order by linking; bookmarks, notes, all are in the same boat as well).
His critique of the Dewey Decimal system was the first thing that came to mind when I read Johnny Decimal. And indeed, it is flawed the same way.
The ideal for me involves hypertext + (local, private) AI-assisted search.
None of this will replace reviewing/pruning your notes regularly (think: Andy Matuschak's fleeting and evergreen notes), but it minimises the friction.
These days I use TMSU via my own Emacs-based UI almost every single day, so thank you for that!
Naming schemes and directory hierarchies have some limited application, but ultimately there will be data which simply won't be shoehorned into any such system, and an externally-managed catalogue tying together disparate elements is required.
(Keeping that catalogue up to date and consistent is a whole 'nother issue.)
I do like the idea of a virtual filesystem in which elements are effectively search dimensions, which leads to an interesting notion that search is identity.
That is, a search will produce one of three possible result sets:
- Null, that is, no matches.
- Plural, that is, a list of matches.
- Unity, that is, one matching item.
In the last case, the search providing a single result is an identity of that result. (It may not be a stable identity over time, but it is at least for the present.)
Where a list is returned, the size of the list determines how usable it is, and how it is usable. Ten items can be quickly scanned to find the relevant item(s), if they exist. 100 or 1,000 items can often still be managed manually, though they'll typically take some time. Somewhere between 100 and a few thousand items, though, you're in the range where automated assessments or filtering becomes necessary.
Large libraries themselves typically have tens of thousands to millions of items. The largest book collections (Library of Congress, British Library) have roughly 150 million books (or equivalents). Other records may exist in greater numbers: periodicals, financial records, databases. Facebook has reported ~5 billion items posted daily for some years now. (I suspect most of those are trivial, but that still leaves a large number of potentially non-trivial items.) Surveillance and other large-scale data collection systems may be larger still.
Yes and no. I was specifically comparing it to Git Annex which is hard to categorize in these terms. It forces every file to become a symbolic link to the actual file living in `.git/annex/` and then every query temporarily mutating the hierarchy of directories storing these symbolic links. I found the latter disruptive enough (in particular for the directory mtimes) that I was actively avoiding doing any such queries. See: https://git-annex.branchable.com/tips/metadata_driven_views/
On the other hand my current setup involves TMSU queries which result in virtual Emacs directory editor (dired) views that don't affect anything else. I don't even use the FUSE functionality of TMSU.
The core problem is that there are formats and storage modes which don't readily allow for the modification of the item itself. Editing PDFs is already a pain, applying metadata to some entry in a database or wiki, or third-party website, isn't possible at all.
The remaining options seem to me analogues of practices with books.
It's possible to "rebind" an item and include biblographic information into the equivalent of fly-leaves of that work, much as a library may rebind a book and apply a label with inventory number(s), call number(s), and/or bibliographic data to that book. Since physical objects are inherently modifiable and enclosable, this makes sense. The digital analogues vary, but are at least theoretically available (e.g., enclosing a work in an archive format which includes metadata. See the WARC (Web ARChive) file format for example: <https://en.wikipedia.org/wiki/WARC_(file_format)>. Epub files and software packaging formats such as RPM and APT are other examples of standardised file structures which encapsulate others.
The other practice of a library is to abstract out the metadata to a catalogue, effectively a metadata index.
In a physical library archive, this involves a cataloguing process, as part of item acquisition workflows. The metadata for many traditionally-published works is already centralised such that it can be obtained from specific organisations such as the US Library of Congress, the British Library, the OCLC (originally the Ohio College Library Center, which has both its own item identifier and manages the Dewey Decimal Classification), the International ISBN Agency or one of its national affiliates (e.g., Bowker, in the US), the International DOI Foundation (for DOI assignments: digital object identifiers, used extensively in academic journals). Circulation of items is managed through a circulation desk, for both external lending and managing, tracking, and reshelving books used within the library itself, but not being borrowed externally.
For digital media, the equivalents would be either some sort of management system, which would require an application-specific interface, or a filesystem which incorporates not only metadata but workflow management. I'm leaning toward the latter concept as more universal, though that also raises the question of how to deal with workflows in which contents leave or enter that filesystem context itself.
But with a filesystem, you have a number of additional possibilities available:
- The pairing of works with metadata is automated.
- Workflows can be integrated into filesystem actions. The act of creating a file would also create the file metadata (to a greater extent than present inode entries do).
- Additionally, introducing the notion of process status means that the filesystem itself could distinguish between works which have no or only default cataloguing data applied, those which have had additional metadata added (say, from an external look-up or automated heuristics based on file contents).
- Renames and deletions are now managed through the filesystem itself rather than third-party tools.
- I'd like to see both versioning (changes to a given file) and relationships (source, derived, referenced, and referencing works) tracked as well.
- Different forms of a work could be tracked together. The markup-language source and generated outputs (PDF, PS, ePub, HTML, plain text, etc.) versions of a text. Translations. Audio formats. Different performances of a work. Optical scans of printed material. Photographs of visual or plastic arts (sculpture), or architecture. See FRBR (Functional Requirements for Biblographic Records), and the Work, Expression, Manifestation, Item distinctions: <https://en.wikipedia.org/wiki/Functional_Requirements_for_Bi...>).
- Ideally, some sort of highly-invariant fingerprinting such that different versions of a work can be identified and matched despite different formats or slight modifications (intentional differences between editions, translations, errors or damage introduced over time). Traditional whole work hashes fail to offer this, though segmented and normalised hashes or vectors might be able to do so.
Again, the largest problem with a filesystem approach is in imports and exports from that filesystem-based archive. That would probably best be achieved through wrapper formats and applications or servers for other organisations or the general public.
Exactly what I needed for my own note storing system where I can tag and describe various pdf's like books, lecture notes, and papers with tags, then use `rga` for searching inside of the pdf's for content!
*If using a unix derived filesystem.
In other news, if anyone knows how to restrict a search in an org mode buffer to search only the headings, that would be very helpful! Right now I'm using an external grep wrapper, restricting to lines that begin with an asterisk.
Having said that, I will be the first to admit. the reason hardlinks are not often used is that it is a real "here be dragons" part of the map.
10 years ago I spent a couple fun weeks categorizing all my music. my schema was something along the lines of
every file was linked to
artist/artist_name/song_name
band/band_name/song_name
song/song_name/band_name
date/YYYY-MM-DD/song_name
genre/genre_name/song_name
playlist/playlist_name/index
etc, etc
then I had fun making up a small set of cli utilities to find tags, create and manage playlists etc. then A. it turns out I don't really listen to music. and B. the whole structure wanted to explode every time I would try and move it to a new host. so it probably still sits there on one of my file servers but now as a million duplicated files.So long as that goal is achieved, the rationality or consistency of that assignment is a strongly-secondary consideration. If the book has a place, and can be located, where it happens to be relative to other related works, or unrelated ones, really doesn't matter.
Searching for works based on similarities or relationships is a problem of the indexing system. I should be able to find works on a similar topic, which reference or are referenced by another, by author, by institution, perhaps by publisher, by publication date, etc. That information would be carried within a catalogue, but need not determine the location of the work within a physical archive.
Fuzzy is ok, not everything needs to be perfect.
> Tagging/labelling with attributes is the only viable solution that I've found.
Categories and tags have different use cases. Categories are the best solution, when there is only one possible location for an object, like with physical objects, which cannot appear at multiple locations at the same time. Tags make sense, if you can offer multiple entry points to reach the goal, and have the option to manage them all on a reliable level.
And nothing speaks against using them both, with different valuesets. Like with digital files, you can have categories for managing the filetypes (books, videos, etc.), but then use tags to manage the content in each category.
> This book is fiction/science-fiction/fantasy.
Isn't this kinda redundant? science-fiction is already fiction, it's in the name. Reads more like a hierarchy. Which you of course you can also with tags. Any fantasy-tag automatically inherits the fiction-tag too.
This is nit-picking the specific example rather than engaging with the GP's point. There are definitely genres that span the fiction/non-fiction divide (like romance, horror, politics, judaica, and many more). Interestingly, hierarchical systems used by libraries tend to ignore genre. The Dewey Decimal system doesn't classify literature by genre but instead by language, though some overarching genres (drama vs poetry) are treated as subcategories of language (e.g., German poetry is 831, French poetry is 841), but only for major European languages.
It's showing that tags can be fuzzy too. So the initial complaint is not avoided, just moved.
They effectively use hierarchical tags, with the set of tags very carefully curated.
The progression usually goes.
"we want to label our things."
"There is a natural hierarchy that we want to be reflected in our labels.
"Some things are not labeling well. instead use multiple labels per thing."
"There is a natural hierarchy that we want to be reflected in our labels."
The unix files system is a good example of the latter. it favors the hierarchy but allows each file to have many names if that is what is required. The main weakness of the unix filesystem is that there is no reverse map, that is, finding a file from its name(s) is simple. finding the names from the file involve searching the whole filesystem.
The only solution I've found is being utterly ruthless about deleting things. YAGNI, YAGNI.
Alongside this, topic-based folders are fine. Not many things are in both "econ" and "sf&f", and when something is relevant to both, such as Krugman's paper on interstellar trade, it's easy to decide where it goes (econ) and where an empty file called "Krugman, Interstellar Trade: see econ" goes.
Empty file with a textual reminder, not a softlink or hardlink. Those are brittle. So are attributes.
Use broad categories, though. "sf&f", not "SF&F - Fantasy - Urban Noir - West Coast - Mermaids". The definition of "broad" depends on your areas of expertise.
Tagging/labelling with attributes is the only viable solution that I've found.
This book is fiction/science-fiction/fantasy.
This document is technical-reference/specification/ratified
I believe this was the notion of the now defunct WinFS built on top of a database engine.