Work migrated to the cloud.
Important for technical users, not so much regular people.
Underappreciated aspect of this is trust. As much as "taxonomy is mostly a search problem", search is in large part a trust problem - trust that the search was done exhaustively, and if it returned no results, it means there aren't any.
This is a problem that IMO most offerings are blind to. One big offender for me is Windows Explorer. It has a search tool that can search both in names, metadata and file content. Yet in the past, it frequently failed to find files I knew were there. Truth is, it may not have been even looking in the right place - it depends on what's been indexed and how, which is information you can find somewhere in the system, but notably not in the search interface itself. I've stopped using it long ago, as I don't trust it at emotional level. I prefer to literally walk the filesystem structure by hand.
And, on Windows, I at least get that option. This is the fallback, the baseline: even if I don't trust a search engine to be exhaustive, I can cope with it as long as I can perform an exhaustive search manually. I.e. as long as I have a way to list everything. But this, then, is almost universally missing from cloud offerings.
(There's a whole rant to be written about the incredibly dumb idea of hiding the filesystem / database from the end-user, but I'll skip it today.)
> there were startups a decade ago trying to address the search problem on the desktop but they all got acquired or folded; I don't understand why. The problem is real and seems quite solvable; yet it seems there's no actual market for it. It's a bit of a mystery.
Nobody really wants to solve this problem anymore, because it conflicts with the major thing vendors want: for you to move your data into their cloud. The desktop, and the ability to own your data, survives only because most cloud systems are still shit. This might eventually change, and in the meantime, the market definitely isn't interested in helping you own your data.
Desktop-focused databases are of course widely available (Paradox, Access, LibreOffice Base etc.) but the volume of data you could manage with them is of course limited. I.e. non-toy datasets will naturally be hosted on some sort of "cloud".
What's the difference?
> Heuristic search is going to be non-exhaustive by definition.
To me, a non-exhaustive search is broken by design, because it will miss results. Making data accessible only through such search means there will be a hole in the system into which some of the data falls, never to be seen again.
https://www.elastic.co/kibana/
At work we have a search interface powered by Postgres that uses the full text index but also uses GIN indexes on arrays to index things based on categories/tags.
In the scenario where you're searching for a file in a group of low-hundreds of files, it's nothing but a bad cop-out excuse.
Maybe, but that doesn't explain why there are no startups on the case, that wouldn't have this conflict of interest.
(1) Performance. Back in the day people frequently turned full text indexing on Windows off because it would slow down their computer too much. People won't be happy with the overheard of a search engine that is always scanning 10's of GBs of document.
(2) Search quality. People are familiar with Google being an effective search engine but they've certainly tried a number with terrible relevance scoring and probably learnt that it is not worth trying the search on a web site or the search on the help of an application. In the case of Elasticsearch the default similarity is BM25 but that has two tunable parameters. There are other similarities you could use but most of them have tunable parameters. It makes a real difference what you choice and there is a methodology for tuning them that is now built into ElasticSearch.
https://www.elastic.co/guide/en/elasticsearch/reference/curr...
I talked to about 20 vendors of full text search products and found that only 2 out of the list regularly evaluated the quality of the results, 1 of them just did it so they could get some advertising by being on the TREC leaderboard. They told over and over again that customers didn't care about search quality, they just wanted to see a list of 350+ data sources that the product could index.
This seems to describe corporate buyers ticking boxes on a form, more than actual users.
The trust issue you mention may be a clue.
Computing is a supply-driven market. The demand is there, it's just partially latent, partially ignored. Vast majority of technology users have no choice but to choose from what's being offered, and the minority of tech-savvy users with opinions are increasingly too small a niche to support "power user" tools.
Most people have Word Documents, PDF files, and other things that need a more complex indexing strategy. Also a lot of people have lots of image and audio files which pose their own challenges, namely indexing textual metadata and possibly some indexing of the content.
And yes, any taxonomy is by definition always a subjective, biased construct. But this is often useful in helping to reveal the preferences and motivations behind the creation and curation of the corpus itself.
It seems what matters is not how files are stored/organized, but how one can find the files one is looking for. Taxonomy is mostly a search problem.
Yet, although one is usually capable of remembering specific or unique details about a file, it's still incredibly hard to search for a file or its contents effectively.
Dropbox, to pick just one example, ditched all its advanced search functionalities a couple of years ago and only allows to search for any keyword (not all of them), with forced stemming, and no filtering of the results.
I seem to remember there were startups a decade ago trying to address the search problem on the desktop but they all got acquired or folded; I don't understand why. The problem is real and seems quite solvable; yet it seems there's no actual market for it. It's a bit of a mystery.