Former librarian here, now at a tech co. This is exactly the domain of information science, and its a salutary tale in two industries talking past one another. Librarians have deep training in the science, philosophy and psychology of information storage and retrieval. Most of the time you think of things like Dewey numbers on library books buts its much more than that. At the dawn of the second internet age (circa 1991, think gopher, WAIS, Archie and a nascent thing called the web) there was a boatload of discussion around what this Internet thing would mean for information.

Then, the tech bros arrived and after a few abortive attempts to catalog things for themselves (webrings, portals, yahoo) the industry collectively shrugged and decided to ignore the problem, assuming that a search engine would always be able to pluck your favorite needle out of the haystack.

Except that, today, it cant. Intranets are essentially corporate graveyards of content. The public web is a webring of 7 or 8 megasites that vacuum up all searches and make it all but impossible to break out of their domain. That article you read in 2005 about XYZ? Forget it, you're never finding that with Google.

Do you have any pointers to things worth reading? I've always sworn if I started a tech company one of the first 10 employees would be a librarian because it needs to be someone's job to organize the information, and you're right the automatic systems for doing it are horrible.

Search isn't enough because search doesn't help you if the philosophy behind how the information is organized doesn't make sense -- if what you need is spread around between 100 slack messages, emails, and unconnected unmaintained wiki pages. You need someone (or ideally a team) whose job it is to one one hand organize that information themselves, and on the other hand create a framework so it's easy for the non-librarians to put things in the right place.

But right now the majority of tech companies are like libraries without librarians, the patrons are just wandering around sticking the books on random shelves and wondering why nobody can ever find anything.

Organizing Knowledge by Patrick Lambe is what I started with.

Cool, thanks. Will check it out.

I'm curious what your thoughts about Cory Doctorow's Metacrap [1] essay which I think summarized a lot of the problems with the semantic, informational organization approaches of the early-mid web. Are you also familiar with research in informational sciences these days?

[1]: https://people.well.com/user/doctorow/metacrap.htm

Yes, please tell us your recommendations for informational retrieval / taxonomy systems - what are the current best practices for the different mediums?

Have you considered going into SEO? Combining SEO expertise w information science sounds like market dynamite. There is probably a 5k/month blog in simply applying information science concepts to SEO in practice, to say nothing of the consulting gigs, etc.

I understand why html is the way it is. I can recall looking at it very early on in its development.

But I think we need more structure to how we represent knowledge get to something like a semantic web.

If the knowledge was easier to parse, it would be easier to break out of the megasites.

I'd agree with knowledge organization. It's either you have to root through academic texts or try and navigate the spammy internet with no really happy medium. It's almost like there's a complete lack of quality middle ground information. It's either total SEO garbage or very low quality entry level information or incredibly specific/dense academic content and the middle ground is missing.

Back when I was in grad school, PhD theses were indispensable for actually learning my way around a topic.

The academic literature itself (even review papers) was way too terse and (I believe, semi-intentionally) obfuscatory.

I would assume that it hasn't changed much since you were in grad school.

Definitely, transfer can only happen when knowledge is organised and understandable by a variety of stakeholders, with different backgrounds (education, languages spoken, years of expertise)