Ask HN: Do you keep a personal knowledge repository?
And if so, which software do you use? I'm fascinated by storing things that I learn outside of my own brain. Ideally I'd love to create my own mini intranet of information, basically a personal wiki, that I can go to first before having to Google things, especially for my field of study.
I've looked into various Wiki packages but none quite seem to offer what I'm looking for (Mac, no need for a webserver, cross device functionality, math markup, searchability, tags)
If you do keep a personal knowledge repository, can you please describe your workflow for using and maintaining one for all the new knowledge you acquire?
(
123 comments
| 136 points
| on HN
)
If it is for the long haul, plain text is the only way to go (org-mode, etc).
Any proprietary commercial solution will fade away in a handful of years. At best, you might be able to extract the content before they shut down but structure and metadata may be lost.
Open source self-hosted solutions will last much longer but even in that case it might get to a point the source has been abandoned so long that it becomes difficult to keep going.
Only plain text will survive. I have plenty of notes in a more or less organized directory tree, dating back to the late 80s.
Furthermore, a personal knowledge repository may contain information that is pretty much timeless. Say you read Rudin to learn analysis in 1988 and took some notes. Why would that be outdated?
Some academics build huge perosnal knowledge repositories that they keep growing during their careers. These will often span longer periods than 1988-2019. See e.g. Luhmann's Zettelkasten.
The advantage of Org is that it is very readable, as it descends from an outliner, and has excellent tooling. There are options for including tables, equations, bibliographies, links to many datatypes, attachments, extensive date support, etc. Lots of export tools, interpreters to show views of your document, and many other things I have not touched upon.
Yet it is quite simple hierarchical plain text. So it will remain accessible in the future. Aside, Emacs has been around for very long, and will probably remain around much longer due to the Lindy Effect.
I create separate decks for different fields and then only if necessary do I create subdecks. I also tag all the cards that I create and make sure to spend a lot of time to ensure that it's made and tagged properly. So, whenever I learn something new and I believe at that point of time that it's worth remembering, I create a card for that.
[1]: http://ankiweb.net/
If you want more theory about Anki / Spaced repetition, https://www.gwern.net/Spaced-repetition is the ultimate resource on topic
[1]: https://jblevins.org/projects/deft/ [2]: https://github.com/ashton314/homebrew-mm
https://github.com/alphapapa/helm-org-rifle
I don't know if Deft recognizes org mode entries, but org-rifle provides entry-based search.
Deft will scan any plain-text file, according to the set of file endings you give it, and it will put your cursor at the point in the buffer where one of the search words occurs, but it won’t automatically extract the ending, AFAIK.
I should mention that deft does fuzzy matching, or at least words don’t have to be contiguous.
There is a great Org Mode tutorial [1] on YouTube by Rainer König.
The only « problem » is that you need to learn Emacs. I recommend Emacs Doom [3]
Also, I'm experimenting with making my Org Mode notes public. Here's a humble beginning [2]. I only publish them once they are cleaned, so it's just a tiny bit for now.
[1]: https://www.youtube.com/playlist?list=PLVtKhBrRV_ZkPnBtt_TD1... [2]: https://zaiste.net/programming/ [3]: https://github.com/hlissner/doom-emacs/
https://gitlab.com/drummyfish/my_text_data
Most people that aren't me would probably be interested in
https://gitlab.com/drummyfish/my_text_data/tree/master/cheat...
and
https://gitlab.com/drummyfish/my_text_data/blob/master/links...
Reading this will change your life: https://www.amazon.com/How-Take-Smart-Notes-Nonfiction/dp/15...
I'm committed to text-only information. No matter how many notes I'm writing, I can always use grep or write my own programs to find what I want.
If you're on a Mac, The Archive is great (https://zettelkasten.de/the-archive/). It's what I'm using
zettelkasten.de has lots of good reading (thanks, Sascha and Christian!), but look to the other sources I've listed to get a more thorough idea of what a zettelkasten is (I call mine a thinking-box) and how to use one.
This setup has worked well for me for over 6 months now. I do not worry about email / git becoming obsolete. Or how I would move my email/git data when I need to switch vendors. I can access it via any device I own or through someone else’s. Stuff I want to share with people are hosted via git pages or sent via email. I have a plethora of client applications to choose from. And above all... it just works.
I can give out more details but I feel anyone’s personal information needs can be met/exceeded with a combination of email + git + calendar.
I do not move too much information from email to wiki. I'm quite mindful of it actually. Wiki gets written and re-written, cross linked and edited. I keep it consistent. Helps quick lookups. I very frequently use markdown files in topic specific directories. Helps store information gathered from deep dives on those topics. Markdown file names are also consistent - YYMMDD-topic-detail.md. Helps sorting by date.
I use the wiki mostly for the free hosting and the links. I could very well use a static site generator when there's enough content in a topic's directory to be shared as a mini website. A link to the wiki on the home screen comes quite handy.
For more general knowledge, Semantic Mediawiki (http://pchr8.net/f/), but it's basically a big link database now. I can sort it by rating / complexity / to read / domains / keywords and found it quite flexible.
Also, Anki for things I want to memorize. https://www.gwern.net/Spaced-repetition describes the theory quite well.
For me, I use Orgmode, plus a mess of PDFs in folders on Google Drive. I've been looking into both Polar and Emacs as potential replacements for document organization.
Right now I have a single folder that I save new things to, and I have a recurring task to go back and file documents regularly. It's not perfect but it works. I don't often reference my pile of PDFs, though; I might use it more with tagging options, which is why I'm looking at replacements.
I used to use Evernote. There were some things I didn't like about it, although it's been so long that I couldn't tell you what they were anymore. It might be a good system now.
Yeah. I am currently using org-mode (plus org-wiki, some bespoke elisp and a few shell scripts) for my own knowledge base, but Oscean/the Nataniev ecosystem in general has impressed me.
plus Zotero for papers/PDFs, plus a pile'o'files stashed using the Universal Decimal Classification for its folder tree, plus assorted odds and ends.
In fact, I spent this week messing around with some of its concepts - Indental plus Runic plus a lightweight markup language loosely based on txt2tags + sexps and which should be easy to render even outside the browser.
The idea is to move away from documents (which org-mode is very good at), and towards smaller, rich chunks of information (which org-mode can't handle that well - I hacked things up with properties, but that namespace is flat).
I've noticed this weakness in org-mode as well. It's really a fantastic tool, but it still has a few gaps here or there. I'm optimistic about its continued development though.
It's good to hear that you liked the Oscean/Nataniev tools; I haven't tried them and it could easily have been all style no substance. They certainly sound impressive. From what I've read, I have a feeling there might be something worthwhile there. And XXIIVV itself looks amazing, although I know that's just one implementation.
Org is amazing, yes, but I'm not too optimistic about this - even if some of its more noticeable warts got fixed (a big one, for me, is the fact that you cannot nest quote blocks), it would still represent a tree of nodes, and trees are somewhat limiting where it comes to knowledge representation.
You can organize information in a tree, but it doesn't neatly fit that model, and just deciding where to put each snippet you captured can be a huge hassle.
> It's good to hear that you liked the Oscean/Nataniev tools; I haven't tried them and it could easily have been all style no substance. They certainly sound impressive. From what I've read, I have a feeling there might be something worthwhile there. And XXIIVV itself looks amazing, although I know that's just one implementation.
I'm not too sure, either, but I'm going to take a stab at implementing something Oscean-like. I want to see where this approach can take me.
Evernote has worked well so far and me trying to switch to OneNote didn't work out as well. I'm still open to alternatives, but Evernote has kept my mind organised without a doubt.
It's fast and has good search, a good editor, and uses simple hierarchical organization + tagging. I like that it gets out of my way and does a good job of handling everything from code snippets to freeform thoughts to recipes.
BookStack is best for things that 'look like' documents. Mobile editing could also be better - you have to click a maximize button to get a usable view of the editor and the editor UI still takes up a lot of space. It is web based, but it is very lightweight and I have found it very easy to host.
[1] https://www.bookstackapp.com/blog/beta-release-v0-26-0/#mobi...
private, a folder called "notes", with almost a thousand short text files. Workflow for it? Use grep to find what I want. I had a system of folders per topic but that became a mess. Grep is good enough. The notes folder is synced with synching to multiple systems and my phone. On the phone I have termux, which has grep and markor as text editor.
As I do stuff, things move from the Current Tasks file to a Logbook file. Beyond 'accomplished todo's, that file gets stuffed with, for example, interesting quotes I run into, command lines I think I'll need again, and a record of what I've been up to. When I need to put together a status report, or figure out for myself where my time went, or recall how to do something intricate that I haven't done in six months, I turn to my Logbook.
With these two pieces, I feel like I've got a handle on what I want to do, and a record of what I've done. Now, if I just could find the time to follow through...
[0] http://freemind.sourceforge.net/wiki/index.php/Main_Page
[1] https://en.wikipedia.org/wiki/Getting_Things_Done
At some point, I ran across the idea that a library card catalog is really just a combined topical index for the entire collection, with each entry being a card. That seemed like a decent model, and I’ve been happy with it so far.
My system is currently a physical card index because I needed the flexibility of pen & paper to figure out how I wanted it to work, but I’ve been slowly computerizing it as I figure things out- you can certainly do something similar with flat text files, spreadsheets, or any other general purpose document software.
So now, I have three interacting systems: a citation list that records where I can find things to read them again, a combined topical index of everything I’ve read, and a flash card deck of things I want to remember without having to look them up.
When I run across an interesting article on the web, for instance, I save a PDF of it to Dropbox for safe keeping and print a copy to read away from my computer. As I read, I’ll make margin notes and have two different colors of highlighter: one to note index headings and another to mark passages to be turned into flash cards.
After my morning flash card review, I go through the material I read the previous day and turn all of the notes and highlights into cards(2). Once everything has been moved into the indexing system, I have no further need for the physical copy, so it goes to the recycler.
(1) Of sufficient length; generally 3 or more printed pages.
(2) Initially by hand, now with the aid of a custom piece of software
By haphazard I mean to focus not only on acquisition but reflection / contemplation: Collecting thoughts is one thing, putting them together into a cohesive and coherent body of knowledge is another. So, that's what I'm working on.
pmwiki for now. Workflow is personal domain, putting browser or window on one side of the screen and what I'm reviewing, or what I'm thinking / text editor (sublime text) of the other side. Putting into a DIY solution based on flask simply because that's flask practice.
I also like pens and paper and whiteboards. So take pictures and use Powerpoint's 'remove background' feature to have a liberal amount of pictures.
A bit rambling, but you asked and asking is usually best! So.. :D !
This way I can edit the files online on github.com, on my desktop and push/pull the changes. On my iOS device I use http://workingcopyapp.com - I can edit and create files as well as push and pull changes.
On my Desktop I can use any tool I want to search and edit the files, on github I have full text search and working copy also features full text search.
Most importantly: I own my data and they don't up in some database I need to worry about.
I also manually maintain an index.md file to categorize things, but it's not really necessary when you can just use grep or your editors search feature to find what you need.
I have been using personal knowledge repositories for more than 20 years now, and if there is one lesson I have learned: never, never ever use any proprietary, platform specific - or worse, web-based - solution, no matter how good it is. Plain text is the only way to go.
My favorite tools in the past were Tinderbox from Eastgate and Curio from Zengobi, but both were Mac only and I had to leave them behind when I switched to Linux a couple of years ago. I collected some experience with Scrivener and used PMWiki a while, but the first is not really supported on Linux and the second wasn't somehow responsive enough for my needs. ZIM wiki on Linux is actually quite nice, but when I discovered orgmode it hit just right and I never looked back for other tools.
My org-file organisation: I have one super-big master org-file in which I collect just about everything regarding software development, technology in general, life lessons and stuff. For customer projects I generally create a dedicated file in which I not only collect knowledge regarding that project, but also do the project management with tasks, documentation and stuff. If one of the topics in the master org-file grows really big (this is for example the case, when I start to use a new programming language full scale for productive purposes) I will migrate that node to its own file. I also keep most of my bookmarks as links in org-files in a proper context, and I like to use links to files in the local filesystem (like PDF-manuals) within my org-files, although this requires some discipline in the organisation of the local folder hierarchy.
The only other tool that I started to use besides org-mode, just a couple of weeks ago is Anki, for stuff that I want to learn by heart.
In what way? I have some pretty enormous ones and they seem fine so far. I'd like it if FreeMind were more actively maintained, but at least it still works basically fine on the latest version of Java.
Preferences -> appearance
Preferences -> behavior Control-rightclick -> hide note windowIt’s good for free-form notes, but also has databases and relations for more structured data. Also has top-of-the-like Markdown and HTML export to get your data out.
You can publish parts of your workspace publicly read-only, or share via email with guests.
I have tried many tools throughout the last couple years in an attempt to declutter my mind and manage my time better each one failed to meet my needs at some aspect: clumsy-UI, not online, not accesible by smartphone/tablet, lacking features and similar.
I can highly recommend this project to anyone skimming the comments - I think it'll suit most people's needs.
https://github.com/jhadjar/Notes
https://github.com/jhadjar/Language-Acquisition
I then experimented with Sphinx. Then a simple alias `kb` (for knowledge base) and `note` to save random notes to an markdown file synced in Dropbox.
Now I use MKdocs. An intern experimented with it and proposed it and that's what I use for now:
https://jhadjar.gitlab.io/kbase/
Original notes are here https://jhadjar.gitlab.io/kbase/notes/
Original knowledge base: https://jhadjar.gitlab.io/kbase/kb/
Notice the timestamps (the alias opened vi and put a timestamp).
I also use TaskWarrior and have an alias to commit and push the changes to the files to a private repo.
I should collect dispersed files on my disk that are the result of reading articles and books (I open a file in the same directory of a paper or a book and start writing thoughts at a coffee shop, which I send to coworkers, especially for papers).
Bottom line, now I use MKdocs with GitLab pages (for now). It works nicely. The alias is good for quickly jotting down to a file and saving, then it's cleaned and edited and incorporated in the proper file in the knowledge base.
I used to use Evernote to write down my knowledge but I quickly found that I never look back at my notes, and quickly lost grasp of my knowledge. That's why I built the automated email digest.
My workflow is simple: when I learn something, I execute a command line program to write it down without leaving the terminal when I am coding. Every weekend, I receive a digest of what I learned in the past. This way, I can successfully maintain all the new knowledge I acquire.
Similarly to your requirements, I often search through my knowledge base in the terminal before doing a web search. Doing so saves me time because my knowledge base is already written in my own words that I understand well.
- [0] https://github.com/dnote/dnote
This helps me to focus on the content instead of thinking how to represent it in markdown/whatever syntax.
What really sold me was the video "Jerry Michalski: Lessons from 15 years using TheBrain" [0], haven't looked back.
[0] https://www.youtube.com/watch?v=EzQkJ4lleKk
I have a few different databases now, the biggest in the millions of rows. A workflow for one of the databases, an ontology of human medical measurements, is as follows: 1) I come across a new term, 2) (if afk I first email it to myself) I simply do a “touch term.hipo” at the shell, or use vim, sublime, or another handy tool. Then I add data in a strongly typed but 0-syntax tree language I created called “hipo”. If I’m in sublime or another editor I get live autocomplete and syntax highlighting 3) I use git to version everything 4) I have a simple DB server which reads the files and allows me to analyze using any tool, from R studio to Vega, etc.
Also, what do you do if you want to put the same node in two different places in the tree? (i.e. you want a graph)
For graph nodes I use an id.
Software wise, I've tryed a few things and settled with MediaWiki and some extensions added. Most notably Semantic MediaWiki (adds structured data storage and querying), Semantic Forms (structured forms) and a Bootstrap based skin. But only the technical setup won't get you far in a wiki, so I've also created a lightweight, flexible "ontology" / structure for knowledge management that was fitting my own needs.
This was not too easy to set up (Finding a structure / ontology / data and query workflow, but also the implementation using the wiki itself), but now I'm quite happy with it as a platform. I really wish there would be a lower barrier to entry for people here. If you're interested, please PM me and I can show you some examples.
As for the workflow of personal knowledge management, I'd be really interested what works for other people as well and I'm still figuring this out.
My current approach is using different phases. It's OK for me to read a book and not note anything or only do kindle notes. When reading articles on the net, I usually bookmark more interesting articles using getpocket. I do notes on paper when I don't want to be distracted or influenced by a computer.
For this year, my resolution is to acquire less new knowledge, but to take some time and to rehearse, summarize and repeat what I've found interesting and worth my time before. What sticks here should go into the wiki.
I want to keep my knowledge base very concise and opinionated. There's no need to create a second wikipedia.org. I'll only add something in the wiki when: * Some idea is truly remarkable on the first spot * I go over my old notes see what was the essentials of it. Those are worth putting into a wiki. * When I'm doing some dedicated, structured research around a topic. Then I also try to get the ontology and structure right first. * Code Snippets that I repeatedly use
I do think a truly useful knowledge repository will fundamentally need to be a graph structure. This roughly reflects the neural connections in the human mind -- knowledge itself is a graph structure.
Draw.io is the best diagramming tool I've found, but it still kind of sucks in a lot of ways. I've started working on an alternative: https://github.com/JesseAldridge/consideration
The key features I'd like to add are multi-user collaboration and search across many diagrams.
Most importantly, the search is intelligent and works instantly... even across my 100,000+ nodes.
My entire life is in Workflowy. I am thrilled to pay for my annual subscription, because the value I receive is outrageous.
The issue I’ve always had is not being able to paste images into it
* Tomoby:
- pros: fast, good search, really liked local editing, integrates with quick launchers like Gnome DO and Albert which I found essential for quick references, especially programming and Linux commands which, when I want them, I want them NOW. type: "<ctrl>+<space>grep examples" and have a reference to my most frequently used patters in 1 sec is SO helpful.
- cons: sync/mobile solutions are hacky, no longer maintained, Ubuntu 19.04 removed it, not encrypted, Copy & paste to/from other apps (Thunderbird) doesn't work like you'd expect, formatting is not preserved.
* Evernote: meh...used it for a couple years. Liked multi-device support. Their web client wasn't that great and they didn't have a Linux client. I paid for it and put maybe 100 notes in it, but eventually stopped using it and converted back to the free version.
* Dropbox Paper:
- pros: nice editor, have the basics right, integrated with Dropbox which I already use, so no additional cost, good mobile client, really easy to share notes with groups of people (family) or one-offs to others with a link.
- cons: some bugs and bad UX still, no encryption (but Dropbox already has most of my files so I've already sold out a bit to them I guess), loading a browser tab is just slower than the Tomboy UX described above...I find I like using Paper for longer docs or things I'll access on mobile, but it's just not a good replacement for those quick notes that when I want them, I want them NOW!
I've tried a number of other ones over the years, never really found a sweet spot. I actually think something like Tomboy is really the best note taking app, but for it to really thrive, it would need better options for mobile and note sync than Tomboy ever had. Adding end-to-end encryption and sharing would also be great.
It lives in a private NextCloud instance, so everything's available from my laptop or phone (although editing markdown on a phone is no fun).
Edit - this is great for detailed note-taking; I also use a custom command-line tool to capture ad hoc notes, ideas, and suggestions without opening vim. They go into a YAML file right now; tooling in process to pull them directly into the wiki, so everything's in one place.
Have you tried Markor? https://github.com/gsantner/markor#screenshots I have found that it's a pretty good Markdown editor for Android. It's available in F-Droid too.
The wiki stores pages in simple text files that are easy to backup, it automatically calculates birthdays of my contacts (stored with semantic annotation in an XML style), and provides an Android App to synchronize all pages on a mobile phone. Many wiki pages are organized hiararchically, but most times I use the full text search.
Technically it runs with Java on any OS, and has a built-in web server.
[1] https://moasdawiki.net/
Here is my workflow of editing and publishing it:
https://wiki.nikitavoloboev.xyz/other/wiki-workflow
It makes it very seamless to make new notes.
And for retrieval I press another global hotkey, a prompt pops up, I start typing and it immediately lists my matching notes (the search query can find the typed words in any order of course), I select from the list with enter and the note is displayed, and can be edited too if needed.
So finding something takes usually at most 5 seconds (pressing the hotkey, typing, selecting). A mouse-based system could not beat this efficency.
I personally use Autohotkey on Windows for the global hotkeys which when pressed activate my running emacs for new note or search (I use orgmode in emacs for notes). And for searching I use the org-rifle package.
- Excerpts (for summaries and excerpts from books and articles)
- Snippets (for code snippets, formulas, etc)
- Reports (for results of research on a subject, e.g. learning new API)
- Papers (copies of research papers)
Dividing notes in such a way also helps to review them with varying frequency. For example, snippets get outdated often (e.g. workarounds for bugs or polyfills) so I have to go over them more often. Reports get updated as I learn new things about a particular subject that has its report. Excerpts are updated only if I decide to re-read the material they cover and extend them. Papers last as long as I'm interested in subject.
Drafts (iOS and Mac) • quick thoughts I want to bang out before they dissolve. Drafts is focussed on being the fastest zero-to-blank-page experience on iOS, which makes it very handy for rapid spur of the moment universal capture.
Quiver (Mac) • I periodically review the above two, and catalogue them here for long term storage. There’s no structure to review, this is just my brain’s external HD.
[1]https://zettelkasten.de/
I want to open source it in the future. It is electron app soon to be written in typescript
E.g. my top levels are: - server administration - networking - programming - telephony - science - mathematics - culture - management - job search
My secret is to transcribe the journals to text files. Then it's searchable, too!
A frequent (daily) transcription process also yields other benefits such as being an intrinsic reminder system.
Works pretty well and is rather cheap.
I came up with my own mental model “Make Knowledge Actionable”. With that, I find it is important to align knowledge you are saving with your goals. This means that someday I might need it knowledge should be kept in a different spot.
If I have some time, I have thought to build a system that lets me find information via mind map and different categories of abstraction.
Now I use a combination of org mode, PDFs and miscellaneous document types replicated in both Dropbox and Google Drive, and on macOS the system search finds things. Dropbox search only hits file names, Drive search is more useful.
In addition to my own notes, I collect PDFs of good papers, copies of books I have bought, etc.
EDIT: forgot about Google Keep and Apple Notes - sometimes use for specific things like vacation travel notes
I have a database for 'research' where items are stored according to the Dewey Decimal categories (no particular reason for choosing this other than it's well organised and well documented).
I then have a separate 'memex' database which contains my journal and other miscellanies.
DevonThink has a rudimentary AI system which attempts to link documents by context. I find it quite useful.
I find myself making so many small notes of how I did things at work or with AWS, zsh etc.
It’s a really powerful system while simple at the same time.
Cherry Tree https://www.giuspen.com/cherrytree/
It was quite impressive on the variety of notes and document types it was able to organize.
Also seemed to have good support for outputting in various formats.
CherryTree is positioned as an open source alternative to OneNote.
https://tiddlywiki.com
Plus a public wiki where I posted things I’m interested in: wiki.secretgeek.net
Also have private places where I store things I learn that are business confidential etc. As well as a password manager for the most protected things.
Seems like something as simple as a firefox add-in would be useful.
A lot od topics
Github would be better, actually....
and evernote, where I occasionally drop ideas, save snippets of code or commandline pipelines along with snipped articles from the web usually recipes or long form articles
I would benefit from doing more
More short term I write whatever is happening on especially long narrow ruled legal pads (8.5 x 14) which often still contain lots of information after a few years
http://john.do/first/