I do something very similar, and have also been doing it for the past 3 years. I started out somewhat structured, not with org-mode but something similar, but I fairly quickly switched to just using plain text.

A few days after I started, I noticed that the vast majority of the things I was writing down were not actually important to be reminded of. I just needed to get them out of my head, and an unstructured plain text file made that somehow easier.

That said, as the number of tasks and appointments I need to keep track of has increased, I might give switching new files to org-mode a try. Integrating agenda items and such might just work for me.