I like this approach and used something similar for a while myself. For me the main drawback was syncing between devices. I had to remember to commit and push on one device and then pull on another. Maybe not a big deal for lecture notes, but a little fussy for just adding a book to a "to read" list.

Also in time I found I never really used any version control features. Backing up to a remote was the only thing I needed.