There were a number of tentative distributed bug trackers a few years ago, they sucked and fizzled out. IIRC Fossil is an attempt at an entire distributed project management system, I don't know how it fares.
> I.e. a naive design just adding a markdown file per issue or for all issues will require manual merging all the time. Still, it seems doable.
That only works for your own personal project where you're the only user and contributor. Bug reporting by editing a markdown file (or even something actually usable like an org-mode file or an sqlite or BDB file) isn't going to scale very high and is way more effort than most bug reporters (even technical ones) will be willing to put in.
And even if your users were willing to subject themselves to that, they still need a way to send back those contributions somehow.
If you don't want special git semantics around it, you'd have to be clever about how you store it so you don't have conflicts within the metadata. I.e. a naive design just adding a markdown file per issue or for all issues will require manual merging all the time. Still, it seems doable.