> a test that fails if a new API endpoint isn't mentioned [...]

That's a great idea. A related thought I had was testing assertions about architecture by looking at the graph of imports or calls.

From your blog post,

> if a change doesn’t update the relevant documentation, point that out in your review!

What you get at but don't seem to state explicitly is that finding the relevant documentation is hard, for both the author and reviewer. Finding any relevant documentation should be easy, but ideally we're finding all relevant documentation, and we need to be reaching sufficient confidence that we're missing little enough that we're able to hold back entropy enough to keep the docs useful.

Your tools address this for some cases! Doctest addresses this for other cases. From TFA here, it sounds like Swimm.io tries to address this for more cases (my gut says the article oversells it but I intend to look more closely).

To get further, an idea I've been toying with is to treat claims (implicit or explicit) in documentation as requiring citation, pointing not at sources but at tests. Ideally the test runner, when a test fails, can then surface all references to that test. In addition to highlighting portions of the docs that may need to change, this seems likely to also provide crucial context when fixing the code and/or the test.

How do you read the in-repo documentation? Search for all files named readme.md? I have never learned about a library from documentation scattered about the repo. There's the readme at the root, and everything else is on a web page, which is a better way to organize and browse documentation.

I use documentation systems that publish the documentation from the repo to a website. Most of my projects use Sphinx and reStructuredText for this, but I recently tried MyST (Markdown for Sphinx) and I like that a lot.

Some examples:

- https://docs.datasette.io serves documentation from https://github.com/simonw/datasette/tree/main/docs - which has documentation unit tests here: https://github.com/simonw/datasette/blob/main/tests/test_doc...

- https://sqlite-utils.datasette.io/ serves from https://github.com/simonw/sqlite-utils/tree/main/docs - unit tests here: https://github.com/simonw/sqlite-utils/blob/main/tests/test_...

- https://django-sql-dashboard.datasette.io/ serves from markdown in https://github.com/simonw/django-sql-dashboard/tree/main/doc... - I don't have documentation unit tests for that yet

Those three are all hosted on https://www.readthedocs.org but I've also used this trick on web app projects that host their own documentation deployed as part of the build process.

This is a good approach and reStructuredText well suited for technical documentation. The only downside I see in reST is, that it is mostly only readable from the standard implementation in Python, which is a custom parser, not a portable grammar for other languages to implement in any parser tool / library. There are some libraries for parsing it, but last I checked those were incomplete.

Emacs org-mode would be a great candidate as well, for things like runnable code inside the documentation as examples and export to many other formats and nice tooling for viewing the files. Unfortunately many git hosts do not support it well and render crappily.

I would recommend against switching to a non-standard Markdown dialect. Why switch away from reST, which offers all of the things nessecary for a good technical documentation, to some Markdown dialect, which has many important features only bolted on?

That said, one thing I noticed happening with this approach is, that people think "Oh, I have my documentation in my comments of the code! I don't need to write anything else!" And then I end up reading documentation like: "def get_a(): ..." Docstring or comment: "Get a." Wow, thanks, how helpful!

In short: There is no simple way to have good documentation, except for writing good documentation. Docstrings probably will not be sufficient, unless you write whole novels in your docstrings. A good documentation needs usage examples and rationale of why something was done in a specific way, what kind of gotchas there are and probably other stuff, that does not come to mind right now.

"Why switch away from reST, which offers all of the things nessecary for a good technical documentation, to some Markdown dialect, which has many important features only bolted on?"

Honestly, the main reason is that I've encountered developers who have an almost alergic reaction to rST - they genuinely hate writing in it, and will be deterred from writing documentation if they have to figure it out.

Custom Markdown flavours are more likely to get buy-in.

I see. Sometimes it seems, that there is some kind of animosity towards any non-markdown format, as if markdown was the one and only. Ha, so far from it ... But everything else must be eradicated with some kind of hostility, it seems. The people exhibiting this kind of behavior often do not know other formats well, nor have they bothered to use another format for a while to find out about what it can express. It is as if children do not want to give up on their favorite toy, even when there is need for something more capable, to write technical documentation.

I have used Markdown, Pandoc Markdown, probably Github Markdown, probably other dialects of Markdown and I kept missing features for writing documents.

I have written a thesis in rST and it worked very well, even though Pandoc at the time did not understand citations and arbitrary document internal references (both of which standard markdown does not have) in rST well enough. I had to write my own pre-parser for it. I could not have written that thesis in some markdown dialect incapable of expressing things, that one simply needs in academic writing. There was Pandoc Markdown, but it felt bolted-on and rather ad-hoc in comparison to rST, which brought all the things out of the box.

I remember also looking at AsciiDoc at some point. Not sure, why I did not choose it.

Then I discovered Org-mode. It has been a journey to uncover more and more of org-mode capabilities. Org-mode does not suffer from the same eco-system split up as markdown does. It does not have this bolted on feeling, that many markdown dialects leave me with. It usually has everything I need. In fact, I have written technical documents in it as well and have discovered things like literate programming. The tooling in Emacs is so good, it is really nice to work with org files. Good, that I can still export to primitive markdown, for anyone not knowledgeable in org-mode. Even though org files are merely human-readable plain text documents. It is unfortunate, that VCS hosts are unable to render them properly.

I would really dislike going back to markdown only. It feels so limiting to me now, that I have developed an aversion against writing documentation in an unsuitable format like markdown (talking standard markdown, probably commonmark). If there is already markdown only documentation, then well OK, I guess I can write it, although it surely will not feel great.

> Sometimes it seems, that there is some kind of animosity towards any non-markdown format

Can't speak for anyone else, but for my part there is animosity towards any format, period. Documentation should be plain ASCII - or, when strictly necessary[0], UTF-8 - text, and readable as such with only human-generated ad hod syntax such as *emphasis* or

  +------------------+
  | ascii-art tables |
  +---+--------------+
  | + | corner       |
  | | | side of cell |
  | - | bottom/top   |
  +---+--------------+

or http://links.to/whereever.written-literally. To the extent that markdown is tolerable, it's because it does not demand any obfuscation in how documentation is written.

Admittedly, I'm not necessarily a repesentative example of anything.

That is an interesting approach as well. What I see as advantages are: that there is no need for a specification and no need for it to be rendered, as one is to view it as plain text.

What I see as disadvantages are: It is not possible to render it properly or have support for it beyond showing plain text. Many users might write things different from each other, which might affect how easily a reader picks up what each document's style or conventions are.

> It is not possible to render it properly or have support for it beyond showing plain text.

Sure you can, and indeed that's what markdown was originally supposed to do (at least as it was first described to me). The catch is that you have to treat the plain text as the authoritative version: you do not edit the plain text in order to produce changes in the rendered document; you edit the plain text for it's own sake, without considering the effect on the rendered version, and the rendered version is derived from that.

> when strictly necessary[0]

Missed this and too late to edit.

0: Which historically meant "not stupid-quotes", with a addendum for stupid-dashes, stupid-ellipses, etc, but now also includes "not emoji".

I agree wholeheartedly.

The most annoying thing about Markdown is that most of the time (90%, I'd say) it's added some kind of formatting, I was not trying to format anything at all.

This is definitely the best approach in my opinion, providing the people writing the docs are capable of contributing directly.

One of my projects[0] builds and deploys a static documentation site[1] on every push to master. The static site generator (Nanoc, in this case) imports the library and uses it to publish its own documentation. All the examples are snippets of code[2] that are both displayed as-is and eval'd into the final output.

The guide can never be out of sync with the library.

[0] https://github.com/dfe-digital/govuk_design_system_formbuild...

[1] https://govuk-form-builder.netlify.app/

[2] https://github.com/DFE-Digital/govuk_design_system_formbuild...

In the PostgreSQL codebase they have readmes scattered about and it’s great to jump into some subfolder and get the details you need in the right context.

If you write your documentation in Emacs org-mode, you could use include [1] to include files on other levels of your repository and then you could export it all to a markdown file for people, who do not know about org-mode and Emacs. This would make documentation anywhere in the repository discoverable / automatically included in your resulting documentation file.

[1] https://orgmode.org/manual/Include-Files.html

The webpage can be generated from markdown files which can be in the repo.

Static site generated from the repo. Can be hosted locally or online. Github pages are usually leveraged for this usecase

If using Django there are tools like django-docs (https://django-docs.readthedocs.io/en/latest/) and the recently released django-sphinx-view (https://noumenal.es/django-sphinx-view/).

You can use the CI to publish you md files with tools like https://docusaurus.io/

Write the documentation as text files.

Include hashtags for topics, e.g. #authentication, #language, or #netscape.

Use the text file indexing and management system to browse and update the documentation.

Check out Backstage, and in particular, TechDocs: https://backstage.io/docs/features/techdocs/techdocs-overvie...

None of this works if the programmer isn't the same person that writes the docs. e.g. if you have a copy-writer come along and write/update the docs before each release, then its not captured in the same commit/branch/etc.

That can still work, in a couple of ways.

You can have the programmer write bad documentation and file an issue for it to be improved. You can then enforce that releases don't go out until those issues have been resolved by the copywriter.

You can also implement new features in a branch with multiple authors. The branch doesn't get merged until the documentation is in good shape.

It still works ok, just not as nicely.

The copy editor updates the repo and while their changes won’t be in the same commit, they should be nearby.

So you still get the benefit of docs history, and being in the same place.

I think you could also make a case that if you've adopted these strict documentation requirements you don't decouple the functional commits from the documentation commits. Your copy editor could work in the same branch as the code changes and then you only approve the PR when the documentation is at the same level of QA as the code. Otherwise a fear separate commits or branches is the thin edge of the wedge.

Sure it does, developers work with domain experts on features all the time. Perhaps you had SDETs adding some test infrastructure, or designers adding assets and layout information. Either you can have feature branches, or stubs/scaffolding for CI. I've seen both work.

The biggest issue I've see is management and product who seem allergic to the actual repos for some reason.

> I'm adamant that the documentation for a project should live in the same repository as the code itself. This is crucial for a number of reasons:

I have a small disagreement on that. I always feel documentation should be outside of code as it might need to be reviewed by people who do not have access access to the code base. Then there are also the "Why are we doing this" part of documentation that is difficult to mark out in code. Documenting in code is great for addressing "how are we doing this" part IMO.

The docs may live in the repo but they should definitely be published somewhere that non-GitHub users in the organization can view.

The higher level strategic stuff can absolutely live elsewhere - in my experience Google Docs or some kind of company-wide wiki often come into play here.

I was thinking more of an enterprise situation where multiple systems might be working together as part of a process. The code for each system might be in different repos in which case the documentation would also be broken. In such a context having the unit level documentation in the repo makes sense, but the overarching process documentation that is constantly evolving and referenced by multiple teams cannot be part of this repo.

I assume the OP meant, or at least how I took it and implement it, it can be in a dir called docs/ off the root of the tree. It doesn't have to be literally scattered about in the actual source, just in the same repo.

Though there are reasons to do both(some docs from the source itself, and some from docs/) but that's a different debate.

Is there any lib/way that automatically generate doc for api/endpoint. I think it is possible to create such generator for graphql api.

You can checkout Swagger and OpenAPI, there are libraries to annotate the endpoints in the code and then generate interactive docs out of that.