Documentation roadmap: How do we build related docs together? What does it look like?

MetaBrainz
, , ,
1

Hi folks! Previously in the Summit, we talked about docs and how to better integrate existing documentation into our code bases and lower the contribution bar of editing docs for the wider community.

So, while the Summit is on-going, I put together a proof-of-concept of how inter-linked Sphinx documentation repos could work. This also comes from an infrastructure POV, how do we assemble and present those pieces in a clean and simple way to the community?

To demonstrate this, I created three GitHub repositories: a parent project and two ā€œsub-projects.ā€ Then I built all three of these on ReadTheDocs.org and connected the two sub-projects into the parent project. The parent project could be a ā€œMetaBrainzā€ parent project, with other sub-projects (MusicBrainz, ListenBrainz, etc.) fitting underneath them. Fortunately, some of this is technically doable in MetaBrainz now even with our existing Sphinx/ReadTheDocs projects.

Here are URLs to three published docs sites:

All three of these repos are built from three separate sources:

If this is interesting, we can squeeze some time somewhere to look at this on Sunday before the Summit ends. :slightly_smiling_face:

(BTWā€”the content on the two proof-of-concept docs sites were real docs taken from the wiki and converted to Markdown with zero changes by me. I left them as-is to show off the ease in how we can use a tool like Pandoc for quick conversion of MediaWiki into Markdown files. Iā€™m working to automate that with this script I am putting together. :slightly_smiling_face: Pandoc covers ~95% of the conversion and the other ~5% requires manual clean-up for things that do not convert nicely (e.g. internal wiki links and wiki category tags).

cc: @Freso @reosarevok @alastairp @rob

5 Likes
2

I also added a short update and a reference back to this thread in a couple of JIRA tickets:

4 Likes
3

Why not keep using our wiki for docs?

3 Likes
4

There are definitely benefits to staying with the wiki.

There are also benefits to moving away from it, towards a purpose-build doc system, including:

  • localization
  • versioning (of the docs themselves)
  • built in functional search
  • markdown-based documents
  • docs in a version control system alongside the source code where changes are more likely to be seen by the devs
6 Likes
5

I donā€™t see how we contribute to this kind of docs.
It is not as obvious and straightforward as in the wiki.

1 Like
6

In a system like Read the Docs (as in typical static site generators) youā€™d edit markdown files on github or wherever the repo is hosted.

2 Likes
7

Hi @jesus2099, thanks for opening this question. At the Summit, we considered the following:

  • ā€œDistanceā€ between development (i.e. git) and documentation (i.e. wiki)
  • Consistency of building and distributing documentation across all MeB projects
  • Flexible export options
  • Translations
  • Versioning

Iā€™ll explain each one in more detail below.

Distance from dev and docs

Project source code lives in GitHub, but MusicBrainz docs are hosted on the wiki. In some cases, some pages also do not reflect changes right away unless a community moderator waves them through. We want a method where code and documentation can be published together, at once, and to always better guarantee that releases and docs are published in step with each other.

As an example, transcluded pages are not published immediately on edit. They must be manually updated to a specific wiki edit.

The problem: Documentation is forgotten or falls out-of-date because they are a distance from the development workflow; or updated docs require a privileged user to flag them through

Hypothesis: Bringing the docs into git helps us maintain more accurate, up-to-date documentation

Consistent methods across all MeB docs

Currently docs are built in a handful of different ways across MetaBrainz. MusicBrainz uses the wiki, which in turn gets rebuilt in an unusual way onto MusicBrainz.org. ListenBrainz uses Sphinx and ReadTheDocs. BookBrainz uses a mix of Sphinx/ReadTheDocs and jsdoc API documentation. While certain use cases might be useful to publish independently (i.e. API docs created from in-line code comments), we want to standardize on how MeB projects should build and publish documentation.

The problem: Docs are built and published differently across MeB projects; itā€™s hard to figure out what the docs pathway is like across different MeB projects.

Hypothesis:

  • Building and publishing (most) docs in one way creates a common user experience of how to create/edit documentation
  • Anyone can edit our docs and make pull requests via the GitHub web editor
  • Maintainers are emailed/notified when a change to docs are made and needs to be reviewed (MeB wiki does not support email notifications)

Flexible export options

Since underlying documentation is either Markdown, ReStructuredText, or a mix of the two, there are many ways to export the documentation out in other places or in other ways. This avoids product-lock for choosing a documentation platform. For example, docs files will still render in the GitHub web viewer at a basic level.

Translations

Sphinx/ReadTheDocs includes more options for translations, including integration with Transifex, which MetaBrainz already uses. It would be possible to make more of our documentation available for communities to translate, and have their changes appear more rapidly alongside the standard English documentation.

Versioning

Leveraging git tags for releases, itā€™s possible to link different versions of documentation based on different versions of a project. For example, if the git tags exist, I can simultaneously look back to documentation built for ListenBrainz API v1 or v2.

This was long-winded but I hope it helps to explain some of the motivations and gains for these changes.

6 Likes
8

Thank you very much @jflory, now I understand better why. :grin::+1:

1 Like
9

Just read this again, and wanted to note that the wiki does not currently correctly send out eā€mail notifications, even if it is (as Iā€™ve understood it) trying. Somewhere in the system something is malfunctioning, which is why eā€mail notifications are not working. Itā€™s not that they arenā€™t supported.

2 Likes
10

I like this proposal a lot. I can see how it can get more difficult to involve some users to contribute to documentation. But on the other hand as far as I can tell this is a lot about documentation that is close to code changes anyway, where the problem is to keep code and docs in sync.

I recently extended the Picard API documentation and that really made me think about using Sphinx for parts of Picardā€™s documentation as well. So Iā€™m happy to see that this would fit well with the strategy for other projects.

6 Likes