Moin2Sphinx

Migration of a MoinMoin v1.x RST-based Wiki to Sphinx

This repository describes a method for migrating an existing Wiki site, implemented as a MoinMoin Wiki engine version 1.9.x (which is based on Python2) to the Sphinx documentation generator.

Requirement of Restructured Text

IMPORTANT: This project only supports Restructured Text (RST) files!

It is assumed that the MoinMoin v1.9.x Wiki content is using Restructured Text (RST) files. Please read the reStructuredText Primer about Sections, Paragraphs and other topics.

Conversion of MoinMoin "wiki-markup" pages to Restructured Text (RST)

The default markup in MoinMoin is the wiki markup, but other markups also exist. For use with Sphinx such pages would need to be converted to Restructured Text (RST).

The pandoc universal markup converter might possibly be used (not confirmed), but this is rather uncertain.

Install pandoc on EL7 or EL8 systems by:

yum install pandoc

Add cross-page links to the MoinMoin pages

While the sphinx-build tool (used below) can parse most MoinMoin RST pages, there are some cases where we have to insert extra information into some MoinMoin RST pages:

• Cross-page links to section headings in other pages need to have a custom anchor inserted above the section heading, for example:

.. _rst-overview:

RST Overview
============

In this way a reference to rst-overview can be made from any page using both MoinMoin as well as Sphinx. See Use a custom anchor for a description. In a Sphinx ref command you then use the anchor text, for example:

This is a link to the RST Overview: :ref:`RST Overview`

NOTE: This is only required for sections that are linked to from other pages, and not for all sections!

Extract RST files from MoinMoin

MoinMoin stores its Wikis in a top-level directory, and we assume here that the /var/moin directory is used. The writefiles.sh script should be edited for the needs of your Wiki:

• First you must edit the script to omit these page patterns and add other pages/patterns that you do not want to convert:

OMIT_PATTERNS="HelpOnMacros WikiCourse AdminGroup BadContent "

• Also, some MoinMoin directories have separators such as (2d), (2e) of (2f), and these are rewritten in the script. Look for the mungedname= line and add any additional rewritings as needed for your Wiki.

Now execute the writefiles.sh script on the server where /var/moin is located:

Moin-server$ ./writefiles.sh <wikiname>

where the <wikiname> is one of the Wikis in /var/moin.

The writefiles.sh script generates a tar-ball <wikiname>.tar.gz of the newest files from /var/moin/<wikiname>/data/pages/. It will ignore all older files (previous page revisions). Since there does not seem to be any way to detect whether a file is in RST format or some other format, we simply copy the files and append an .rst extension.

Setup a Python3 virtual environment

We will use a Python3 virtual environment to process RST files from MoinMoin. On EL8 we recommend to use Python 3.9:

dnf install python39
python3.9 -m venv venv

Activate it and install Sphinx :

. venv/bin/activate
pip3.9 install --upgrade pip
pip3.9 install sphinx

Check the Sphinx version by:

sphinx-build --version

Upgrade an existing installation of Sphinx :

pip3.9 install --upgrade sphinx

There are also externally provided themes such as the sphinx-rtd-theme which you can install with:

pip3.9 install sphinx-rtd-theme

Alternatively, install Python3 RPM packages from the OS:

yum install python3 python3-pip
pip3 install sphinx

Customized Sphinx Makefile

The Makefile.new_t template for the Sphinx Makefile adds some new rules to the file generated by default:

• Web-server top-level directory WEBTOPDIR which you should customize for your site.

• A default rule of dirhtml for the HTML pages generation.

• A check for the Python virtual environment.

• An rsync rule for copying HTML pages.

Convert the RST files into Sphinx format

Copy the tar-ball <wikiname>.tar.gz generated above to the current directory.

Define some predefined author names as an environment variable:

export AUTHOR="Fullname1[,Fullname2...]"

Run the script moin2sphinx.sh on <wikiname>:

./moin2sphinx.sh <wikiname>

This script will perform these steps:

• Unpacks the tar-ball files to a subfolder <wikiname>

• Creates Sphinx files in a subfolder named <wikiname>-sphinx.

• Calls the sphinx-quickstart tool that asks some questions about your project and then generates a complete documentation directory and sample Makefile to be used with sphinx-build. We are going to set the Sphinx Project name to <wikiname> (with first letter capitalized), and the version to 1.0.

• Runs the script moin2sphinx.py to convert the MoinMoin RST files in subfolder <wikiname> to Sphinx format into the folder <wikiname>-sphinx.

Edit the table of contents

Go to the <wikiname>-sphinx folder and edit the file index.rst:

• Delete any page names which you do not want in the Sphinx documentation.

• Reorder page names in a logical way for the project.

Change Sphinx page theme (optional)

Sphinx provides a number of builders for HTML and HTML-based formats, see Sphinx themes.

The sphinx-quickstart tool may work with a number of templates, see the manual section on Project templating options for sphinx-quickstart.

The Sphinx themes may be defined in the template file conf.py_t. You can copy a conf.py (created as above) to the top-level directory and use it for all future Wiki pages:

cp <wikiname>-sphinx/conf.py conf.py_t

and edit this file to select a different theme (the default is alabaster), for example:

html_theme = 'classic'

There are also externally provided themes such as the sphinx-rtd-theme which you can install with:

pip3.9 install sphinx-rtd-theme

and configure it in conf.py or conf.py_t with (notice that the _ replaces -):

extensions = [ 'sphinx_rtd_theme', ]
html_theme = 'sphinx_rtd_theme'

See also the Sphinx Configuration page.

When you run the moin2sphinx.sh script, it will use the current directory . for sphinx-quickstart templates like:

sphinx-quickstart --templatedir . <further options...>

and pick up the conf.py_t template file and use it when creating a new <wikiname>-sphinx/conf.py.

Open external links in new tabs of the browser in Sphinx HTML documents

Installing the sphinx-new-tab-link project in the virtual environment will open external links in new tabs of the browser in Sphinx HTML documents:

pip3 install sphinx-new-tab-link

Then edit conf.py to use this extension.

extensions = [
    "sphinx_new_tab_link",
]

Generate HTML pages

Go to the <wikiname>-sphinx folder and and generate HTML pages using sphinx-build:

make dirhtml

The make dirhtml command makes sphinx-build build HTML pages with a single directory per document in _build/dirhtml/. This makes for prettier URLs (no .html) if served from a webserver. Also, this will make the web-page URLs have the same names as with MoinMoin (otherwise you will have to use .html URLs).

The HTML pages will be built in the _build/html/ and _build/dirhtml/ subfolders with an index.html file which you can use in a browser, for example:

firefox file://<path-to-project>/<wikiname>-sphinx/_build/dirhtml/index.html

or copy the files to a web-server using this customized Makefile rule from the above Makefile.new_t:

make rsync

To start over the HTML page generation from scratch in the <wikiname>-sphinx folder:

rm -rf _build
make dirhtml

You can also delete the <wikiname>-sphinx folder and repeat the moin2sphinx.sh script starting as above.

Correcting Sphinx errors with MoinMoin RST files

There may be several errors and warnings from Sphinx while HTML pages are generated:

• ERROR: Unknown target name:
  See Add cross-page links to the MoinMoin pages above and Use a custom anchor for a description.

• ERROR: Anonymous hyperlink mismatch:
  See the section Downloadable attachment files below.

These types of errors are simplest to correct by manually editing the Sphinx RST file.

Downloadable attachment files

In MoinMoin attachment files can be added to a Wiki page, You refer to the attachment like this:

The latest version of FooBar is available as FooBar__
__ attachment:FooBar

This seems to be a special feature of MoinMoin.

Sphinx has a different method (role) to reference downloadable files. The way to include downloadable attachment files is:

The latest version of FooBar is available as :download:`FooBar <FooBar>`

where the file referenced must be found in the current directory.