Use AsciiDoctor for the 3D Tiles specification

[ptrgags]

To follow the lead of glTF, it might be good to upgrade our spec from Markdown to AsciiDoctor for nicer HTML and PDF output. It sounds like https://github.com/CesiumGS/wetzel has been updated within the past year. it would be good to see how well it fits 3D Tiles.

Some notes from @emackey about the latest details of wetzel:

• It can convert JSON directly to AsciiDoctor now.
• It has a new option to skip writing the TOC (because AsciiDoctor does its own TOC).
• A parent AsciiDoctor file could use multiple include statements to deal with the multiple roots issue.

Over in glTF land we're using it, check out the Makefile that calls wetzel from npx on a command line, and the CI.yml that runs this Makefile from within GitHub Actions.

wetzel is run as part of the GitHub Actions CI, to convert JSON schemas directly to AsciiDoctor files You can go into the "Actions" tab and download build artifacts from the latest run.

This should give you a file spec-outputs.zip that has one self-contained PDF and one self-contained HTML file included. Extract and double-click either one to see how the output is shaping up. Section 5 is the main reference section that's auto-generated from the schema files. There's also an Appendix A that has verbatim copies of the schema (also handled by wetzel now), so that if the PDF were sent to ISO, the real schemas would be included in the standardization process. The remainder of the glTF specification document does not come from schemas, and was manually translated from original source Markdown to AsciiDoctor by Khronos veteran Jon Leech.

[emackey]

There are some additional details that have been added to the wetzel readme.

[javagl]

First experiments for the conversion to AsciiDoc have been started in #682. On top of the state in this PR, I locally created a state that is even more experimental, and will merge that back into the visible one at some point.

There are specific points that are related to #631 and #632 (comment) :

• I assume that the directory structure should largely remain as it is, and that each README.md should be converted into a README.adoc. Alternatives would be to just "flatten" everything into a single asciiDoc/chapters directory or so. AsciiDoc files can be previewed on GitHub (see the draft state of Specification.adoc as an example). But there might be glitches or things that do not work perfectly when viewing them on GitHub.
• The overall structure of the single specification document will largely be aligned with that. The current (local) state is shown in this image: Cesium AsciiDoc Spec 2022-04-14. Many details will likely change soon. I'm omitting many details, e.g. about the right location for figures folders etc. But everybody can make suggestions here.

---

The general approach for building the specification still has to be decided upon. Some preliminary notes about the process of converting from Markdown to AsciiDoc, as well as basic instructions for the PDF/HTML generation, are summarized at https://github.com/javagl/3d-tiles/blob/ac729d4f67e68852b47fbf8ce900fcd24d2bd3bd/BUILDING.md , but these will become proper "build instructions" later. But some things to consider:

• Other AsciiDoc-based specifications use Makefiles (yes, these things from the 90`s) to generate the specifications. For example, Vulkan or glTF. While this does offer some nice possibilites (e.g. regarding wetzel - more on that below), it adds a complexity that I'd rather like to avoid. Khronos went the extra mile here, and wrapped all that into a Docker container (see Vulkan Build documentation).
• The single HTML/PDF file for the OGC standardization process should follow some "template" that is described in the OGC submission process. Now, we're talking about documents with dozens of pages that describe how documents should look like, and the process for creating them is - you guessed it - so complicated that they wrapped it into a Docker container.

I do not yet have a clear idea whether we want to be "inspired" by that (starting with the shallow question of "Which Docker container should we use?")

---

The inclusion of the "Property Reference" that can be generated with wetzel raises some questions. While wetzel has been updated to generate AsciiDoc, the overall process has to be sorted out. First, we need to check whether wetzel can generate the reference for the current 3D Tiles schema. Then, we have to integrate that into the full specification document. (That's where Docker+Makefiles might have to come into play...)

---

The conversion from Markdown to AsciiDoc involves a ton of manual work. While the "bulk conversion" can be done with kramdown, the resulting .adoc files require manual polishing. Particularly, each and every link (source and target) will have to be checked. For example, a Markdown link like [Example](DirectoryWithReadme) will by default point to a GitHub directory, and not to the .adoc file that was created from the README.md that it contained. On top of that: The naming pattern for #anchors based on sections is different between GitHub/Markdown and AsciiDoc. We might have to manually add custom IDs to the sections. (glTF and Vulkan are doing this as well) - otherwise, resolving ambiguities might be hard, and links to auto-generated anchors like _overview_7_ might easily break due to the slightest restructuring....

[javagl]

One broad and generic question is how we want to handle links to repositories.

Right now, the spec usually contains sections like

Something

Defined in something.schema.json

where the link points directly into the repo. Similarly, there are some links like

Bounding Volumes

~It's also possible to use S2, see https://github.com/CesiumGS/3d-tiles/tree/main/extensions/3DTILES_bounding_volume_S2

These links are (very) convenient for the reader on GitHub. But that does not play well with the idea of having a single, self-contained document.

The links to the schema files could just be removed, but that would be a pity. It could be better to replace them with a link to the "Property Reference", but the process of generating and integrating that is still to be investigated.

The handling of other repo-links might be decided on a case-by-case basis, and I can do a first pass here (specific cases that are unclear could be tracked in #682 )

[javagl]

On top of the question above (how to link to the schema from the spec), there are also links from the schema to the spec - e.g. in style.booleanExpression.schema.json . These could/should be replaced with a short inlined text, accordingly.

I did a few experiments with wetzel,

• The Replaced single-element allOf with ref commit turns out to be an issue for wetzel as well. There is still no definite answer to the question of whether this is "correct and valid" (beyond "it seems to be"), but causing some trouble. Maybe it should be undone.
• Large parts of wetzel make certain assumptions, for example:
  • There is only a single 'searchPath', which does not play well with our schema subdirectories
  • Even when copying all the files into a single directory: The generated documentation wants to link to the actual schema.json files, and there is no way of handling this when they are in different directories
• Apparently, wetzel does not play well with how definitions are used in some of the schema files. (The workaround is: "Ignore the error and just call replaceRef twice" 😬 )
• Even if all the property references could be generated: There will be sections that have the same name, causing a
  WARNING: Styling/PROPERTY_REFERENCE.adoc: line 104: id assigned to section already in use: reference-extension
  It would be necessary to disambiguate these...

A more high-level question: Should we try to move all property references from their individual pages into one large one that is added as an appendix?

---

And a high-level question, unrelated to the above: I wondered in how far 'Styling' us actually part of the specification. It was part of 1.0, so certainly is a part, but in many ways unrelated to the format itself. (I just moved it into an appendix locally, but that might not be the right thing to do here...)

[lilleyse]

Hm we might need to dig into wetzel a bit for these.

The Replaced single-element allOf with ref commit turns out to be an issue for wetzel as well. There is still no definite answer to the question of whether this is "correct and valid" (beyond "it seems to be"), but causing some trouble. Maybe it should be undone.

I believe it's valid according to the JSON Schema 2019-09 Release Notes and seems like it should be quick enough to fix in wetzel, at least compared to the other things here.

There is only a single 'searchPath', which does not play well with our schema subdirectories

This seems hard to fix from the 3D Tiles side. How much effort is it to update wetzel?

• Apparently, wetzel does not play well with how definitions are used in some of the schema files. (The workaround is: "Ignore the error and just call replaceRef twice" 😬 )

Also curious what the level of effort is for updating wetzel.

A more high-level question: Should we try to move all property references from their individual pages into one large one that is added as an appendix?

That makes sense. It keeps the rest of the specification tidy and those sections can link to the other page / appendix.

And a high-level question, unrelated to the above: I wondered in how far 'Styling' us actually part of the specification. It was part of 1.0, so certainly is a part, but in many ways unrelated to the format itself. (I just moved it into an appendix locally, but that might not be the right thing to do here...)

It would be its own section near the end of the document, but not an appendix, similar to https://github.com/CesiumGS/3d-tiles/blob/main/specification/specification.pdf

[javagl]

Regardless of the Property Reference, there will the difficulties for internal links. For example, we have

\ImplicitTiling\README.adoc
\Matadata\README.adoc

and the first one is supposed to link to the second, but this is apparently not possible.

[javagl]

The workaround that I'm currently using in #682 for cross-linking between documents in sibling directories is described at https://github.com/javagl/3d-tiles/blob/67a5b447f1c25c13f54fc17b3fcb511ad323c55a/BUILDING.md#cross-linking-between-files . This might well break some "Best Practices", but if anybody comes up with a solution that works for both the GitHub preview and the generated HTML/PDF, I'd apply that solution accordingly.

[javagl]

The specification of 3D Tiles 1.1 is now written in AsciiDoc. (That's the format name. AsciiDoctor is apparently only the tool that is most commonly used for handling AsciiDoc).