Handling documentation

[iboB]

How are we going to do it?

We're interested in tools which scrape the source files and produce documentation from them, but we're about to add multiple languages... in multiple repos, no less.

Combining the docs

Should documentation pages have a single page per API entry with tabs for different languages?

Should we look for a tool which can handle multiple languages?

Should we produce something custom? An example of something custom would be using Doxygen to produce XML from C and C++, something else which produces an intermediate format from say swift sources, then parse the intermediates to join into a single documentation (with tabs for language for example).

The API is expected to have slight language-specific differences per entry. How would we deal with them?

In this case the doc tooling should be in a separate repo which has submodules for the relevant implementations.

Splitting the docs

Alternatively we could just have different pages for the different languages, using their own language-specific doc generators. And few (if any) crossrefs.

In this case a lot of text would have to be pasted between the different codebases for the (presumably significant number of) entries which are the same or very similar.

[iboB]

Blocking #44

[iboB]

Perhaps discussing this could also weigh in on #1

[IvanFilipov]

Despite combining the docs sounding reasonable, at this very moment I have the feeling that it will
cost us much more effort.

Yes, splitting the docs, will bring up many duplications,
yet I can think of example documentation, I have recently used which handles the case by
using somewhat of placeholders.

Refs:
https://learn.microsoft.com/en-us/azure/iot-central/core/tutorial-connect-device?pivots=programming-language-ansi-c
https://gstreamer.freedesktop.org/documentation/fmp4/cmafmux.html?gi-language=python

One can easily toggle between programming languages.
Maybe GStreamer I find easy to use as a developer.
We can investigate how they are generating it.

[IvanFilipov]

Another thing on top of my mind - more as a sharing of personal point of view:
I am a huge fan of visual representations as-a documentation. For example graphs & other diagram based ones.

We can think of something in that direction, but of course not for the API itself, which is the main discussion here.
Maybe as a kind of explanation of how exactly our stack achieves efficient inference time.

[tzanko-matev]

We will need to use separate documentation generators for the different languages, since there is no generator which covers all of them. Here's a list of options:

• Doxygen - Used for C/C++ Also has support for Java, Python
• Sphinx - Main documentation tool used for Python
• Javadoc - Main documentation tool used for Java
• Swift-docc - Main documentation tool used for Swift
• Jsdoc, esdoc - used for JavaScript

When dealing with each language we need to target the developers who are familiar with the language's ecosystem. So for example a Doxygen-generated website might not be so familiar to Python programmers, unlike Sphinx.

I think we need to consult with the frontend dev who will be responsible for the frontend part, in order to see how to integrate output from several documentation generators. For now I can create a simple Doxygen website which covers C/C++ only.

[iboB]

I don't accept this argument that we should take into account the familiarity of developers with a particular doc generator.

For repos we maintain it will be our job to generate and host the documentation. External contributors will not be expected to generate documentation, much less host it.

Users will consume the API and documentation however provided. It is important to make the API natural to users, but the documentation should be whatever works best for us.

I'm not saying that we should use Doxygen for the Java wrapper (or, say, C# if we do it), but this particular argument should not play a role in our decision.

[iboB]

Even splitting the documentation can share document text. Splitting or combining doesn't have to affect the text source.

Both can use a pattern like:

namespace ac {
class Provider {
    /// ac.provider.create_model
    void createModel(...);
};
}

package ac;
class Provider {
    /// ac.provider.create_model
    public void createModel();
}

Then at the documentation source the symbol ac.provider.create_model can be documented and reused when generating docs for the appropriate language

[tzanko-matev]

Would this mean that the documentation would have to be in a separate repo from the source code? Are we going to use a single repo or multiple repos for the wrappers? Ideally the change of source code and the corresponding change of documentation should happen with a single PR in a single repo. Otherwise development speed goes down and chances of errors and out-of-date docs goes up.

Maybe if we use several repos we can try to implement the following architecture:

• we keep docs relevant to some source code in the source-code repo
• We generate documentation artefacts for each repo. Those artefacts can refer to each other across repos
• We build the final documentation from the artefacts.

One way to do that is to make a 'docs' repo which contains as submodules all source-code repos. I need to figure out if it is possible to integrate output from the different documentation generators.

[iboB]

Alternatively, if we choose such an approach, one repo could be the source (containing text and id) and the others are consumers, which only have ids and use the text from the source

[tzanko-matev]

If we do it like that a single change in the API will need two changes in repos: One PR to change the source and one PR to update the documentation text to match the new source. If the text relevant to the source is in the same repo we can do both with a single PR

[iboB]

We have to do that anyway since we have two repos for API and wrapper. Not for the documentation but for the code itself (and I think we can assume that doc-only changes come with a lower velocity than api+doc changes)

[iboB]

We should also add docs for models - generate some doc pages based on a model manifest.

Since the model manifest is either actually or equivalent to JSON, perhaps we should invest in client-side code to generate the docs from JSON, thus it could be used for third party manifests as well

[tzanko-matev]

Yes. Actually the model documentation is the key piece of documentation that users will need. The current API is rather simple and it doesn't need that much documentation. The key pieces of information that the user needs are:

• What are the ops for a given model. What do they do
• What is the meaning of the dict params
• What is the meaning of the result dict
• Example uses for this model for the different languages so that the user can copy-paste code

The best scenario for the user is to have a central place which stores all model documentation. Is this feasible? Are all models going to be public?

If the necessary info is part of the model manifest we could create a model_manifest --> docs generator. I don't think that it needs to run on mobile devices (if that's what you mean by client-side). So to progress on this issue we need to work on #55

[iboB]

By client side I meant in the browser itself as dynamic content.

[iboB]

I will remove the doc generation from CMake. Even though CMake does provide some helpful utils for Doxygen (via find_package(Doxygen) it is not appropriate for our use case.

We don't have (and I'd say we shouldn't have) docs which depend on the current CMake configuration. To burden the binary build with docs seems like a bad idea.

Moreover we will very likely not use Doxygen for the Java documentation and with absolute certainty we won't use it for Swift one and many other wrappers that we provide. This means that the nice features which CMake provides for Doxygen will have to be (clumsily) reimplemented for other doc generators. It simply makes no sense.

We will have documentation generation independent from CMake. We will employ similar (in terms of tooling scope) solutions for the SDK itself and the wrappers.

[iboB]

Experimenting with doxygen I was able to make the in-source documentation less obnoxious. Unfortunately the generated XML doesn't structure the information enough. We could switch to sphinx for C and C++ (and all languages supported by sphinx in the future). Doxygen gives us Java though. Javadoc can't output an intermediate format on its own (there's this doclet, but it hasn't been touched in 9 years... could it still be relevant?)

We could end-up in a weird situation where we use sphinx for C++ and doxygen for java.

Anyway. We need to start creating content. My current plan is to generate xml for the apis we have: C++, C, Java, Swift in separate directories and use an external script to parse the xml output and arrange it into a web page. Exernal as in "not part of this repo". I think it should be part of the website repo, where data from multiple sources can safely be merged.