Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Central API Docs #1041

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Central API Docs #1041

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
51e368b
add apis
bschifferer Jul 3, 2023
af70516
add changes tox.ini
bschifferer Jul 3, 2023
ab6e416
trigger
bschifferer Jul 3, 2023
ab94230
update docs
bschifferer Jul 4, 2023
23ba7d3
Merge branch 'main' into centra_apidocs
bschifferer Jul 5, 2023
205a8c8
Merge branch 'main' into centra_apidocs
bschifferer Jul 6, 2023
216fed0
fix dep
bschifferer Jul 6, 2023
d0d0c97
Merge branch 'main' into centra_apidocs
bschifferer Jul 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

clean:
rm -rf build source/examples
rm -rf build source/examples NVTabular core dataloader models systems Transformers4Rec source/api_core/ source/api_dataloader.rst source/api_models.rst source/api_nvtabular.rst source/api_systems.rst source/api_transformers4rec/

.PHONY: help clean Makefile

Expand Down
4 changes: 4 additions & 0 deletions docs/source/api_overview.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
API Documentation
=====================

NVIDIA Merlin is an open source framework. It consists a collection of libraries, designed to integrate well together.
2 changes: 2 additions & 0 deletions docs/source/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

from natsort import natsorted

sys.path.insert(0, os.path.abspath("/models/"))
docs_dir = os.path.dirname(__file__)
repodir = os.path.abspath(os.path.join(__file__, r"../../.."))
gitdir = os.path.join(repodir, r".git")
Expand All @@ -26,6 +27,7 @@
"sphinx_multiversion",
"sphinx_rtd_theme",
"sphinx.ext.autodoc",
"sphinx.ext.autosummary",
"sphinx.ext.coverage",
"sphinx.ext.githubpages",
"sphinx.ext.napoleon",
Expand Down
21 changes: 20 additions & 1 deletion docs/source/toc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,5 +52,24 @@ subtrees:
title: Deploy the HugeCTR Model with Triton
- file: examples/scaling-criteo/04-Triton-Inference-with-Merlin-Models-TensorFlow.ipynb
title: Deploy the TensorFlow Model with Triton
- file: api_overview
title: API Documentation
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs become a little dense with this all under one TOC item. If you are open to experimenting, consider creating another caption like line 3 with "API Documentation" or "Merlin API" or something along those lines.

Centralizing the API is certainly better than how I left it, but I'm also wondering if adding titles would help the reader choose the correct API rather than browse.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel the structure looks easy to follow - it is structured in the way how we would import Classes, functions or modules

merlin.dag
merlin.loader
merlin.io
merlin.models
merlin.schema
merlin.systems
nvtabular

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might still be nice to organize the packages into themes or something though, because I suspect this makes the most sense to people who are already familiar with what's in each sub-package (e.g. people on the Merlin team.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@karlhigley do you have a suggestions? I thought organizing it by import statements is the way a user will search for API documentation?

If I want to know something about ColumnsSelector, I know it is in merlin.schema and I will look there first?

entries:
- file: api_core/merlin.dag
title: merlin.dag
- file: api_dataloader
title: merlin.loader
- file: api_core/merlin.io
title: merlin.io
- file: api_models
title: merlin.models
- file: api_core/merlin.schema
title: merlin.schema
- file: api_systems
title: merlin.systems
- file: api_nvtabular
title: nvtabular
- file: ./api_transformers4rec/modules
bschifferer marked this conversation as resolved.
Show resolved Hide resolved
title: transformers4re
- file: containers.rst
- file: support_matrix/index.rst
- file: support_matrix/index.rst
41 changes: 38 additions & 3 deletions tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,14 +61,49 @@ commands =
; Generates documentation with sphinx. There are other steps in the Github Actions workflow
; to publish the documentation on release.
changedir = {toxinidir}
deps = -rdocs/requirements-doc.txt
deps =
-rdocs/requirements-doc.txt
tensorflow
torch
torchmetrics
git+https://github.com/NVIDIA-Merlin/core.git
git+https://github.com/NVIDIA-Merlin/dataloader.git
git+https://github.com/NVIDIA-Merlin/models.git
git+https://github.com/NVIDIA-Merlin/NVTabular.git
git+https://github.com/NVIDIA-Merlin/systems.git
git+https://github.com/NVIDIA-Merlin/Transformers4Rec.git

allowlist_externals =
git
cp
commands =
git clone --depth 1 https://github.com/NVIDIA-Merlin/core.git ./docs/core
cp -r ./docs/core/docs/source/api ./docs/source/api_core/
git clone --depth 1 https://github.com/NVIDIA-Merlin/dataloader.git ./docs/dataloader
cp ./docs/dataloader/docs/source/api.rst ./docs/source/api_dataloader.rst
git clone --depth 1 https://github.com/NVIDIA-Merlin/models.git ./docs/models
cp ./docs/models/docs/source/api.rst ./docs/source/api_models.rst
git clone --depth 1 https://github.com/NVIDIA-Merlin/NVTabular.git ./docs/NVTabular
cp ./docs/NVTabular/docs/source/api.rst ./docs/source/api_nvtabular.rst
git clone --depth 1 https://github.com/NVIDIA-Merlin/systems.git ./docs/systems
cp ./docs/systems/docs/source/api.rst ./docs/source/api_systems.rst
git clone --depth 1 https://github.com/NVIDIA-Merlin/Transformers4Rec.git ./docs/Transformers4Rec
cp -r ./docs/Transformers4Rec/docs/source/api ./docs/source/api_transformers4rec/
python -m sphinx.cmd.build -E -P -b html docs/source docs/build/html
Comment on lines +66 to 82
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If tox alone can get you what you need, I think I'd just go with that. (And, tbh, it escaped my imagination that tox would make it so simple.)

The only alternative that I'd consider is using one of the containers because the containers already have the tagged source in the container.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure if the documentation is build inside a container - in addition, I want to pull the latest documentation code from every repository



[testenv:docs-multi]
; Run the multi-version build that is shown on GitHub Pages.
changedir = {toxinidir}
deps = -rdocs/requirements-doc.txt
deps =
-rdocs/requirements-doc.txt
git+https://github.com/NVIDIA-Merlin/models.git
git+https://github.com/NVIDIA-Merlin/NVTabular.git
git+https://github.com/NVIDIA-Merlin/core.git
git+https://github.com/NVIDIA-Merlin/systems.git
git+https://github.com/NVIDIA-Merlin/dataloader.git
allowlist_externals =
git clone https://github.com/NVIDIA-Merlin/core.git
commands =
sphinx-multiversion --dump-metadata docs/source docs/build/html | jq "keys"
sphinx-multiversion docs/source docs/build/html
sphinx-multiversion docs/source docs/build/html