Skip to content

Commit

Permalink
Prelim docs (#74)
Browse files Browse the repository at this point in the history
This PR fixes sc-20959 which requested some preliminary PyEnsign docs
that we could host on read the docs and link to from the ensign docs.
This should also make it easier for PyEnsign contributors to contribute
to the PyEnsign docs!

I have made the following changes:

1. Set up a new [sphinx
project](https://www.sphinx-doc.org/en/master/usage/quickstart.html#setting-up-the-documentation-sources)
inside the `docs/` folder
2. Added some preliminary content, but mainly just to show how we can
integrate our own text with the PyEnsign code using the `autodoc`
feature in Sphinx.
  • Loading branch information
rebeccabilbro authored Oct 6, 2023
1 parent bf5be2e commit 20e86f2
Show file tree
Hide file tree
Showing 14 changed files with 286 additions and 1 deletion.
20 changes: 20 additions & 0 deletions docs/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build

# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
27 changes: 27 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# PyEnsign Documentation

*Welcome to the PyEnsign docs!*

## reStructuredText

PyEnsign uses [Sphinx](http://www.sphinx-doc.org/en/master/index.html) to build our documentation. With Sphinx, docstrings used to describe PyEnsign functions can be automatically included when the documentation is built via [autodoc](http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#sphinx.ext.autodoc).

To take advantage of these features, our documentation must be written in reStructuredText (or "rst"). reStructuredText is similar to markdown, but not identical, and does take some getting used to. For instance, styling for things like codeblocks, external hyperlinks, internal cross references, notes, and fixed-width text are all unique in rst.

If you would like to contribute to our documentation and do not have prior experience with rst, we recommend you make use of these resources:

- [A reStructuredText Primer](http://docutils.sourceforge.net/docs/user/rst/quickstart.html)
- [rst notes and cheatsheet](https://cheat.readthedocs.io/en/latest/rst.html)

*Note: If you are on an M1 mac and you want to [install sphinx using brew](https://www.sphinx-doc.org/en/master/usage/installation.html#homebrew), make sure you have the [correct brew installation](https://stackoverflow.com/a/64997047).


## Building the Docs

To build the documents locally, first install any documentation-specific requirements with `pip` using the `requirements.txt` file in the `docs` directory:

```bash
$ pip install -r docs/requirements.txt
```

You will then be able to build the documentation from inside the `docs` directory by running `make html`; the documentation will be built and rendered in the `_build/html` directory. You can view it by opening `_build/html/index.html` then navigating to your documentation in the browser.
16 changes: 16 additions & 0 deletions docs/api/connecting.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
.. -*- mode: rst -*-
Client Connection
=================

In order to write or read data from an underlying data system (like a database or event stream), you need a client to connect to the data system and interact with it as needed (such as reading and writing data). This connection often looks something like `conn = DBConnection(credentials)`, and after creating the conn variable, subsequent lines of code can leverage it to perform the kinds of data interactions you wish to make.

*Note: To establish a client in Ensign you need an API key . By default Ensign will read credentials from the ENSIGN_CLIENT_ID and ENSIGN_CLIENT_SECRET environment variables. If you include these in your bash profile, you can connect to Ensign with the following without having to specify your credentials in code.*

API Reference
-------------

.. automodule:: pyensign.ensign
:members: Ensign
:undoc-members:
:show-inheritance:
16 changes: 16 additions & 0 deletions docs/api/eventing.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
.. -*- mode: rst -*-
Events
======

In an event-driven or microservice architecture, an event is the atomic element of data.

An events can have many different datatypes, which are then wrapped in an object or struct that provides some schema information to help Ensign know how to serialize and deserialize your data.

API Reference
-------------

.. automodule:: pyensign.api.v1beta1.event
:members: wrap, unwrap
:undoc-members:
:show-inheritance:
21 changes: 21 additions & 0 deletions docs/api/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
.. -*- mode: rst -*-
The PyEnsign API
================

This part of the documentation describes the PyEnsign API, including:

- :doc:`eventing`: How PyEnsign conceptualizes events and how to work with them.
- :doc:`connecting`: Establishing a PyEnsign client connection.
- :doc:`publishing`: Publishing data to a topic stream.
- :doc:`subscribing`: Subscribing to a topic to read off a topic stream.
- :doc:`querying`: How to query your Ensign stream data using PyEnsign, including historical data!

.. toctree::
:maxdepth: 2

eventing
connecting
publishing
subscribing
querying
16 changes: 16 additions & 0 deletions docs/api/publishing.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
.. -*- mode: rst -*-
Publishing
==========

A `Publisher` is responsible for emitting events to a topic stream.

In Ensign, you can create a `Publisher` once you have established a client. On publish, the client checks to see if it has an open publish stream created and if it doesn’t, it opens a stream to the correct Ensign node.

API Reference
-------------

.. automodule:: pyensign.publisher
:members: Publisher
:undoc-members:
:show-inheritance:
16 changes: 16 additions & 0 deletions docs/api/querying.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
.. -*- mode: rst -*-
Queries
=======

Ready to query data from your Ensign streams? Ensign stores chronological changes to all objects, which you can query with SQL-like syntax to get change vectors in addition to static snapshots.

This page describes how to format your PyEnsign queries to get the data you want.

API Reference
-------------

.. automodule:: pyensign.api.v1beta1.query
:members: format_query
:undoc-members:
:show-inheritance:
14 changes: 14 additions & 0 deletions docs/api/subscribing.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
.. -*- mode: rst -*-
Subscribing
===========

A `Subscriber` is a downstream component that is listening for events published by a `Publisher` onto a topic.

API Reference
-------------

.. automodule:: pyensign.subscriber
:members: Subscriber
:undoc-members:
:show-inheritance:
49 changes: 49 additions & 0 deletions docs/conf.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html

# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information

project = 'PyEnsign'
copyright = '2023, Rotational Labs'
author = 'Rotational Labs'

# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

extensions = []

templates_path = ['_templates']
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']

# The master toctree document.
master_doc = "index"

# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
# source_suffix = ['.rst', '.md']
source_suffix = ".rst"

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or custom ones.
extensions = [
"sphinx.ext.autodoc",
"sphinx.ext.autosummary",
"sphinx.ext.intersphinx",
"sphinx.ext.coverage",
"sphinx.ext.mathjax",
"sphinx.ext.viewcode",
"sphinx.ext.todo",
]


# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output

html_theme = 'alabaster'
html_static_path = ['_static']

# The name of the Pygments (syntax highlighting) style to use.
pygments_style = "sphinx"
44 changes: 44 additions & 0 deletions docs/index.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
.. PyEnsign documentation master file, created by
sphinx-quickstart on Fri Oct 6 11:07:32 2023.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
Welcome to PyEnsign's documentation!
====================================

PyEnsign is an open source library for interacting with `Ensign <https://ensign.rotational.dev/>`_, an eventing tool for data scientists.

The source code for PyEnsign is `available on GitHub <https://github.com/rotationalio/pyensign>`_.

Using PyEnsign, you can:
- convert your trained, pickled machine learning model into `an API <https://youtu.be/w69glRpOBD4?si=AXsoCM3QBez207Wn>`_.
- bootstrap a HuggingFace LLM with `transfer learning <https://rotational.io/blog/streaming-nlp-with-llms-and-ensign/>`_.
- transform static data APIs into `change flows <https://rotational.io/data-playground/>`_.
- and much, much more!

Recommended Learning Path
-------------------------

1. Check out the :doc:`quickstart`!
2. Explore the `examples repo <https://github.com/rotationalio/ensign-examples/tree/main/python>`_.

Find a Bug?
-----------

Check if there's already an `open issue <https://github.com/rotationalio/pyensign/issues/>`_ on the topic. If you don't find it there, open a `new issue <https://github.com/rotationalio/pyensign/issues/new/choose>`_ to let us know!

.. toctree::
:maxdepth: 2
:caption: Contents:

quickstart
api/index



Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
35 changes: 35 additions & 0 deletions docs/make.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
@ECHO OFF

pushd %~dp0

REM Command file for Sphinx documentation

if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build

%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.https://www.sphinx-doc.org/
exit /b 1
)

if "%1" == "" goto help

%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end

:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%

:end
popd
1 change: 0 additions & 1 deletion docs/quickstart.md

This file was deleted.

6 changes: 6 additions & 0 deletions docs/quickstart.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
.. -*- mode: rst -*-
Quickstart
==========

Coming soon!
6 changes: 6 additions & 0 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
pyensign>=0.10b0

# Documentation Dependencies
Sphinx>=3.4
sphinx-rtd-theme>=0.5.1
numpydoc>=1.1

0 comments on commit 20e86f2

Please sign in to comment.