Update Sphinx docs

README.md

@@ -2,6 +2,7 @@ Tools for the iTEM databases
 ============================
 [![Build Status](https://travis-ci.org/transportenergy/database.svg?branch=master)](https://travis-ci.org/transportenergy/database)
 ![Codecov](https://img.shields.io/codecov/c/gh/transportenergy/database.svg)
+[![Documentation Status](https://readthedocs.org/projects/transportenergy/badge/?version=latest)](https://transportenergy.readthedocs.io/en/latest/?badge=latest)
 
 [iTEM](https://transportenergy.org) maintains two databases:
 
@@ -17,57 +18,4 @@ This repository contains tools for *both* databases, in *both* Python and R. The
 - Members of the [iTEM organizing group](https://transportenergy.org/participants/#organizing-group) use the tools to prepare templates for data submission, and to aggregate & clean submitted data.
 - iTEM participants, if they choose, maintain code that *transparently & reproducibly* derives the iTEM variables from immediate output files of their models.
 - The tools are well-tested and yield reproducible results.
-
-Contents
---------
-
-- `python/` — Python code for accessing & maintaining the database. See [`python/README.md`](https://github.com/transportenergy/database/blob/master/python/README.md).
-  - `item/data/` — *Meta*data about the iTEM databases. Actual data is stored separately; see below.
-    - `model/` — Metadata about the iTEM model database. Many of these files are in [YAML format](http://www.yaml.org/spec/1.2/spec.html) ([Wikipedia](https://en.wikipedia.org/wiki/YAML)), with more detailed comments in-line.
-      - `models.yaml` — description of the [iTEM participating models & teams](https://transportenergy.org/participants/).
-      - `dimensions/` — information about the [data dimensions of the model database](https://transportenergy.org/database/).
-      - Other directories:
-        - are named to match the keys in `models.yaml`,
-        - contain model-specific metadata, including but not limited to:
-          - `regions.yaml`: regional aggregation.
-          - `scenarios.yaml`: descriptions of scenarios submitted to each iTEM workshop.
-    - `concepts.yaml`, `measures.yaml`, `spec.yaml` — specifications of data to be submitted for for the 3rd iTEM MIP.
-    - `item_config_example.yaml` — an example configuration file.
-- `R/` — R code for accessing & maintaining the database. See [`R/README.md`](https://github.com/transportenergy/database/blob/master/R/README.md).
-
-
-Usage
------
-
-The following instructions are not language-specific.
-Both the Python and R tools will operate on data stored in the following way; see the language `README`s for specific information.
-
-1. Clone this repository and/or install the code in one or both languages.
-
-2. Create 1 or more separate directory to contain the input and output files.
-   A simple way to do this is to call `item mkdirs PATH`, where `PATH` is a base directory; this will create a tree of directories.
-   Use `item mkdirs --help` for more information.
-
-3. Copy the file [`item_config_example.yaml`](https://github.com/transportenergy/database/blob/master/python/item/data/item_config_example.yaml`) to `item_config.yaml` in any working directory where you intend to use this code.
-   Edit the file (see the inline comments) to point to the directories created in #2 above.
-
-4. Use the tools through the command-line interface:
-
-   ```
-   $ item
-   Usage: item [OPTIONS] COMMAND [ARGS]...
-
-     Command-line interface for the iTEM databases.
-
-   Options:
-     --path <KEY> <PATH>  Override data paths (multiple allowed).
-     --help               Show this message and exit.
-
-   Commands:
-     debug     Show debugging information, including paths.
-     help      Show extended help for the command-line tool.
-     mkdirs    Create a directory tree for the database.
-     model     Manipulate the model database.
-     stats     Manipulate the stats database.
-     template  Generate the MIP3 submission template.
-   ```
+- The tools are [fully documented](https://transportenergy.readthedocs.io).

doc/conf.py

@@ -187,7 +187,7 @@
 
 # Example configuration for intersphinx: refer to the Python standard library.
 intersphinx_mapping = {
-    'python': ('https://docs.python.org/', None),
+    'python': ('https://docs.python.org/3/', None),
     'pandas': ('https://pandas.pydata.org/pandas-docs/stable/', None),
     }
 

doc/historical.rst

@@ -0,0 +1,9 @@
+Historical & statistical data (:mod:`item.stats`)
+*************************************************
+
+This module contains the code that implements the `iTEM Open Data project <https://transportenergy.org/data/historical/>`_, the broader aims of which are described on the main iTEM website.
+
+The present documentation covers the code.
+
+.. automodule:: item.stats
+   :members:

doc/index.rst

@@ -9,12 +9,12 @@ International Transport Energy Modeling (iTEM) toolkit
 `iTEM`_ maintains two databases:
 
 1. A **model database** of transport energy projections assembled as part of
-   the `iTEM workshops`_, of which there have been four so far, with a fourth
-   planned for late 2019. Because of restrictions from workshop participants,
+   the `iTEM workshops`_, of which there have been four so far, with a fifth
+   planned for early 2020. Because of restrictions from workshop participants,
    the model database is currently *not public*, but only available to iTEM
    workshop participants.
 
-2. A **statistics database** to form a common, public, “best available”
+2. A **historical database** to form a common, public, “best available”
    baseline for model calibration and projections. The statistics database is
    currently *under development*.
 
@@ -40,11 +40,11 @@ This repository contains tools for *both* databases, in *both* Python and R. The
    :maxdepth: 2
    :caption: Contents:
 
+   usage
    model
-   stats
+   historical
+   reference
 
-Indices and tables
-==================
 
 * :ref:`genindex`
 * :ref:`modindex`

doc/model.rst

@@ -1,5 +1,5 @@
-``item.model``: Transport energy model projections
-==================================================
+Transport energy model projections (:mod:`item.model`)
+******************************************************
 
 .. automodule:: item.model
    :members:

doc/reference.rst

@@ -0,0 +1,24 @@
+Reference
+*********
+
+Repository organization
+-----------------------
+
+- ``python/`` — Python code for accessing & maintaining the database.
+
+  - ``item/data/`` — *Meta* -data about the iTEM databases. Actual data is stored separately; see below.
+
+    - ``model/`` — Metadata about the iTEM model database. Many of these files are in `YAML format <http://www.yaml.org/spec/1.2/spec.html>`_ (`Wikipedia <https://en.wikipedia.org/wiki/YAML>`_), with more detailed comments in-line.
+
+      - ``models.yaml`` — description of the `iTEM participating models & teams <https://transportenergy.org/participants/>`_.
+      - ``dimensions/`` — information about the `data dimensions of the model database <https://transportenergy.org/database/>`_.
+      - Other directories:
+
+        - are named to match the keys in ``models.yaml``,
+        - contain model-specific metadata, including but not limited to:
+
+          - ``regions.yaml``: regional aggregation.
+          - ``scenarios.yaml``: descriptions of scenarios submitted to each iTEM workshop.
+    - ``concepts.yaml``, ``measures.yaml``, ``spec.yaml`` — specifications of data to be submitted for for the 3rd iTEM MIP.
+    - ``item_config_example.yaml`` — an example configuration file.
+- ``R/`` — R code for accessing & maintaining the database.

doc/usage.rst

@@ -0,0 +1,129 @@
+Usage
+*****
+
+The following instructions are not language-specific.
+Both the Python and R tools will operate on data stored in the following way; see the language-specific sections below for more details.
+
+1. Clone the repository and/or install the code in one or both languages.
+
+2. Create 1 or more separate directory to contain the input and output files.
+   A simple way to do this is to call ``$ item mkdirs PATH``, where ``PATH`` is a base directory; this will create a tree of directories.
+   Use ``item mkdirs --help`` for more information.
+
+3. Copy the file ``item_config_example.yaml`` to ``item_config.yaml`` in any working directory where you intend to use this code.
+   Edit the file (see the inline comments) to point to the directories created in #2 above.
+
+4. Use the tools through the command-line interface::
+
+    $ item
+    Usage: item [OPTIONS] COMMAND [ARGS]...
+
+      Command-line interface for the iTEM databases.
+
+    Options:
+      --path <KEY> <PATH>  Override data paths (multiple allowed).
+      --help               Show this message and exit.
+
+    Commands:
+      debug     Show debugging information, including paths.
+      help      Show extended help for the command-line tool.
+      mkdirs    Create a directory tree for the database.
+      model     Manipulate the model database.
+      stats     Manipulate the stats database.
+      template  Generate the MIP3 submission template.
+
+Python
+======
+
+Installation
+------------
+Use `pip <https://pip.pypa.io/en/stable/>`_.
+From source (for instance, to develop the code locally)::
+
+    $ git clone [email protected]:transportenergy/database.git
+    $ pip install --editable database/python
+
+Or, without cloning the repository::
+
+    $ pip install --editable git://github.com/transportenergy/database#egg=item@subdirectory=python
+
+
+Usage
+-----
+
+From Python scripts::
+
+    import item
+
+    data = item.load_model_data(1)
+
+Run tests
+---------
+
+Tests in ``python/item/tests`` can be run with `py.test <https://pytest.org/>`_.
+The command-line option ``--local-data`` must be defined in order for these tests to work::
+
+    $ py.test --local-data=../../data/model/database item
+    ================ test session starts ================
+    …
+
+
+R
+===
+
+Installation
+------------
+
+Use `devtools <https://cran.r-project.org/package=devtools>`_.
+From source (for instance, to develop the code locally)::
+
+    $ git clone [email protected]:transportenergy/database.git
+    $ Rscript -e "devtools::install_local('database/R/item')"
+
+Or without cloning the repository::
+
+    devtools::install_github('transportenergy/database/R/item')
+
+
+Usage
+-----
+
+From R scripts::
+
+    library(item)
+
+    # Load version 1 of the iTEM models database
+    data <- item::load_model_data(1)
+
+From the command-line: ``run`` is an executable that invokes ``item::cli()``.
+It can be used without installing the package::
+
+    $ ./run
+    Loading item
+    Usage: ./run [OPTIONS] COMMAND
+    Command-line interface for the iTEM databases.
+
+    …
+
+    Commands:
+      mkdirs
+      debug
+      load_model_data
+
+
+Development
+-----------
+
+Code conventions and packaging follow the `“R packages” book <http://r-pkgs.had.co.nz/>`_.
+
+``test`` is an executable that runs the tests in ``R/item/tests/testthat``.
+The environment variable ``ITEM_TEST_DATA`` must be defined in order for these tests to work::
+
+  $ export ITEM_TEST_DATA=../../data/model/database
+  $ ./test
+  Loading item
+  Loading required package: testthat
+  Testing item
+  Model database: .
+
+  DONE ============================================