From 2e6f1f9f66dc78a1e658f10b18818ac02d6e9a0d Mon Sep 17 00:00:00 2001 From: Paul Natsuo Kishimoto Date: Tue, 25 Aug 2020 17:06:40 +0200 Subject: [PATCH 1/4] Update install instructions (closes #196; closes #344) --- doc/source/conf.py | 1 + doc/source/install.rst | 172 ++++++++++++++++++++++++++--------------- 2 files changed, 109 insertions(+), 64 deletions(-) diff --git a/doc/source/conf.py b/doc/source/conf.py index 9b501bff9..1e5b2b41f 100644 --- a/doc/source/conf.py +++ b/doc/source/conf.py @@ -99,6 +99,7 @@ intersphinx_mapping = { 'dask': ('https://docs.dask.org/en/stable/', None), 'jpype': ('https://jpype.readthedocs.io/en/latest', None), + 'message_ix': ('https://docs.messageix.org/en/latest/', None), 'pandas': ('https://pandas.pydata.org/pandas-docs/stable/', None), 'pint': ('https://pint.readthedocs.io/en/stable/', None), 'python': ('https://docs.python.org/3/', None), diff --git a/doc/source/install.rst b/doc/source/install.rst index 790324087..b74ce1803 100644 --- a/doc/source/install.rst +++ b/doc/source/install.rst @@ -1,22 +1,31 @@ Installation ************ -A desktop or laptop computer is sufficient for most purposes using :mod:`ixmp`. Most users will have :mod:`ixmp` installed automatically when `installing MESSAGEix`_. +The sections below cover other use cases. -The sections below cover other use cases: +Ensure you have first read the :doc:`prerequisites ` for understanding and using |MESSAGEix|. +These include specific points of knowledge that are necessary to understand these instructions and choose among different installation options. -- Installing *ixmp* to be used alone (i.e., with models or frameworks other than |MESSAGEix|): +Use cases for installing ixmp directly include: - - see the sections `Install system dependencies`_, - - then `Install ixmp via Anaconda`_. +- Installing *ixmp* to be used alone (i.e., with models or frameworks other than |MESSAGEix|). + Follow the sections: -- Installing *ixmp* from source, for development purposes: see `Install ixmp from source`_. + - `Install system dependencies`_, then + - `Using Anaconda`_. -- Installing the R API to *ixmp*: +- Installing *ixmp* from source, for development purposes, e.g. to be used with a source install of :mod:`message_ix`. + Follow the sections: + + - `Install system dependencies`_, then + - `From source`_. + +- Installing the :doc:`rixmp ` R package, to use the :mod:`ixmp`, alone, from R. + Again, to use |MESSAGEix| from R, it is sufficient to install ``rmessageix``; and not necessary to also/separately install ``rximp``. - Start with `Install system dependencies`_. - - Then install *ixmp* either via Anaconda, or from source. + - Then install :mod:`ixmp` from source. - Finally, see `Install rixmp`_. **Contents:** @@ -27,12 +36,12 @@ The sections below cover other use cases: Install system dependencies =========================== -GAMS ----- +GAMS (required) +--------------- :mod:`ixmp` requires `GAMS`_. -1. Download GAMS for your operating system; either the `latest version`_ or `version 29`_ (see note below). +1. Download GAMS for your operating system; either the `latest version`_ or, for users not familiar with GAMS licenses, `version 29`_ (see note below). 2. Run the installer. @@ -59,40 +68,62 @@ GAMS If you only have a license for an older version of GAMS, install both the older and the latest versions. -Graphviz --------- +Graphviz (optional) +------------------- :meth:`ixmp.reporting.Reporter.visualize` uses `Graphviz`_, a program for graph visualization. Installing ixmp causes the python :mod:`graphviz` package to be installed. If you want to use :meth:`.visualize` or run the test suite, the Graphviz program itself must also be installed; otherwise it is **optional**. -If you `Install ixmp via Anaconda`_, Graphviz is installed automatically via `its conda-forge package`_. +If you install :mod:`ixmp` using Anaconda, Graphviz is installed automatically via `its conda-forge package`_. For other methods of installation, see the `Graphviz download page`_ for downloads and instructions for your system. -Install ``ixmp`` via Anaconda -============================= +Install :mod:`ixmp` +=================== + +Using Anaconda +-------------- After installing GAMS, we recommend that new users install Anaconda, and then use it to install :mod:`ixmp`. Advanced users may choose to install :mod:`ixmp` from source code (next section). -4. Install Python via `Anaconda`_. +4. Install Python via either `Miniconda`_ or `Anaconda`_. [1]_ We recommend the latest version; currently Python 3.8. 5. Open a command prompt. - We recommend Windows users use the “Anaconda Prompt” to avoid permissions issues when installing and using :mod:`ixmp`. + We recommend Windows users use the “Anaconda Prompt” to avoid issues with permissions and environment variables when installing and using :mod:`ixmp`. This program is available in the Windows Start menu after installing Anaconda. -6. Install the ``ixmp`` package:: +6. Configure conda to install :mod:`ixmp` from the conda-forge channel [2]_:: + + $ conda config --prepend channels conda-forge + +7. Create a new conda enviroment. + This step is **required** if using Anaconda, but *optional* if using Miniconda. + This example uses the name ``ixmp_env``, but you can use any name of your choice:: + + $ conda create --name ixmp_env + $ conda activate ixmp_env + +6. Install the ``ixmp`` package into the current environment (either ``base``, or another name from step 7, e.g. ``ixmp_env``):: $ conda install -c conda-forge ixmp +.. [1] See the `conda glossary`_ for the differences between Anaconda and Miniconda, and the definitions of the terms ‘channel’ and ‘environment’ here. +.. [2] The ‘$’ character at the start of these lines indicates that the command text should be entered in the terminal or prompt, depending on the operating system. + Do not retype the ‘$’ character itself. + +.. note:: When using Anaconda (not Miniconda), steps (5) through (8) can also be performed using the graphical Anaconda Navigator. + See the `Anaconda Navigator documentation`_ for how to perform the various steps. -Install ``ixmp`` from source -============================ + +From source +----------- 4. (Optional) If you intend to contribute changes to *ixmp*, first register a Github account, and fork the `ixmp repository `_. This will create a new repository ``/ixmp``. + (Please also see :message_ix:doc:`contributing`.) 5. Clone either the main repository, or your fork; using the `Github Desktop`_ client, or the command line:: @@ -113,83 +144,96 @@ Install ``ixmp`` from source $ pytest -Install development tools -------------------------- +Install ``rixmp`` +================= -Developers making changes to the :mod:`ixmp` source **may** need one or more of the following tools. -Users developing models using existing functionality **should not** need these tools. +``rixmp`` is the R interface to :mod:`ixmp`; see :doc:`its documentaiton `. +You only need to install ``rixmp`` if you intend to use :mod:`ixmp` from R, rather than from Python. -Git - Use one of: +Install :mod:`ixmp` **from source**, per the section above. +Then: - - https://git-scm.com/downloads - - https://desktop.github.com - - https://www.gitkraken.com +8. `Install R `_. + Ensure that your ``PATH`` environment variable is configured correctly so that the ``Rscript`` executable is available. - In addition, set up an account at https://github.com, and familiarize yourself with forking and cloning repositories, as well as pulling, committing and pushing changes. + .. warning:: + Ensure the the R version installed is either 32- *or* 64-bit (and >= 3.5.0), consistently with GAMS and Java. + Having both 32- and 64-bit versions of R, or mixed 32- and 64-bit versions of different packages, can cause errors. -Java Development Kit (JDK) - - Install the Java Development Kit (JDK) for Java SE version 8 from https://www.oracle.com/technetwork/java/javase/downloads/index.html +9. Open a command prompt in the :file:`ixmp/` directory. + Type the following commands to build, then install, ``rixmp`` and its dependencies, including reticulate_:: - .. note:: At this point, ixmp is not compatible with JAVA SE 9. + $ R CMD build rixmp - - Follow the `JDK website instructions `_ to set the ``JAVA_HOME`` environment variable; if ``JAVA_HOME`` does not exist, add it as a new system variable. +10. Check that there is only one :file:`*.tar.gz` or :file:`.zip` file in the folder, then run:: - - Update your ``PATH`` environment variable to point to the JRE binaries and server installation (e.g., :file:`C:\\Program Files\\Java\\jdk[YOUR JDK VERSION]\\jre\\bin\\`, :file:`C:\\Program Files\\Java\\jdk[YOUR JDK VERSION]\\jre\\bin\\server`). + # On Windows + $ R CMD INSTALL rixmp_*.zip - .. warning:: Do not overwrite the existing ``PATH`` environment variable, but add to the list of existing paths. + # Other operating systems + $ R CMD INSTALL rixmp_*.tar.gz +11. (Optional) Install `IRKernel`_, which allows running R code in Jupyter notebooks (see the link for instructions). +12. (Optional) Check that the R interface works by using the built-in test suite to run the R tutorial notebooks:: -Install ``rixmp`` -================= + $ pytest -m rixmp -See also the :ref:`rixmp documentation `. -1. `Install R `_. +Troubleshooting +=============== + +Run ``ixmp show-versions`` on the command line to check that you have all dependencies installed, or when reporting issues. - .. warning:: - Ensure the the R version installed is either 32- *or* 64-bit (and >= 3.5.0), consistently with GAMS and Java. - Having both 32- and 64-bit versions of R, or mixed 32- and 64-bit versions of different packages, can cause errors. +For Anaconda users experiencing problems during installation of ixmp, check that the following paths are part of the ``PATH`` environment variable, and add them if missing:: -2. Enter the directory ``rixmp/`` and use R to build and install the package and its dependencies, including reticulate_:: + C:\[YOUR ANACONDA LOCATION]\Anaconda3; + C:\[YOUR ANACONDA LOCATION]\Anaconda3\Scripts; + C:\[YOUR ANACONDA LOCATION]\Anaconda3\Library\bin; - $ cd rixmp - $ Rscript -e "install.packages(c('knitr', 'reticulate'), repos='http://cran.rstudio.com/')" - $ R CMD build . -3. Check that there is only one ``*tar.gz`` in the folder, then run:: +Install development tools +========================= - $ R CMD INSTALL rixmp_* +Developers making changes to the :mod:`ixmp` source **may** need one or more of the following tools. +Users developing models using existing functionality **should not** need these tools. -4. (Optional) Run the built-in test suite to check that *ixmp* and *rixmp* functions, as in *Install ixmp from source 6.* (installing - the R ``devtools`` package might be a pre-requisite). In the ``ixmp`` directory type:: +Git + Use one of: - $ pytest -m rixmp + - https://git-scm.com/downloads + - https://desktop.github.com + - https://www.gitkraken.com -5. (Optional) To use rixmp in Jupyter notebooks, install the `IR kernel `_. +Java Development Kit (JDK) + - Install the Java Development Kit (JDK) for Java SE version 8 from https://www.oracle.com/technetwork/java/javase/downloads/index.html -6. (Optional) Install `Rtools `_ and add the path to the environment variables. + .. note:: At this point, ixmp is not compatible with JAVA SE 9. -.. _reticulate: https://rstudio.github.io/reticulate/ + - Follow the `JDK website instructions`_ to set the ``JAVA_HOME`` environment variable; if ``JAVA_HOME`` does not exist, add it as a new system variable. + - Update your ``PATH`` environment variable to point to the JRE binaries and server installation (e.g., :file:`C:\\Program Files\\Java\\jdk[YOUR JDK VERSION]\\jre\\bin\\`, :file:`C:\\Program Files\\Java\\jdk[YOUR JDK VERSION]\\jre\\bin\\server`). -Troubleshooting -=============== + .. warning:: Do not overwrite the existing ``PATH`` environment variable, but add to the list of existing paths. -Run ``ixmp show-versions`` on the command line to check that you have all dependencies installed, or when reporting issues. +Rtools + https://cran.r-project.org/bin/windows/Rtools/ -For Anaconda users experiencing problems during installation of ixmp, check that the following paths are part of the ``PATH`` environment variable, and add them if missing:: + For installing or modifying some R packages on Windows. - C:\[YOUR ANACONDA LOCATION]\Anaconda3; - C:\[YOUR ANACONDA LOCATION]\Anaconda3\Scripts; - C:\[YOUR ANACONDA LOCATION]\Anaconda3\Library\bin; .. _`installing MESSAGEix`: https://docs.messageix.org/en/latest/getting_started.html .. _`Anaconda`: https://www.continuum.io/downloads .. _`GAMS`: http://www.gams.com .. _`latest version`: https://www.gams.com/download/ .. _`version 29`: https://www.gams.com/29/ +.. _Graphviz: https://www.graphviz.org .. _`its conda-forge package`: https://anaconda.org/conda-forge/graphviz -.. _`Graphviz download page`: https://www.graphviz.org/download/ +.. _Graphviz download page: https://www.graphviz.org/download/ +.. _Miniconda: https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html +.. _conda glossary: https://docs.conda.io/projects/conda/en/latest/glossary.html +.. _Anaconda Navigator documentation: https://docs.anaconda.com/anaconda/navigator/ .. _`Github Desktop`: https://desktop.github.com +.. _reticulate: https://rstudio.github.io/reticulate/ +.. _IRkernel: https://irkernel.github.io/installation/ +.. _JDK website instructions: https://docs.oracle.com/cd/E19182-01/820-7851/inst_cli_jdk_javahome_t/ From 598e0d577d37f3c1066cf9b951dee866e6c5b1d4 Mon Sep 17 00:00:00 2001 From: Paul Natsuo Kishimoto Date: Tue, 25 Aug 2020 17:18:11 +0200 Subject: [PATCH 2/4] Document support for in-memory HyperSQL databases (closes #224) --- doc/source/api-backend.rst | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/doc/source/api-backend.rst b/doc/source/api-backend.rst index 855d49eb4..081caea0f 100644 --- a/doc/source/api-backend.rst +++ b/doc/source/api-backend.rst @@ -25,10 +25,19 @@ Provided backends .. autoclass:: ixmp.backend.jdbc.JDBCBackend :members: read_file, write_file - JDBCBackend supports: + JDBCBackend **supports**: - Databases in local files (HyperSQL) using ``driver='hsqldb'`` and the *path* argument. - Remote, Oracle databases using ``driver='oracle'`` and the *url*, *username* and *password* arguments. + - Temporary, in-memory databases using ``driver='hsqldb'`` and the *url* argument. + Use the `url` parameter with the format ``jdbc:hsqldb:mem:[NAME]``, where [NAME] is any string:: + + mp = ixmp.Platform( + backend="jdbc", + driver="hsqldb", + url="jdbc:hsqldb:mem:temporary platform", + ) + JDBCBackend caches values in memory to improve performance when repeatedly reading data from the same items with :meth:`.par`, :meth:`.equ`, or :meth:`.var`. @@ -41,10 +50,17 @@ Provided backends .. tip:: Modifying an item by adding or deleting elements invalidates its cache. - JDBCBackend has the following limitations: + JDBCBackend has the following **limitations**: - The `comment` argument to :meth:`Platform.add_unit` is limited to 64 characters. + JDBCBackend's implementation allows the following kinds of file input and output: + + .. autosummary:: + + read_file + write_file + .. automethod:: ixmp.backend.jdbc.start_jvm Backend API From 3a35386fea2dcabc569f3c0e20f9424f4617ff43 Mon Sep 17 00:00:00 2001 From: Paul Natsuo Kishimoto Date: Tue, 25 Aug 2020 17:19:52 +0200 Subject: [PATCH 3/4] Update PR template to match message_ix --- .github/PULL_REQUEST_TEMPLATE.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 67ea66253..1258d535e 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -30,12 +30,12 @@ the user behaviour, e.g. new features or fixes to existing behaviour. They are In ambiguous cases, strike them out and add a short explanation, e.g. -- [x] ~Tests added.~ No change in behaviour, simply refactoring. +- ~Add or expand tests.~ No change in behaviour, simply refactoring. --> -- [ ] Tests added. -- [ ] Documentation added. -- [ ] Release notes updated. +- [ ] Add or expand tests. +- [ ] Add, expand, or update documentation. +- [ ] Update release notes.