From e578fd4f714284add132fd863f9c09162dcc82d2 Mon Sep 17 00:00:00 2001 From: danielballan Date: Wed, 24 May 2017 08:59:56 -0400 Subject: [PATCH 01/10] DOC: Mention scripts and conda envs. --- source/fresh-installation.rst | 37 +++++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/source/fresh-installation.rst b/source/fresh-installation.rst index b1fd26e..41742d9 100644 --- a/source/fresh-installation.rst +++ b/source/fresh-installation.rst @@ -17,6 +17,43 @@ one at least one machine, skip to the next section. class that installs mongo 3.x some dedicated server and start the mongo daemon. +Create Conda Environments ++++++++++++++++++++++++++ + +A puppet class should install conda into ``/opt/conda`` on machines designated +``*-srv*`` ("server") or ``*-ws*`` ("workstation") as soon as they are on the +NSLS-II network and working with puppet. It should also create an empty +directory, ``/opt/conda_envs`` for root-controlled conda environments. The +configuration in ``/opt/conda/.condarc`` designates this location as the second +place conda should look for environments, after ``~/conda_envs``. + +Environments for data collection and data anlysis should be installed into +``/opt/conda_envs``. In the second operating cycle of 2017, the commands to do +this were: + +.. code-block:: bash + + sudo conda create -p /opt/conda_envs/collection-17Q2.0 -c nsls2-tag -y collection + sudo conda create -p /opt/conda_envs/analysis-17Q2.0 -c nsls2-tag -y analysis + fix_conda_privileges.sh + +where ``collection-17Q2.0`` and ``analysis-17Q2.0`` are the names of conda +metapackages and the names of the environments (under ``/opt/conda_envs``) +where these metapackages are installed. + +Install Scripts ++++++++++++++++ + +Two convenience scripts, ``fix_conda_privileges.sh`` and ``bsui`` should be +installed in ``/usr/local/bin``, also by puppet. + +* ``fix_conda_privileges.sh`` works around a bug in conda wherein users cannot + properly access root-installed conda packages. It should be run after + creating or updating and conda environments in ``/opt/conda_envs``, as shown + above. +* ``bsui`` is a shortcut script that activates a conda environment and starts + IPython with the 'collection' profile. + Create New IPython Profile ++++++++++++++++++++++++++ From 73aff741652398fc06308c3291b87d68d3534003 Mon Sep 17 00:00:00 2001 From: danielballan Date: Wed, 24 May 2017 14:46:48 -0400 Subject: [PATCH 02/10] Fix typo --- source/fresh-installation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/fresh-installation.rst b/source/fresh-installation.rst index 41742d9..713ad05 100644 --- a/source/fresh-installation.rst +++ b/source/fresh-installation.rst @@ -103,7 +103,7 @@ server where the mongo daemon is running. 'port': 27017, 'database': 'filestore-production-v1', mds = MDS(mds_config) - mds_readonly = MDS(mds_config) + mds_readonly = MDSRO(mds_config) fs_readonly = FileStoreRO(fs_config) db = Broker(mds_readonly, fs_readonly) From 2a8cdb762d27caa84a47171208dae816d07227e0 Mon Sep 17 00:00:00 2001 From: danielballan Date: Wed, 24 May 2017 14:46:58 -0400 Subject: [PATCH 03/10] Update suggested insert API --- source/fresh-installation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/fresh-installation.rst b/source/fresh-installation.rst index 713ad05..03608bf 100644 --- a/source/fresh-installation.rst +++ b/source/fresh-installation.rst @@ -110,7 +110,7 @@ server where the mongo daemon is running. # Subscribe metadatastore to documents. # If this is removed, data is not saved to metadatastore. from bluesky.global_state import gs - gs.RE.subscribe('all', mds.insert) + gs.RE.subscribe('all', db.insert) # Import matplotlib and put it in interactive mode. import matplotlib.pyplot as plt From b7ec47da5e21c25927aa5e23d13bd9f2cd005961 Mon Sep 17 00:00:00 2001 From: danielballan Date: Wed, 24 May 2017 14:47:11 -0400 Subject: [PATCH 04/10] typo --- source/fresh-installation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/fresh-installation.rst b/source/fresh-installation.rst index 03608bf..6933382 100644 --- a/source/fresh-installation.rst +++ b/source/fresh-installation.rst @@ -138,7 +138,7 @@ server where the mongo daemon is running. # ophyd.logger.setLevel(logging.DEBUG) # logging.basicConfig(level=logging.DEBUG) -Create a Beamling GitHub Organization +Create a Beamline GitHub Organization +++++++++++++++++++++++++++++++++++++ 1. Create a username on github.com if you don't have one. Create a new From f604c14af3475173a07caf95f0859db9d8cd7a3e Mon Sep 17 00:00:00 2001 From: danielballan Date: Wed, 24 May 2017 14:47:19 -0400 Subject: [PATCH 05/10] typo --- source/fresh-installation.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/source/fresh-installation.rst b/source/fresh-installation.rst index 6933382..ae093f2 100644 --- a/source/fresh-installation.rst +++ b/source/fresh-installation.rst @@ -161,8 +161,8 @@ Create a Beamline GitHub Organization .. code-block:: bash - git remote add https://github.com/NSLS-II-XXX/profile_collection.git - git push -u origin master + git remote add upstream https://github.com/NSLS-II-XXX/profile_collection.git + git push -u upstream master Configure the Olog From 457040a8dad3dd1164819fabf22b0bdd65a66495 Mon Sep 17 00:00:00 2001 From: danielballan Date: Wed, 24 May 2017 14:47:34 -0400 Subject: [PATCH 06/10] Udpate conda explanation --- source/fresh-installation.rst | 37 ++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 18 deletions(-) diff --git a/source/fresh-installation.rst b/source/fresh-installation.rst index ae093f2..4a1ef38 100644 --- a/source/fresh-installation.rst +++ b/source/fresh-installation.rst @@ -254,7 +254,8 @@ New Workstation for Data Collection or Analysis puppet. 2. Create configuration files for metadatastore and filestore. As root user, - compose two new files: + compose two new files. The ``hostname`` should be the host where the mongo + service running (conventionally, the ``*-ca1`` machine, as noted above). .. code-block:: bash @@ -284,12 +285,18 @@ Add the following to the user's ``~/.bashrc`` file. export no_proxy=cs.nsls2.local export PATH=/opt/conda/bin:$PATH -The first three lines are local NSLS-II controls network configuration. +The first three lines are local NSLS-II controls network configuration. They +should already be set at the system level but in practice they are often not. Conda has already been installed on all NSLS-II workstations (ws) and servers (srv) in a shared location. The last line adds conda to the user's PATH so that it overrides any system-installed Python, IPython, etc. +Convenience Script ``bsui`` ++++++++++++++++++++++++++++ + +The script ``bsui`` + Custom User Environments ++++++++++++++++++++++++ @@ -297,39 +304,33 @@ Any user can create a conda environment, a set of binaries and Python packages completely under their control. User conda environments are stored under ``~/conda_envs/``. -This command creates a new environment called ``collection`` with the latest -"tagged" (i.e., stable) versions of ophyd, bluesky, pyolog, and xray_vision. +This command creates a new environment called ``my-env`` with all the versions +of the collection software used for the second operating cycle of 2017. .. code-block:: bash - conda create -c nsls2-tag -n collection ophyd bluesky pyolog xray_vision + conda create -c nsls2-tag -n my-env collection-17Q2 To test the new environment, activate it: .. code-block:: bash - source activate collection + source activate my-env -Check that ``which ipython`` point to a path with the word ``collection`` it -in (not ``/usr/bin/python``, as a counterexample). To troubleshoot, you -might need to refresh bash with the command ``hash -r``. +Troubleshooting: Check that ``which ipython`` point to a path with the word +``my-env`` it in (not ``/usr/bin/python``, as a counterexample). To +troubleshoot, you might need to refresh bash with the command ``hash -r``. To get "development" versions that are maybe less stable but contain the latest bug fixes and features, use the ``nsls2-dev`` channel in place of ``nsls2-tag``. - -Create or Updating Shared (Root) Environments -+++++++++++++++++++++++++++++++++++++++++++++ +Creating or Updating Shared (Root) Environments ++++++++++++++++++++++++++++++++++++++++++++++++ Administrators with sudo access can create or update conda environments that users can use ("activate") but only administrators can edit. These environments -are located in ``/opt/conda_envs``. All beamlines that use JupyterHub -(notebook.nsls2.bnl.gov) have an environment at ``/opt/conda_envs/analysis``. -Some beamlines also have a shared collection environment at -``/opt/conda_envs/collection``. Others prefer to install the collection -software individually per user. In either case, it always possible for users to -create custom environments as desired. +are located in ``/opt/conda_envs``. .. note:: From b07fbdea84e4aeb17ca769f4422649e1e791c146 Mon Sep 17 00:00:00 2001 From: Stuart Campbell Date: Wed, 24 May 2017 15:16:56 -0400 Subject: [PATCH 07/10] Update fresh-installation.rst --- source/fresh-installation.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/fresh-installation.rst b/source/fresh-installation.rst index 4a1ef38..61679e8 100644 --- a/source/fresh-installation.rst +++ b/source/fresh-installation.rst @@ -22,9 +22,9 @@ Create Conda Environments A puppet class should install conda into ``/opt/conda`` on machines designated ``*-srv*`` ("server") or ``*-ws*`` ("workstation") as soon as they are on the -NSLS-II network and working with puppet. It should also create an empty -directory, ``/opt/conda_envs`` for root-controlled conda environments. The -configuration in ``/opt/conda/.condarc`` designates this location as the second +NSLS-II network and have a puppet client configured and running. It should also +create an empty directory, ``/opt/conda_envs`` for root-controlled conda environments. +The configuration in ``/opt/conda/.condarc`` designates this location as the second place conda should look for environments, after ``~/conda_envs``. Environments for data collection and data anlysis should be installed into From 323397ff9964069c606ff09d587ef89f61f76efd Mon Sep 17 00:00:00 2001 From: Stuart Campbell Date: Wed, 24 May 2017 15:17:58 -0400 Subject: [PATCH 08/10] Update fresh-installation.rst --- source/fresh-installation.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/fresh-installation.rst b/source/fresh-installation.rst index 61679e8..e96b3c7 100644 --- a/source/fresh-installation.rst +++ b/source/fresh-installation.rst @@ -317,7 +317,7 @@ To test the new environment, activate it: source activate my-env -Troubleshooting: Check that ``which ipython`` point to a path with the word +Troubleshooting: Check that ``which ipython`` points to a path with the word ``my-env`` it in (not ``/usr/bin/python``, as a counterexample). To troubleshoot, you might need to refresh bash with the command ``hash -r``. From 7ebe859a8b3f02ee7881118a78d86deb9062cb26 Mon Sep 17 00:00:00 2001 From: Stuart Campbell Date: Wed, 24 May 2017 15:23:19 -0400 Subject: [PATCH 09/10] Added sentence about conda on other machines Such as ca or gpu machines. --- source/fresh-installation.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/source/fresh-installation.rst b/source/fresh-installation.rst index e96b3c7..274d570 100644 --- a/source/fresh-installation.rst +++ b/source/fresh-installation.rst @@ -22,7 +22,9 @@ Create Conda Environments A puppet class should install conda into ``/opt/conda`` on machines designated ``*-srv*`` ("server") or ``*-ws*`` ("workstation") as soon as they are on the -NSLS-II network and have a puppet client configured and running. It should also +NSLS-II network and have a puppet client configured and running. It should also be +noted that there are some special cases where other 'classes' of machine also receive +the installation (i.e. on some ``-ca*`` and ``-gpu*`` machines). Puppet should also create an empty directory, ``/opt/conda_envs`` for root-controlled conda environments. The configuration in ``/opt/conda/.condarc`` designates this location as the second place conda should look for environments, after ``~/conda_envs``. From 1ba3b2327ab3216f962c53bc5148061285e76d86 Mon Sep 17 00:00:00 2001 From: danielballan Date: Thu, 25 May 2017 13:00:37 -0400 Subject: [PATCH 10/10] BLD: Install doctr from PR branch instead of specific commit on that branch. --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index bb4f185..dd33148 100644 --- a/.travis.yml +++ b/.travis.yml @@ -10,5 +10,5 @@ script: - set -e - make html - pip uninstall -y doctr - - pip install git+https://github.com/danielballan/doctr@30e8e715be8386108eddd9188c60f4ad742159d8 + - pip install git+https://github.com/danielballan/doctr@other-master - doctr deploy --deploy-repo NSLS-II/NSLS-II.github.io --deploy-branch-name master .