diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 000000000..d8bd0a9db --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "galaxy/roles/galaxyproject.galaxyextras"] + path = galaxy/roles/galaxyprojectdotorg.galaxyextras + url = https://github.com/galaxyproject/ansible-galaxy-extras diff --git a/README.md b/README.md index dcb1cb626..d7441d070 100644 --- a/README.md +++ b/README.md @@ -7,22 +7,28 @@ One of the main goals is to make the access to entire tool suites as easy as pos this includes the setup of a public available webservice that needs to be maintained, or that the Tool-user needs to either setup a Galaxy Server by its own or to have Admin access to a local Galaxy server. With docker, tool developers can create their own Image with all dependencies and the user only needs to run it within docker. -The Image is based on [Debian/wheezy](http://www.debian.org/). and all recommended Galaxy requirements are installed. +The Image is based on [Ubuntu 14.04 LTS](http://releases.ubuntu.com/14.04/) and all recommended Galaxy requirements are installed. The following chart should illustrate the [Docker](http://www.docker.io) image hierarchy we have build to make is as easy as possible to build on different layers of our stack and create many exciting Galaxy flavours. + +![Docker hierarchy](chart.png) Usage ===== -At first you need to install docker. Please follow the instruction on https://www.docker.io/gettingstarted/#h_installation +At first you need to install docker. Please follow the [very good instructions](https://docs.docker.com/installation/) from the Docker project. After the successful installation, all what you need to do is: -``docker run -d -p 8080:80 -p 8021:21 bgruening/galaxy-stable`` + ```sh + docker run -d -p 8080:80 -p 8021:21 bgruening/galaxy-stable + ``` I will shortly explain the meaning of all the parameters. For a more detailed description please consult the [docker manual](http://docs.docker.io/), it's really worth reading. Let's start: ``docker run`` will run the Image/Container for you. In case you do not have the Container stored locally, docker will download it for you. ``-p 8080:80`` will make the port 80 (inside of the container) available on port 8080 on your host. Same holds for port 8021, that can be used to transfer data via the FTP protocol. Inside the container a Apache Webserver is running on port 80 and that port can be bound to a local port on your host computer. With this parameter you can access your Galaxy instance via ``http://localhost:8080`` immediately after executing the command above. ``bgruening/galaxy-stable`` is the Image/Container name, that directs docker to the correct path in the [docker index](https://index.docker.io/u/bgruening/galaxy-stable/). ``-d`` will start the docker container in daemon mode. For an interactive session, you can execute: -``docker run -i -t -p 8080:80 bgruening/galaxy-stable /bin/bash`` + ```sh + docker run -i -t -p 8080:80 bgruening/galaxy-stable /bin/bash + ``` and run the ``` startup ``` script by yourself, to start PostgreSQL, Apache and Galaxy. @@ -30,7 +36,9 @@ Docker images are "read-only", all your changes inside one session will be lost Fortunately, this is as easy as: -``docker run -d -p 8080:80 -v /home/user/galaxy_storage/:/export/ bgruening/galaxy-stable`` + ```sh + docker run -d -p 8080:80 -v /home/user/galaxy_storage/:/export/ bgruening/galaxy-stable + ``` With the additional ``-v /home/user/galaxy_storage/:/export/`` parameter, docker will mount the folder ``/home/user/galaxy_storage`` into the Container under ``/export/``. A ``startup.sh`` script, that is usually starting Apache, PostgreSQL and Galaxy, will recognize the export directory with one of the following outcomes: @@ -45,85 +53,163 @@ Enabling Interactive Environments in Galaxy Interactive Environments (IE) are sophisticated ways to extend Galaxy with powerful services, like IPython, in a secure and reproducible way. For this we need to be able to launch Docker containers inside our Galaxy Docker container. At least docker 1.3 is needed on the host system. -``docker run -d -p 8080:80 -p 8021:21 -p 8800:8800 --privileged=true -v /home/user/galaxy_storage/:/export/ bgruening/galaxy-stable`` + ```bash + docker run -d -p 8080:80 -p 8021:21 -p 8800:8800 --privileged=true \ + -v /home/user/galaxy_storage/:/export/ bgruening/galaxy-stable + ``` The port 8800 is the proxy port that is used to handle Interactive Environments. ``--privileged`` is needed to start docker containers inside docker. Using Parent docker ------------------- -On some linux distributions, Docker-In-Docker can run into issues (such as running out of loopback interfaces). If this is an issue, -you can use a 'legacy' mode that use a docker socket for the parent docker installation mounted inside the container. To engage, set the -environmental variable DOCKER_PARENT +On some linux distributions, Docker-In-Docker can run into issues (such as running out of loopback interfaces). If this is an issue, you can use a 'legacy' mode that use a docker socket for the parent docker installation mounted inside the container. To engage, set the environmental variable `DOCKER_PARENT` + + ```bash + docker run -p 8080:80 -p 8021:21 -p 8800:8800 \ + --privileged=true -e DOCKER_PARENT=True \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /home/user/galaxy_storage/:/export/ \ + bgruening/galaxy-stable + ``` + +Galaxy Report Webapp +-------------------- + +For admins wishing to have more information on the status of a galaxy instance, the Galaxy Report Webapp is served on `http://localhost:8080/reports`. As default this site is password protected with `admin:admin`. You can change this by providing a `reports_htpasswd` file in `/home/user/galaxy_storage/`. + +You can disable the Report Webapp entirely by providing the environment variable `NONUSE` during container startup. + + ```bash + docker run -p 8080:80 -e "NONUSE=reports" bgruening/galaxy-stable + ``` + +Galaxy's config settings +------------------------ + +Every Galaxy configuration setting can be overwritten by a given environment variable during startup. For example by default the `admin_users`, `master_api_key` and the `brand` variable it set to: + + ```sh + GALAXY_CONFIG_ADMIN_USERS=admin@galaxy.org + GALAXY_CONFIG_MASTER_API_KEY=HSNiugRFvgT574F43jZ7N9F3 + GALAXY_CONFIG_BRAND="Galaxy Docker Build" + ``` -``docker run -d -p 8080:80 -p 8021:21 -p 8800:8800 --privileged=true -e DOCKER_PARENT=True -v /var/run/docker.sock:/var/run/docker.sock -v /home/user/galaxy_storage/:/export/ bgruening/galaxy-stable`` +You can and should overwrite these during launching your container: -Enabling the Galaxy Report Tool -------------------------------- + ```bash + docker run -p 8080:80 \ + -e "GALAXY_CONFIG_ADMIN_USERS=albert@einstein.gov" \ + -e "GALAXY_CONFIG_MASTER_API_KEY=83D4jaba7330aDKHkakjGa937" \ + -e "GALAXY_CONFIG_BRAND='My own Galaxy flavour'" \ + bgruening/galaxy-stable + ``` -For admins wishing to have more information on the status of a galaxy instance, you can start this image with ``-p 9001:9001`` to serve the Galaxy Report Tool -on port 9001 on your host system. +Personalize your Galaxy +----------------------- -``docker run -d -p 8080:80 -p 8021:21 -p 9001:9001 -v /home/user/galaxy_storage/:/export/ bgruening/galaxy-stable`` +The Galaxy welcome screen can be changed by providing a `welcome.hml` page in `/home/user/galaxy_storage/`. All files starting with `welcome` will be copied during starup and served as indroduction page. If you want to include images or other media, name them `welcome_*` and link them relative to your `welcome.html` ([example](`https://github.com/bgruening/docker-galaxy-stable/blob/master/galaxy/welcome.html`)). + + +Deactivating services +--------------------- + +Non-essential services can be deactivated during startup. Set the environment variable `NONUSE` to a comma separated list of services. Currently, `nodejs`, `proftp` and `reports` are supported. + + ```bash + docker run -d -p 8080:80 -p 8021:21 -p 9002:9002 \ + -e "NONUSE=nodejs,proftp,reports" bgruening/galaxy-stable + ``` + +A graphical user interface, to start and stop your services, is available on port `9002` if you run your container like above. Restarting Galaxy ----------------- -If you want to restart Galaxy without restarting the entire Galaxy container we can use `docker exec` (docker > 1.3). +If you want to restart Galaxy without restarting the entire Galaxy container you can use `docker exec` (docker > 1.3). + + ```sh + docker exec supervisorctl restart galaxy: + ``` -```docker exec supervisorctl restart galaxy:``` +In addition you start/stop every supersisord process using a webinterface on port `9002`. Start your container with: + ```sh + docker run -p 9002:9002 bgruening/galaxy-stable + ``` -Advanced logging +Advanced Logging ---------------- You can set the environment variable $GALAXY_LOGGING to FULL to access all logs from supervisor. For example start your container with: -``docker run -d -p 8080:80 -p 8021:21 -e "GALAXY_LOGGING=full" bgruening/galaxy-stable`` + ```sh + docker run -d -p 8080:80 -p 8021:21 -e "GALAXY_LOGGING=full" bgruening/galaxy-stable + ``` + +In addition you can access the supersisord webinterface on port `9002` and get access to log files. Start your container with: + ```sh + docker run -d -p 8080:80 -p 8021:21 -p 9002:9002 -e "GALAXY_LOGGING=full" bgruening/galaxy-stable + ``` -Extending the docker Image +Extending the Docker Image ========================== -If you have your Tools already included in the Tool Shed, building your own personalised Galaxy docker Image can be done using the following steps: +If your tools are already included in the Tool Shed, building your own personalised Galaxy docker Image (Galaxy flavour) can be done using the following steps: 1. Create a file the name ``Dockerfile`` 2. Include ``FROM bgruening/galaxy-stable`` at the top of the file. This means that you use the Galaxy Docker Image as base Image and build your own extensions on top of it. 3. Install your Tools from the Tool Shed via the ``install_tool_shed_repositories.py`` script. 4. execute ``docker build -t='my-docker-test'`` - 5. run your container with ``docker run -d -p 8080:80 my-docker-test`` + 5. run your container with ``docker run -p 8080:80 my-docker-test`` 6. open your web browser on ``http://localhost:8080`` For example have a look at the [deepTools](http://deeptools.github.io/) or the [ChemicalToolBox](https://github.com/bgruening/galaxytools/tree/master/chemicaltoolbox) Dockerfile's. - -https://github.com/bgruening/docker-recipes/blob/master/galaxy-deeptools/Dockerfile -https://github.com/bgruening/docker-recipes/blob/master/galaxy-chemicaltoolbox/Dockerfile + * https://github.com/bgruening/docker-recipes/blob/master/galaxy-deeptools/Dockerfile + * https://github.com/bgruening/docker-recipes/blob/master/galaxy-chemicaltoolbox/Dockerfile ``` # Galaxy - deepTools # -# VERSION 0.1 +# VERSION 0.2 FROM bgruening/galaxy-stable MAINTAINER Björn A. Grüning, bjoern.gruening@gmail.com +ENV GALAXY_CONFIG_BRAND deepTools + WORKDIR /galaxy-central -RUN service postgresql start && service apache2 start && ./run.sh --daemon && sleep 120 && python ./scripts/api/install_tool_shed_repositories.py --api admin -l http://localhost:8080 --url -# Mark one folders as imported from the host. -VOLUME ["/export/"] +# Install deepTools +RUN install-repository \ + "--url https://toolshed.g2.bx.psu.edu/ -o bgruening --name deeptools" -# Expose port 80 to the host +# Mark folders as imported from the host. +VOLUME ["/export/", "/data/", "/var/lib/docker"] + +# Expose port 80 (webserver), 21 (FTP server), 8800 (Proxy) EXPOSE :80 +EXPOSE :21 +EXPOSE :8800 # Autostart script that is invoked during container start CMD ["/usr/bin/startup"] ``` +List of Galaxy flavours +----------------------- + + * [docker-galaxy-blast](https://github.com/bgruening/docker-galaxy-blast) + * [ChemicalToolBox](https://github.com/bgruening/docker-recipes/blob/master/galaxy-chemicaltoolbox) + * [ballaxy](https://github.com/anhi/docker-scripts/tree/master/ballaxy) + * [docker-galaxy-deeptools](https://github.com/bgruening/docker-recipes/blob/master/galaxy-deeptools) + * [docker-galaxyp](https://github.com/bgruening/docker-galaxyp) + Users & Passwords -================ +----------------- The Galaxy Admin User has the username ``admin@galaxy.org`` and the password ``admin``. The PostgreSQL username is ``galaxy``, the password is ``galaxy`` and the database name is ``galaxy`` (I know I was really creative ;)). @@ -132,16 +218,34 @@ If you want to create new users, please make sure to use the ``/export/`` volume The proftpd server is configured to use the main galaxy PostgreSQL user to access the database and select the username and password. If you want to run the docker container in production, please do not forget to change the user credentials in /etc/proftp/proftpd.conf too. +The Galaxy Report Webapp is `htpasswd` protected with username and password st to `admin`. + + +Development +----------- + +This repository uses a git submodule to include [Ansible roles](https://github.com/galaxyproject/ansible-galaxy-extras) maintained by the Galaxy project. + +You can clone this repository and the Ansible submodule with: + + ```sh + git clone --recursive https://github.com/bgruening/docker-galaxy-stable.git + ``` + +Updating already existing submodules is possible with: + + ```sh + git submodule update --remote + ``` Requirements -============ +------------ - [docker](https://www.docker.io/gettingstarted/#h_installation) - History -======= +------- - 0.1: Initial release! - with Apache2, PostgreSQL and Tool Shed integration @@ -150,10 +254,15 @@ History - 0.3: Add Interactive Environments - IPython in docker in Galaxy in docker - advanged logging + - 0.4: + - base the image on toolshed/requirements with all required Galaxy dependencies + - use Ansible roles to build large parts of the image + - export the supervisord webinterface on port 9002 + - enable Galaxy reports webapp Support & Bug Reports -===================== +--------------------- -You can file an issue here https://github.com/bgruening/docker-galaxy-stable/issues or ask -us on the Galaxy development list http://lists.bx.psu.edu/listinfo/galaxy-dev +You can file an [github issue](https://github.com/bgruening/docker-galaxy-stable/issues) or ask +us on the [Galaxy development list](http://lists.bx.psu.edu/listinfo/galaxy-dev). diff --git a/chart.png b/chart.png new file mode 100644 index 000000000..12ff77fb1 Binary files /dev/null and b/chart.png differ diff --git a/chart.svg b/chart.svg new file mode 100644 index 000000000..64348907b --- /dev/null +++ b/chart.svg @@ -0,0 +1,721 @@ + + + + + + + + + + + + + + + + image/svg+xml + + + + + + + docker-ipython-notebook + + + + + + + + + + + + + + + + + + galaxy-stable + docker-galaxy-stable + + + + + + + + + + galaxy-proteomics + docker-recipes + + + + + + chemicaltoolbox + docker-recipes + + rna-workbench + galaxy-rna-workbench + + + docker-ipython-notebook + + + docker-ipython-notebook plus + docker-recipes + + + + + + + + + + + + + + + + + + + + + + + + + docker-ipython-notebook + + docker-galaxy-requirements + + + + + + + + + + diff --git a/galaxy/Dockerfile b/galaxy/Dockerfile index 5b60add6f..e75401ffc 100644 --- a/galaxy/Dockerfile +++ b/galaxy/Dockerfile @@ -2,7 +2,7 @@ # # VERSION Galaxy-central -FROM ubuntu:14.04 +FROM toolshed/requirements MAINTAINER Björn A. Grüning, bjoern.gruening@gmail.com @@ -11,32 +11,71 @@ MAINTAINER Björn A. Grüning, bjoern.gruening@gmail.com # * Enable the @natefoo magic # Web server infrastructure matching usegalaxy.org - supervisor, uwsgi, and nginx. -ENV DEBIAN_FRONTEND noninteractive - -RUN apt-get -qq update && apt-get install --no-install-recommends -y apt-transport-https software-properties-common && \ +RUN apt-get -qq update && apt-get install --no-install-recommends -y apt-transport-https software-properties-common && \ apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 && \ sh -c "echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list" && \ + apt-add-repository -y ppa:ansible/ansible && \ + apt-add-repository -y ppa:galaxyproject/nginx && \ apt-get update -qq && apt-get upgrade -y && \ - add-apt-repository ppa:galaxyproject/nginx && apt-get -qq update && \ + apt-get install --no-install-recommends -y mercurial python-psycopg2 postgresql-9.3 sudo samtools python-virtualenv wget \ + nginx-extras uwsgi uwsgi-plugin-python supervisor lxc-docker-1.4.1 slurm-llnl slurm-llnl-torque libswitch-perl \ + slurm-drmaa-dev proftpd proftpd-mod-pgsql libyaml-dev nodejs-legacy npm aufs-tools ansible \ + nano nmap lynx vim curl python-pip && \ apt-get purge -y software-properties-common && \ - apt-get install --no-install-recommends -y autoconf automake build-essential gfortran cmake wget \ - git-core libatlas-base-dev libblas-dev liblapack-dev mercurial subversion python-dev pkg-config \ - openjdk-7-jre-headless python-setuptools python-psycopg2 postgresql-9.3 sudo samtools python-virtualenv \ - nginx-extras uwsgi uwsgi-plugin-python supervisor lxc-docker slurm-llnl slurm-llnl-torque libswitch-perl \ - slurm-drmaa-dev zlib1g-dev proftpd proftpd-mod-pgsql libyaml-dev nodejs-legacy npm aufs-tools && \ apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* -RUN groupadd -r galaxy -g 450 && \ - useradd -u 451 -r -g galaxy -d /home/galaxy -c "Galaxy user" galaxy && \ - mkdir /export/ /home/galaxy/ && chown -R galaxy:galaxy /home/galaxy /export/ && \ - gpasswd -a galaxy docker +ENV GALAXY_RELEASE=release_15.03_docker \ +GALAXY_REPO=https://github.com/jmchilton/galaxy/ \ +GALAXY_ROOT=/galaxy-central \ +GALAXY_CONFIG_DIR=/etc/galaxy + +ENV GALAXY_CONFIG_FILE=$GALAXY_CONFIG_DIR/galaxy.ini \ +GALAXY_CONFIG_JOB_CONFIG_FILE=$GALAXY_CONFIG_DIR/job_conf.xml \ +GALAXY_CONFIG_JOB_METRICS_CONFIG_FILE=$GALAXY_CONFIG_DIR/job_metrics_conf.xml \ +GALAXY_VIRTUALENV=/home/galaxy/venv \ +GALAXY_USER=galaxy \ +GALAXY_UID=1450 \ +GALAXY_GID=1450 \ +GALAXY_HOME=/home/galaxy \ +GALAXY_DEFAULT_ADMIN_USER=admin@galaxy.org \ +GALAXY_DEFAULT_ADMIN_PASSWORD=admin \ +GALAXY_DEFAULT_ADMIN_KEY=admin \ +EXPORT_DIR=/export + +RUN groupadd -r $GALAXY_USER -g $GALAXY_GID && \ + useradd -u $GALAXY_UID -r -g $GALAXY_USER -d $GALAXY_HOME -c "Galaxy user" $GALAXY_USER && \ + mkdir $EXPORT_DIR $GALAXY_HOME && chown -R $GALAXY_USER:$GALAXY_USER $GALAXY_HOME $EXPORT_DIR && \ + gpasswd -a $GALAXY_USER docker + +# Download latest stable release of Galaxy. +RUN mkdir $GALAXY_ROOT && wget -q -O - $GALAXY_REPO/archive/$GALAXY_RELEASE.tar.gz | tar xzf - --strip-components=1 -C $GALAXY_ROOT && chown -R $GALAXY_USER:$GALAXY_USER $GALAXY_ROOT + +# TODO: ensure virtualenv as part of galaxy role +RUN su $GALAXY_USER -c "virtualenv $GALAXY_VIRTUALENV" + +# Setup Galaxy configuration files. +RUN mkdir -p $GALAXY_CONFIG_DIR $GALAXY_CONFIG_DIR/web && chown -R $GALAXY_USER:$GALAXY_USER $GALAXY_CONFIG_DIR + +RUN su $GALAXY_USER -c "cp $GALAXY_ROOT/config/galaxy.ini.sample $GALAXY_CONFIG_FILE" +ADD ./reports_wsgi.ini.sample $GALAXY_CONFIG_DIR/reports_wsgi.ini +ADD ./htpasswd /etc/nginx/htpasswd +ADD ./servers.ini $GALAXY_CONFIG_DIR/servers.ini +RUN cat $GALAXY_CONFIG_DIR/servers.ini >> $GALAXY_CONFIG_FILE && rm $GALAXY_CONFIG_DIR/servers.ini +ADD roles/ /tmp/ansible/roles +ADD provision.yml /tmp/ansible/provision.yml +RUN ansible-playbook /tmp/ansible/provision.yml \ + --extra-vars galaxy_user_name=$GALAXY_USER \ + --extra-vars galaxy_config_file=$GALAXY_CONFIG_FILE \ + --extra-vars galaxy_config_dir=$GALAXY_CONFIG_DIR \ + --extra-vars galaxy_job_conf_path=$GALAXY_CONFIG_JOB_CONFIG_FILE \ + --extra-vars galaxy_job_metrics_conf_path=$GALAXY_CONFIG_JOB_METRICS_CONFIG_FILE \ + --tags=galaxyextras -c local && \ + apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* -# Download and update Galaxy to the latest stable release -RUN hg clone --rev latest_2015.01.13 https://bitbucket.org/galaxy/galaxy-central/ && \ - cd /galaxy-central/ && \ - rm /galaxy-central/.hg/ -rf && chown galaxy:galaxy /galaxy-central/ -R +ADD roles/ /tmp/ansible/roles +ADD provision.yml /tmp/ansible/provision.yml -RUN mkdir /shed_tools && chown galaxy:galaxy /shed_tools +RUN mkdir /shed_tools && chown $GALAXY_USER:$GALAXY_USER /shed_tools # The following commands will be executed as User galaxy USER galaxy @@ -44,68 +83,55 @@ USER galaxy WORKDIR /galaxy-central # Configure Galaxy to use the Tool Shed -RUN cp /galaxy-central/config/galaxy.ini.sample /galaxy-central/config/galaxy.ini -RUN cp /galaxy-central/config/reports_wsgi.ini.sample /galaxy-central/config/reports_wsgi.ini -RUN mkdir /galaxy-central/tool_deps - -RUN virtualenv /home/galaxy/venv +RUN mkdir $GALAXY_ROOT/tool_deps # Fetching all Galaxy python dependencies -RUN . /home/galaxy/venv/bin/activate && python scripts/fetch_eggs.py && python scripts/fetch_eggs.py -e drmaa +RUN . $GALAXY_VIRTUALENV/bin/activate && python scripts/fetch_eggs.py -c $GALAXY_CONFIG_FILE && python scripts/fetch_eggs.py -e drmaa -c $GALAXY_CONFIG_FILE # Updating genome informations from UCSC #RUN export GALAXY=/galaxy-central && sh ./cron/updateucsc.sh.sample -ENV GALAXY_CONFIG_DATABASE_CONNECTION postgresql://galaxy:galaxy@localhost:5432/galaxy -ENV GALAXY_CONFIG_TOOL_DEPENDENCY_DIR ./tool_deps -ENV GALAXY_CONFIG_ADMIN_USERS admin@galaxy.org -ENV GALAXY_CONFIG_MASTER_API_KEY HSNiugRFvgT574F43jZ7N9F3 -ENV GALAXY_CONFIG_BRAND Galaxy Docker Build -ENV GALAXY_CONFIG_STATIC_ENABLED False -ENV GALAXY_CONFIG_JOB_WORKING_DIRECTORY /export/galaxy-central/database/job_working_directory -ENV GALAXY_CONFIG_FILE_PATH /export/galaxy-central/database/files -ENV GALAXY_CONFIG_NEW_FILE_PATH /export/galaxy-central/database/files -ENV GALAXY_CONFIG_TEMPLATE_CACHE_PATH /export/galaxy-central/database/compiled_templates -ENV GALAXY_CONFIG_CITATION_CACHE_DATA_DIR /export/galaxy-central/database/citations/data -ENV GALAXY_CONFIG_CLUSTER_FILES_DIRECTORY /export/galaxy-central/database/pbs -ENV GALAXY_CONFIG_FTP_UPLOAD_DIR /export/galaxy-central/database/ftp -ENV GALAXY_CONFIG_FTP_UPLOAD_SITE galaxy.docker.org -ENV GALAXY_CONFIG_USE_PBKDF2 False -ENV GALAXY_CONFIG_NGINX_X_ACCEL_REDIRECT_BASE /_x_accel_redirect -ENV GALAXY_CONFIG_NGINX_X_ARCHIVE_FILES_BASE /_x_accel_redirect -ENV GALAXY_CONFIG_NGINX_UPLOAD_STORE database/tmp/upload_store -ENV GALAXY_CONFIG_NGINX_UPLOAD_PATH /_upload -ENV GALAXY_CONFIG_DYNAMIC_PROXY_MANAGE False -ENV GALAXY_CONFIG_VISUALIZATION_PLUGINS_DIRECTORY config/plugins/visualizations -ENV GALAXY_CONFIG_TRUST_IPYTHON_NOTEBOOK_CONVERSION True +ENV GALAXY_CONFIG_DATABASE_CONNECTION=postgresql://galaxy:galaxy@localhost:5432/galaxy \ +GALAXY_CONFIG_TOOL_DEPENDENCY_DIR=./tool_deps \ +GALAXY_CONFIG_ADMIN_USERS=admin@galaxy.org \ +GALAXY_CONFIG_MASTER_API_KEY=HSNiugRFvgT574F43jZ7N9F3 \ +GALAXY_CONFIG_BRAND="Galaxy Docker Build" \ +GALAXY_CONFIG_STATIC_ENABLED=False \ +GALAXY_CONFIG_JOB_WORKING_DIRECTORY=/export/galaxy-central/database/job_working_directory \ +GALAXY_CONFIG_FILE_PATH=/export/galaxy-central/database/files \ +GALAXY_CONFIG_NEW_FILE_PATH=/export/galaxy-central/database/files \ +GALAXY_CONFIG_TEMPLATE_CACHE_PATH=/export/galaxy-central/database/compiled_templates \ +GALAXY_CONFIG_CITATION_CACHE_DATA_DIR=/export/galaxy-central/database/citations/data \ +GALAXY_CONFIG_CLUSTER_FILES_DIRECTORY=/export/galaxy-central/database/pbs \ +GALAXY_CONFIG_FTP_UPLOAD_DIR=/export/galaxy-central/database/ftp \ +GALAXY_CONFIG_FTP_UPLOAD_SITE=galaxy.docker.org \ +GALAXY_CONFIG_USE_PBKDF2=False \ +GALAXY_CONFIG_NGINX_X_ACCEL_REDIRECT_BASE=/_x_accel_redirect \ +GALAXY_CONFIG_NGINX_X_ARCHIVE_FILES_BASE=/_x_accel_redirect \ +GALAXY_CONFIG_NGINX_UPLOAD_STORE=/tmp/nginx_upload_store \ +GALAXY_CONFIG_NGINX_UPLOAD_PATH=/_upload \ +GALAXY_CONFIG_DYNAMIC_PROXY_MANAGE=False \ +GALAXY_CONFIG_VISUALIZATION_PLUGINS_DIRECTORY=config/plugins/visualizations \ +GALAXY_CONFIG_TRUST_IPYTHON_NOTEBOOK_CONVERSION=True \ +GALAXY_CONFIG_TOOLFORM_UPGRADE=True \ # Next line allow child docker container for viz to find this docker container. -ENV GALAXY_CONFIG_GALAXY_INFRASTRUCTURE_URL http://$HOST_IP/ -ENV GALAXY_CONFIG_SANITIZE_ALL_HTML False -ENV GALAXY_CONFIG_TOOLFORM_UPGRADE True -ENV GALAXY_CONFIG_OVERRIDE_DEBUG False +GALAXY_CONFIG_GALAXY_INFRASTRUCTURE_URL=http://$HOST_IP/ \ +GALAXY_CONFIG_SANITIZE_ALL_HTML=False \ +GALAXY_CONFIG_TOOLFORM_UPGRADE=True \ +GALAXY_CONFIG_WELCOME_URL=$GALAXY_CONFIG_DIR/web/welcome.html \ +GALAXY_CONFIG_OVERRIDE_DEBUG=False # Define the default postgresql database path -ENV PG_DATA_DIR_DEFAULT /var/lib/postgresql/9.3/main/ -ENV PG_DATA_DIR_HOST /export/postgresql/9.3/main/ - -ADD ./servers.ini /galaxy-central/config/servers.ini -RUN cat /galaxy-central/config/servers.ini >> /galaxy-central/config/galaxy.ini && rm /galaxy-central/config/servers.ini - -ADD ./job_conf.xml /galaxy-central/config/job_conf.xml - -# can I remove this? -#RUN mkdir /galaxy-central/database/job_working_directory/ /galaxy-central/database/files/ +ENV PG_DATA_DIR_DEFAULT=/var/lib/postgresql/9.3/main/ \ +PG_DATA_DIR_HOST=/export/postgresql/9.3/main/ -# Add optional watchdog dependency to Galaxy's environment. -# We need the psutil library to configure SLURM at startup. It's not needed by Galaxy. -RUN . /home/galaxy/venv/bin/activate && pip install watchdog ipython pygments jinja2 psutil # Install all required Node dependencies. This is required to get proxy support to work for Interactive Environments RUN cd /galaxy-central/lib/galaxy/web/proxy/js && npm install # Container Style -ADD GalaxyDocker.png /galaxy-central/static/welcome/GalaxyDocker.png -ADD welcome.html /galaxy-central/static/welcome.html +ADD GalaxyDocker.png $GALAXY_CONFIG_DIR/web/welcome_image.png +ADD welcome.html $GALAXY_CONFIG_DIR/web/welcome.html # Switch back to User root USER root @@ -114,61 +140,40 @@ USER root ENV LC_ALL en_US.UTF-8 RUN locale-gen en_US.UTF-8 && dpkg-reconfigure locales -ADD ./nginx.conf /etc/nginx/nginx.conf -ADD ./supervisor.conf /etc/supervisor/conf.d/galaxy.conf - -ADD ./proftpd.conf /etc/proftpd/proftpd.conf -RUN echo 'LoadModule mod_sql.c' >> /etc/proftpd/modules.conf && \ - echo 'LoadModule mod_sql_passwd.c' >> /etc/proftpd/modules.conf && \ - echo 'LoadModule mod_sql_postgres.c' >> /etc/proftpd/modules.conf - # Include all needed scripts from the host -ADD ./setup_postgresql.py /galaxy-central/setup_postgresql.py -ADD ./create_galaxy_user.py /galaxy-central/create_galaxy_user.py -ADD ./export_user_files.py /galaxy-central/export_user_files.py +ADD ./setup_postgresql.py /usr/local/bin/setup_postgresql.py +ADD ./export_user_files.py /usr/local/bin/export_user_files.py + +# workaround for a Docker AUFS bug: https://github.com/docker/docker/issues/783#issuecomment-56013588 +#RUN mkdir /etc/ssl/private-copy; mv /etc/ssl/private/* /etc/ssl/private-copy/; rm -r /etc/ssl/private; mv /etc/ssl/private-copy /etc/ssl/private; chmod -R 0700 /etc/ssl/private; chown -R postgres /etc/ssl/private # Configure PostgreSQL # 1. Remove all old configuration # 2. Create DB-user 'galaxy' with password 'galaxy' in database 'galaxy' # 3. Create Galaxy Admin User 'admin@galaxy.org' with password 'admin' and API key 'admin' -RUN service postgresql stop -RUN rm $PG_DATA_DIR_DEFAULT -rf -RUN python setup_postgresql.py --dbuser galaxy --dbpassword galaxy --db-name galaxy --dbpath $PG_DATA_DIR_DEFAULT -RUN service postgresql start && sh create_db.sh -RUN service postgresql start && sleep 5 && python create_galaxy_user.py --user admin@galaxy.org --password admin --key admin -RUN service postgresql start && sudo -u galaxy -i -- sh -c "cd /galaxy-central/ && sh run.sh --daemon && sleep 60 && sh run.sh --stop-daemon" -# nginx and uwsgi Will be controller by supervisor -RUN update-rc.d -f uwsgi remove && update-rc.d -f nginx remove && update-rc.d -f proftpd remove +RUN rm $PG_DATA_DIR_DEFAULT -rf && python /usr/local/bin/setup_postgresql.py --dbuser galaxy --dbpassword galaxy --db-name galaxy --dbpath $PG_DATA_DIR_DEFAULT +RUN service postgresql start && sh create_db.sh -c $GALAXY_CONFIG_FILE +ADD ./create_galaxy_user.py /usr/local/bin/create_galaxy_user.py +RUN service postgresql start && sleep 10 && python /usr/local/bin/create_galaxy_user.py --user $GALAXY_DEFAULT_ADMIN_USER --password $GALAXY_DEFAULT_ADMIN_PASSWORD -c $GALAXY_CONFIG_FILE --key $GALAXY_DEFAULT_ADMIN_KEY ADD ./startup.sh /usr/bin/startup # Script that enables easier downstream installation of tools (e.g. for different Galaxy Docker flavours) ADD install_repo_wrapper.sh /usr/bin/install-repository RUN chmod +x /usr/bin/install-repository /usr/bin/startup -# This needs to happen here and not above, otherwise the Galaxy start +# This needs to happen here and not above, otherwise the Galaxy start # (without running the startup.sh script) will crash because integrated_tool_panel.xml could not be found. ENV GALAXY_CONFIG_INTEGRATED_TOOL_PANEL_CONFIG /export/galaxy-central/integrated_tool_panel.xml -# Setup SLURM -# TODO use fixed key -RUN /usr/sbin/create-munge-key -RUN mkdir -p /var/run/munge && chown root:root /var/lib/munge/ /var/log/munge/ /var/run/munge /etc/munge/ /etc/munge/munge.key -ADD ./configure_slurm.py /usr/sbin/configure_slurm.py -# SLURM don't like world writeable StateSaveLocation folders -RUN mkdir /tmp/slurm && chown galaxy:galaxy /tmp/slurm - ADD ./cgroupfs_mount.sh /root/cgroupfs_mount.sh -# Expose port 80 (webserver), 21 (FTP server), 8800 (Proxy), 9001 (Galaxy report app) +# Expose port 80 (webserver), 21 (FTP server), 8800 (Proxy), 9002 (supvisord web app) EXPOSE :80 EXPOSE :21 EXPOSE :8800 -EXPOSE :9001 - -RUN apt-get -qq update && apt-get install --no-install-recommends -y nano nmap lynx vim curl +EXPOSE :9002 -RUN mkdir -p /galaxy-central/database/tmp/upload_store && chown galaxy:galaxy /galaxy-central/database/tmp/upload_store # We need to set $HOME for some Tool Shed tools (e.g Perl libs with $HOME/.cpan) ENV HOME /home/galaxy diff --git a/galaxy/configure_slurm.py b/galaxy/configure_slurm.py deleted file mode 100644 index 523090e4d..000000000 --- a/galaxy/configure_slurm.py +++ /dev/null @@ -1,96 +0,0 @@ -from socket import gethostname -from string import Template -from subprocess import call -from getpass import getuser -from os import environ -import psutil - -SLURM_CONFIG_TEMPLATE = ''' -# slurm.conf file generated by configurator.html. -# Put this file on all nodes of your cluster. -# See the slurm.conf man page for more information. -# -ControlMachine=$hostname -#ControlAddr= -#BackupController= -#BackupAddr= -# -AuthType=auth/munge -CacheGroups=0 -#CheckpointType=checkpoint/none -CryptoType=crypto/munge -MpiDefault=none -#PluginDir= -#PlugStackConfig= -#PrivateData=jobs -ProctrackType=proctrack/pgid -#Prolog= -#PrologSlurmctld= -#PropagatePrioProcess=0 -#PropagateResourceLimits= -#PropagateResourceLimitsExcept= -ReturnToService=1 -#SallocDefaultCommand= -SlurmctldPidFile=/var/run/slurmctld.pid -SlurmctldPort=6817 -SlurmdPidFile=/var/run/slurmd.pid -SlurmdPort=6818 -SlurmdSpoolDir=/tmp/slurmd -SlurmUser=$user -#SlurmdUser=root -#SrunEpilog= -#SrunProlog= -StateSaveLocation=/tmp/slurm -SwitchType=switch/none -#TaskEpilog= -TaskPlugin=task/none -#TaskPluginParam= -#TaskProlog= -InactiveLimit=0 -KillWait=30 -MinJobAge=300 -#OverTimeLimit=0 -SlurmctldTimeout=120 -SlurmdTimeout=300 -#UnkillableStepTimeout=60 -#VSizeFactor=0 -Waittime=0 -FastSchedule=1 -SchedulerType=sched/backfill -SchedulerPort=7321 -SelectType=select/cons_res -SelectTypeParameters=CR_Core_Memory -AccountingStorageType=accounting_storage/none -#AccountingStorageUser= -AccountingStoreJobComment=YES -ClusterName=cluster -#DebugFlags= -#JobCompHost= -#JobCompLoc= -#JobCompPass= -#JobCompPort= -JobCompType=jobcomp/none -#JobCompUser= -JobAcctGatherFrequency=30 -JobAcctGatherType=jobacct_gather/none -SlurmctldDebug=3 -#SlurmctldLogFile= -SlurmdDebug=3 -#SlurmdLogFile= -NodeName=$hostname CPUs=$cpus RealMemory=$memory State=UNKNOWN -PartitionName=debug Nodes=$hostname Default=YES MaxTime=INFINITE State=UP -''' - - -def main(): - template_params = { - "hostname": gethostname(), - "user": 'galaxy', - "cpus": environ.get("SLURM_CPUS", psutil.cpu_count()), - "memory": environ.get("SLURM_MEMORY", int(psutil.virtual_memory().total / (1024*1024))) - } - config_contents = Template(SLURM_CONFIG_TEMPLATE).substitute(template_params) - open("/etc/slurm-llnl/slurm.conf", "w").write(config_contents) - -if __name__ == "__main__": - main() diff --git a/galaxy/create_galaxy_user.py b/galaxy/create_galaxy_user.py index f62a50c65..aac27f451 100644 --- a/galaxy/create_galaxy_user.py +++ b/galaxy/create_galaxy_user.py @@ -1,3 +1,7 @@ +import sys +sys.path.insert(1,'/galaxy-central') +sys.path.insert(1,'/galaxy-central/lib') + from scripts.db_shell import * from galaxy.util.bunch import Bunch from galaxy.security import GalaxyRBACAgent diff --git a/galaxy/export_user_files.py b/galaxy/export_user_files.py index 4b79ce490..436b37141 100644 --- a/galaxy/export_user_files.py +++ b/galaxy/export_user_files.py @@ -24,8 +24,8 @@ def change_path( src ): os.makedirs(dest_dir) shutil.move( src, dest ) os.symlink( dest, src.rstrip('/') ) - os.chown( src, 451, 450 ) - subprocess.call('chown -R 451:450 %s' % dest, shell=True) + os.chown( src, int(os.environ['GALAXY_UID']), int(os.environ['GALAXY_GID']) ) + subprocess.call( 'chown -R %s:%s %s' % ( os.environ['GALAXY_UID'], os.environ['GALAXY_GID'], dest ), shell=True ) # if destination exists (e.g. continuing a previous session), remove source and symlink else: if os.path.isdir( src ): @@ -46,16 +46,43 @@ def change_path( src ): if os.path.exists( '/export/.distribution_config/' ): shutil.rmtree( '/export/.distribution_config/' ) shutil.copytree( '/galaxy-central/config/', '/export/.distribution_config/' ) + + + # Copy all files starting with "welcome" + # This enables a flexible start page design. + for filename in os.listdir('/export/'): + if filename.startswith('welcome'): + export_file = os.path.join( '/export/', filename) + image_file = os.path.join('/etc/galaxy/web/', filename) + shutil.copy(export_file, image_file) + if not os.path.exists( '/export/galaxy-central/' ): os.makedirs("/export/galaxy-central/") - os.chown("/export/galaxy-central/", 451, 450) + os.chown( "/export/galaxy-central/", int(os.environ['GALAXY_UID']), int(os.environ['GALAXY_GID']) ) + change_path('/galaxy-central/config/') - change_path('/galaxy-central/static/welcome.html') + + # copy image defaults to config/.docker_sample to base derivatives on, + # and if there is a realized version of these files in the export directory + # replace Galaxy's copy with these. Use symbolic link instead of copying so + # deployer can update and reload Galaxy and changes will be reflected. + for config in [ 'galaxy.ini', 'job_conf.xml' ]: + image_config = os.path.join('/etc/galaxy/', config) + export_config = os.path.join( '/export/galaxy-central/config', config ) + export_sample = export_config + ".docker_sample" + shutil.copy(image_config, export_sample) + if os.path.exists(export_config): + subprocess.call('ln -s -f %s %s' % (export_config, image_config), shell=True) + change_path('/galaxy-central/integrated_tool_panel.xml') change_path('/galaxy-central/display_applications/') change_path('/galaxy-central/tool_deps/') change_path('/galaxy-central/tool-data/') change_path('/shed_tools/') + + if os.path.exists('/export/reports_htpasswd'): + shutil.copy('/export/reports_htpasswd', '/etc/nginx/htpasswd') + try: change_path('/var/lib/docker/') except: @@ -70,7 +97,7 @@ def change_path( src ): # copy the postgresql data folder to the new location subprocess.call('cp -R %s/* %s' % (PG_DATA_DIR_DEFAULT, PG_DATA_DIR_HOST), shell=True) # copytree needs an non-existing dst dir, how annoying :( - #shutil.copytree(PG_DATA_DIR_DEFAULT, PG_DATA_DIR_HOST) + # shutil.copytree(PG_DATA_DIR_DEFAULT, PG_DATA_DIR_HOST) subprocess.call('chown -R postgres:postgres /export/postgresql/', shell=True) subprocess.call('chmod -R 0755 /export/', shell=True) subprocess.call('chmod -R 0700 %s' % PG_DATA_DIR_HOST, shell=True) @@ -80,4 +107,3 @@ def change_path( src ): new_data_directory = "'%s'" % PG_DATA_DIR_HOST cmd = 'sed -i "s|data_directory = .*|data_directory = %s|g" %s' % (new_data_directory, PG_CONF) subprocess.call(cmd, shell=True) - diff --git a/galaxy/htpasswd b/galaxy/htpasswd new file mode 100644 index 000000000..1663ffbc8 --- /dev/null +++ b/galaxy/htpasswd @@ -0,0 +1 @@ +admin:WiBKbsJTSQ8dc diff --git a/galaxy/install_repo_wrapper.sh b/galaxy/install_repo_wrapper.sh index 0868f87f3..4cff5286b 100644 --- a/galaxy/install_repo_wrapper.sh +++ b/galaxy/install_repo_wrapper.sh @@ -1,11 +1,35 @@ #!/bin/sh # start Galaxy -/etc/init.d/postgresql start -./run.sh --daemon -sleep 60 +service postgresql start +install_log='galaxy_install.log' +sudo -E -u galaxy ./run.sh --daemon --log-file=$install_log --pid-file=galaxy_install.pid + +galaxy_install_pid=`cat galaxy_install.pid` + +while : ; do + tail -n 2 $install_log | grep -E -q "Removing PID file galaxy_install.pid|Daemon is already running" + if [ $? -eq 0 ] ; then + echo "Galaxy could not be started." + echo "More information about this failure may be found in the following log snippet from galaxy_install.log:" + echo "========================================" + tail -n 60 $install_log + echo "========================================" + echo $1 + exit 1 + fi + tail -n 2 $install_log | grep -q "Starting server in PID $galaxy_install_pid" + if [ $? -eq 0 ] ; then + echo "Galaxy is running." + break + fi +done + +for repository in "$@"; do + echo "Processing:\t $repository" + python ./scripts/api/install_tool_shed_repositories.py --api admin -l http://localhost:8080 --tool-deps --repository-deps $repository +done -python ./scripts/api/install_tool_shed_repositories.py --api admin -l http://localhost:8080 --tool-deps --repository-deps $1 exit_code=$? if [ $exit_code != 0 ] ; then @@ -13,5 +37,6 @@ if [ $exit_code != 0 ] ; then fi # stop everything -./run.sh --stop-daemon +sudo -E -u galaxy ./run.sh --stop-daemon --log-file=$install_log --pid-file=galaxy_install.pid +rm $install_log service postgresql stop diff --git a/galaxy/job_conf.xml b/galaxy/job_conf.xml deleted file mode 100644 index 51773cebc..000000000 --- a/galaxy/job_conf.xml +++ /dev/null @@ -1,21 +0,0 @@ - - - - - /usr/lib/slurm-drmaa/lib/libdrmaa.so - - - - - - - - - - false - galaxy - - - - - \ No newline at end of file diff --git a/galaxy/nginx.conf b/galaxy/nginx.conf deleted file mode 100644 index 902899e5f..000000000 --- a/galaxy/nginx.conf +++ /dev/null @@ -1,110 +0,0 @@ -user galaxy; -worker_processes 1; -daemon off; - -events { - worker_connections 1024; -} - -http { - include mime.types; - default_type application/octet-stream; - - sendfile on; - - keepalive_timeout 65; - types_hash_max_size 2048; - - gzip on; - gzip_vary on; - gzip_proxied any; - gzip_comp_level 6; - gzip_buffers 16 8k; - gzip_http_version 1.1; - gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript; - - client_max_body_size 50g; - uwsgi_read_timeout 300; - - #server { - # listen 9000; - - # location / { - # uwsgi_pass 127.0.0.1:9001; - # include uwsgi_params; - # } - #} - - server { - listen 80 default_server; - server_name localhost; - - # pass to uWSGI by default - location / { - uwsgi_pass 127.0.0.1:4001; - include uwsgi_params; - } - - # serve static content - location /static { - alias /galaxy-central/static; - gzip on; - gzip_types text/plain text/xml text/javascript text/css application/x-javascript; - expires 24h; - } - location /static/style { - alias /galaxy-central/static/style/blue; - gzip on; - gzip_types text/plain text/xml text/javascript text/css application/x-javascript; - expires 24h; - } - location /static/scripts { - alias /galaxy-central/static/scripts/packed; - gzip on; - gzip_types text/plain text/javascript application/x-javascript; - expires 24h; - } - - # delegated downloads - location /_x_accel_redirect { - internal; - alias /; - } - - location ~ ^/plugins/visualizations/ipython/static/(?.*?)$ { - alias /galaxy-central/config/plugins/interactive_environments/ipython/static/$static_file; - } - location ~ ^/plugins/visualizations/(?.+?)/static/(?.*?)$ { - alias /galaxy-central/config/plugins/visualizations/$vis_name/static/$static_file; - } - - - # delegated uploads - location /_upload { - upload_store /galaxy-central/database/tmp/upload_store; - upload_store_access user:rw; - upload_pass_form_field ""; - upload_set_form_field "__${upload_field_name}__is_composite" "true"; - upload_set_form_field "__${upload_field_name}__keys" "name path"; - upload_set_form_field "${upload_field_name}_name" "$upload_file_name"; - upload_set_form_field "${upload_field_name}_path" "$upload_tmp_path"; - upload_pass_args on; - upload_pass /_upload_done; - } - location /_upload_done { - set $dst /api/tools; - if ($args ~ nginx_redir=([^&]+)) { - set $dst $1; - } - rewrite "" $dst; - } - - error_page 502 /502.html; - location = /502.html { - root /root/; - proxy_intercept_errors on; - } - - - } -} diff --git a/galaxy/proftpd.conf b/galaxy/proftpd.conf deleted file mode 100644 index 704b9f9a9..000000000 --- a/galaxy/proftpd.conf +++ /dev/null @@ -1,63 +0,0 @@ -# Includes DSO modules -Include /etc/proftpd/modules.conf - - -# Basics, some site-specific -ServerName "Public Galaxy FTP" -ServerType standalone -DefaultServer on -Port 21 -Umask 022 -#SyslogFacility DAEMON -#SyslogLevel debug -MaxInstances 30 -User nobody -Group nogroup - -# Passive port range for the firewall -PassivePorts 30000 40000 - -# Cause every FTP user to be "jailed" (chrooted) into their home directory -DefaultRoot ~ - -# Automatically create home directory if it doesn't exist -CreateHome on dirmode 700 - -# Allow users to overwrite their files -AllowOverwrite on - -# Allow users to resume interrupted uploads -AllowStoreRestart on - -# Bar use of SITE CHMOD - - DenyAll - - -# Bar use of RETR (download) since this is not a public file drop - - DenyAll - - -# Do not authenticate against real (system) users -AuthPAM off - -# Set up mod_sql_password - Galaxy passwords are stored as hex-encoded SHA1 -SQLPasswordEngine on -SQLPasswordEncoding hex - -# Set up mod_sql to authenticate against the Galaxy database -SQLEngine on -SQLBackend postgres -SQLConnectInfo galaxy@localhost galaxy galaxy -SQLAuthTypes SHA1 -SQLAuthenticate users - -# An empty directory in case chroot fails -SQLDefaultHomedir /var/opt/local/proftpd - -# Define a custom query for lookup that returns a passwd-like entry. UID and GID should match your Galaxy user. -SQLUserInfo custom:/LookupGalaxyUser -SQLNamedQuery LookupGalaxyUser SELECT "email,password,'galaxy','galaxy','/export/galaxy-central/database/ftp/%U','/bin/bash' FROM galaxy_user WHERE email='%U'" - - diff --git a/galaxy/provision.yml b/galaxy/provision.yml new file mode 100644 index 000000000..a25473c98 --- /dev/null +++ b/galaxy/provision.yml @@ -0,0 +1,5 @@ +- hosts: localhost + connection: local + roles: + - role: galaxyprojectdotorg.galaxyextras + tags: galaxyextras diff --git a/galaxy/reports_wsgi.ini.sample b/galaxy/reports_wsgi.ini.sample new file mode 100644 index 000000000..8bac450a0 --- /dev/null +++ b/galaxy/reports_wsgi.ini.sample @@ -0,0 +1,100 @@ +# ---- HTTP Server ---------------------------------------------------------- + +[server:main] + +use = egg:Paste#http +port = 9001 +host = 127.0.0.1 +use_threadpool = true +threadpool_workers = 10 + +# ---- Filters -------------------------------------------------------------- + +# Filters sit between Galaxy and the HTTP server. + +# These filters are disabled by default. They can be enabled with +# 'filter-with' in the [app:main] section below. + +# Define the proxy-prefix filter. +[filter:proxy-prefix] +use = egg:PasteDeploy#prefix +prefix = /reports + +# ---- Galaxy Webapps Report Interface ------------------------------------------------- + +[app:main] + +# -- Application and filtering + +# If running behind a proxy server and Galaxy is served from a subdirectory, +# enable the proxy-prefix filter and set the prefix in the +# [filter:proxy-prefix] section above. +filter-with = proxy-prefix + +# If proxy-prefix is enabled and you're running more than one Galaxy instance +# behind one hostname, you will want to set this to the same path as the prefix +# in the filter above. This value becomes the "path" attribute set in the +# cookie so the cookies from each instance will not clobber each other. +#cookie_path = None + +# -- Report + +# Specifies the factory for the universe WSGI application +paste.app_factory = galaxy.webapps.reports.buildapp:app_factory +log_level = DEBUG + +# Database connection +# Galaxy reports are intended for production Galaxy instances, so sqlite is not supported. +# You may use a SQLAlchemy connection string to specify an external database. +# database_connection = postgres:///galaxy_test?user=postgres&password=postgres + +# Where dataset files are saved +#file_path = database/files +# Temporary storage for additional datasets, this should be shared through the cluster +#new_file_path = database/tmp + +# Mako templates are compiled as needed and cached for reuse, this directory is +# used for the cache +#template_cache_path = database/compiled_templates/reports + +# Session support (beaker) +use_beaker_session = True +session_type = memory +session_data_dir = %(here)s/database/beaker_sessions +session_key = galaxysessions +session_secret = changethisinproduction + +# Configuration for debugging middleware +# debug = true +use_lint = false + +# NEVER enable this on a public site (even test or QA) +# use_interactive = true + +# path to sendmail +sendmail_path = /usr/sbin/sendmail + +# Address to join mailing list +mailing_join_addr = galaxy-user-join@bx.psu.edu + +# Write thread status periodically to 'heartbeat.log' (careful, uses disk space rapidly!) +## use_heartbeat = True + +# Profiling middleware (cProfile based) +## use_profile = True + +# Mail +# smtp_server = yourserver@yourfacility.edu +# error_email_to = your_bugs@bx.psu.edu + +# Use the new iframe / javascript based layout +use_new_layout = true + +# Serving static files (needed if running standalone) +# static_enabled = True +# static_cache_time = 360 +# static_dir = %(here)s/static/ +# static_images_dir = %(here)s/static/images +# static_favicon_dir = %(here)s/static/favicon.ico +# static_scripts_dir = %(here)s/static/scripts/ +# static_style_dir = %(here)s/static/june_2007_style/blue diff --git a/galaxy/roles/galaxyprojectdotorg.galaxyextras b/galaxy/roles/galaxyprojectdotorg.galaxyextras new file mode 160000 index 000000000..056e26452 --- /dev/null +++ b/galaxy/roles/galaxyprojectdotorg.galaxyextras @@ -0,0 +1 @@ +Subproject commit 056e26452f9733be46f3f2f6cb67cb36e9c81b09 diff --git a/galaxy/startup.sh b/galaxy/startup.sh index 7f63486fc..8dd148441 100644 --- a/galaxy/startup.sh +++ b/galaxy/startup.sh @@ -5,38 +5,70 @@ cd /galaxy-central/ # symlinks will point from the original location to the new path under /export/ # If /export/ is not given, nothing will happen in that step umount /var/lib/docker -python ./export_user_files.py $PG_DATA_DIR_DEFAULT +python /usr/local/bin/export_user_files.py $PG_DATA_DIR_DEFAULT + # Configure SLURM with runtime hostname. -/home/galaxy/venv/bin/python /usr/sbin/configure_slurm.py +python /usr/sbin/configure_slurm.py + +# $NONUSE can be set to include proftp, reports or nodejs +# if included we will _not_ start these services. +function start_supersisor { + /usr/bin/supervisord + sleep 5 + if [[ $NONUSE != *"proftp"* ]] + then + echo "Starting ProFTP" + supervisorctl start proftpd + fi + if [[ $NONUSE != *"reports"* ]] + then + echo "Starting Galaxy reports webapp" + supervisorctl start reports + fi + if [[ $NONUSE != *"nodejs"* ]] + then + echo "Starting nodejs" + supervisorctl start galaxy:galaxy_nodejs_proxy + fi +} + # Try to guess if we are running under --privileged mode if mount | grep "/proc/kcore"; then echo "Disable Galaxy Interactive Environments. Start with --privileged to enable IE's." export GALAXY_CONFIG_INTERACTIVE_ENVIRONMENT_PLUGINS_DIRECTORY="" - /usr/bin/supervisord - sleep 5 + start_supersisor else echo "Enable Galaxy Interactive Environments." export GALAXY_CONFIG_INTERACTIVE_ENVIRONMENT_PLUGINS_DIRECTORY="config/plugins/interactive_environments" if [ x$DOCKER_PARENT == "x" ]; then #build the docker in docker environment bash /root/cgroupfs_mount.sh - /usr/bin/supervisord - sleep 5 + start_supersisor supervisorctl start docker else #inheriting /var/run/docker.sock from parent, assume that you need to #run docker with sudo to validate echo "galaxy ALL = NOPASSWD : ALL" >> /etc/sudoers - /usr/bin/supervisord - sleep 5 + start_supersisor fi fi +# Enable verbose output if [ `echo ${GALAXY_LOGGING:-'no'} | tr [:upper:] [:lower:]` = "full" ] - then + then tail -f /var/log/supervisor/* /var/log/nginx/* /home/galaxy/*.log else tail -f /home/galaxy/*.log fi + +# Disable authentication of Galaxy reports +if [ "x$DISABLE_REPORTS_AUTH" != "x" ] + then + # disable authentification by deleting the htpasswd file + echo "Disable Galaxy reports authentification " + rm /etc/nginx/htpasswd +fi + + diff --git a/galaxy/supervisor.conf b/galaxy/supervisor.conf deleted file mode 100644 index b0b06d656..000000000 --- a/galaxy/supervisor.conf +++ /dev/null @@ -1,100 +0,0 @@ -[supervisord] -nodaemon=false - -[program:munge] -user=root -command=/usr/sbin/munged -F -redirect_stderr=true - -[program:slurmctld] -user=root -command=/usr/sbin/slurmctld -D -L /home/galaxy/slurmctld.log -redirect_stderr=true - -[program:slurmd] -user=root -command=/usr/sbin/slurmd -D -L /home/galaxy/slurmd.log -redirect_stderr=true - -[program:postgresql] -user = postgres -command = /usr/lib/postgresql/9.3/bin/postmaster -D "/export/postgresql/9.3/main" -process_name = %(program_name)s -stopsignal = INT -autostart = true -autorestart = true -redirect_stderr = true - -[program:proftpd] -command = /usr/sbin/proftpd -n -c /etc/proftpd/proftpd.conf -autorestart = true -autorestart = true - -[program:nginx] -command = /usr/sbin/nginx -directory = / -umask = 022 -autostart = true -autorestart = unexpected -startsecs = 5 -exitcodes = 0 -user = root - -[program:galaxy_uwsgi] -command = /usr/bin/uwsgi --plugin python --ini-paste /galaxy-central/config/galaxy.ini -directory = /galaxy-central -umask = 022 -autostart = true -autorestart = true -startsecs = 10 -user = galaxy -environment = PATH=/home/galaxy/venv:/home/galaxy/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,PYTHON_EGG_CACHE=/home/galaxy/.python-eggs,PYTHONPATH=/galaxy-central/eggs/PasteDeploy-1.5.0-py2.7.egg -numprocs = 1 -stopsignal = INT - -[program:handler] -command = /home/galaxy/venv/bin/python ./scripts/paster.py serve config/galaxy.ini --server-name=handler%(process_num)s --pid-file=/home/galaxy/handler%(process_num)s.pid --log-file=/home/galaxy/handler%(process_num)s.log -directory = /galaxy-central -process_name = handler%(process_num)s -numprocs = 2 -umask = 022 -autostart = true -autorestart = true -startsecs = 15 -user = galaxy -environment = PYTHON_EGG_CACHE=/home/galaxy/.python-eggs - -[program:reports] -command = /home/galaxy/venv/bin/python ./scripts/paster.py serve config/reports_wsgi.ini --server-name=main --pid-file=/home/galaxy/reports.pid --log-file=/home/galaxy/reports.log -directory = /galaxy-central -process_name = reports -umask = 022 -autostart = false -autorestart = true -startsecs = 5 -user = galaxy -environment = PYTHON_EGG_CACHE=/home/galaxy/.python-eggs - -[program:galaxy_nodejs_proxy] -directory = /galaxy-central -command = /galaxy-central/lib/galaxy/web/proxy/js/lib/main.js --sessions database/session_map.sqlite --ip 0.0.0.0 --port 8800 -autostart = true -autorestart = unexpected -user = galaxy -startsecs = 5 -redirect_stderr = true - - -[program:docker] -directory = / -command = /usr/bin/docker -d -autostart = false -autorestart = true -user = root -startsecs = 5 -redirect_stderr = true - -[group:galaxy] -programs = handler, galaxy_uwsgi, galaxy_nodejs_proxy - - diff --git a/galaxy/welcome.html b/galaxy/welcome.html index 3f0659973..767a4b056 100644 --- a/galaxy/welcome.html +++ b/galaxy/welcome.html @@ -3,18 +3,19 @@ - + +
Hello world! Your Galaxy Docker container is running...
- To customize this page edit static/welcome.html + To customize this page you can create a welcome.html page in your directory mounted to /export.

- +