From 29df3f6d00cdd5ddf22509513f8475217177f3e8 Mon Sep 17 00:00:00 2001 From: Brett Date: Wed, 8 Feb 2023 10:57:56 +0100 Subject: [PATCH 1/4] Update REAME.md --- README.md | 49 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 42 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 39df3edf..b324eca1 100644 --- a/README.md +++ b/README.md @@ -2,13 +2,16 @@ * [Overview](#overview) +* [docker compose vs docker-compose](#docker-compose-vs-docker-compose) * [Quick start](#quick-start) +* [The ckanext-envvars extension](#envvars) * [Development mode](#development-mode) * [Create an extension](#create-an-extension) * [CKAN images](#ckan-images) * [Extending the base images](#extending-the-base-images) * [Applying patches](#applying-patches) * [Debugging with pdb](#pdb) +* [Datastore and Datapusher](#Datastore-and-datapusher) * [NGINX](#nginx) * [Known Issues](#known-issues) @@ -29,6 +32,12 @@ The non-CKAN images are as follows: The site is configured using environment variables that you can set in the `.env` file. +## docker compose vs docker-compose + +All Docker Compose commands in this README will use the V2 version of Compose ie: `docker compose`. The older version (V1) +used the `docker-compose` command. Please see [Docker Compose](https://docs.docker.com/compose/compose-v2/) for +more information. + ## Quick start Copy the included `.env.example` and rename it to `.env` - modify it depending on your own needs. @@ -37,16 +46,35 @@ Using the default values on the `.env.example` file will get you a working CKAN To build the images: - docker-compose build + docker compose build To start the containers: - docker-compose up + docker compose up At the end of the container start sequence there should be 6 containers running ![Screenshot 2022-12-12 at 10 36 21 am](https://user-images.githubusercontent.com/54408245/207012236-f9571baa-4d99-4ffe-bd93-30b11c4829e0.png) +## envvars + +The ckanext-envvars extension is used in the CKAN Docker base repo to build the base images. +This extension checks for environmental variables conforming to an expected format and updates the corresponding CKAN config settings with its value. + +For the extension to correctly identify which env var keys map to the format used for the config object, env var keys should be formatted in the following way: + + All uppercase + Replace periods ('.') with two underscores ('__') + Keys must begin with 'CKAN' or 'CKANEXT' + +For example: + + CKAN__PLUGINS="envvars image_view text_view recline_view datastore datapusher" + CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000 + +These parameters can be added to the `.env` file + +For more information please see [ckanext-envvars](https://github.com/okfn/ckanext-envvars) ## Development mode @@ -54,11 +82,11 @@ To develop local extensions use the `docker-compose.dev.yml` file: To build the images: - docker-compose -f docker-compose.dev.yml build + docker compose -f docker-compose.dev.yml build To start the containers: - docker-compose -f docker-compose.dev.yml up + docker compose -f docker-compose.dev.yml up See [CKAN Images](#ckan-images) for more details of what happens when using development mode. @@ -67,7 +95,7 @@ See [CKAN Images](#ckan-images) for more details of what happens when using deve You can use the ckan [extension](https://docs.ckan.org/en/latest/extensions/tutorial.html#creating-a-new-extension) instructions to create a CKAN extension, only executing the command inside the CKAN container and setting the mounted `src/` folder as output: - docker-compose -f docker-compose.dev.yml exec ckan-dev /bin/bash -c "ckan generate extension --output-dir /srv/app/src_extensions" + docker compose -f docker-compose.dev.yml exec ckan-dev /bin/bash -c "ckan generate extension --output-dir /srv/app/src_extensions" The new extension files and directories will be created in the `src/` folder. You might need to change the owner of its folder to have the appropiate permissions. @@ -82,14 +110,14 @@ The Docker image config files used to build your CKAN project are located in the * `Dockerfile`: this is based on `ckan/ckan-base:`, a base image located in the DockerHub repository, that has CKAN installed along with all its dependencies, properly configured and running on [uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/) (production setup) * `Dockerfile.dev`: this is based on `ckan/ckan-base:-dev` also located located in the DockerHub repository, and extends `ckan/ckan-base:` to include: - * Any extension cloned on the `src` folder will be installed in the CKAN container when booting up Docker Compose (`docker-compose up`). This includes installing any requirements listed in a `requirements.txt` (or `pip-requirements.txt`) file and running `python setup.py develop`. + * Any extension cloned on the `src` folder will be installed in the CKAN container when booting up Docker Compose (`docker compose up`). This includes installing any requirements listed in a `requirements.txt` (or `pip-requirements.txt`) file and running `python setup.py develop`. * CKAN is started running this: `/usr/bin/ckan -c /srv/app/ckan.ini run -H 0.0.0.0`. * Make sure to add the local plugins to the `CKAN__PLUGINS` env var in the `.env` file. ### Extending the base images -You can modify the docker files to build your own customized image tailored to your project, installing any extensions and extra requirements needed. Here is where you would update to use a different CKAN base image ie: CKAN 2.9.7 +You can modify the docker files to build your own customized image tailored to your project, installing any extensions and extra requirements needed. For example here is where you would update to use a different CKAN base image ie: `ckan/ckan-base:` To perform extra initialization steps you can add scripts to your custom images and copy them to the `/docker-entrypoint.d` folder (The folder should be created for you when you build the image). Any `*.sh` and `*.py` file in that folder will be executed just after the main initialization script ([`prerun.py`](https://github.com/ckan/ckan-docker-base/blob/main/ckan-2.9/base/setup/prerun.py)) is executed and just before the web server and supervisor processes are started. @@ -124,6 +152,8 @@ RUN pip install -e git+https://github.com/frictionlessdata/ckanext-validation.gi COPY docker-entrypoint.d/* /docker-entrypoint.d/ ``` +NB: There are a number of extension examples commented out in the Dockerfile.dev file + ### Applying patches When building your project specific CKAN images (the ones defined in the `ckan/` folder), you can apply patches @@ -158,6 +188,11 @@ Debug with pdb (example) - Interact with `docker attach $(docker container ls -q command: `python -m pdb /usr/lib/ckan/venv/bin/ckan --config /srv/app/ckan.ini run --host 0.0.0.0 --passthrough-errors` +## Datastore and datapusher + +The Datastore database and user is created as part of the entrypoint scripts for the db container. There is also a Datapusher container +running the latest version of Datapusher. + ## NGINX * The base Docker Compose configuration uses an NGINX image as the front-end (ie: reverse proxy). It includes HTTPS running on port number 8443 and an HTTP port (81). A "self-signed" SSL certificate is generated beforehand and the server certificate and key files are included. The NGINX `server_name` directive and the `CN` field in the SSL certificate have been both set to 'localhost'. This should obviously not be used for production. From ec9ec421f63a110d4ccb845a47eedcde14d4e653 Mon Sep 17 00:00:00 2001 From: Brett Date: Wed, 8 Feb 2023 12:27:55 +0100 Subject: [PATCH 2/4] Updates to README, delete old directory --- .env.template => .env.example | 0 README.md | 57 +++-- .../OLD.README.txt | 66 ------ .../build-ckan-without-compose.sh | 71 ------- .../cleanup-docker-environment.sh | 16 -- .../environ | 34 --- .../old stuff to delete/2.7/Dockerfile | 97 --------- .../2.7/setup/ckan-uwsgi.ini | 15 -- .../old stuff to delete/2.7/setup/prerun.py | 197 ------------------ .../2.7/setup/start_ckan.sh | 47 ----- .../2.7/setup/supervisord.conf | 23 -- .../old stuff to delete/2.7/setup/uwsgi.conf | 2 - .../old stuff to delete/2.7/setup/wsgi.py | 9 - .../old stuff to delete/2.8/Dockerfile | 95 --------- .../2.8/setup/ckan-uwsgi.ini | 15 -- .../old stuff to delete/2.8/setup/prerun.py | 197 ------------------ .../2.8/setup/start_ckan.sh | 47 ----- .../2.8/setup/supervisord.conf | 23 -- .../old stuff to delete/2.8/setup/uwsgi.conf | 2 - .../old stuff to delete/2.8/setup/wsgi.py | 9 - .../old stuff to delete/Dockerfile.dev | 37 ---- .../ckan-dev/2.7/Dockerfile | 20 -- .../ckan-dev/2.8/Dockerfile | 20 -- .../ckan-dev/2.9/Dockerfile | 29 --- .../ckan-dev/setup/start_ckan_development.sh | 82 -------- .../old stuff to delete/dev-requirements.txt | 25 --- 26 files changed, 34 insertions(+), 1201 deletions(-) rename .env.template => .env.example (100%) delete mode 100644 auxilliary stuff (delete before going live)/OLD.README.txt delete mode 100755 auxilliary stuff (delete before going live)/build-ckan-without-compose.sh delete mode 100755 auxilliary stuff (delete before going live)/cleanup-docker-environment.sh delete mode 100644 auxilliary stuff (delete before going live)/environ delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/2.7/Dockerfile delete mode 100644 auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/ckan-uwsgi.ini delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/prerun.py delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/start_ckan.sh delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/supervisord.conf delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/uwsgi.conf delete mode 100644 auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/wsgi.py delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/2.8/Dockerfile delete mode 100644 auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/ckan-uwsgi.ini delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/prerun.py delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/start_ckan.sh delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/supervisord.conf delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/uwsgi.conf delete mode 100644 auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/wsgi.py delete mode 100644 auxilliary stuff (delete before going live)/old stuff to delete/Dockerfile.dev delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.7/Dockerfile delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.8/Dockerfile delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.9/Dockerfile delete mode 100755 auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/setup/start_ckan_development.sh delete mode 100644 auxilliary stuff (delete before going live)/old stuff to delete/dev-requirements.txt diff --git a/.env.template b/.env.example similarity index 100% rename from .env.template rename to .env.example diff --git a/README.md b/README.md index b324eca1..b35a1b5a 100644 --- a/README.md +++ b/README.md @@ -3,8 +3,8 @@ * [Overview](#overview) * [docker compose vs docker-compose](#docker-compose-vs-docker-compose) -* [Quick start](#quick-start) -* [The ckanext-envvars extension](#envvars) +* [Installing Docker](#installing-docker) +* [Install CKAN plus dependencies](#install-ckan-plus-dependencies) * [Development mode](#development-mode) * [Create an extension](#create-an-extension) * [CKAN images](#ckan-images) @@ -13,6 +13,7 @@ * [Debugging with pdb](#pdb) * [Datastore and Datapusher](#Datastore-and-datapusher) * [NGINX](#nginx) +* [The ckanext-envvars extension](#envvars) * [Known Issues](#known-issues) @@ -38,7 +39,14 @@ All Docker Compose commands in this README will use the V2 version of Compose ie used the `docker-compose` command. Please see [Docker Compose](https://docs.docker.com/compose/compose-v2/) for more information. -## Quick start +## Installing Docker + +Install Docker by following the following instructions: [Install Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/) + +To verify a successful Docker installation, run `docker run hello-world` and `docker version`. These commands should output +versions for client and server. + +## Install CKAN plus dependencies Copy the included `.env.example` and rename it to `.env` - modify it depending on your own needs. @@ -56,25 +64,8 @@ At the end of the container start sequence there should be 6 containers running ![Screenshot 2022-12-12 at 10 36 21 am](https://user-images.githubusercontent.com/54408245/207012236-f9571baa-4d99-4ffe-bd93-30b11c4829e0.png) -## envvars - -The ckanext-envvars extension is used in the CKAN Docker base repo to build the base images. -This extension checks for environmental variables conforming to an expected format and updates the corresponding CKAN config settings with its value. - -For the extension to correctly identify which env var keys map to the format used for the config object, env var keys should be formatted in the following way: - - All uppercase - Replace periods ('.') with two underscores ('__') - Keys must begin with 'CKAN' or 'CKANEXT' +After this step, CKAN should be running at `CKAN_SITE_URL`. -For example: - - CKAN__PLUGINS="envvars image_view text_view recline_view datastore datapusher" - CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000 - -These parameters can be added to the `.env` file - -For more information please see [ckanext-envvars](https://github.com/okfn/ckanext-envvars) ## Development mode @@ -195,12 +186,32 @@ running the latest version of Datapusher. ## NGINX -* The base Docker Compose configuration uses an NGINX image as the front-end (ie: reverse proxy). It includes HTTPS running on port number 8443 and an HTTP port (81). A "self-signed" SSL certificate is generated beforehand and the server certificate and key files are included. The NGINX `server_name` directive and the `CN` field in the SSL certificate have been both set to 'localhost'. This should obviously not be used for production. +The base Docker Compose configuration uses an NGINX image as the front-end (ie: reverse proxy). It includes HTTPS running on port number 8443 and an HTTP port (81). A "self-signed" SSL certificate is generated beforehand and the server certificate and key files are included. The NGINX `server_name` directive and the `CN` field in the SSL certificate have been both set to 'localhost'. This should obviously not be used for production. Creating the SSL cert and key files as follows: `openssl req -new -newkey rsa:4096 -days 365 -nodes -x509 -subj "/C=DE/ST=Berlin/L=Berlin/O=None/CN=localhost" -keyout ckan-local.key -out ckan-local.crt` The `ckan-local.*` files will then need to be moved into the nginx/setup/ directory +## envvars + +The ckanext-envvars extension is used in the CKAN Docker base repo to build the base images. +This extension checks for environmental variables conforming to an expected format and updates the corresponding CKAN config settings with its value. + +For the extension to correctly identify which env var keys map to the format used for the config object, env var keys should be formatted in the following way: + + All uppercase + Replace periods ('.') with two underscores ('__') + Keys must begin with 'CKAN' or 'CKANEXT' + +For example: + + * `CKAN__PLUGINS="envvars image_view text_view recline_view datastore datapusher"` + * `CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000` + +These parameters can be added to the `.env` file + +For more information please see [ckanext-envvars](https://github.com/okfn/ckanext-envvars) + ## Known Issues -* Running the tests: Running the tests for CKAN or an extension inside the container will delete your current database. We need to patch CKAN core in our image to work around that. +Running the tests: Running the tests for CKAN or an extension inside the container will delete your current database. We need to patch CKAN core in our image to work around that. diff --git a/auxilliary stuff (delete before going live)/OLD.README.txt b/auxilliary stuff (delete before going live)/OLD.README.txt deleted file mode 100644 index 70d1f75e..00000000 --- a/auxilliary stuff (delete before going live)/OLD.README.txt +++ /dev/null @@ -1,66 +0,0 @@ -### Please note that this re-purposed CKAN Docker repo is a WORK IN PROGRESS ### -### It should not be used to install CKAN via Docker until this page is updated ### - -------------------------------------- -(From Adria) -Be limited in scope, and act as a base that users can extend to their own needs -Be opinionated, and provide one way to do things -Be automatically tested -------------------------------------- - -should we have an nginx container included? maybe just instructions on how to include one - -ARG should be used for sensitive variables so as to keep the values of these variables -out of the build and hence will not show with an "inspect image" command - -Do we pre-build a CKAN image and use that (and extend) as the base image OR - just build it from scratch and while saving to storage what we need to - -To use local storage for the ckan.ini file and the CKAN src code, do the following: - -# docker-compose up -d --build -# mkdir local -# docker cp ckan:/srv/app/ckan.ini ./local/ckan.ini -# docker cp ckan:/srv/app/src ./local/src -Stop/Remove ckan container -Start ckan container as follows -Use a bind mount for the config file (ckan.ini) -# docker run -p 0.0.0.0:5000:5000 --net ckan-docker_default --hostname ckan --name ckan \ - --env-file=./environ --mount type=bind,source=$(pwd)/local/ckan.ini,target=/srv/app/ckan.ini \ - --mount type=bind,source=$(pwd)/local/src,target=/srv/app/src \ - -d ckan-docker_ckan - -Maybe include a script to replace docker-compose if required - -Dev Mode (OKFN) - -The differences between Docker dev abd Docker base is as the following: - -docker-compose.dev.yml - solr: explicitly puts in ports (8983:8983) - db: Doesn't pass in environment and arg values - ckan: has extra volume bind mount (./src:/srv/app/src_extensions) - -Dockerfile.dev - Takes the base image and - Adds a new directory (SRC_EXTENSIONS_DIR=/srv/app/src_extensions) - installs libffi-dev - installs dev-requirements.txt - runs different start script (start_ckan_development.sh) which installs any extension located in SRC_EXTENSIONS_DIR - runs a typical Dev install - pip install -r pip-requirements.txt - pip install -r requirements.txt - pip install -r dev-requirements.txt - python3 setup.py develop - ckan config-tool test.ini - ckan config-tool $CKAN_INI -s DEFAULT "debug = true" - ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS" - ckan config-tool $SRC_DIR/ckan/test-core.ini \ - "sqlalchemy.url = $TEST_CKAN_SQLALCHEMY_URL" \ - "ckan.datastore.write_url = $TEST_CKAN_DATASTORE_WRITE_URL" \ - "ckan.datastore.read_url = $TEST_CKAN_DATASTORE_READ_URL" \ - "solr_url = $TEST_CKAN_SOLR_URL" \ - "ckan.redis.url = $TEST_CKAN_REDIS_URL" - - All other steps are similar to the base Dockerfile - diff --git a/auxilliary stuff (delete before going live)/build-ckan-without-compose.sh b/auxilliary stuff (delete before going live)/build-ckan-without-compose.sh deleted file mode 100755 index 791bb287..00000000 --- a/auxilliary stuff (delete before going live)/build-ckan-without-compose.sh +++ /dev/null @@ -1,71 +0,0 @@ -#!/bin/bash - -# -# Create the network first -# -docker network create ckan-net -# - - -# Redis - Run only (image already built) -# -REDIS_CONTAINERNAME=redis -REDIS_HOSTNAME=redis - -docker run -d -it --net ckan-net --name ${REDIS_CONTAINERNAME} --hostname ${REDIS_HOSTNAME} redis:latest - -# -# Solr - Run/Build -# -SOLR_CONTAINERNAME=solr -SOLR_HOSTNAME=solr -SOLR_IMAGENAME=solr - -cd solr -docker image build -t ${SOLR_IMAGENAME} . -docker run -d -it --net ckan-net --name ${SOLR_CONTAINERNAME} --hostname ${SOLR_HOSTNAME} \ - -v solr_core:/opt/solr/server/solr ${SOLR_IMAGENAME} -cd - - -# -# PostgreSQL (DB) - Run/Build -# -DB_CONTAINERNAME=db -DB_HOSTNAME=db -DB_IMAGENAME=db - -cd postgresql -docker image build -t ${DB_IMAGENAME} . -docker run -d -it --net ckan-net --name ${DB_CONTAINERNAME} --hostname ${DB_HOSTNAME} --env-file=../environ \ - -v pg_data:/var/lib/postgresql/data ${DB_IMAGENAME} -cd - - -# -# CKAN - Run/Build -# -CKAN_CONTAINERNAME=ckan -CKAN_HOSTNAME=ckan -CKAN_IMAGENAME=ckan - -cd ckan -docker image build -t ${CKAN_IMAGENAME} . -docker run -d -it --net ckan-net --name ${CKAN_CONTAINERNAME} --hostname ${CKAN_HOSTNAME} --env-file=../environ \ - -v ckan_storage:/var/lib/ckan ${CKAN_IMAGENAME} -cd - - -# -# Nginx - Run/Build -# -NGINX_CONTAINERNAME=nginx -NGINX_HOSTNAME=nginx -NGINX_IMAGENAME=nginx - -cd nginx -docker image build -t ${NGINX_IMAGENAME} . -docker run -d -it --net ckan-net --name ${NGINX_CONTAINERNAME} --hostname ${NGINX_HOSTNAME} -p 0.0.0.0:80:80 ${NGINX_IMAGENAME} -cd - - -# -# PostgreSQL client -# -docker run -d -it --net ckan-net --name postgres-client --hostname postgres-client -e POSTGRES_PASSWORD=ckan -p 5432:5432 postgres diff --git a/auxilliary stuff (delete before going live)/cleanup-docker-environment.sh b/auxilliary stuff (delete before going live)/cleanup-docker-environment.sh deleted file mode 100755 index d1a8d1ff..00000000 --- a/auxilliary stuff (delete before going live)/cleanup-docker-environment.sh +++ /dev/null @@ -1,16 +0,0 @@ -#!/bin/bash - -# Stop all containers -docker kill $(docker ps -q) - -# Remove all stopped containers -docker rm -f $(docker ps -aq) - -# Remove all docker images -docker rmi $(docker images -q) - -# Remove all volumes -docker volume rm $(docker volume ls -q) - -# Remove ckan-docker_default network -docker network rm ckan-docker_default diff --git a/auxilliary stuff (delete before going live)/environ b/auxilliary stuff (delete before going live)/environ deleted file mode 100644 index df4e7f38..00000000 --- a/auxilliary stuff (delete before going live)/environ +++ /dev/null @@ -1,34 +0,0 @@ -REDIS_CONTAINERNAME=redis -REDIS_HOSTNAME=redis - -SOLR_CONTAINERNAME=solr -SOLR_HOSTNAME=solr -SOLR_IMAGENAME=solr - -DB_CONTAINERNAME=db -DB_HOSTNAME=db -DB_IMAGENAME=db -POSTGRES_PASSWORD=ckan -DS_RO_PASS=datastore -PGDATA=/var/lib/postgresql/data/db - -CKAN_CONTAINERNAME=ckan -CKAN_HOSTNAME=ckan -CKAN_IMAGENAME=ckan - -CKAN_SQLALCHEMY_URL=postgresql://ckan:ckan@db/ckan -CKAN_DATASTORE_WRITE_URL=postgresql://ckan:ckan@db/datastore -CKAN_DATASTORE_READ_URL=postgresql://datastore_ro:datastore@db/datastore -CKAN_SOLR_URL=http://solr:8983/solr/ckan -CKAN_REDIS_URL=redis://redis:6379/1 -CKAN_SITE_URL=http://localhost -CKAN_MAX_UPLOAD_SIZE_MB=32 -POSTGRES_PASSWORD=ckan -DS_RO_PASS=datastore -CKAN_PORT=5000 -CKAN_DATAPUSHER_URL=http://datapusher:8800 -CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000 - -NGINX_CONTAINERNAME=nginx -NGINX_HOSTNAME=nginx -NGINX_IMAGENAME=nginx diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/Dockerfile b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/Dockerfile deleted file mode 100755 index 0904522c..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/Dockerfile +++ /dev/null @@ -1,97 +0,0 @@ -FROM alpine:3.7 - -# Internals, you probably don't need to change these -ENV APP_DIR=/srv/app -ENV SRC_DIR=/srv/app/src -ENV CKAN_INI=${APP_DIR}/production.ini -ENV PIP_SRC=${SRC_DIR} -ENV CKAN_STORAGE_PATH=/var/lib/ckan -ENV GIT_URL=https://github.com/ckan/ckan.git -# CKAN version to build -ENV GIT_BRANCH=ckan-2.7.12 -# Customize these on the .env file if needed -ENV CKAN_SITE_URL=http://localhost:5000 -ENV CKAN__PLUGINS image_view text_view recline_view datastore datapusher envvars - -WORKDIR ${APP_DIR} - -# Install necessary packages to run CKAN -RUN apk add --no-cache tzdata \ - git \ - gettext \ - postgresql-client \ - python \ - apache2-utils \ - libxml2 \ - libxslt \ - musl-dev \ - uwsgi \ - uwsgi-http \ - uwsgi-corerouter \ - uwsgi-python \ - py2-gevent \ - uwsgi-gevent \ - libmagic \ - curl \ - sudo && \ - # Packages to build CKAN requirements and plugins - apk add --no-cache --virtual .build-deps \ - postgresql-dev \ - gcc \ - make \ - g++ \ - autoconf \ - automake \ - libtool \ - python-dev \ - libxml2-dev \ - libxslt-dev \ - linux-headers - # Create SRC_DIR - RUN mkdir -p ${SRC_DIR} && \ - # Install pip, supervisord and uwsgi - curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/pip/2.7/get-pip.py && \ - python ${SRC_DIR}/get-pip.py 'pip==20.3.3' && \ - pip install supervisor && \ - mkdir /etc/supervisord.d && \ - #pip wheel --wheel-dir=/wheels uwsgi gevent && \ - rm -rf ${SRC_DIR}/get-pip.py - -COPY 2.7/setup/supervisord.conf /etc - -# Install CKAN -RUN pip install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan && \ - cd ${SRC_DIR}/ckan && \ - cp who.ini ${APP_DIR} && \ - # Workaround to solve https://github.com/psycopg/psycopg2/issues/594 in Alpine 3.7 - sed -i -e "s/psycopg2==2.4.5/psycopg2==2.7.3.2/g" requirements.txt && \ - pip install --no-binary :all: -r requirements.txt && \ - # Install CKAN envvars to support loading config from environment variables - pip install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars && \ - # Create and update CKAN config - paster --plugin=ckan make-config ckan ${CKAN_INI} && \ - paster --plugin=ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}" && \ - paster --plugin=ckan config-tool ${CKAN_INI} "ckan.site_url = ${CKAN__SITE_URL}" - -# Create a local user and group to run the app -RUN addgroup -g 92 -S ckan && \ - adduser -u 92 -h /srv/app -H -D -S -G ckan ckan - -# Create local storage folder -RUN mkdir -p $CKAN_STORAGE_PATH && \ - chown -R ckan:ckan $CKAN_STORAGE_PATH - -# Copy setup files -COPY 2.7/setup ${APP_DIR} -RUN mv ${APP_DIR}/supervisord.conf /etc/supervisord.conf - -# Create entrypoint directory for children image scripts -ONBUILD RUN mkdir /docker-entrypoint.d - -RUN chown ckan -R /srv/app - -EXPOSE 5000 - -HEALTHCHECK --interval=10s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1 - -CMD ["/srv/app/start_ckan.sh"] diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/ckan-uwsgi.ini b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/ckan-uwsgi.ini deleted file mode 100644 index 2361f366..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/ckan-uwsgi.ini +++ /dev/null @@ -1,15 +0,0 @@ -[uwsgi] -http-socket = :5000 -uid = ckan -guid = ckan -plugins = python3 -wsgi-file = /srv/app/wsgi.py -virtualenv = /srv/app -module = wsgi:application -master = true -processes = 5 -pidfile = /tmp/%n.pid -harakiri = 50 -max-requests = 5000 -vacuum = true -callable = application diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/prerun.py b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/prerun.py deleted file mode 100755 index 7ebacfa5..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/prerun.py +++ /dev/null @@ -1,197 +0,0 @@ -import os -import sys -import subprocess -import psycopg2 -import urllib2 -import time -import re - -ckan_ini = os.environ.get('CKAN_INI', '/srv/app/production.ini') - -RETRY = 5 - -def update_plugins(): - - plugins = os.environ.get('CKAN__PLUGINS', '') - print('[prerun] Setting the following plugins in {}:'.format(ckan_ini)) - print(plugins) - cmd = ['paster', '--plugin=ckan', 'config-tool', - ckan_ini, 'ckan.plugins = {}'.format(plugins)] - subprocess.check_output(cmd, stderr=subprocess.STDOUT) - print '[prerun] Plugins set.' - - -def check_main_db_connection(retry=None): - - conn_str = os.environ.get('CKAN_SQLALCHEMY_URL') - if not conn_str: - print '[prerun] CKAN_SQLALCHEMY_URL not defined, not checking db' - return check_db_connection(conn_str, retry) - - -def check_datastore_db_connection(retry=None): - - conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL') - if not conn_str: - print '[prerun] CKAN_DATASTORE_WRITE_URL not defined, not checking db' - return check_db_connection(conn_str, retry) - - -def check_db_connection(conn_str, retry=None): - - if retry is None: - retry = RETRY - elif retry == 0: - print '[prerun] Giving up after 5 tries...' - sys.exit(1) - - try: - connection = psycopg2.connect(conn_str) - - except psycopg2.Error as e: - print str(e) - print '[prerun] Unable to connect to the database, waiting...' - time.sleep(10) - check_db_connection(conn_str, retry=retry - 1) - else: - connection.close() - - -def check_solr_connection(retry=None): - - if retry is None: - retry = RETRY - elif retry == 0: - print '[prerun] Giving up after 5 tries...' - sys.exit(1) - - url = os.environ.get('CKAN_SOLR_URL', '') - search_url = '{url}/select/?q=*&wt=json'.format(url=url) - - try: - connection = urllib2.urlopen(search_url) - except urllib2.URLError as e: - print str(e) - print '[prerun] Unable to connect to solr, waiting...' - time.sleep(10) - check_solr_connection(retry=retry - 1) - else: - eval(connection.read()) - - -def init_db(): - - db_command = ['paster', '--plugin=ckan', 'db', - 'init', '-c', ckan_ini] - print '[prerun] Initializing or upgrading db - start' - try: - subprocess.check_output(db_command, stderr=subprocess.STDOUT) - print '[prerun] Initializing or upgrading db - end' - except subprocess.CalledProcessError, e: - if 'OperationalError' in e.output: - print e.output - print '[prerun] Database not ready, waiting a bit before exit...' - time.sleep(5) - sys.exit(1) - else: - print e.output - raise e - - -def init_datastore_db(): - - conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL') - if not conn_str: - print '[prerun] Skipping datastore initialization' - return - - datastore_perms_command = ['paster', '--plugin=ckan', 'datastore', - 'set-permissions', '-c', ckan_ini] - - connection = psycopg2.connect(conn_str) - cursor = connection.cursor() - - print '[prerun] Initializing datastore db - start' - try: - datastore_perms = subprocess.Popen( - datastore_perms_command, - stdout=subprocess.PIPE) - - perms_sql = datastore_perms.stdout.read() - # Remove internal pg command as psycopg2 does not like it - perms_sql = re.sub('\\\\connect \"(.*)\"', '', perms_sql) - cursor.execute(perms_sql) - for notice in connection.notices: - print notice - - connection.commit() - - print '[prerun] Initializing datastore db - end' - print datastore_perms.stdout.read() - except psycopg2.Error as e: - print '[prerun] Could not initialize datastore' - print str(e) - - except subprocess.CalledProcessError, e: - if 'OperationalError' in e.output: - print e.output - print '[prerun] Database not ready, waiting a bit before exit...' - time.sleep(5) - sys.exit(1) - else: - print e.output - raise e - finally: - cursor.close() - connection.close() - - -def create_sysadmin(): - - name = os.environ.get('CKAN_SYSADMIN_NAME') - password = os.environ.get('CKAN_SYSADMIN_PASSWORD') - email = os.environ.get('CKAN_SYSADMIN_EMAIL') - - if name and password and email: - - # Check if user exists - command = ['paster', '--plugin=ckan', 'user', name, '-c', ckan_ini] - - out = subprocess.check_output(command) - if 'User:None' not in re.sub(r'\s', '', out): - print '[prerun] Sysadmin user exists, skipping creation' - return - - # Create user - command = ['paster', '--plugin=ckan', 'user', 'add', - name, - 'password=' + password, - 'email=' + email, - '-c', ckan_ini] - - subprocess.call(command) - print '[prerun] Created user {0}'.format(name) - - # Make it sysadmin - command = ['paster', '--plugin=ckan', 'sysadmin', 'add', - name, - '-c', ckan_ini] - - subprocess.call(command) - print '[prerun] Made user {0} a sysadmin'.format(name) - - -if __name__ == '__main__': - - maintenance = os.environ.get('MAINTENANCE_MODE', '').lower() == 'true' - - if maintenance: - print '[prerun] Maintenance mode, skipping setup...' - else: - check_main_db_connection() - init_db() - update_plugins() - check_datastore_db_connection() - init_datastore_db() - check_solr_connection() - create_sysadmin() \ No newline at end of file diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/start_ckan.sh b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/start_ckan.sh deleted file mode 100755 index d1c67261..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/start_ckan.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/bash - -# Run the prerun script to init CKAN and create the default admin user -sudo -u ckan -EH python prerun.py - -# Run any startup scripts provided by images extending this one -if [[ -d "/docker-entrypoint.d" ]] -then - for f in /docker-entrypoint.d/*; do - case "$f" in - *.sh) echo "$0: Running init file $f"; . "$f" ;; - *.py) echo "$0: Running init file $f"; python "$f"; echo ;; - *) echo "$0: Ignoring $f (not an sh or py file)" ;; - esac - echo - done -fi - -# Set the common uwsgi options -UWSGI_OPTS="--plugins http,python,gevent --socket /tmp/uwsgi.sock --uid 92 --gid 92 --http :5000 --master --enable-threads --paste config:/srv/app/production.ini --paste-logger --lazy-apps --gevent 2000 -p 2 -L" - -# Check whether http basic auth password protection is enabled and enable basicauth routing on uwsgi respecfully -if [ $? -eq 0 ] -then - if [ "$PASSWORD_PROTECT" = true ] - then - if [ "$HTPASSWD_USER" ] || [ "$HTPASSWD_PASSWORD" ] - then - # Generate htpasswd file for basicauth - htpasswd -d -b -c /srv/app/.htpasswd $HTPASSWD_USER $HTPASSWD_PASSWORD - # Start supervisord - supervisord --configuration /etc/supervisord.conf & - # Start uwsgi with basicauth - sudo -u ckan -EH uwsgi --ini /srv/app/uwsgi.conf --pcre-jit $UWSGI_OPTS - else - echo "Missing HTPASSWD_USER or HTPASSWD_PASSWORD environment variables. Exiting..." - exit 1 - fi - else - # Start supervisord - supervisord --configuration /etc/supervisord.conf & - # Start uwsgi - sudo -u ckan -EH uwsgi $UWSGI_OPTS - fi -else - echo "[prerun] failed...not starting CKAN." -fi diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/supervisord.conf b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/supervisord.conf deleted file mode 100755 index a3f66711..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/supervisord.conf +++ /dev/null @@ -1,23 +0,0 @@ -[unix_http_server] -file = /tmp/supervisor.sock -chmod = 0777 -chown = nobody:nogroup - -[supervisord] -logfile = /tmp/supervisord.log -logfile_maxbytes = 50MB -logfile_backups=10 -loglevel = info -pidfile = /tmp/supervisord.pid -nodaemon = true -umask = 022 -identifier = supervisor - -[supervisorctl] -serverurl = unix:///tmp/supervisor.sock - -[rpcinterface:supervisor] -supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface - -[include] -files = /etc/supervisord.d/*.conf diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/uwsgi.conf b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/uwsgi.conf deleted file mode 100755 index 6321d6df..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/uwsgi.conf +++ /dev/null @@ -1,2 +0,0 @@ -[uwsgi] -route = ^(?!/api).*$ basicauth:Restricted,/srv/app/.htpasswd diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/wsgi.py b/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/wsgi.py deleted file mode 100644 index b37d80e3..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.7/setup/wsgi.py +++ /dev/null @@ -1,9 +0,0 @@ -import os -from ckan.config.middleware import make_app -from ckan.cli import CKANConfigLoader -from logging.config import fileConfig as loggingFileConfig -config_filepath = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'ckan.ini') -abspath = os.path.join(os.path.dirname(os.path.abspath(__file__))) -loggingFileConfig(config_filepath) -config = CKANConfigLoader(config_filepath).get_config() -application = make_app(config) diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/Dockerfile b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/Dockerfile deleted file mode 100755 index 09d8806e..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/Dockerfile +++ /dev/null @@ -1,95 +0,0 @@ -FROM alpine:3.7 - -# Internals, you probably don't need to change these -ENV APP_DIR=/srv/app -ENV SRC_DIR=/srv/app/src -ENV CKAN_INI=${APP_DIR}/production.ini -ENV PIP_SRC=${SRC_DIR} -ENV CKAN_STORAGE_PATH=/var/lib/ckan -ENV GIT_URL=https://github.com/ckan/ckan.git -# CKAN version to build -ENV GIT_BRANCH=ckan-2.8.9 -# Customize these on the .env file if needed -ENV CKAN_SITE_URL=http://localhost:5000 -ENV CKAN__PLUGINS image_view text_view recline_view datastore datapusher envvars - -WORKDIR ${APP_DIR} - -# Install necessary packages to run CKAN -RUN apk add --no-cache tzdata \ - git \ - gettext \ - postgresql-client \ - python \ - apache2-utils \ - libxml2 \ - libxslt \ - musl-dev \ - uwsgi \ - uwsgi-http \ - uwsgi-corerouter \ - uwsgi-python \ - py2-gevent \ - uwsgi-gevent \ - libmagic \ - curl \ - sudo && \ - # Packages to build CKAN requirements and plugins - apk add --no-cache --virtual .build-deps \ - postgresql-dev \ - gcc \ - make \ - g++ \ - autoconf \ - automake \ - libtool \ - python-dev \ - libxml2-dev \ - libxslt-dev \ - linux-headers - # Create SRC_DIR - RUN mkdir -p ${SRC_DIR} && \ - # Install pip, supervisord and uwsgi - curl -o ${SRC_DIR}/get-pip.py https://bootstrap.pypa.io/pip/2.7/get-pip.py && \ - python ${SRC_DIR}/get-pip.py 'pip==20.3.3' && \ - pip install supervisor && \ - mkdir /etc/supervisord.d && \ - #pip wheel --wheel-dir=/wheels uwsgi gevent && \ - rm -rf ${SRC_DIR}/get-pip.py - -COPY 2.8/setup/supervisord.conf /etc - -# Install CKAN -RUN pip install -e git+${GIT_URL}@${GIT_BRANCH}#egg=ckan && \ - cd ${SRC_DIR}/ckan && \ - cp who.ini ${APP_DIR} && \ - pip install --no-binary :all: -r requirements.txt && \ - # Install CKAN envvars to support loading config from environment variables - pip install -e git+https://github.com/okfn/ckanext-envvars.git#egg=ckanext-envvars && \ - # Create and update CKAN config - paster --plugin=ckan make-config ckan ${CKAN_INI} && \ - paster --plugin=ckan config-tool ${CKAN_INI} "ckan.plugins = ${CKAN__PLUGINS}" && \ - paster --plugin=ckan config-tool ${CKAN_INI} "ckan.site_url = ${CKAN__SITE_URL}" - -# Create a local user and group to run the app -RUN addgroup -g 92 -S ckan && \ - adduser -u 92 -h /srv/app -H -D -S -G ckan ckan - -# Create local storage folder -RUN mkdir -p $CKAN_STORAGE_PATH && \ - chown -R ckan:ckan $CKAN_STORAGE_PATH - -# Copy setup files -COPY 2.8/setup ${APP_DIR} -RUN mv ${APP_DIR}/supervisord.conf /etc/supervisord.conf - -# Create entrypoint directory for children image scripts -ONBUILD RUN mkdir /docker-entrypoint.d - -RUN chown ckan -R /srv/app - -EXPOSE 5000 - -HEALTHCHECK --interval=10s --timeout=5s --retries=5 CMD curl --fail http://localhost:5000/api/3/action/status_show || exit 1 - -CMD ["/srv/app/start_ckan.sh"] diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/ckan-uwsgi.ini b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/ckan-uwsgi.ini deleted file mode 100644 index 2361f366..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/ckan-uwsgi.ini +++ /dev/null @@ -1,15 +0,0 @@ -[uwsgi] -http-socket = :5000 -uid = ckan -guid = ckan -plugins = python3 -wsgi-file = /srv/app/wsgi.py -virtualenv = /srv/app -module = wsgi:application -master = true -processes = 5 -pidfile = /tmp/%n.pid -harakiri = 50 -max-requests = 5000 -vacuum = true -callable = application diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/prerun.py b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/prerun.py deleted file mode 100755 index 7ebacfa5..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/prerun.py +++ /dev/null @@ -1,197 +0,0 @@ -import os -import sys -import subprocess -import psycopg2 -import urllib2 -import time -import re - -ckan_ini = os.environ.get('CKAN_INI', '/srv/app/production.ini') - -RETRY = 5 - -def update_plugins(): - - plugins = os.environ.get('CKAN__PLUGINS', '') - print('[prerun] Setting the following plugins in {}:'.format(ckan_ini)) - print(plugins) - cmd = ['paster', '--plugin=ckan', 'config-tool', - ckan_ini, 'ckan.plugins = {}'.format(plugins)] - subprocess.check_output(cmd, stderr=subprocess.STDOUT) - print '[prerun] Plugins set.' - - -def check_main_db_connection(retry=None): - - conn_str = os.environ.get('CKAN_SQLALCHEMY_URL') - if not conn_str: - print '[prerun] CKAN_SQLALCHEMY_URL not defined, not checking db' - return check_db_connection(conn_str, retry) - - -def check_datastore_db_connection(retry=None): - - conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL') - if not conn_str: - print '[prerun] CKAN_DATASTORE_WRITE_URL not defined, not checking db' - return check_db_connection(conn_str, retry) - - -def check_db_connection(conn_str, retry=None): - - if retry is None: - retry = RETRY - elif retry == 0: - print '[prerun] Giving up after 5 tries...' - sys.exit(1) - - try: - connection = psycopg2.connect(conn_str) - - except psycopg2.Error as e: - print str(e) - print '[prerun] Unable to connect to the database, waiting...' - time.sleep(10) - check_db_connection(conn_str, retry=retry - 1) - else: - connection.close() - - -def check_solr_connection(retry=None): - - if retry is None: - retry = RETRY - elif retry == 0: - print '[prerun] Giving up after 5 tries...' - sys.exit(1) - - url = os.environ.get('CKAN_SOLR_URL', '') - search_url = '{url}/select/?q=*&wt=json'.format(url=url) - - try: - connection = urllib2.urlopen(search_url) - except urllib2.URLError as e: - print str(e) - print '[prerun] Unable to connect to solr, waiting...' - time.sleep(10) - check_solr_connection(retry=retry - 1) - else: - eval(connection.read()) - - -def init_db(): - - db_command = ['paster', '--plugin=ckan', 'db', - 'init', '-c', ckan_ini] - print '[prerun] Initializing or upgrading db - start' - try: - subprocess.check_output(db_command, stderr=subprocess.STDOUT) - print '[prerun] Initializing or upgrading db - end' - except subprocess.CalledProcessError, e: - if 'OperationalError' in e.output: - print e.output - print '[prerun] Database not ready, waiting a bit before exit...' - time.sleep(5) - sys.exit(1) - else: - print e.output - raise e - - -def init_datastore_db(): - - conn_str = os.environ.get('CKAN_DATASTORE_WRITE_URL') - if not conn_str: - print '[prerun] Skipping datastore initialization' - return - - datastore_perms_command = ['paster', '--plugin=ckan', 'datastore', - 'set-permissions', '-c', ckan_ini] - - connection = psycopg2.connect(conn_str) - cursor = connection.cursor() - - print '[prerun] Initializing datastore db - start' - try: - datastore_perms = subprocess.Popen( - datastore_perms_command, - stdout=subprocess.PIPE) - - perms_sql = datastore_perms.stdout.read() - # Remove internal pg command as psycopg2 does not like it - perms_sql = re.sub('\\\\connect \"(.*)\"', '', perms_sql) - cursor.execute(perms_sql) - for notice in connection.notices: - print notice - - connection.commit() - - print '[prerun] Initializing datastore db - end' - print datastore_perms.stdout.read() - except psycopg2.Error as e: - print '[prerun] Could not initialize datastore' - print str(e) - - except subprocess.CalledProcessError, e: - if 'OperationalError' in e.output: - print e.output - print '[prerun] Database not ready, waiting a bit before exit...' - time.sleep(5) - sys.exit(1) - else: - print e.output - raise e - finally: - cursor.close() - connection.close() - - -def create_sysadmin(): - - name = os.environ.get('CKAN_SYSADMIN_NAME') - password = os.environ.get('CKAN_SYSADMIN_PASSWORD') - email = os.environ.get('CKAN_SYSADMIN_EMAIL') - - if name and password and email: - - # Check if user exists - command = ['paster', '--plugin=ckan', 'user', name, '-c', ckan_ini] - - out = subprocess.check_output(command) - if 'User:None' not in re.sub(r'\s', '', out): - print '[prerun] Sysadmin user exists, skipping creation' - return - - # Create user - command = ['paster', '--plugin=ckan', 'user', 'add', - name, - 'password=' + password, - 'email=' + email, - '-c', ckan_ini] - - subprocess.call(command) - print '[prerun] Created user {0}'.format(name) - - # Make it sysadmin - command = ['paster', '--plugin=ckan', 'sysadmin', 'add', - name, - '-c', ckan_ini] - - subprocess.call(command) - print '[prerun] Made user {0} a sysadmin'.format(name) - - -if __name__ == '__main__': - - maintenance = os.environ.get('MAINTENANCE_MODE', '').lower() == 'true' - - if maintenance: - print '[prerun] Maintenance mode, skipping setup...' - else: - check_main_db_connection() - init_db() - update_plugins() - check_datastore_db_connection() - init_datastore_db() - check_solr_connection() - create_sysadmin() \ No newline at end of file diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/start_ckan.sh b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/start_ckan.sh deleted file mode 100755 index d1c67261..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/start_ckan.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/bin/bash - -# Run the prerun script to init CKAN and create the default admin user -sudo -u ckan -EH python prerun.py - -# Run any startup scripts provided by images extending this one -if [[ -d "/docker-entrypoint.d" ]] -then - for f in /docker-entrypoint.d/*; do - case "$f" in - *.sh) echo "$0: Running init file $f"; . "$f" ;; - *.py) echo "$0: Running init file $f"; python "$f"; echo ;; - *) echo "$0: Ignoring $f (not an sh or py file)" ;; - esac - echo - done -fi - -# Set the common uwsgi options -UWSGI_OPTS="--plugins http,python,gevent --socket /tmp/uwsgi.sock --uid 92 --gid 92 --http :5000 --master --enable-threads --paste config:/srv/app/production.ini --paste-logger --lazy-apps --gevent 2000 -p 2 -L" - -# Check whether http basic auth password protection is enabled and enable basicauth routing on uwsgi respecfully -if [ $? -eq 0 ] -then - if [ "$PASSWORD_PROTECT" = true ] - then - if [ "$HTPASSWD_USER" ] || [ "$HTPASSWD_PASSWORD" ] - then - # Generate htpasswd file for basicauth - htpasswd -d -b -c /srv/app/.htpasswd $HTPASSWD_USER $HTPASSWD_PASSWORD - # Start supervisord - supervisord --configuration /etc/supervisord.conf & - # Start uwsgi with basicauth - sudo -u ckan -EH uwsgi --ini /srv/app/uwsgi.conf --pcre-jit $UWSGI_OPTS - else - echo "Missing HTPASSWD_USER or HTPASSWD_PASSWORD environment variables. Exiting..." - exit 1 - fi - else - # Start supervisord - supervisord --configuration /etc/supervisord.conf & - # Start uwsgi - sudo -u ckan -EH uwsgi $UWSGI_OPTS - fi -else - echo "[prerun] failed...not starting CKAN." -fi diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/supervisord.conf b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/supervisord.conf deleted file mode 100755 index a3f66711..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/supervisord.conf +++ /dev/null @@ -1,23 +0,0 @@ -[unix_http_server] -file = /tmp/supervisor.sock -chmod = 0777 -chown = nobody:nogroup - -[supervisord] -logfile = /tmp/supervisord.log -logfile_maxbytes = 50MB -logfile_backups=10 -loglevel = info -pidfile = /tmp/supervisord.pid -nodaemon = true -umask = 022 -identifier = supervisor - -[supervisorctl] -serverurl = unix:///tmp/supervisor.sock - -[rpcinterface:supervisor] -supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface - -[include] -files = /etc/supervisord.d/*.conf diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/uwsgi.conf b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/uwsgi.conf deleted file mode 100755 index 6321d6df..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/uwsgi.conf +++ /dev/null @@ -1,2 +0,0 @@ -[uwsgi] -route = ^(?!/api).*$ basicauth:Restricted,/srv/app/.htpasswd diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/wsgi.py b/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/wsgi.py deleted file mode 100644 index b37d80e3..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/2.8/setup/wsgi.py +++ /dev/null @@ -1,9 +0,0 @@ -import os -from ckan.config.middleware import make_app -from ckan.cli import CKANConfigLoader -from logging.config import fileConfig as loggingFileConfig -config_filepath = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'ckan.ini') -abspath = os.path.join(os.path.dirname(os.path.abspath(__file__))) -loggingFileConfig(config_filepath) -config = CKANConfigLoader(config_filepath).get_config() -application = make_app(config) diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/Dockerfile.dev b/auxilliary stuff (delete before going live)/old stuff to delete/Dockerfile.dev deleted file mode 100644 index 6086f6ea..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/Dockerfile.dev +++ /dev/null @@ -1,37 +0,0 @@ -FROM ckan/ckan-dev:testing-only.2.9 - -LABEL maintainer="brett@kowh.ai" - -# Set up environment variables -ENV APP_DIR=/srv/app -ENV TZ=UTC -RUN echo ${TZ} > /etc/timezone - -# Make sure both files are not exactly the same -RUN if ! [ /usr/share/zoneinfo/${TZ} -ef /etc/localtime ]; then \ - cp /usr/share/zoneinfo/${TZ} /etc/localtime ;\ - fi ; - -# Install any extensions needed by your CKAN instance -# (Make sure to add the plugins to CKAN__PLUGINS in the .env file) -# For instance: -#RUN pip install -e git+https://github.com/ckan/ckanext-pages.git#egg=ckanext-pages && \ -# pip install -e git+https://github.com/ckan/ckanext-dcat.git@v0.0.6#egg=ckanext-dcat && \ -# pip install -r https://raw.githubusercontent.com/ckan/ckanext-dcat/v0.0.6/requirements.txt - -# Clone the extension(s) your are writing for your own project in the `src` folder -# to get them mounted in this image at runtime - -# Apply any patches needed to CKAN core or any of the built extensions (not the -# runtime mounted ones) -# See https://github.com/okfn/docker-ckan#applying-patches - -COPY patches ${APP_DIR}/patches - -RUN for d in ${APP_DIR}/patches/*; do \ - if [ -d $d ]; then \ - for f in `ls $d/*.patch | sort -g`; do \ - cd $SRC_DIR/`basename "$d"` && echo "$0: Applying patch $f to $SRC_DIR/`basename $d`"; patch -p1 < "$f" ; \ - done ; \ - fi ; \ - done diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.7/Dockerfile b/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.7/Dockerfile deleted file mode 100755 index a421a599..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.7/Dockerfile +++ /dev/null @@ -1,20 +0,0 @@ -FROM ckan/ckan-base:testing-only.2.7 - -LABEL maintainer="brett@kowh.ai" - -ENV APP_DIR=/srv/app -ENV SRC_EXTENSIONS_DIR=/srv/app/src_extensions - -# Install packages needed by the dev requirements -RUN apk add --no-cache libffi-dev - -# Install CKAN dev requirements -RUN pip install --no-binary :all: -r https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/dev-requirements.txt - -# Create folder for local extensions sources -RUN mkdir $SRC_EXTENSIONS_DIR - -COPY setup/start_ckan_development.sh ${APP_DIR} - - -CMD ["/srv/app/start_ckan_development.sh"] diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.8/Dockerfile b/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.8/Dockerfile deleted file mode 100755 index 6142a8da..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.8/Dockerfile +++ /dev/null @@ -1,20 +0,0 @@ -FROM ckan/ckan-base:testing-only.2.8 - -LABEL maintainer="brett@kowh.ai" - -ENV APP_DIR=/srv/app -ENV SRC_EXTENSIONS_DIR=/srv/app/src_extensions - -# Install packages needed by the dev requirements -RUN apk add --no-cache libffi-dev - -# Install CKAN dev requirements -RUN pip install --no-binary :all: -r https://raw.githubusercontent.com/ckan/ckan/${GIT_BRANCH}/dev-requirements.txt - -# Create folder for local extensions sources -RUN mkdir $SRC_EXTENSIONS_DIR - -COPY setup/start_ckan_development.sh ${APP_DIR} - - -CMD ["/srv/app/start_ckan_development.sh"] diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.9/Dockerfile b/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.9/Dockerfile deleted file mode 100755 index fc170ee9..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/2.9/Dockerfile +++ /dev/null @@ -1,29 +0,0 @@ -FROM ckan/ckan-base:testing-only.2.9 - -LABEL maintainer="brett@kowh.ai" - -ENV APP_DIR=/srv/app -ENV SRC_EXTENSIONS_DIR=/srv/app/src_extensions - -# Install packages needed by the dev requirements -RUN apk add --no-cache libffi-dev - -# Set up Python3 virtual environment -RUN cd ${APP_DIR} && \ - source ${APP_DIR}/bin/activate - -# Virtual environment binaries/scripts to be used first -ENV PATH=${APP_DIR}/bin:${PATH} - -# Install CKAN dev requirements -# Will need to change this eventually - when CKAN 2.9 is out -# wget https://raw.githubusercontent.com/ckan/ckan/master/dev-requirements.txt -# RUN pip3 install --no-binary :all: -r https://raw.githubusercontent.com/ckan/ckan/master/dev-requirements.txt -RUN pip3 install -r https://raw.githubusercontent.com/ckan/ckan/master/dev-requirements.txt - -# Create folder for local extensions sources -RUN mkdir $SRC_EXTENSIONS_DIR - -COPY setup/start_ckan_development.sh ${APP_DIR} - -CMD ["/srv/app/start_ckan_development.sh"] diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/setup/start_ckan_development.sh b/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/setup/start_ckan_development.sh deleted file mode 100755 index 331266c5..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/ckan-dev/setup/start_ckan_development.sh +++ /dev/null @@ -1,82 +0,0 @@ -#!/bin/bash - -# Install any local extensions in the src_extensions volume -echo "Looking for local extensions to install..." -echo "Extension dir contents:" -ls -la $SRC_EXTENSIONS_DIR -for i in $SRC_EXTENSIONS_DIR/* -do - if [ -d $i ]; - then - - if [ -f $i/pip-requirements.txt ]; - then - pip install -r $i/pip-requirements.txt - echo "Found requirements file in $i" - fi - if [ -f $i/requirements.txt ]; - then - pip install -r $i/requirements.txt - echo "Found requirements file in $i" - fi - if [ -f $i/dev-requirements.txt ]; - then - pip install -r $i/dev-requirements.txt - echo "Found dev-requirements file in $i" - fi - if [ -f $i/setup.py ]; - then - cd $i - python $i/setup.py develop - echo "Found setup.py file in $i" - cd $APP_DIR - fi - - # Point `use` in test.ini to location of `test-core.ini` - if [ -f $i/test.ini ]; - then - echo "Updating \`test.ini\` reference to \`test-core.ini\` for plugin $i" - paster --plugin=ckan config-tool $i/test.ini "use = config:../../src/ckan/test-core.ini" - fi - fi -done - -# Set debug to true -echo "Enabling debug mode" -ckan config-tool $CKAN_INI -s DEFAULT "debug = true" - -# Update the plugins setting in the ini file with the values defined in the env var -echo "Loading the following plugins: $CKAN__PLUGINS" -ckan config-tool $CKAN_INI "ckan.plugins = $CKAN__PLUGINS" - -# Update test-core.ini DB, SOLR & Redis settings -echo "Loading test settings into test-core.ini" -ckan config-tool $SRC_DIR/ckan/test-core.ini \ - "sqlalchemy.url = $TEST_CKAN_SQLALCHEMY_URL" \ - "ckan.datstore.write_url = $TEST_CKAN_DATASTORE_WRITE_URL" \ - "ckan.datstore.read_url = $TEST_CKAN_DATASTORE_READ_URL" \ - "solr_url = $TEST_CKAN_SOLR_URL" \ - "ckan.redis_url = $TEST_CKAN_REDIS_URL" - -# Run the prerun script to init CKAN and create the default admin user -sudo -u ckan -EH python prerun.py - -# Run any startup scripts provided by images extending this one -if [[ -d "/docker-entrypoint.d" ]] -then - for f in /docker-entrypoint.d/*; do - case "$f" in - *.sh) echo "$0: Running init file $f"; . "$f" ;; - *.py) echo "$0: Running init file $f"; python "$f"; echo ;; - *) echo "$0: Ignoring $f (not an sh or py file)" ;; - esac - echo - done -fi - -# Start supervisord -supervisord --configuration /etc/supervisord.conf & - -# Start the development server with automatic reload -# Check the --reloader options sudo -u ckan -EH ckan -c $CKAN_INI run --reloader -sudo -u ckan -EH ckan -c $CKAN_INI run diff --git a/auxilliary stuff (delete before going live)/old stuff to delete/dev-requirements.txt b/auxilliary stuff (delete before going live)/old stuff to delete/dev-requirements.txt deleted file mode 100644 index 902fe5fb..00000000 --- a/auxilliary stuff (delete before going live)/old stuff to delete/dev-requirements.txt +++ /dev/null @@ -1,25 +0,0 @@ -# These are packages that are required by ckan developers - for running ckan in debug mode, running ckan tests, building the docs and to pip-compile the requirements.in file. - -beautifulsoup4==4.5.1 -coveralls #Let Unpinned - Requires latest coveralls -docutils==0.12 -factory-boy==2.1.1 -Flask-DebugToolbar==0.10.1 -freezegun==0.3.15 -responses==0.10.6 -mock==2.0.0 -pycodestyle==2.5.0 -pip-tools==2.0.2 -pyfakefs==3.2 -Sphinx==1.8.5 -sphinx-rtd-theme==0.3.1 -cookiecutter==1.6.0 - -# nose==1.3.0 # already in requirements.txt -pytest==4.6.5 -pytest-split-tests==1.0.9 -pytest-cov==2.7.1 -pytest-freezegun==0.4.1 -pytest-rerunfailures==8.0 - -towncrier==19.2.0 From b84187912911d526c65e2c0dc6d1881b33a20398 Mon Sep 17 00:00:00 2001 From: Brett Date: Wed, 8 Feb 2023 14:05:16 +0100 Subject: [PATCH 3/4] Update README (moving sections) --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index b35a1b5a..7b13cee9 100644 --- a/README.md +++ b/README.md @@ -2,8 +2,8 @@ * [Overview](#overview) -* [docker compose vs docker-compose](#docker-compose-vs-docker-compose) * [Installing Docker](#installing-docker) +* [docker compose vs docker-compose](#docker-compose-vs-docker-compose) * [Install CKAN plus dependencies](#install-ckan-plus-dependencies) * [Development mode](#development-mode) * [Create an extension](#create-an-extension) @@ -33,12 +33,6 @@ The non-CKAN images are as follows: The site is configured using environment variables that you can set in the `.env` file. -## docker compose vs docker-compose - -All Docker Compose commands in this README will use the V2 version of Compose ie: `docker compose`. The older version (V1) -used the `docker-compose` command. Please see [Docker Compose](https://docs.docker.com/compose/compose-v2/) for -more information. - ## Installing Docker Install Docker by following the following instructions: [Install Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/) @@ -46,6 +40,12 @@ Install Docker by following the following instructions: [Install Docker Engine o To verify a successful Docker installation, run `docker run hello-world` and `docker version`. These commands should output versions for client and server. +## docker compose vs docker-compose + +All Docker Compose commands in this README will use the V2 version of Compose ie: `docker compose`. The older version (V1) +used the `docker-compose` command. Please see [Docker Compose](https://docs.docker.com/compose/compose-v2/) for +more information. + ## Install CKAN plus dependencies Copy the included `.env.example` and rename it to `.env` - modify it depending on your own needs. From 0ba0aa7e1a469c41cfc89ea465f6cd1a52db9c28 Mon Sep 17 00:00:00 2001 From: Brett Date: Wed, 8 Feb 2023 14:20:56 +0100 Subject: [PATCH 4/4] more updates to README --- README.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 7b13cee9..9f59714d 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,6 @@ * [Datastore and Datapusher](#Datastore-and-datapusher) * [NGINX](#nginx) * [The ckanext-envvars extension](#envvars) -* [Known Issues](#known-issues) ## Overview @@ -48,7 +47,7 @@ more information. ## Install CKAN plus dependencies -Copy the included `.env.example` and rename it to `.env` - modify it depending on your own needs. +Copy the included `.env.example` and rename it to `.env`. Modify it depending on your own needs. Using the default values on the `.env.example` file will get you a working CKAN instance. There is a sysadmin user created by default with the values defined in `CKAN_SYSADMIN_NAME` and `CKAN_SYSADMIN_PASSWORD`(`ckan_admin` and `test1234` by default). This should be obviously changed before running this setup as a public CKAN instance. @@ -60,6 +59,9 @@ To start the containers: docker compose up +This will start up the containers in the current window. By default the containers will log direct to this window with each container +using a different. You could also use the -d "detach mode" option if you wished to use the current window for something else ie: `docker compose up -d` + At the end of the container start sequence there should be 6 containers running ![Screenshot 2022-12-12 at 10 36 21 am](https://user-images.githubusercontent.com/54408245/207012236-f9571baa-4d99-4ffe-bd93-30b11c4829e0.png) @@ -211,7 +213,3 @@ For example: These parameters can be added to the `.env` file For more information please see [ckanext-envvars](https://github.com/okfn/ckanext-envvars) - -## Known Issues - -Running the tests: Running the tests for CKAN or an extension inside the container will delete your current database. We need to patch CKAN core in our image to work around that.