Merge pull request #114 from epiphany-platform/develop

0.2.0 Release

.dockerignore

@@ -11,5 +11,5 @@ certs/
 tests/
 Dockerfile
 
-data/
-build/
+core/data/
+core/build/

.markdownlint.json

@@ -1,5 +1,7 @@
 {
     "MD013": false,
     "MD029": false,
+    "MD031": false,
+    "MD032": false,
     "MD034": false
 }

CHANGELOG-0.1.2.md

@@ -1,7 +1,7 @@
-# Changelog 0.1.1
+# Changelog 0.1.2
 
 ## [0.1.2] - 2018-12-05
 
 ### Changed
 
-- Kubernetes v1.11.5 installation
+-   Kubernetes v1.11.5 installation

CHANGELOG-0.2.0.md

@@ -0,0 +1,21 @@
+# Changelog 0.2.0
+
+## [0.2.0] - 2019-02-19
+
+### Changed
+
+-   Kubernetes v1.13.0 installation
+-   Filebeat 6.5.4 installation
+-   RabbitMQ installation inside Kubernetes (clustered RabbitMQ) #17
+-   RabbitMQ installation outside of Kubernetes (VM) #17
+-   PostgreSQL installation with replication #16
+-   Authentication service installation (Keycloak) inside Kubernetes
+-   Automatic untainting Kubernetes Master when single Master deployed without Nodes #22
+-   Example applications added to /examples
+    -   Keycloak authentication (dotnet, java, python, javascript) #19
+    -   RabbitMQ/Kafka (dotnet) #50, #39
+-   Documentation updates
+
+### Fixed
+
+-   Filebeat memory consumption when Elasticsearch does not accept data #61

CHANGELOG.md

@@ -7,9 +7,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ## Current release
 
-- [CHANGELOG-0.1.2.md](./CHANGELOG-0.1.2.md)
+-   [CHANGELOG-0.2.0.md](./CHANGELOG-0.2.0.md)
 
 ## Older releases
 
-- [CHANGELOG-0.1.1.md](./CHANGELOG-0.1.1.md)
-- [CHANGELOG-0.1.0.md](./CHANGELOG-0.1.0.md)
+-   [CHANGELOG-0.1.2.md](./CHANGELOG-0.1.2.md)
+-   [CHANGELOG-0.1.1.md](./CHANGELOG-0.1.1.md)
+-   [CHANGELOG-0.1.0.md](./CHANGELOG-0.1.0.md)

README.md

@@ -2,39 +2,46 @@
 
 ## Overview
 
-Epiphany at it's core is full automation of Kubernetes and Docker plus additional builtin services such as Kafka for high speed messaging/events, Prometheus for monitoring and Graphana for dashboards, Elasticsearch and Kibana for centralized logging. Other optional services are being evaluated now.
+Epiphany at its core is a full automation of Kubernetes and Docker plus additional builtin services such as Kafka for high speed messaging/events, Prometheus for monitoring and Graphana for dashboards, Elasticsearch and Kibana for centralized logging. Other optional services are being evaluated now.
 
 Epiphany can run on as few as one node (laptop, desktop, server) but the real value comes from running 3 or more nodes for scale and HA. Nodes can be added or removed at will depending on data in the manifest. Everything is data driven so simply changing the manifest data and running the automation will modify the environment.
 
 We currently use Terraform and Ansible for our automation orchestration. All automation is idempotent so you can run it as many times as you wish and it will maintain the same state unless you change the data. If someone makes a "snow flake" change to the environment (you should never do this) then simply running the automation again will put the environment back to the desired state.
 
-## Table of Contents
+## Quickstart
 
-<!-- TOC -->
+Fork the `epiphany` repository and modify the yaml's under `core/data/` directory. For example in `data/azure/infrastructure/epiphany-playground/basic-data.yaml` file you will need to modify a few values (like you Azure subscription name, directory path for ssh keys). Once you are done done with `basic-data.yaml` you can execute Epiphany with the command:
+
+```shell
+./epiphany -a -b -i -f infrastructure/epiphany-playground -t infrastructure/epiphany-template
+```
+
+This setup works on a simplified file that is fine to start with, if you need more control over the infrastructure created you should look at `data/azure/infrastructure/epiphany-bld-apps/data.yaml`.
+Execution of this full profile would look like:
 
-- [Epiphany project](docs/home/README.md)
+```shell
+./epiphany -a -b -i -f infrastructure/epiphany-bld-apps
+```
 
-- [How-to guides](docs/home/HOWTO.md)
+Find more information using table of contents below - especially the [How-to guides](docs/home/HOWTO.md).
 
-- [Troubleshooting](docs/home/TROUBLESHOOTING.md)
+## Documentation
 
+<!-- TOC -->
+
+- Platform
+  - [Resources](docs/home/RESOURCES.md)
+  - [How-to guides](docs/home/HOWTO.md)
+  - [Troubleshooting](docs/home/TROUBLESHOOTING.md)
 - Architecture
   - [Logical View](docs/architecture/logical-view.md)
-
   - [Process View](docs/architecture/process-view.md)
-
   - [Physical View](docs/architecture/physical-view.md)
-
-- [How-to contribute](docs/home/CONTRIBUTING.md)
-
-- [Workflow to follow](docs/home/GITWORKFLOW.md)
-
-- [Governance model](docs/home/GOVERNANCE.md)
-
-- [Changelog](CHANGELOG.md)
-
-- [Project layout](docs/project_layout.md)
+- Project
+  - [How-to contribute](docs/home/CONTRIBUTING.md)
+  - [Workflow to follow](docs/home/GITWORKFLOW.md)
+  - [Governance model](docs/home/GOVERNANCE.md)
+  - [Notices](docs/home/NOTICES.md)
+  - [Changelog](CHANGELOG.md)
 
 <!-- TOC -->
-
----

core/README.md

@@ -1,11 +1,5 @@
-# Epiphany
+# Core
 
-## Overview
+This folder contains the Epiphany core.
 
-Epiphany at it's core is full automation of Kubernetes and Docker plus additional builtin services such as Kafka for high speed messaging/events, Prometheus for monitoring and Graphana for dashboards, Elasticsearch and Kibana for centralized logging. Other optional services are being evaluated now.
-
-Epiphany can run on as few as one node (laptop, desktop, server) but the real value comes from running 3 or more nodes for scale and HA. Nodes can be added or removed at will depending on data in the manifest. Everything is data driven so simply changing the manifest data and running the automation will modify the environment.
-
-We currently use Terraform and Ansible for our automation orchestration. All automation is idempotent so you can run it as many times as you wish and it will maintain the same state unless you change the data. If someone makes a "snow flake" change to the environment (you should never do this) then simply running the automation again will put the environment back to the desired state.
-
-For the full story, go to [Epiphany documentation](https://github.com/epiphany-platform/docs/README.md).
+For the full story, go to [Epiphany documentation](../README.md).

core/core/src/ansible/deployments.yml

@@ -0,0 +1,11 @@
+---
+# Ansible playbook that deploys applications on Kubernetes after cluster creation
+
+- hosts: all
+  tasks: [ ]  
+
+- hosts: deployments
+  become: true
+  become_method: sudo
+  roles:
+    - deployments

core/core/src/ansible/filebeat.yml

@@ -1,5 +1,8 @@
 ---
-# Ansible playbook that makes sure the base items for all nodes are installed
+# Ansible playbook that installs and configures Filebeat
+
+- hosts: elasticsearch:kibana # To gather facts
+  tasks: [ ]
 
 - hosts: filebeat
   become: true

core/core/src/ansible/roles/common/tasks/main.yml

@@ -73,14 +73,6 @@
     - disableswap
   notify: restart waagent
 
-- name: Manage {{ admin_user.name }} key
-  copy:
-    src: "{{ admin_user.key_path }}"
-    dest: /home/{{ admin_user.name }}/.ssh/id_rsa
-    mode: 0600
-    owner: "{{ admin_user.name }}"
-    group: "{{ admin_user.name }}"
-
 - name: Disable SELinux at next reboot
   selinux:
     state: disabled

core/core/src/ansible/roles/deployments/defaults/main.yml

@@ -0,0 +1,3 @@
+postgresql_database_port: "5432"
+configuration_temp_directory: "/home/{{ admin_user.name }}/epiphany-auth"
+default_namespace: "default"

core/core/src/ansible/roles/deployments/tasks/applications/auth-service/cleanup.yml

@@ -0,0 +1,4 @@
+- name: Clean configs path
+  file:
+    state: absent
+    path: "{{ configuration_temp_directory }}/"

core/core/src/ansible/roles/deployments/tasks/applications/auth-service/configure-database.yml

@@ -0,0 +1,15 @@
+- name: Create database for Epiphany Auth Service
+  delegate_to: "{{ groups['postgresql'] | first }}"
+  become_user: postgres
+  postgresql_db:
+    name: "{{ data.database.name }}"
+
+- name: Add postresql user
+  delegate_to: "{{ groups['postgresql'] | first }}"
+  become_user: postgres
+  postgresql_user:
+    db: "{{ data.database.name }}"
+    name: "{{ data.database.user }}"
+    password: "{{ data.database.password }}"
+    role_attr_flags: CREATEDB,NOSUPERUSER
+    expires: infinity

core/core/src/ansible/roles/deployments/tasks/applications/auth-service/create-service.yml

@@ -0,0 +1,18 @@
+- name: Create directory for files
+  file: 
+    path: "{{ configuration_temp_directory }}"
+    state: directory
+  become_user: "{{ admin_user.name }}"
+
+- name: Upload auth-service yaml
+  become: yes
+  template:
+    dest: "{{ configuration_temp_directory }}/{{ auth_service_name }}-{{ namespace_name }}.yml"
+    src: auth-service/auth-service.yml.j2
+    owner: "{{ admin_user.name }}"
+    group: "{{ admin_user.name }}"
+    mode: 0644
+
+- name: Apply Epiphany Auth Service deployment
+  shell: kubectl apply --kubeconfig=/home/{{ admin_user.name }}/.kube/config -f {{ configuration_temp_directory }}/{{ auth_service_name }}-{{ namespace_name }}.yml
+  become_user: "{{ admin_user.name }}"

core/core/src/ansible/roles/deployments/tasks/applications/auth-service/main.yml

@@ -0,0 +1,15 @@
+- name: Include validation 
+  include_tasks: "validation.yml"
+
+- name: Include database configuration when Epiphany storage used
+  include_tasks: "configure-database.yml"
+  when: data.database.address is undefined 
+
+- name: Include variables setup
+  include_tasks: "set-variables.yml"
+
+- name: Include service creation
+  include_tasks: "create-service.yml"
+
+- name: Include cleanup tasks
+  include_tasks: "cleanup.yml"

core/core/src/ansible/roles/deployments/tasks/applications/auth-service/set-variables.yml

@@ -0,0 +1,21 @@
+- name: Set storage address if not set
+  set_fact:
+    auth_service_db_address: "{{ data.database.address | default(hostvars[groups['postgresql'][0]]['ansible_default_ipv4']['address']) }}"
+
+- name: Set storage port if not set
+  set_fact:
+    auth_service_db_port: "{{ data.database.port | default(postgresql_database_port) }}"
+
+- name: Set namespace variable
+  set_fact:
+    namespace_name: "{{ data.service.namespace | default(default_namespace) | lower }}" # Kubernetes (as of v1.11) accepts only lowercase namespace names. 
+
+- name: Set service name variable if set
+  set_fact:
+    auth_service_name: "{{ data.service.name | lower }}" # Kubernetes (as of v1.11) accepts only lowercase service names. 
+  when: data.service.name is defined 
+
+- name: Set service name variable for default
+  set_fact:
+    auth_service_name: "as-{{ namespace_name }}" # Kubernetes (as of v1.11) accepts only lowercase service names. 
+  when: data.service.name is undefined 

core/core/src/ansible/roles/deployments/tasks/applications/auth-service/validation.yml

@@ -0,0 +1,4 @@
+- name: Verify storage configuration
+  assert:
+    that: "not ({{ groups['postgresql'] | length }} == 0 and data.database.address is not defined)"
+    msg: "At least one postgresql component or database address has to be defined."

core/core/src/ansible/roles/deployments/tasks/applications/rabbitmq/cleanup.yml

@@ -0,0 +1,5 @@
+---
+- name: Clean configs path
+  file:
+    state: absent
+    path: "{{ configuration_temp_directory }}/"

core/core/src/ansible/roles/deployments/tasks/applications/rabbitmq/create-namespace.yml

@@ -0,0 +1,13 @@
+---
+- name: Upload namespace yaml definition
+  become: yes
+  template:
+    dest: "{{ configuration_temp_directory }}/{{ rabbitmq_service_name }}-ns-{{ namespace_name }}.yml"
+    src: rabbitmq/rabbitmq-namespace.yml.j2
+    owner: "{{ admin_user.name }}"
+    group: "{{ admin_user.name }}"
+    mode: 0644
+
+- name: Apply namespace
+  shell: kubectl apply --kubeconfig=/home/{{ admin_user.name }}/.kube/config -f {{ configuration_temp_directory }}/{{ rabbitmq_service_name }}-ns-{{ namespace_name }}.yml
+  become_user: "{{ admin_user.name }}"

core/core/src/ansible/roles/deployments/tasks/applications/rabbitmq/create-rbac.yml

@@ -0,0 +1,13 @@
+---
+- name: Upload rbac yaml definition
+  become: yes
+  template:
+    dest: "{{ configuration_temp_directory }}/{{ rabbitmq_service_name }}-rbac-{{ namespace_name }}.yml"
+    src: rabbitmq/rabbitmq-rbac.yml.j2
+    owner: "{{ admin_user.name }}"
+    group: "{{ admin_user.name }}"
+    mode: 0644
+
+- name: Apply rbac
+  shell: kubectl apply --kubeconfig=/home/{{ admin_user.name }}/.kube/config -f {{ configuration_temp_directory }}/{{ rabbitmq_service_name }}-rbac-{{ namespace_name }}.yml
+  become_user: "{{ admin_user.name }}"

core/core/src/ansible/roles/deployments/tasks/applications/rabbitmq/create-service.yml

@@ -0,0 +1,13 @@
+---
+- name: Upload RabbitMQ service yaml
+  become: yes
+  template:
+    dest: "{{ configuration_temp_directory }}/{{ rabbitmq_service_name }}-{{ namespace_name }}.yml"
+    src: rabbitmq/rabbitmq.yml.j2
+    owner: "{{ admin_user.name }}"
+    group: "{{ admin_user.name }}"
+    mode: 0644
+
+- name: Apply RabbitMQ deployment
+  shell: kubectl apply --kubeconfig=/home/{{ admin_user.name }}/.kube/config -f {{ configuration_temp_directory }}/{{ rabbitmq_service_name }}-{{ namespace_name }}.yml
+  become_user: "{{ admin_user.name }}"

core/core/src/ansible/roles/deployments/tasks/applications/rabbitmq/main.yml

@@ -0,0 +1,24 @@
+---
+- name: Include variables setup
+  include_tasks: "set-variables.yml"
+
+- name: Create temp directory for files
+  file:
+    path: "{{ configuration_temp_directory }}"
+    state: directory
+  become_user: "{{ admin_user.name }}"
+
+- name: Include namespace creation
+  include_tasks: "create-namespace.yml"
+
+- name: Include rbac creation
+  include_tasks: "create-rbac.yml"
+
+- name: Include erlang cookie handling
+  include_tasks: "set-erlang-cookie.yml"
+
+- name: Include service creation
+  include_tasks: "create-service.yml"
+
+- name: Include cleanup tasks
+  include_tasks: "cleanup.yml"