Releases: uselagoon/lagoon
lagoon-core v2.5.0
This release is built on the https://github.com/uselagoon/lagoon-images/releases/tag/22.2.0 images
New in this release
There are three main features debuting in this release, two of which are still under development, but are in pre-release and in active use already
Bulk Deployments
This allows a Lagoon user to trigger the simultaneous deployment of multiple sites at once, and for those deployments to be automatically allocated to lagoon-remotes and stampede protection/QoS implemented in the remote-controller. These deployments come with additional updates to the UI, linking bulk deployments together, and providing an overview screen for easy tracking.
Insights (pre-release)
Insights is a remote-to-core system that collects data (currently SBOM and image data per service) from Lagoon Builds (into configMaps), and then transmits it back to a handler that stores that data into S3, and processes "key facts" into the API, stored against the environment. Additional functionality will be added to be able to analyse this data for vulnerabilities and inconsistencies, triggering alerts and data to the API. The key facts are still in development, but the underlying data model isn't expected to change.
Workflows (pre-release)
Workflows is an extension of the tasks system that allows more control over when tasks are run, what pre-conditions must exist. It's still in development, but is already in use.
Other updates
There are a number of other fixes in here, including improvements to DBaaS detection, configuration for single-node clusters, Kibana integration, rootless migration updates, GitLab MR labels, and some improvements to task logging, build logging and error tracking in deployments.
Deprecations and Updates
- A large amount of legacy (pre-RBAC) code has been removed - this was no longer functional, and was adjudged safe to be removed
- Kubernetes 1.22 comaptible updates have been made to the Ingresses created by Lagoon. More 1.22 work on other Lagoon aspects is also underway
- Alpine 3.15 has been rolled out where possible to Lagoon services.
What's Changed
- Feature/workflows by @bomoko in #2943
- updating advanced task resolver and permissions by @timclifford in #2955
- Workflows/add and update workflows ap ichanges by @timclifford in #2969
- Fix the check for mongodb-dbaas by @shreddedbacon in #3032
- fix: handle symlink to charts directory other than a local subdirectory by @smlx in #3043
- reduce logging verbosity some more by @shreddedbacon in #3037
- feat: add configuration-complete readiness signalling to keycloak by @smlx in #3042
- implement kubectl check for lagoon-build and add more tests by @tobybellwood in #3039
- check if scc is present and prevent patching the build pod by @shreddedbacon in #3040
- allow rwx volumes to be changed to rwo using a feature flag by @shreddedbacon in #3038
- Add cluster permission to allow downloading of reports from Kibana by @twardnw in #3034
- Adjust permissions on openshift:view by @shreddedbacon in #3031
- Upgrade upstream images to Alpine 3.15 by @tobybellwood in #3041
- networking.k8s.io Ingress deprecations by @tobybellwood in #2815
- Add support for ssh-portal / service-api by @smlx in #2941
- fix: improve rootless migration logic by @smlx in #3051
- Add additional information for failed pods during a build by @shreddedbacon in #3049
- Cleanup pre-rbac data by @rocketeerbkw in #2871
- Adding docker inspect insights gathering by @timclifford in #3033
- remove ssh_portal and tests for now by @tobybellwood in #3053
- fix: set the idling values in the build spec correctly by @shreddedbacon in #3055
- Feature: Bulk deployments by @shreddedbacon in #3046
- Adds summary functionality to environment facts by @bomoko in #3062
- Use actual Merge Request number for GitLab MRs by @tobybellwood in #3060
- Adds fact summary functionality to environment fact resolver by @bomoko in #3063
- environment_fact table updates - service column and unique constraint by @timclifford in #3064
- Feature/workflows by @bomoko in #3045
- fix workflows tests by @tobybellwood in #3071
- Fix up task logs for environment names with slash in the name by @shreddedbacon in #3068
- Fixes error with argument names in advanced tasks by @bomoko in #3070
- Removes logic error where workflows exit before processing all regist… by @bomoko in #3067
Full Changelog: v2.4.1...v2.5.0
lagoon-core v2.4.1
This image is built on the https://github.com/uselagoon/lagoon-images/releases/tag/22.1.0 release
This release introduces a number of new and improved features:
- SBOM generation per-service into namespace ConfigMap
- Integration with the latest amazeeio/dbaas-operator - to dynamically check for presence of dbaas providers
- Incremental build log generation - logs are sent to S3 at a number of relevant build stages
- Collecting the pod logs from failed deployments, to help diagnose failures.
- Retries to skopeo docker image commands to overcome transient read/write issues
- UI updates to show DeployTarget configs, and expose metadata about lagoon-remote clusters as they pertain to environments.
- Addition of a python-persistent helm-chart
- Removal of the legacy billing code from the API
- Conversion of the stored API DB procedures into knex.
What's Changed (since 2.4.0)
- Fix docker-compose-build to execute checkDBaaSHealth by @twardnw in #3027
- feat: wrap sbom generation in feature flag by @shreddedbacon in #3028
- Zip and check size of SBOM configMap before posting it by @timclifford in #3025
What's Changed (since 2.3.0)
- Update docs about
monitoring-path
by @rocketeerbkw in #2976 - use request-timeout for storage calculator by @tobybellwood in #2986
- fix: resolver for environment openshift needs to do a project lookup by @shreddedbacon in #2973
- add idling values to the build spec for the controller to inject by @shreddedbacon in #2979
- Updating SSH docs to include SCP instructions. by @cdchris12 in #2996
- add retries to skopeo commands by @tobybellwood in #2977
- add Blackfire variables documentation to php variables by @Schnitzel in #2999
- Removing Billing Code from API by @justinlevi in #2837
- Collect pod logs for failed deployments by @shreddedbacon in #3011
- install latest fluent-plugins by @tobybellwood in #3009
- Support checking the dbaas-operator http endpoint for provider support by @shreddedbacon in #3007
- Some minor changes and typos by @timclifford in #2992
- Fix/advanced task permissions and queries by @bomoko in #2993
- Add image scanning and SBOM creation back to lagoon tag publishing by @tobybellwood in #3013
- Fix Storybook/Chromatic integration by @timclifford in #3014
- Resolving alertmanager config creation bug by @cdchris12 in #2974
- Additional kubernetes resource fields and supported UI changes by @shreddedbacon in #3010
- Add python-persistent helm chart. by @steveworley in #2989
- Add middleware to UI for security headers. by @steveworley in #2381
- Removes stored proc calls from resolvers by @bomoko in #3006
- Add scroll-to-top-bottom arrow to deployments page by @timclifford in #3019
- Assorted OpenDistro and OpenSearch fixes by @tobybellwood in #3017
- Custom task arguments by @bomoko in #2920
- Use local cached copy of svcat for oc image by @tobybellwood in #3020
- SBOM generation into configmap on build by @timclifford in #3012
- Fixes rogue semicolon in custom task creation by @bomoko in #3021
- add buildName label to sbom configMap by @tobybellwood in #3023
- remove Minishift/k3d from local dev setup by @tobybellwood in #3024
Full Changelog: v2.3.0...v2.4.1
lagoon-core v2.4.0 - USE v2.4.1 INSTEAD
The following changes are all incorporated in the 2.4.1 release - that release also contains two hotfixes to build-deploys that can cause build failures.
Use of this release may result in some mariadb services incorrectly being allocated container resources (mariadb-single) instead of the expected dbaas ones. In addition, sites that generated large ConfigMaps of their SBOM may have their builds incorrectly reported as "failing". Both these issues are resolved in v2.4.1
What's Changed
- Update docs about
monitoring-path
by @rocketeerbkw in #2976 - use request-timeout for storage calculator by @tobybellwood in #2986
- fix: resolver for environment openshift needs to do a project lookup by @shreddedbacon in #2973
- add idling values to the build spec for the controller to inject by @shreddedbacon in #2979
- Updating SSH docs to include SCP instructions. by @cdchris12 in #2996
- add retries to skopeo commands by @tobybellwood in #2977
- add Blackfire variables documentation to php variables by @Schnitzel in #2999
- Removing Billing Code from API by @justinlevi in #2837
- Collect pod logs for failed deployments by @shreddedbacon in #3011
- install latest fluent-plugins by @tobybellwood in #3009
- Support checking the dbaas-operator http endpoint for provider support by @shreddedbacon in #3007
- Some minor changes and typos by @timclifford in #2992
- Fix/advanced task permissions and queries by @bomoko in #2993
- Add image scanning and SBOM creation back to lagoon tag publishing by @tobybellwood in #3013
- Fix Storybook/Chromatic integration by @timclifford in #3014
- Resolving alertmanager config creation bug by @cdchris12 in #2974
- Additional kubernetes resource fields and supported UI changes by @shreddedbacon in #3010
- Add python-persistent helm chart. by @steveworley in #2989
- Add middleware to UI for security headers. by @steveworley in #2381
- Removes stored proc calls from resolvers by @bomoko in #3006
- Add scroll-to-top-bottom arrow to deployments page by @timclifford in #3019
- Assorted OpenDistro and OpenSearch fixes by @tobybellwood in #3017
- Custom task arguments by @bomoko in #2920
- Use local cached copy of svcat for oc image by @tobybellwood in #3020
- SBOM generation into configmap on build by @timclifford in #3012
- Fixes rogue semicolon in custom task creation by @bomoko in #3021
- add buildName label to sbom configMap by @tobybellwood in #3023
- remove Minishift/k3d from local dev setup by @tobybellwood in #3024
Full Changelog: v2.3.0...v2.4.0
lagoon-core v2.3.0
This is the most recent scheduled release of Lagoon, built from the https://github.com/uselagoon/lagoon-images/releases/tag/21.12.1 images
There are three main items here:
-
Support for deifining services in routerPatterns (#2953) - this will allow users (particularly those with multi-clusters) to define their own router patterns. The Lagoon default is
${service}.${environment}.${project}.clusterURL
- but this can cause issues with some certificate authorities when used to secure Autogenerated routes. This PR allows the service, environment and project combination to be defined per project (or per cluster) - commonly to${service}-${environment}-${project}.clusterURL
-
Support for Routes defined via the API (#2940) - this will allow Administrators to override, or add routes to projects without the need for them to be added to the project's .lagoon.yml file. This is especially handy from a support point of view, as well as in Polysite or Multisite applications.
-
Images from previous deployments available as cache in the build step (#2919) - this exposes some new environment variables into the Lagoon Build that provide the image reference for the previous deployment's images. These can then be loaded into your dockerfile as a cache, especially useful for builds that have submodules. There is a brief example we use for testing at https://github.com/uselagoon/lagoon/blob/main/tests/files/image-cache/Dockerfile#L17 but we will publish more information shortly
Other smaller fixes include improved logic for Drush sql-dumps, log verbosity improvements, our documentation change, cronjob fixes, storage-calculator improvements and some improvements to docker-host management.
What's Changed
- fix: actually check the project defined routerpattern exists by @shreddedbacon in #2926
- Documentation Fixes - Make uselagoon/* images easier discoverable by @dasrecht in #2934
- add fallback true to docker_pull by @tobybellwood in #2942
- Refactor drupal tests by @tobybellwood in #2936
- wrap HELM_ARGUMENTS in setx by @tobybellwood in #2928
- Adds image cache images to build by @bomoko in #2919
- Ensure to only capture single digest by @tobybellwood in #2944
- Adds tests for cache and fixes env var format by @bomoko in #2945
- feat: support routes from the api in environment variables by @shreddedbacon in #2940
- Reduce concurrent tests per run - adding third suite temporarily by @tobybellwood in #2948
- Fix error running addKubernetes with empty id by @rocketeerbkw in #2929
- Passing no-tablespaces flag into drush sql-dump for task by @timclifford in #2939
- gitignore stern by @shreddedbacon in #2963
- fix: use the project name instead of the environment name by @shreddedbacon in #2954
- first batch of updated docker images by @tobybellwood in #2938
- change to mkdocs from gitbook by @tobybellwood in #2968
- feat: support for service being defined in routerPattern by @shreddedbacon in #2953
- Adding startingDeadlineSeconds to all cli native cronjob templates by @cdchris12 in #2831
- Refactor Storage Calculator to be more robust by @seanhamlin in #2947
- prune dangling images on docker-host automatically by @twardnw in #2967
- docker-host - added environment variable PRUNE_IMAGES_UNTIL to prune-… by @dasrecht in #2960
Full Changelog: v2.2.4...v2.3.0
lagoon-core v2.2.4
What's Changed
- fix: populate pullrequests not branches by @shreddedbacon in #2912
- fix: always create a backupS3Config and use it by @shreddedbacon in #2914
- fix: order that routerpattern is checked for deploytarget config by @shreddedbacon in #2915
- fix: routerpattern can be null, so check if undefined by @shreddedbacon in #2917
- fix: populate the values.yaml for fastly ingress support by @shreddedbacon in #2921
- feat: add build step patching, and time and duration messaging by @shreddedbacon in #2922
- pin test pip dependencies by @tobybellwood in #2924
Full Changelog: v2.2.3...v2.2.4
lagoon-core v2.2.3
This is a hotfix release - it fixes two regressions since v2.2.0 and a long-running dashboard task incompatibility with newer Drush versions.
What's Changed
- Environment check fixes by @shreddedbacon in #2909
- fix: use a sql query to get project vars for restores rather than graphql by @shreddedbacon in #2910
- Feature/support drush gt 8 by @bomoko in #2906
Full Changelog: v2.2.2...v2.2.3
lagoon-core v2.2.2
What's Changed
- fix: use the correct environmentname when sourcing logs by @shreddedbacon in #2894
- Add pre-deploy lagoon.yml to configmap by @shreddedbacon in #2896
- Update linter to version v0.5.0 by @tobybellwood in #2895
- feat: make the linter errors clearer by @smlx in #2897
- Additional fixes for pre/post deploy lagoon-yaml configmap by @shreddedbacon in #2900
- feat: add verbose linter message on success by @tobybellwood in #2901
Full Changelog: v2.2.1...v2.2.2
lagoon-core v2.2.1
This release is built on the https://github.com/uselagoon/lagoon-images/releases/tag/21.10.0 images
Hotfix release for v2.2.0
What's Changed
- fix: don't touch the sentinel file if it already exists by @smlx in #2891
- Bump lagoon-linter version to address tls-acme validation issue by @smlx in #2893
- Change updateEnvironment data handling by @rocketeerbkw in #2464
Full Changelog: v2.2.0...v2.2.1
lagoon-core v2.2.0
This release is built on the https://github.com/uselagoon/lagoon-images/releases/tag/21.10.0 images
Three important Alpha stability features here:
Add default Kubernetes network policy support by @smlx in #2536
In order to better provide namespace isolation, a NetworkPolicy has been implemented to prevent inter-namespace communication. This can be enabled in a number of fashions:
- Forced for all namespaces in the controller (via variable
LAGOON_FEATURE_FLAG_FORCE_ISOLATION_NETWORK_POLICY=true
) - Individually per project or environment (via variable
LAGOON_FEATURE_FLAG_ISOLATION_NETWORK_POLICY=true
) - Set as default for all namespaces in the controller (via variable
LAGOON_FEATURE_FLAG_DEFAULT_ISOLATION_NETWORK_POLICY=true
)
Implement rootless workloads by @smlx in #2481
In order to better provide protection against workloads running as root, a SecurityContext has been set for services, along with an init container that will ensure namespaces have the correct permissions in their file storage. This can be enabled in a number of fashions:
- Forced for all namespaces in the controller (via variable
LAGOON_FEATURE_FLAG_FORCE_ROOTLESS_WORKLOAD=true
) - Individually per project or environment (via variable
LAGOON_FEATURE_FLAG_ROOTLESS_WORKLOAD=true
) - Set as default for all namespaces in the controller (via variable
LAGOON_FEATURE_FLAG_DEFAULT_ROOTLESS_WORKLOAD=true
)
Validate ingress annotation snippets against an allow-list by @tobybellwood in #2889
There is now a lagoon-linter step that runs as part of the build & deploy process that will inspect defined routes for correct configuration of nginx annotation snippets (in response to CVE-2021-25742. Instead of disallowing snippets entirely (which is the current recommended remediation), Lagoon has opted to utilise a linter (https://github.com/uselagoon/lagoon-linter) to process an allowlist of defined snippets. The catch here is that the linter will not lint files that are not valid YAML.
To check a .lagoon.yml file yourself, download and extract the binary from https://github.com/uselagoon/lagoon-linter/releases and run it against your .lagoon.yml file locally. If the linter exits successfully (no output), the file is ok.
What's Changed
- update refs in makefile by @tobybellwood in #2884
- docs: make clear where .env files are loaded from by @pmelab in #2886
- Add default Kubernetes network policy support by @smlx in #2536
- Additional SSH service changes by @shreddedbacon in #2881
- Validate ingress annotation snippets against an allow-list by @tobybellwood in #2889
- Document new feature flags by @smlx in #2541
- Add fastly configuration to autogenerated routes if enabled by @shreddedbacon in #2883
- Implement rootless workloads by @smlx in #2481
New Contributors
Full Changelog: v2.1.0...v2.2.0
lagoon-core v2.1.0
This release is built on the https://github.com/uselagoon/lagoon-images/releases/tag/21.9.0 images
Create new variable for project seed by @shreddedbacon in #2859
The Lagoon team identified a situation that may arise when a jwtsecret is rotated (following good security practice). The jwtsecret was used to create a password for the k8up repository, but also used to generate that password on each run. Changing the jwtsecret changes the password, so instead we have provided a projectseed instead, that doesn't need rotating, and is therefore safer to use to create these passwords reliably. We have added backward compatibility shims both in Lagoon and in the charts used to deploy Lagoon-core that will create this projectseed from the existing jwtsecret to ensure backwards compatibility
Experimental Support for multiple deployment targets per project by @shreddedbacon in #2829
Some of this functionality is in an early release phase, and all API schema calls are marked accordingly. The primary impact is that the storage of the DeployTarget (Kubernetes cluster) is now stored against the individual environment, as opposed to the project. This allows admins to configure a set of rules that determine which environments in a project deploy to specific clusters. Once an environment is allocated a DeployTarget, that is where that environment will always deploy. To manage the implementation of this, a procedure has been added to the api-db to ensure that all environments have the correct current DeployTarget added to them.
Please ensure you run the rerun_initdb.sh script after update to update all the projects for this change.
Improvements to build and task log stability by @shreddedbacon in #2862
In this release, a new service has been added to retrieve build and task logs from the lagoon-logs exchange and upload them to the S3 files bucket (along with task uploads). The API (and the UI) then retrieve these logs from that bucket instead of Elasticsearch. The logs are still currently configured to upload to Elasticsearch as well as S3. Additionally, a minio service is configured to hold the logs in local development.
Removing defaultMeta from all user activity logs by @timclifford in #2856
We picked up a situation where the user performing an action could occasionally be attributed to future actions in the audit logs. Making the user logging action more thread-safe fixes this, and removes the erroneous replication.
add X-Robots-Tag noindex, nofollow server-snippet to all ingresses by @tobybellwood in #2867
Previously all robots control was performed in the nginx base image. This PR has brought this forward into the creation of all the auto-generated ingresses (for all service types, not just nginx). This means that development environments, and production internal URLs are all covered automatically. There is a note in the documentation about use of additional server-snippets in .lagoon.yml.
What's Changed
- Fix Kibana hyperlink in Logging page by @christopher-hopper in #2852
- Fix broken markup in README.md by @ndouglas in #2860
- Removing defaultMeta from all user activity logs by @timclifford in #2856
- Trim trailing periods in custom route shortening logic by @smlx in #2784
- Add Login Required title to Error/unauthorized page by @dan2k3k4 in #2453
- Improvements to build and task log stability by @shreddedbacon in #2862
- Create new variable for project seed by @shreddedbacon in #2859
- More verbose logging in ssh service by @shreddedbacon in #2866
- Support for multiple deployment targets per project by @shreddedbacon in #2829
- Clean up any deploytarget configurations when removing projects by @shreddedbacon in #2872
- reference environment.id correctly in createTaskTask taskData by @tobybellwood in #2870
- Fix retry loop logic in testing by @tobybellwood in #2873
- Fix deleting projects with a name greater than 50 characters by @rocketeerbkw in #2869
- Add random string to the end of restore names to be more unique by @shreddedbacon in #2875
- add X-Robots-Tag noindex, nofollow server-snippet to all ingresses by @tobybellwood in #2867
- Clean up any deploytarget configurations when removing projects v2 by @shreddedbacon in #2876
- Use IS NULL in the deploytarget migration by @shreddedbacon in #2878
- Declare DeployTargetConfig related api fields as unstable by @rocketeerbkw in #2877
- Fixing PHP8 links in documentation by @dasrecht in #2879
- Adds fact limit to to graphql by @timclifford in #2868
New Contributors
- @christopher-hopper made their first contribution in #2852
- @ndouglas made their first contribution in #2860
Full Changelog: v2.0.0...v2.1.0