snowplow · spenes · Mar 19, 2024 · Mar 13, 2024 · Mar 24, 2023 · Nov 1, 2023
diff --git a/CHANGELOG b/CHANGELOG
@@ -1,3 +1,21 @@
+Version 6.0.0 (2024-03-18)
+--------------------------
+Bump AWS SDK to 1.12.677 (#1344)
+Bump commons-compress to 1.26.0 (#1344)
+Bump nimbus-jose-jwt to 9.37.2 (#1344)
+Add mandatory SLULA license acceptance flag (#1344)
+Bump schema-ddl to 0.22.1 (#1342)
+Bump AWS SDK to 2.23.17 (#1339)
+pubsub transformer: increase subscriber's awaitTermiantePeriod (#1328)
+pubsub transformer: Increase default value of minDurationPerAckExtension (#1326)
+Loader: Fix column names for shredded tables (#1332)
+Redshift loader: send statsd metrics for recovery tables (#1331)
+Quote column names in Redshift load statements (#1330)
+Loader: Report recovery table names in load_succeeded payload (#1318)
+Loader: Fix table name in COPY logs (#1316)
+Upgrade schema-ddl to 0.20.0 (#1265)
+Move to Snowplow Limited Use License (#1345)
+
 Version 5.7.5 (2023-12-04)
 --------------------------
 Bump zookeeper to 3.7.2 (#1325)

diff --git a/README.md b/README.md
@@ -46,11 +46,12 @@ At the moment, we have loader applications for Redshift, Databricks and Snowflak
 | ![i1][techdocs-image]      | ![i2][setup-image]   | ![i3][roadmap-image]   |
 | [Technical Docs][techdocs] | [Setup Guide][setup] | [Roadmap][roadmap]     |
 
-## Copyright and License
+
+## Copyright and license
 
 Copyright (c) 2012-present Snowplow Analytics Ltd. All rights reserved.
 
-Licensed under the [Snowplow Community License](https://docs.snowplow.io/community-license-1.0). _(If you are uncertain how it applies to your use case, check our answers to [frequently asked questions](https://docs.snowplow.io/docs/contributing/community-license-faq/).)_
+Licensed under the [Snowplow Limited Use License Agreement][license]. _(If you are uncertain how it applies to your use case, check our answers to [frequently asked questions][faq].)_
 
 
 [techdocs-image]: https://d3i6fms1cm1j0i.cloudfront.net/github/images/techdocs.png
@@ -65,8 +66,10 @@ Licensed under the [Snowplow Community License](https://docs.snowplow.io/communi
 [build-image]: https://github.com/snowplow/snowplow-rdb-loader/workflows/CI/badge.svg
 [build]: https://github.com/snowplow/snowplow-rdb-loader/actions/workflows/ci.yml
 
-[release-image]: https://img.shields.io/badge/release-5.7.5-blue.svg?style=flat
+[release-image]: https://img.shields.io/badge/release-6.0.0-blue.svg?style=flat
 [releases]: https://github.com/snowplow/snowplow-rdb-loader/releases
 
-[license]: https://docs.snowplow.io/docs/contributing/community-license-faq/
-[license-image]: https://img.shields.io/badge/license-Snowplow--Community-blue.svg?style=flat
+[license]: https://docs.snowplow.io/limited-use-license-1.0
+[license-image]: https://img.shields.io/badge/license-Snowplow--Limited-Use-blue.svg?style=flat
+
+[faq]: https://docs.snowplow.io/docs/contributing/limited-use-license-faq/
diff --git a/SNOWPLOW-LICENSE.md b/SNOWPLOW-LICENSE.md
@@ -1,30 +1,36 @@
-# Snowplow Community License Agreement
+# Snowplow Limited Use License Agreement
 
-_Version 1.0, January 2023_
+_Version 1.0, January 2024_
 
-This Snowplow Community License Agreement, Version 1.0 (the “Agreement”) sets forth the terms on which Snowplow Analytics, Ltd. (“Snowplow”) makes available certain software made available by Snowplow under this Agreement (the “Software”). BY INSTALLING, DOWNLOADING, ACCESSING, OR USING ANY OF THE SOFTWARE, YOU AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO SUCH TERMS AND CONDITIONS, YOU MUST NOT USE THE SOFTWARE. IF YOU ARE RECEIVING THE SOFTWARE ON BEHALF OF A LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU HAVE THE ACTUAL AUTHORITY TO AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT ON BEHALF OF SUCH ENTITY. “Licensee” means you, an individual, or the entity on whose behalf you are receiving the Software.
+This Snowplow Limited Use License Agreement, Version 1.0 (the “Agreement”) sets forth the terms on which Snowplow Analytics, Ltd. (“Snowplow”) makes available certain software (the “Software”). BY INSTALLING, DOWNLOADING, ACCESSING, OR USING ANY OF THE SOFTWARE, YOU AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT. IF YOU DO NOT AGREE TO SUCH TERMS AND CONDITIONS, YOU MUST NOT USE THE SOFTWARE. IF YOU ARE RECEIVING THE SOFTWARE ON BEHALF OF A LEGAL ENTITY, YOU REPRESENT AND WARRANT THAT YOU HAVE THE ACTUAL AUTHORITY TO AGREE TO THE TERMS AND CONDITIONS OF THIS AGREEMENT ON BEHALF OF SUCH ENTITY. “Licensee” means you, an individual, or the entity on whose behalf you are receiving the Software.
 
 ## LICENSE GRANT AND CONDITIONS
 
-**1.1 License.** Subject to the terms and conditions of this Agreement, Snowplow hereby grants to Licensee a non-exclusive, royalty-free, worldwide, non-transferable, non-sublicenseable license during the term of this Agreement to: (a) use the Software; (b) prepare modifications and derivative works of the Software; and (c) reproduce copies of the Software (the “License”). No right to distribute or make available the Software is granted under this License. Licensee is not granted the right to, and Licensee shall not, exercise the License for an Excluded Purpose. For purposes of this Agreement, “Excluded Purpose” means making available any on-premises or distributed software product, software-as-a-service, platform-as-a-service, infrastructure-as-a-service, or other similar online service, that competes with any products or services that Snowplow or any of its affiliates provides using the Software.
+**1.1 License.** Subject to the terms and conditions of this Agreement, Snowplow hereby grants to Licensee a non-exclusive, royalty-free, worldwide, non-transferable, non-sublicensable license during the term of this Agreement to: (a) use the Software; (b) prepare modifications and derivative works of the Software; and (c) reproduce copies of the Software (the “License”). No right to distribute or make available the Software is granted under this License. Licensee is not granted the right to, and Licensee shall not, exercise the License for any Excluded Purpose.
 
-**1.2 Conditions.** In consideration of the License, Licensee’s distribution of the Software is subject to the following conditions:
+**1.2** For purposes of this Agreement, an “Excluded Purpose” is any use that is either a Competing Use or a Highly-Available Production Use, or both of them.
+
+* **1.2.1** A “Competing Use” is making available any on-premises or distributed software product, or any software-as-a-service, platform-as-a-service, infrastructure-as-a-service, or other similar online service, that competes with any products or services that Snowplow or any of its affiliates provides using the Software.
+
+* **1.2.2** Highly-Available Production Use is any highly-available use, including without limitation any use where multiple instances of any Software component run concurrently to avoid a single point of failure, in a production environment, where production means use on live data.
+
+**1.3 Conditions.** In consideration of the License, Licensee’s use of the Software is subject to the following conditions:
 
 * **a.** Licensee must cause any Software modified by Licensee to carry prominent notices stating that Licensee modified the Software.
 
 * **b.** On each Software copy, Licensee shall reproduce and not remove or alter all Snowplow or third party copyright or other proprietary notices contained in the Software, and Licensee must include the notice below on each copy.
 
   ```
   This software is made available by Snowplow Analytics, Ltd.,
-  under the terms of the Snowplow Community License Agreement, Version 1.0
-  located at https://docs.snowplow.io/community-license-1.0
-  BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY OF THE SOFTWARE,
-  YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
+  under the terms of the Snowplow Limited Use License Agreement, Version 1.0
+  located at https://docs.snowplow.io/limited-use-license-1.0
+  BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
+  OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
   ```
 
-**1.3 Licensee Modifications.** Licensee may add its own copyright notices to modifications made by Licensee.
+**1.4 Licensee Modifications.** Licensee may add its own copyright notices to modifications made by Licensee.
 
-**1.4 No Sublicensing.** The License does not include the right to sublicense the Software, however, each recipient to which Licensee provides the Software may exercise the Licenses so long as such recipient agrees to the terms and conditions of this Agreement.
+**1.5 No Sublicensing.** The License does not include the right to sublicense the Software, however, each recipient to which Licensee provides the Software may exercise the Licenses so long as such recipient agrees to the terms and conditions of this Agreement.
 
 ## TERM AND TERMINATION
 
@@ -40,12 +46,12 @@ SNOWPLOW HEREBY DISCLAIMS ANY AND ALL WARRANTIES AND CONDITIONS, EXPRESS, IMPLIE
 
 ## LIMITATION OF LIABILITY
 
-SNOWPLOW WILL NOT BE LIABLE FOR ANY DAMAGES OF ANY KIND, INCLUDING BUT NOT LIMITED TO, LOST PROFITS OR ANY CONSEQUENTIAL, SPECIAL, INCIDENTAL, INDIRECT, OR DIRECT DAMAGES, HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, ARISING OUT OF THIS AGREEMENT. THE FOREGOING SHALL APPLY TO THE EXTENT PERMITTED BY APPLICABLE LAW.
+SNOWPLOW WILL NOT BE LIABLE FOR ANY DAMAGES OF ANY KIND, INCLUDING BUT NOT LIMITED TO LOST PROFITS OR ANY CONSEQUENTIAL, SPECIAL, INCIDENTAL, INDIRECT, OR DIRECT DAMAGES, HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, ARISING OUT OF THIS AGREEMENT. THE FOREGOING SHALL APPLY TO THE EXTENT PERMITTED BY APPLICABLE LAW.
 
 ## GENERAL
 
 **6.1 Governing Law.** This Agreement will be governed by and interpreted in accordance with the laws of the state of Delaware, without reference to its conflict of laws principles. If Licensee is located within the United States, all disputes arising out of this Agreement are subject to the exclusive jurisdiction of courts located in Delaware, USA. If Licensee is located outside of the United States, any dispute, controversy or claim arising out of or relating to this Agreement will be referred to and finally determined by arbitration in accordance with the JAMS International Arbitration Rules. The tribunal will consist of one arbitrator. The place of arbitration will be in the State of Delaware, USA. The language to be used in the arbitral proceedings will be English. Judgment upon the award rendered by the arbitrator may be entered in any court having jurisdiction thereof.
 
 **6.2. Assignment.** Licensee is not authorized to assign its rights under this Agreement to any third party. Snowplow may freely assign its rights under this Agreement to any third party.
 
-**6.3. Other.** This Agreement is the entire agreement between the parties regarding the subject matter hereof. No amendment or modification of this Agreement will be valid or binding upon the parties unless made in writing and signed by the duly authorized representatives of both parties. In the event that any provision, including without limitation any condition, of this Agreement is held to be unenforceable, this Agreement and all licenses and rights granted hereunder will immediately terminate. Waiver by Snowplow of a breach of any provision of this Agreement or the failure by Snowplow to exercise any right hereunder will not be construed as a waiver of any subsequent breach of that right or as a waiver of any other right.
+**6.3. Other.** This Agreement is the entire agreement between the parties regarding the subject matter hereof. No amendment or modification of this Agreement will be valid or binding upon the parties unless made in writing and signed by the duly authorized representatives of both parties. In the event that any provision, including without limitation any condition, of this Agreement is held to be unenforceable, this Agreement and all licenses and rights granted hereunder will immediately terminate. Waiver by Snowplow of a breach of any provision of this Agreement or the failure by Snowplow to exercise any right hereunder will not be construed as a waiver of any subsequent breach of that right or as a waiver of any other right.
diff --git a/build.sbt b/build.sbt
@@ -1,9 +1,12 @@
 /*
- * Copyright (c) 2012-present Snowplow Analytics Ltd. All rights reserved.
+ * Copyright (c) 2012-present Snowplow Analytics Ltd.
+ * All rights reserved.
  *
- * This program is licensed to you under the Snowplow Community License Version 1.0,
- * and you may not use this file except in compliance with the Snowplow Community License Version 1.0.
- * You may obtain a copy of the Snowplow Community License Version 1.0 at https://docs.snowplow.io/community-license-1.0
+ * This software is made available by Snowplow Analytics, Ltd.,
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
+ * located at https://docs.snowplow.io/limited-use-license-1.0
+ * BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
+ * OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
  */
 
 /**

diff --git a/config/loader/aws/databricks.config.minimal.hocon b/config/loader/aws/databricks.config.minimal.hocon
@@ -1,4 +1,7 @@
 {
+  "license": {
+    "accept": true
+  }
   "messageQueue": "test-queue",
   "storage" : {
     "host": "abc.cloud.databricks.com"

diff --git a/config/loader/aws/databricks.config.reference.hocon b/config/loader/aws/databricks.config.reference.hocon
@@ -1,4 +1,9 @@
 {
+  # Full license text available in SNOWPLOW-LICENSE.md
+  "license": {
+    "accept": true
+  }
+
   # Data Lake (S3) region
   # This field is optional if it can be resolved with AWS region provider chain.
   # It checks places like env variables, system properties, AWS profile file.

diff --git a/config/loader/aws/redshift.config.minimal.hocon b/config/loader/aws/redshift.config.minimal.hocon
@@ -1,4 +1,7 @@
 {
+  "license": {
+    "accept": true
+  }
   "messageQueue": "test-queue"
   "storage" : {
     "type": "redshift"

diff --git a/config/loader/aws/redshift.config.reference.hocon b/config/loader/aws/redshift.config.reference.hocon
@@ -1,4 +1,9 @@
 {
+  # Full license text available in SNOWPLOW-LICENSE.md
+  "license": {
+    "accept": true
+  }
+
   # Specifies the cloud provider that application will be deployed into
   "cloud": "aws"
   # Data Lake (S3) region
@@ -266,4 +271,14 @@
     # Version of the terraform module that deployed the app
     "moduleVersion": "1.0.0"
   }
+
+  # Optional. Enable features that are still in beta, or which are here to enable smoother upgrades
+  "featureFlags": {
+    # List of SchemaKey with partial SchemaVer to disable recoveries for, redshift only
+    # Redshift Loader will disable all migration and recovery table creation for the tables
+    # which belongs to provided schema keys
+    # e.g. [ "iglu:com.example/myschema1/jsonschema/1-*-*", "iglu:com.example/myschema2/jsonschema/1-*-*"]
+    # Optional, empty list by default
+    "disableRecovery": []
+  }
 }
diff --git a/config/loader/aws/snowflake.config.minimal.hocon b/config/loader/aws/snowflake.config.minimal.hocon
@@ -1,4 +1,7 @@
 {
+  "license": {
+    "accept": true
+  }
   "messageQueue": "test-queue",
   "storage" : {
     "type": "snowflake",

diff --git a/config/loader/aws/snowflake.config.reference.hocon b/config/loader/aws/snowflake.config.reference.hocon
@@ -1,4 +1,9 @@
 {
+  # Full license text available in SNOWPLOW-LICENSE.md
+  "license": {
+    "accept": true
+  }
+
   # Data Lake (S3) region
   # This field is optional if it can be resolved with AWS region provider chain.
   # It checks places like env variables, system properties, AWS profile file.

diff --git a/config/loader/azure/databricks.config.minimal.hocon b/config/loader/azure/databricks.config.minimal.hocon
@@ -1,4 +1,7 @@
 {
+  "license": {
+    "accept": true
+  }
   "blobStorageEndpoint": "https://accountName.blob.core.windows.net/container-name"
   "messageQueue": {
     "type": "kafka"

diff --git a/config/loader/azure/databricks.config.reference.hocon b/config/loader/azure/databricks.config.reference.hocon
@@ -1,4 +1,8 @@
 {
+  # Full license text available in SNOWPLOW-LICENSE.md
+  "license": {
+    "accept": true
+  }
 
   # Azure Blob Storage endpoint, should contain container with transformer's output
   "blobStorageEndpoint": "https://accountName.blob.core.windows.net/container-name"

diff --git a/config/loader/azure/snowflake.config.minimal.hocon b/config/loader/azure/snowflake.config.minimal.hocon
@@ -1,4 +1,7 @@
 {
+  "license": {
+    "accept": true
+  }
   "blobStorageEndpoint": "https://accountName.blob.core.windows.net/container-name"
   "messageQueue": {
     "type": "kafka"

diff --git a/config/loader/azure/snowflake.config.reference.hocon b/config/loader/azure/snowflake.config.reference.hocon
@@ -1,4 +1,8 @@
 {
+  # Full license text available in SNOWPLOW-LICENSE.md
+  "license": {
+    "accept": true
+  }
 
   # Azure Blob Storage endpoint, should contain container with transformer's output
   "blobStorageEndpoint": "https://accountName.blob.core.windows.net/container-name"

diff --git a/config/loader/gcp/databricks.config.minimal.hocon b/config/loader/gcp/databricks.config.minimal.hocon
@@ -1,4 +1,7 @@
 {
+  "license": {
+    "accept": true
+  }
   "messageQueue": {
     "type": "pubsub"
     "subscription": "projects/project-id/subscriptions/subscription-id"

diff --git a/config/loader/gcp/databricks.config.reference.hocon b/config/loader/gcp/databricks.config.reference.hocon
@@ -1,4 +1,9 @@
 {
+  # Full license text available in SNOWPLOW-LICENSE.md
+  "license": {
+    "accept": true
+  }
+
   # Pubsub subscription used by Transformer and Loader to communicate
   "messageQueue": {
     "type": "pubsub"

diff --git a/config/loader/gcp/snowflake.config.minimal.hocon b/config/loader/gcp/snowflake.config.minimal.hocon
@@ -1,4 +1,7 @@
 {
+  "license": {
+    "accept": true
+  }
   "messageQueue": {
     "type": "pubsub"
     "subscription": "projects/project-id/subscriptions/subscription-id"

diff --git a/config/loader/gcp/snowflake.config.reference.hocon b/config/loader/gcp/snowflake.config.reference.hocon
@@ -1,4 +1,9 @@
 {
+  # Full license text available in SNOWPLOW-LICENSE.md
+  "license": {
+    "accept": true
+  }
+
   # Pubsub subscription used by Transformer and Loader to communicate
   "messageQueue": {
     "type": "pubsub"

diff --git a/config/transformer/aws/transformer.batch.config.minimal.hocon b/config/transformer/aws/transformer.batch.config.minimal.hocon
@@ -1,4 +1,7 @@
 {
+  "license": {
+    "accept": true
+  }
   "input": "s3://bucket/input/"
   "output": {
     "path": "s3://bucket/transformed/"

diff --git a/config/transformer/aws/transformer.batch.config.reference.hocon b/config/transformer/aws/transformer.batch.config.reference.hocon
@@ -1,4 +1,9 @@
 {
+  # Full license text available in SNOWPLOW-LICENSE.md
+  "license": {
+    "accept": true
+  }
+
   # Path to enriched archive (must be populated separately with run=YYYY-MM-DD-hh-mm-ss-UUID directories) for S3 input
   "input": "s3://bucket/input/",
 

diff --git a/config/transformer/aws/transformer.kinesis.config.minimal.hocon b/config/transformer/aws/transformer.kinesis.config.minimal.hocon
@@ -1,4 +1,7 @@
 {
+  "license": {
+    "accept": true
+  }
   "input": {
     "streamName": "enriched-events"
   }

diff --git a/config/transformer/aws/transformer.kinesis.config.reference.hocon b/config/transformer/aws/transformer.kinesis.config.reference.hocon
@@ -1,4 +1,9 @@
 {
+  # Full license text available in SNOWPLOW-LICENSE.md
+  "license": {
+    "accept": true
+  }
+
   "input": {
     # kinesis and file are the only options for transformer-kinesis
     # Optional, default value kinesis

diff --git a/config/transformer/azure/transformer.kafka.config.minimal.hocon b/config/transformer/azure/transformer.kafka.config.minimal.hocon
@@ -1,4 +1,7 @@
 {
+  "license": {
+    "accept": true
+  }
   "input": {
     "topicName": "enriched"
     "bootstrapServers": "localhost:9092"

diff --git a/config/transformer/azure/transformer.kafka.config.reference.hocon b/config/transformer/azure/transformer.kafka.config.reference.hocon
@@ -1,4 +1,9 @@
 {
+  # Full license text available in SNOWPLOW-LICENSE.md
+  "license": {
+    "accept": true
+  }
+
   "input": {
     "type": "kafka"
 

diff --git a/config/transformer/gcp/transformer.pubsub.config.minimal.hocon b/config/transformer/gcp/transformer.pubsub.config.minimal.hocon
@@ -1,4 +1,7 @@
 {
+  "license": {
+    "accept": true
+  }
   "input": {
     "subscription": "projects/project-id/subscriptions/subscription-id"
   }

diff --git a/config/transformer/gcp/transformer.pubsub.config.reference.hocon b/config/transformer/gcp/transformer.pubsub.config.reference.hocon
@@ -1,4 +1,9 @@
 {
+  # Full license text available in SNOWPLOW-LICENSE.md
+  "license": {
+    "accept": true
+  }
+
   "input": {
     # Name of the Pubsub subscription with the enriched events
     "subscription": "projects/project-id/subscriptions/subscription-id"
@@ -12,6 +17,13 @@
 
     # Optional. Default value '1 hour'. The maximum period a message ack deadline will be extended.
     "maxAckExtensionPeriod": "1 hour"
+
+    # Optional. Sets a lower-bound on how the pubsub Subscriber extends ack deadlines.  Most relevant after the app
+    # first starts up, until the underlying Subscriber has metrics on how long we take to process an event.
+    "minDurationPerAckExtension": "60 seconds"
+
+    # Optional. How long we allow the pubsub Subscriber to ack any outstanding events during clean shutdown
+    "awaitTerminatePeriod": "60 seconds"
   }
 
   # Path to transformed archive

diff --git a/...s/aws/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/aws/EC2ParameterStore.scala b/...s/aws/src/main/scala/com/snowplowanalytics/snowplow/rdbloader/aws/EC2ParameterStore.scala
@@ -1,9 +1,12 @@
 /*
- * Copyright (c) 2012-present Snowplow Analytics Ltd. All rights reserved.
+ * Copyright (c) 2012-present Snowplow Analytics Ltd.
+ * All rights reserved.
  *
- * This program is licensed to you under the Snowplow Community License Version 1.0,
- * and you may not use this file except in compliance with the Snowplow Community License Version 1.0.
- * You may obtain a copy of the Snowplow Community License Version 1.0 at https://docs.snowplow.io/community-license-1.0
+ * This software is made available by Snowplow Analytics, Ltd.,
+ * under the terms of the Snowplow Limited Use License Agreement, Version 1.0
+ * located at https://docs.snowplow.io/limited-use-license-1.0
+ * BY INSTALLING, DOWNLOADING, ACCESSING, USING OR DISTRIBUTING ANY PORTION
+ * OF THE SOFTWARE, YOU AGREE TO THE TERMS OF SUCH LICENSE AGREEMENT.
  */
 package com.snowplowanalytics.snowplow.rdbloader.aws