Skip to content
This repository has been archived by the owner on Oct 23, 2023. It is now read-only.

Feature/integration testing #34

Merged
merged 40 commits into from
Dec 3, 2017
Merged
Changes from 1 commit
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
d30fe19
Upgrading KPL/KCL libraries, replacing deprecated shutdown calls
Aug 30, 2017
a0a69ea
auto format from compile
Aug 30, 2017
a7179f0
Merge branch 'master' of github.com:WW-Digital/reactive-kinesis into …
markglh Sep 7, 2017
8c1211a
restructure the ProducerConf into its own file and spec
markglh Sep 7, 2017
1c9fa2d
further refactoring of Producer to use ProducerConf
markglh Sep 7, 2017
06f2cba
tests pass - ish
markglh Sep 7, 2017
7098a97
improve test exception for manager
markglh Sep 7, 2017
316ca57
Fix Intermittent failing test - #10
markglh Sep 7, 2017
90fa3f5
refactored producer to remove pointless trait, updated readme and specs
markglh Sep 8, 2017
32fe708
Merge branch 'master' into feature/Issue28-TestKPLProperties
markglh Sep 8, 2017
777f375
removed unused config
markglh Sep 8, 2017
a758a85
fix scalafmt
markglh Sep 8, 2017
efae23a
address potters comments
markglh Sep 8, 2017
2997479
address formatting issues
markglh Sep 8, 2017
9f60a4c
added ConsumerConfig test plus missing consumer fields
markglh Sep 8, 2017
6fb2fa5
improved thread reference docs
markglh Sep 9, 2017
d77b0d1
removed comment
markglh Sep 9, 2017
1ed73af
added docker compose localstack environment
markglh Sep 10, 2017
a9362c0
fixed readme
markglh Sep 10, 2017
0ec6bcc
more readme tweaks
markglh Sep 10, 2017
7257f8f
readme fixes
markglh Sep 10, 2017
b19ad1a
readme fixes
markglh Sep 10, 2017
500c521
added integration test for producer, pending CI infra
markglh Sep 11, 2017
9cac2e3
Merge branch 'master' of github.com:WW-Digital/reactive-kinesis into …
markglh Sep 11, 2017
d4daa54
fix formatting
markglh Sep 11, 2017
f302f9f
moved localstack stuff to new image
markglh Sep 13, 2017
f433b3a
fix previous unintended change
markglh Sep 13, 2017
a104e12
formatting
markglh Sep 13, 2017
42e6c5c
removed localstack readme
markglh Dec 2, 2017
8606d96
reworked integration test setup
markglh Dec 2, 2017
bc4ea0d
fix formatting
markglh Dec 2, 2017
a7d59d0
added travis docker-compose
markglh Dec 2, 2017
d0c7335
tweaked travis compose
markglh Dec 2, 2017
8a31827
upgrade docker compose
markglh Dec 2, 2017
2b89d2b
upgrade docker compose
markglh Dec 2, 2017
7473195
fixed travis yaml
markglh Dec 2, 2017
04d9384
run docker compose as daemon
markglh Dec 2, 2017
f8e8166
added integration tests to build
markglh Dec 2, 2017
fb393b2
remove debug, update scala, improve readme
markglh Dec 3, 2017
47275c6
Merge branch 'master' into feature/IntegrationTesting
markglh Dec 3, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
moved localstack stuff to new image
markglh committed Sep 13, 2017
commit f302f9f37b5e56f24b6784d0ccb52fa97f75fb3e
11 changes: 0 additions & 11 deletions localstack/Dockerfile

This file was deleted.

48 changes: 41 additions & 7 deletions localstack/README.md
Original file line number Diff line number Diff line change
@@ -1,24 +1,58 @@
# Running
`docker-compose up` will start the stack with the services defined by `$LOCALSTACK_SERVICES` in the `docker-compose.yml` or `.env` overrides.
# Initialised Locastack
This image extends the functionality of the default image provided by the awesome [localstack](https://github.com/localstack/localstack).

# Bootstrapping
In a nutshell, localstack enabled you to run a number of AWS services locally for testing.

This image extends that functionality to allow you to start the image fully initialised with your configuration. For example Kinesis Streams, SQS queues, Dynamo Tables, etc.

You can do this by providing either a CloudFormation template or a script running `awslocal` Cli commands.

## Running
`docker-compose up` will start the stack with the services defined by `SERVICES` in the `docker-compose.yml` or `LOCALSTACK_SERVICES` in the `.env` overrides.

## Bootstrapping
Scripts are copied to `/opt/bootstrap/scripts`.

Templates are copied to `/opt/bootstrap/templates`.

By default the `init.sh` script creates an AWS stack using the CloudFormation template located in `/opt/bootstrap/templates`.

## Runtime overrides
Note that the CloudFormation template functionality provided by localstack isn't feature complete, [this](https://github.com/localstack/localstack/tree/master/tests/integration/templates) example `test` templates directory from localstack gives an indication of the currently supported featureset.

### Runtime overrides
Two options for overriding this at runtime:
- To just use a different CloudFormation template mount a Volume over `/opt/bootstrap/templates` containing a `cftemplate.yaml` template.
- To directly use `awslocal` on the Cli, mount a Volume over `/opt/bootstrap/scripts` containing an `init.sh` script.

[awslocal](https://github.com/localstack/awscli-local) is installed and used for bootstrapping scripts.

# Environment
The `.env` file will overwrite environment variables defined in the `docker-compose.yml`.
# docker-compose
Here's an example compose file for running the container with kinesis, dynamodb, cloudwatch & Cloudformation.
This mounts over the `cftemplate.yml` with a template in the same directory as the compose file:

```yaml
version: "3.3"

services:
localstack:
image: markglh/initialised-localstack:latest
environment:
- "SERVICES=${LOCALSTACK_SERVICES:-kinesis,dynamodb,cloudwatch,cloudformation}"
- "DEFAULT_REGION=${AWS_REGION:-us-east-1}"
- "HOSTNAME=${LOCALSTACK_HOSTNAME:-localhost}"
- "HOSTNAME_EXTERNAL=${LOCALSTACK_HOSTNAME_EXTERNAL:-localhost}"
- "USE_SSL=true"
#- "DATA_DIR=${LOCALSTACK_DATA_DIR:-/tmp/localstack/data}" # uncomment if you want to persist data between runs
volumes:
- ./templates:/opt/bootstrap/templates
ports:
- "4567-4582:4567-4582"
- "8080:8080"
```

Note that the environment variables supply default values but can be overridden using a `.env` file.

# Example
# Example query against the container
From your host, either install `awslocal` or pass the appropriate endpoint overrides to the aws Cli.

```bash
59 changes: 0 additions & 59 deletions localstack/bootstrap/scripts/init.sh

This file was deleted.

11 changes: 0 additions & 11 deletions localstack/bootstrap/supervisord-init.conf

This file was deleted.

10 changes: 4 additions & 6 deletions localstack/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -2,14 +2,12 @@ version: "3.3"

services:
localstack:
build:
context: .
args:
LOCALSTACK_DOCKER_IMAGE_TAG: 0.7.5
container_name: templated-localstack
image: markglh/initialised-localstack:latest
volumes:
- ./templates:/opt/bootstrap/templates
#network_mode: "host"
environment:
- "SERVICES=${LOCALSTACK_SERVICES:-kinesis,dynamodb,sqs,cloudwatch,cloudformation}"
- "SERVICES=${LOCALSTACK_SERVICES:-kinesis,dynamodb,cloudwatch,cloudformation}"
- "DEFAULT_REGION=${AWS_REGION:-us-east-1}"
- "HOSTNAME=${LOCALSTACK_HOSTNAME:-localhost}"
- "HOSTNAME_EXTERNAL=${LOCALSTACK_HOSTNAME_EXTERNAL:-localhost}"
Original file line number Diff line number Diff line change
@@ -16,3 +16,8 @@ Resources:
Properties:
Name: int-test-stream-3
ShardCount: 1
KinesisStream4:
Type: AWS::Kinesis::Stream
Properties:
Name: int-test-stream-4
ShardCount: 1
Original file line number Diff line number Diff line change
@@ -60,7 +60,12 @@ object ProducerConf {

val streamName = producerConfig.getString("stream-name")
require(
!streamName.isEmpty,

val kplConfig = producerConfig.getConfig("kpl")
val kplLibConfiguration: KinesisProducerConfiguration =
buildKPLConfig(kplConfig, credentialsProvider)

!streamName.isEmpty,
s"Config field `stream-name` missing, a value must be provided to start the Kinesis Producer!"
)

@@ -74,11 +79,6 @@ object ProducerConf {
else
Some(dispatcherProp)
}

val kplConfig = producerConfig.getConfig("kpl")
val kplLibConfiguration: KinesisProducerConfiguration =
buildKPLConfig(kplConfig, credentialsProvider)

new ProducerConf(streamName,
kplLibConfiguration,
dispatcher,