Skip to content
This repository has been archived by the owner on Oct 23, 2023. It is now read-only.

support for aws / local config initialisation #21

Closed
fommil opened this issue Aug 18, 2017 · 15 comments
Closed

support for aws / local config initialisation #21

fommil opened this issue Aug 18, 2017 · 15 comments

Comments

@fommil
Copy link

fommil commented Aug 18, 2017

AWS seems to use magic to provide the host/port and creds, whereas using a local docker mock of kinesis requires other information. It would be good if all this setup was abstracted away such that evil

if (tests) { ... }
else { ... untested prod codebase ...}

branches are minimised / localised to this project and tested in the field with the long hard stick of production support.

That's a guaranteed bug, right there.

@markglh
Copy link
Contributor

markglh commented Aug 18, 2017

@fommil You can configure all Kinesis settings/endpoints in the application.conf - could you give me an example of what you're hoping for as I suspect I'm not fully understanding this one.

@fommil
Copy link
Author

fommil commented Aug 18, 2017

yeah, but not publicly.

@markglh
Copy link
Contributor

markglh commented Aug 18, 2017

->>> slack :)

@etspaceman
Copy link
Contributor

etspaceman commented Sep 6, 2017

This issue confused me initially because I thought that the library couldn't support local kinesis streams. I've been able to successfully implement LocalStack + this library; just need to consider the following:

  • LocalStack must be set up with USE_SSL=true because the KPL only supports SSL traffic
  • Verification of the SSL certificates need to be disabled through 2 means:
    -- KPL "VerifyCertificate" needs to be set to false
    -- The following system property is needed for the KCL (can be passed in as a Java Option): -Dcom.amazonaws.sdk.disableCertChecking=true
  • AWS_CBOR_DISABLE needs to be set to true
  • Endpoints need to be overridden to localstack endpoints (e.g. https://localhost:4568 for KCL)

CC @markglh and @fommil.

@markglh
Copy link
Contributor

markglh commented Sep 6, 2017

Thanks @etspaceman - I know @fommil used Kinesalite directly to ease creation of streams on startup. I plan to look into this more and add a testkit module to the library

@j-potts
Copy link

j-potts commented Sep 6, 2017

@etspaceman Thanks for posting this bit, it was blocking us for a while! 😄

Verification of the SSL certificates need to be disabled through 2 means:
-- KPL "VerifyCertificate" needs to be set to false
-- The following system property is needed for the KCL (can be passed in as a Java Option): -Dcom.amazonaws.sdk.disableCertChecking=true

@markglh
Copy link
Contributor

markglh commented Sep 6, 2017

I do like how consistent Amazon is with the KPL and KCL libraries ;)

@etspaceman
Copy link
Contributor

Happy to help @j-potts! I had to dig through some KCL/KPL issues to really find what was going wrong. Was bugging me for days.

@etspaceman
Copy link
Contributor

@markglh just an FYI - Kinesisalite is actually what LocalStack uses so this should work for that too.

@markglh
Copy link
Contributor

markglh commented Sep 6, 2017

Yeah it wraps it - it's a nice entrypoint for all the different AWS services. Hopefully we can get it working using that as it'll make working across services in docker much cleaner. I'm not sure of the exact details as I'm yet to dig into it and compare what @fommil did with this:
https://hub.docker.com/r/fommil/docker-kinesis/~/dockerfile/

@etspaceman
Copy link
Contributor

I've actually got mine working in docker today, across containers, using the localstack/localstack image + their awscli-local solution. Would be happy to post a Gist!

Mine uses --net host though, so if you are wanting to use the docker networking it'll need some tweaks like link/port references.

@markglh
Copy link
Contributor

markglh commented Sep 6, 2017

Ooh yeah that would be awesome!

@etspaceman
Copy link
Contributor

etspaceman commented Sep 6, 2017

Here's the gist. Use either docker-compose.yml or create_localstack_container.sh, then run create_kinesis_stream.sh. See the local.env.template for a .env template on your setup.

I also included a reference.conf file uses the same environment variables. Our app has startup scripts that export the .env file into environment variables for either docker or a local run, which makes its way into the reference.conf.

https://gist.github.com/etspaceman/137f7f540af32bf106873813c830a699

You can get pretty fancy with the docker compose setup. If you tie in the create kinesis stream to the container startup command, you can chain the localstack creation along with your service.

Be sure to install awscli-local via PIP: https://github.com/localstack/awscli-local

@fernando-torterolo
Copy link
Contributor

I would want to add that if you don't want avoid verify-ssl in you local env you could add a kinesalite-server-crt-pem file and config your aws profile with this line

'ca_bundle = path/server-crt.pem'

also you should add a new entry host

echo "127.0.0.1 kinesalite" >> /etc/hosts

so instead @fommil suggestion

aws kinesis create-stream --stream-name foo --shard-count 3 --endpoint-url https://localhost:4567 --no-verify-ssl

you could create a stream

aws kinesis create-stream --stream-name foo --shard-count 1 --profile kinesalite --endpoint-url https://kinesalite:4567

certificate could be found here:
https://github.com/fernando-oktana/core-kinesalite
https://github.com/mhart/kinesalite

@markglh markglh mentioned this issue Dec 3, 2017
4 tasks
@markglh
Copy link
Contributor

markglh commented Dec 3, 2017

@markglh markglh closed this as completed Dec 3, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Development

No branches or pull requests

5 participants