Programmatic usage & Ansible integration

[mattvonrocketstein]

I'm interested in building an ansible module for sceptre. This requires me to work with sceptre's API, because I won't benefit from my usage of custom sceptre resolvers if I just naively grab the config yaml from the file system.

Here's some working code I came up with, but I noticed that the imports would be broken with the current development version of sceptre, and the docs specifically mention not to directly use sceptre.cli.

import sceptre
assert sceptre.__version__=='1.3.4'
env = sceptre.cli.get_env('/sceptre/root', '/env_name', {})
stack = env.stacks['stack_name']

# gives back custom `stack_name` value or correctly computes default in the usual sceptre way
print stack.external_name; 

# gives back fully sceptre-resolved python dictionary
print stack.template.sceptre_user_data 

# gives back cloudformation outputs
print stack.describe_outputs()

So where did get_env() go in current master, and what is the best way to accomplish this in a way that will be stable across many sceptre versions in the recent past and moving forward? If I can get some guidance on this I'll follow up with a pull request to the docs.. I think what I'm trying to do is the first thing everyone who is interested in the API will want to do.

[stolendog]

there is ansible cloudformation module which I haven't tried http://docs.ansible.com/ansible/latest/modules/cloudformation_module.html
but doesn't it do the same work as sceptre does? it looks like it has stacks configs, supports jinja template language, tags and more?

[mattvonrocketstein]

I'm not looking to reimplement sceptre in Ansible (even though that would probably be possible); we use Ansible ultimately for CM and not IaC. That said there are still opportunities for integration, and to keep vars DRY I need to build a bridge between the Ansible runtime, sceptre inputs, cloudformation outputs.

Sceptre already shines at basically being a replacement client/UX for cloudformation, insists on project layouts that are scalable and easy to understand, and provides a bunch of niceties like the hooks/custom resolvers. Cloudforming with Ansible directly would trash a lot of those good things by default, removing the powerful conventions that parameter/template files must be positioned in predictable locations, and especially the client UX, because Ansible cannot stream stdout in realtime.

[njsnx]

@mattvonrocketstein current master branch is the WIP for Sceptre V2. If you are looking to work with the current live version, take a look at the v1 branch.

[mattvonrocketstein]

@nmyster do we think that the v1/v2 split means my previous question has no answer? To paraphrase: what is the best way to accomplish [basic programmatic usage] in a way that will be stable across many sceptre versions in the recent past and moving forward?

[ngfgrant]

@mattvonrocketstein V2 will unfortunately introduce breaking changes. If you want to have your code work for previous versions then it is probably best to write something branching off v1 and once version 2 is complete amend as appropriate for v2.

This is something we would probably include int the contrib directory in v1.

[mattvonrocketstein]

I'm still interested in the question of basic programmatic usage, but I'm more familiar at this point with writing Sceptre hooks, and rather than driving Sceptre from Ansible I can see it would possible to drive Ansible from Sceptre.

Here's a few more possible integrations I'm brain storming about. I have code for much of this, I'm now looking for feedback about whether I ultimately might need to fork, or if there is interest in mainstreaming it. Because from sceptre import contrib doesn't actually work, contributing upstream there might not accomplish much for me over a fork, and what to do might also depend on the time table for v2 and whether it's currently considered stable enough for me and my org to try to work with / contribute to directly.

1. Pulling Sceptre Stack Outputs into Ansible. This is possible by just capturing stdout from sceptre CLI sceptre describe-stack-outputs, loading the YAML into ansible's runtime with |from_yaml, then using |json_query to drill down. It feels kind of gross, but might be smart if the CLI is planned to be completely stable across v1 and v2. If Sceptre exports an IP address like this, Ansible can add it to inventory dynamically and begin working against the sceptre-created host. Downside is this doesn't help me reuse for example a custom Sceptre SSM resolver. (I just saw that committed to contrib a few hours ago, and I wrote that myself already too 😄 )

2. Ansible as a Sceptre Hook. . Inside a stack config, things would look something like this:

# sceptre/config/env/stack.yaml

hooks:
  after_update:
    - !provision ansible

provisioners:
  ansible:
    playbook: ansible-playbook.yml
    host: !stack_output this_env/this_stack::HostIP
    vars:
      dns_name: !stack_output this_env/this_stack::HostDNS

sceptre_user_data:
  sceptre_var1:  !stack_output other_env/other_stack::ExportName

Here the familiar (and fully rendered) contents of Sceptre's stack_config and environment_config variables are also available to Ansible under the same variable names, and Ansible's runtime will have access to the results from all your custom sceptre resolvers. Given the config above, Ansible would be happy to render a template like {{stack_config.sceptre_user_data.sceptre_var1}} for you.

3. Ansible Embedded directly in Sceptre

Many Ansible modules by themselves could be useful directly as sceptre hooks, but to avoid making a hook for each it's possible to just allow Sceptre to use embedded Ansible. In that scenario, things could look something like this:

# sceptre/config/env/stack.yaml

hooks:
  before_update:
    - !inlined_ansible make_slack_noise
  after_update:
    - !inlined_ansible make_newrelic_noise

make_newrelic_noise:
  # https://docs.ansible.com/ansible/devel/modules/newrelic_deployment_module.html#examples
  - newrelic_deployment:
      token: !ssm /my/token/path
      app_name: myapp
      user: ansible
      revision: '1.0'

make_slack_noise:
  # https://docs.ansible.com/ansible/2.5/modules/slack_module.html
  - slack:
      token: !ssm /my/token/path
      msg: 'finished sceptre update'
      channel: #infrastructure
      username: ansible

[ngfgrant]

Definitely happy for this to progress but it's not something I'll be able to help with it the near-mid term. I think it would be reasonable to introduce this as a new feature or plugin somewhere in post v2.0 release if there is enough interest etc.

Re from sceptre import contrib. If you'd created your directory here as a python module with a __init__.py then I doubt the import will work. If you are looking to get this into v1 then contrib/ would be the way to get it into the mainstream as it is probably not something we'd support given that the main focus of development will be on v2.

Regarding version 2, currently in master, it is probably stable enough to experiment with but not to use in production. As it stands V2 is waiting for a merge to rename environments, a refactor of stack config dependencies and then it's mainly a documentation update to do. V2 is likely to come together over the next few weeks.

[ngfgrant]

@mattvonrocketstein has your thinking evolved any more on this as time has gone on? V2 is now out and provides a cleaner api if you are wanting to look at this from that angle again?

[mattvonrocketstein]

I still have code for a lot of what I described, and I'd still like to contribute upstream, but it's probably too early for me to move to v2. Maybe the question now becomes, what is the best-practice way to interact with sceptre programmatically in v2, and can I backport at least a compatible interface to v1 (even if the implementation is very different)?

[Vince-Cercury]

Has this progressed?

I'm interested in the ability to create ephemeral infrastructure without having to create a new folder and files.

So having a folder in config, something like 'dev-like'. And Ansible or Jenkins to take a copy of that and find/replace some values and folder name.
So that I can create "vince-dev-env"
Just for testing a feature branch, I wouldn't want to create a folder by hand and go replace all values. And I'm not interested in putting that in git.
However for static long lived environment, sceptre folders pushed in Git is still interesting as it can assist with following GitOps principle (where git is a mirror of what is on AWS account)

[ngfgrant]

Sceptre can be used programmatically already.

V3 which is still a WIP separates all file handling from core sceptre functionality. Eg in v3 sceptre core instantiates with an object and that can be from file system or created manually or programmatically. There is no date for go live with v3