Citizen

A private fork of outsideris/citizen

A Private Terraform Module and Terraform Provider registry in the early stages of development.

Requirements

• Node.js 10+

• HTTPS - Terraform module registry only support HTTPS.

citizen server

To launch the registry server

$ ./citizen server

It will be launched at http://localhost:3000. You can check it at http://localhost:3000/health.

Because Terraform CLI works with only HTTPS server, you should set up HTTPS in front of the registry server.

If you want to test it at local, you need a tool which provides HTTPS like ngrok.

Environment variables:

• CITIZEN_DATABASE: Backend provider for registry metadata. Set to mongodb to use MongoDB. Leaving unset will use local nedb file.

• CITIZEN_MONGO_DB_URI: MongoDB database URI if using MongoDB backend. URI format is mongodb://username:password@host:port/database?options…​. Default is mongodb://localhost:27017/citizen

• CITIZEN_DB_DIR: A directory to save database file if using local backend storage. The default is data directory in a current working directory (absolute/relative path can be used).

• CITIZEN_STORAGE : Storage type to store module files. You can use file or s3 type.

• CITIZEN_STORAGE_PATH: A directory to save module files only if CITIZEN_STORAGE is file (absolute/relative path can be used).

• CITIZEN_AWS_S3_BUCKET`: A S3 bucket to save module files only if CITIZEN_STORAGE is s3.

• AWS_ACCESS_KEY_ID: Your AWS access key only if CITIZEN_STORAGE is s3.

• AWS_SECRET_ACCESS_KEY: Your AWS secret access key only if CITIZEN_STORAGE is s3.

Storage

File Storage

AWS (s3)

Source Reference

Source Reference will not directly save to any file system. Instead, it will rely on an assumption that the CITIZEN_MODULE_SOURCE environment variable is a valid GitHub repository, and a release was created matching the version parameter of the publish command. Using that information, it will build a URL for a tar.gz file attached to that release.

Example

In GitHub, a new release for version 0.1.0 is created. That version is then published to the citizen server.

$ CITIZEN_ADDR=https://registry.example.com \
  CITIZEN_MODULE_OWNER=dev-team\
  CITIZEN_MODULE_SOURCE=https://github.com/dev-team/terraform-aws-alb \
  citizen publish module dev-team alb aws 0.1.0

When version 0.1.0 is requested for download, citizen responds with the following URL:

https://github.com/dev-team/terraform-aws-alb/archive/0.1.0.tar.gz

MongoDB Authentication

There are two possible authentication methods.

Option 1: Standard URI

To use a standard mongodb URI connection string, simply set the CITIZEN_MONGO_DB_URI.

Option 2: Auth Inside Options

To connect to mongoDB with the auth block inside options, multiple environment variables are needed.

• CITIZEN_MONGO_DB_NAME: default terraform-registry

• CITIZEN_MONGO_DB_HOST: defualt localhost

• CITIZEN_MONGO_DB_PORT: default 27017

• CITIZEN_MONGO_DB_OPTIONS

• CITIZEN_MONGO_DB_USER

• CITIZEN_MONGO_DB_PASSWORD

This will build a build a mongoose connection with one of four structures, based on the variables provided.

URI: mongodb://$CITIZEN_MONGO_DB_HOST:$CITIZEN_MONGO_DB_PORT/$CITIZEN_MONGO_DB_NAME
OPTIONS: {
  useUnifiedTopology: true,
  useNewUrlParser: true,
  useFindAndModify: false
}

URI: mongodb://$CITIZEN_MONGO_DB_HOST:$CITIZEN_MONGO_DB_PORT/$CITIZEN_MONGO_DB_NAME?$CITIZEN_MONGO_DB_OPTIONS
OPTIONS: {
  useUnifiedTopology: true,
  useNewUrlParser: true,
  useFindAndModify: false
}

URI: mongodb://$CITIZEN_MONGO_DB_HOST:$CITIZEN_MONGO_DB_PORT/$CITIZEN_MONGO_DB_NAME
OPTIONS: {
  useUnifiedTopology: true,
  useNewUrlParser: true,
  useFindAndModify: false,
  auth: {
    user: $CITIZEN_MONGO_DB_USER,
    password: $CITIZEN_MONGO_DB_PASSWORD
  }
}

URI: mongodb://$CITIZEN_MONGO_DB_HOST:$CITIZEN_MONGO_DB_PORT/$CITIZEN_MONGO_DB_NAME?$CITIZEN_MONGO_DB_OPTIONS
OPTIONS: {
  useUnifiedTopology: true,
  useNewUrlParser: true,
  useFindAndModify: false,
  auth: {
    user: $CITIZEN_MONGO_DB_USER,
    password: $CITIZEN_MONGO_DB_PASSWORD
  }
}

citizen publish module

Since official Terraform Module Registry is integrated with GitHub, users can publish terraform modules if they just push it on GitHub.

Citizen provides a special command to publish a module onto citizen registry server instead integrating GitHub.

In a module directory, you can publish your terraform module via a command below:

$ ./citizen publish module <namespace> <name> <provider> <version>

You should set CITIZEN_ADDR as citizen registry server address which you will publish your modules to. e.g. https://registry.example.com.

Additional module properties can be set via optional environment variables - CITIZEN_MODULE_OWNER: Set the owner property - CITIZEN_MODULE_SOURCE: Set the source property, generally the URL of the source code repository

Examples

If you have ALB module in ./alb directory and your registry server is launched at https://registry.example.com, you run below command in ./alb directory to publish ALB module.

$ CITIZEN_ADDR=https://registry.example.com \
  citizen publish module dev-team alb aws 0.1.0

With owner and source set

$ CITIZEN_ADDR=https://registry.example.com \
  CITIZEN_MODULE_OWNER=dev-team\
  CITIZEN_MODULE_SOURCE=https://github.com/dev-team/terraform-aws-alb \
  citizen publish module dev-team alb aws 0.1.0

Then, you can define it in your terraform file like this:

module "alb" {
  source = "registry.example.com/dev-team/alb/aws"
  version = "0.1.0"
}

citizen publish provider

Citizen provides a special command to publish providers onto citizen.

You must first build, package and gpg sign your provider, citizen expects following files in the provider location:

• terraform-provider-<type>_<version>_<os>_<os>.zip (one per os/arch combination)

• terraform-provider-<type>_<version>_SHA256SUMS

• terraform-provider-<type>_<version>_SHA256SUMS.sig

Where <type> is a name of the provider and version is a provider version in the MAJOR.MINOR.PATCH version format.

SHA256 sums file can be generated using following shell command (linux/macos):

shasum -a *.zip > terraform-provider-<type>_<version>_SHA256SUMS

SHA256.sig file is a GPG signature of the SHA256SUMS file, generate using following command:

gpg --detach-sign terraform-provider-<type>_<version>_SHA256SUMS

• You need to publish your GPG public key to terraform registry otherwise terraform will refuse to install providers. You can publish the key using citizen publish publisher command documented below.

In a provider directory, you can publish your terraform provider via a command below:

$ ./citizen publish provider <namespace> <type> <version>

You should set CITIZEN_ADDR as citizen registry server address which you will publish your modules to. e.g. https://registry.example.com.

Examples

If you have ALB provider in ./utilities directory and your registry server is launched at https://registry.example.com, you run below command in ./utilities directory to publish utilities provider.

$ CITIZEN_ADDR=https://registry.example.com \
  citizen publish provider dev-team utilities 0.1.0

Then, you can define it in your terraform file like this:

provider "utilities" {
}

terraform {
  required_providers {
    utilities = {
      source = "registry.example.com/dev-team/utilities"
      version = "0.1.0"
    }
  }
}

citizen publish publisher

Citizen provider registry requires to have at least one trusted provider publisher. citizen publisher publisher command uploads a public GPG key from local GPG store using gpg command.

Find GPG key id you want to publisy using gpg --list-keys command and extract public key (long text in hex format e.g. CE1E75EC86B9F2). Then run citizen command to publish the key:

$ ./citizen publish publisher CE1E75EC86B9F2

You should set CITIZEN_ADDR as citizen registry server address which you will publish your modules to. e.g. https://registry.example.com.

Development

Set environment variables, see above.

$ ./bin/citizen server
$ ./bin/citizen publish

Test

Set at least a storage path and the s3 bucket name variables for the tests to succeed. You need to be able to access the bucket, so you probably want to have an active aws or aws-vault profile.

Run mongodb first like:

$ docker run --rm -p 27017:27017 --name mongo mongo

Run the tests:

$ npm test

Run the tests with the environment variables prefixed:

$ CITIZEN_STORAGE_PATH=storage CITIZEN_AWS_S3_BUCKET=terraform-registry-modules npm test

Build distributions

$ npm run build

Under dist/, citizen binaries for linux, darwin and windows made.