Skip to content

Commit

Permalink
Vault aws auth (#9)
Browse files Browse the repository at this point in the history
* add additional auth-methods supported by vault-api
* refactor config-parser into its own package; resolve relative paths in configuration-files while parsing
* change AWS-env-vars and add AWS_SESSION_TOKEN
  • Loading branch information
Argelbargel authored Sep 15, 2023
1 parent 50ee232 commit 38780f6
Show file tree
Hide file tree
Showing 39 changed files with 1,844 additions and 940 deletions.
171 changes: 161 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,11 @@ uploaders:
### Environment variables
Vault Raft Snapshot Agent supports configuration with environment variables. For some common options there are shortcuts defined:
- `VAULT_ADDR` configures the url to the vault-server (same as `vault.url`)
- `AWS_ACCESS_KEY_ID` configures the access key for the AWS uploader (same as `uploaders.aws.credentials.key`)
- `SECRET_ACCESS_KEY` configures the access secret for the AWS uploader (same as `uploaders.aws.credentials.secret`)
- `AWS_ACCESS_KEY_ID` configures the access key for the AWS uploader (same as `uploaders.aws.credentials.key`) and AWS EC2 authentication
- `AWS_SECRET_ACCESS_KEY` configures the access secret for the AWS uploader (same as `uploaders.aws.credentials.secret`) and AWS EC2 authentication
- `AWS_SESSION_TOKEN` configures the session-token for AWS EC2 authentication
- `AWS_SHARED_CREDENTIALS_FILE` configures AWS EC2 authentication from a file


Any other option can be set by prefixing `VRSA_` to the uppercased path to the key and replacing `.` with `_`. For example `VRSA_SNAPSHOTS_FREQUENCY=<value>` configures the snapshot-frequency and `VRSA_VAULT_AUTH_TOKEN=<value>` configures the token authentication for vault.

Expand Down Expand Up @@ -146,20 +149,116 @@ An AppRole allows the snapshot agent to automatically rotate tokens to avoid lon
vault:
auth:
approle:
id: "<role-id>
role: "<role-id>"
secret: "<secret-id>"
```

##### Configuration options
- `id` **(required)** - specifies the role_id used to call the Vault API. See the authentication steps below
- `role` **(required)** - specifies the role_id used to call the Vault API. See the authentication steps below
- `secret` **(required)** - specifies the secret_id used to call the Vault API
- `path` *(default: approle)* - specifies the backend-name used to select the login-endpoint (`auth/<path>/login`)

To allow the App-Role access to the snapshots you should run the following commands on your vault-cluster:
```
vault write auth/approle/role/snapshot token_policies="snapshots"
vault read auth/approle/role/snapshot/<role-id>
vault write -f auth/approle/role/snapshot/<secret-id>
vault write auth/<path>/role/snapshot token_policies=snapshots
vault read auth/<path>/role/snapshot/<role-id>
vault write -f auth/<path>/role/snapshot/<secret-id>
```

#### AWS authentication

Uses AWS for authentication (see the [Vault docs](https://developer.hashicorp.com/vault/docs/auth/aws)).


##### Minimal configuration
```
vault:
auth:
aws:
role: "<role>"
```

##### Configuration options
- `role` **(required)** - specifies the role used to call the Vault API. See the authentication steps below
- `ec2Nonce` - enables EC2 authentication and sets the required nonce
- `ec2SignatureType` *(default: pkcs7)* - changes the signature-type for EC2 authentication; valid values are `identity`, `pkcs7` and `rs2048`
- `iamServerIdHeader` - specifies the server-id-header when using IAM authtype
- `region` - specifies the aws region to use
- `path` *(default: aws)* - specifies the backend-name used to select the login-endpoint (`auth/<path>/login`)

By default AWS authentication uses the iam authentication type unless `ec2Nonce` is set. The credentials for IAM authentication must be provided via environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` or `AWS_SHARED_CREDENTIALS_FILE`). While relative paths normally are resolved relative to the configuration-file, `AWS_SHARED_CREDENTIALS_FILE` must be specified as an absolute path.

To allow the access to the snapshots you should run the following commands on your vault-cluster:
```
# for ec2 authentication
vault write auth/<path>/role/<role> auth_type=ec2 bound_ami_id=<ami-id> policies=snapshots max_ttl=500h
# for iam authentication
vault write auth/<path>/role/<role> auth_type=iam bound_iam_principal_arn=<princial-arn> policies=snapshots max_ttl=500h
```

#### Azure authentication

Authentication using Azure (see [the Vault docs](https://developer.hashicorp.com/vault/docs/auth/azure)).


##### Minimal configuration
```
vault:
auth:
azure:
role: "<role-id>"
```

##### Configuration options
- `role` **(required)** - specifies the role used to call the Vault API. See the authentication steps below
- `resource` - optional azure resource
- `path` *(default: azure)* - specifies the backend-name used to select the login-endpoint (`auth/<path>/login`)

To allow the access to the snapshots you should run the following commands on your vault-cluster:
```
vault write auth/<path>/role/<role> \
policies="snapshots" \
bound_subscription_ids=<subscription-ids> \
bound_resource_groups=<resource-group>
```

#### Google Cloud authentication

Authentication using Google Cloud GCE or IAM authentication (see [the Vault docs](https://developer.hashicorp.com/vault/docs/auth/gcp)).


##### Minimal configuration
```
vault:
auth:
gcp:
role: "<role>"
```

##### Configuration options
- `role` **(required)** - specifies the role used to call the Vault API. See the authentication steps below
- `serviceAccountEmail` - activates iam authentication and s specifies the service-account to use
- `path` *(default: gcp)* - specifies the backend-name used to select the login-endpoint (`auth/<path>/login`)

By default Google Cloud authentication uses the gce authentication type unless `serviceAccountEmail` is set.

To allow the access to the snapshots you should run the following commands on your vault-cluster:
```
# for iam authentication type
vault write auth/<path>/role/<role> \
type="iam" \
policies="snapshots" \
bound_service_accounts="<service-account-email>"
# for gce authentication type
vault write auth/<path>/role/<role> \
type="gce" \
policies="snapshots" \
bound_projects="<projects>" \
bound_zones="<zones>" \
bound_labels="<labels>" \
bound_service_accounts="<service-acoount-email>"
```


Expand All @@ -177,14 +276,39 @@ vault:
##### Configuration options
- `role` **(required)** - specifies vault k8s auth role
- `path` *(default: kubernetes)* - specifies the backend-name used to select the login-endpoint (`auth/<path>/login`)
- `jwtPath` *(default: /var/run/secrets/kubernetes.io/serviceaccount/token)* - specifies the path to the file with the JWT-Token for the kubernetes Service-Account
- `jwtPath` *(default: /var/run/secrets/kubernetes.io/serviceaccount/token)* - specifies the path to the file with the JWT-Token for the kubernetes service-account. You may specify the path relative to the location of the configuration file.

To allow kubernetes access to the snapshots you should run the following commands on your vault-cluster:
```
kubectl -n <your-vault-namespace> exec -it <vault-pod-name> -- vault write auth/<kubernetes.path>/role/<kubernetes.role> bound_service_account_names=* bound_service_account_namespaces=<namespace of your vault-raft-snapshot-agent-pod> policies=snapshots ttl=24h
kubectl -n <your-vault-namespace> exec -it <vault-pod-name> -- vault write auth/<path>/role/<kubernetes.role> bound_service_account_names=* bound_service_account_namespaces=<namespace of your vault-raft-snapshot-agent-pod> policies=snapshots ttl=24h
```
Depending on your setup you can restrict access to specific service-account-names and/or namespaces.

#### LDAP authentication
Authentication using LDAP (see [the Vault docs](https://developer.hashicorp.com/vault/docs/auth/ldap)).

##### Minimal configuration
```
vault:
auth:
ldap:
role: "test"
```

##### Configuration options
- `username` **(required)** - the username
- `password` **(required)** - the password
- `path` *(default: ldap)* - specifies the backend-name used to select the login-endpoint (`auth/<path>/login`)

To allow access to the snapshots you should run the following commands on your vault-cluster:
```
# allow access for a specific user
vault write auth/<path>/users/<username> policies=snapshot
# allow access based on group
vault write auth/<path>/groups/<group> policies=snapshots
```


#### Token authentication
##### Minimal configuration
Expand All @@ -198,6 +322,32 @@ vault:
- `token` **(required)** - specifies the token used to login


#### User and Password authentication
Authentication using username and password (see [the Vault docs](https://developer.hashicorp.com/vault/docs/auth/userpass)).

##### Minimal configuration
```
vault:
auth:
userpass:
username: "<username>"
password: "<password>"
```

##### Configuration options
- `username` **(required)** - the username
- `password` **(required)** - the password
- `path` *(default: userpass)* - specifies the backend-name used to select the login-endpoint (`auth/<path>/login`)

To allow access to the snapshots you should run the following commands on your vault-cluster:

```
vault write auth/<path>/users/<username> \
password=<password> \
policies=snapshots
```


### Snapshot configuration
```
snapshots:
Expand Down Expand Up @@ -260,7 +410,7 @@ uploaders:
secret: <secret>
```
- `key` **(required)** - specifies the access key. It's recommended to use the standard `AWS_ACCESS_KEY_ID` env var, though
- `secret` **(required)** - specifies the secret It's recommended to use the standard `SECRET_ACCESS_KEY` env var, though
- `secret` **(required)** - specifies the secret It's recommended to use the standard `AWS_SECRET_ACCESS_KEY` env var, though


#### Azure Storage
Expand All @@ -284,3 +434,4 @@ uploaders:
## Contributors
- Vault Raft Snapshot Agent was originally developed by [@Lucretius](https://github.com/Lucretius/vault_raft_snapshot_agent/)
- This build contains improvements done by [@Boostport](https://github.com/Boostport/vault_raft_snapshot_agent/)
- support for additional authentication methods based on code from [@alexeiser](https://github.com/Lucretius/vault_raft_snapshot_agent/pull/25)
21 changes: 11 additions & 10 deletions cmd/vault-raft-snapshot-agent/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ Usage:
The flags are:
-v, -version
Prints version information and exits
-v, -version
Prints version information and exits
The options are:
Expand Down Expand Up @@ -44,6 +44,12 @@ import (
var Version = "development"
var Platform = "linux/amd64"

var snapshotterOptions internal.SnapshotterOptions = internal.SnapshotterOptions{
ConfigFileName: "snapshots",
ConfigFileSearchPaths: []string{"/etc/vault.d/", "."},
EnvPrefix: "VRSA",
}

type quietBoolFlag struct {
cli.BoolFlag
}
Expand Down Expand Up @@ -99,17 +105,12 @@ Options:
}

func startSnapshotter(configFile cli.Path) {
config, err := internal.ReadConfig(configFile)
if err != nil {
log.Fatalf("Could not read configuration: %s\n", err)
}

snapshotter, err := internal.CreateSnapshotter(config)
snapshotterOptions.ConfigFilePath = configFile
snapshotter, err := internal.CreateSnapshotter(snapshotterOptions)
if err != nil {
log.Fatalf("Cannot instantiate snapshotter: %s\n", err)
log.Fatalf("Cannot create snapshotter: %s\n", err)
}

internal.WatchConfigAndReconfigure(snapshotter)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()

Expand Down
55 changes: 35 additions & 20 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,20 @@ require github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0
// GCP-Uploader
require (
cloud.google.com/go/storage v1.31.0
google.golang.org/api v0.129.0
google.golang.org/api v0.140.0
)

// Vault
require github.com/hashicorp/vault/api v1.9.2
require (
github.com/hashicorp/vault/api v1.10.0
github.com/hashicorp/vault/api/auth/approle v0.5.0
github.com/hashicorp/vault/api/auth/aws v0.5.0
github.com/hashicorp/vault/api/auth/azure v0.5.0
github.com/hashicorp/vault/api/auth/gcp v0.5.0
github.com/hashicorp/vault/api/auth/kubernetes v0.5.0
github.com/hashicorp/vault/api/auth/ldap v0.5.0
github.com/hashicorp/vault/api/auth/userpass v0.5.0
)

// helpers
require (
Expand All @@ -41,18 +50,19 @@ require (
)

// ensure up-to-date versions (because of known vulnerabilities etc.)
require golang.org/x/crypto v0.12.0 // indirect
require golang.org/x/crypto v0.13.0 // indirect

// testing
require github.com/stretchr/testify v1.8.4

require (
cloud.google.com/go v0.110.2 // indirect
cloud.google.com/go/compute v1.19.3 // indirect
cloud.google.com/go v0.110.7 // indirect
cloud.google.com/go/compute v1.23.0 // indirect
cloud.google.com/go/compute/metadata v0.2.3 // indirect
cloud.google.com/go/iam v1.1.0 // indirect
cloud.google.com/go/iam v1.1.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.7.1 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.3.0 // indirect
github.com/aws/aws-sdk-go v1.45.8 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.13 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.11 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.41 // indirect
Expand All @@ -78,18 +88,21 @@ require (
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/s2a-go v0.1.4 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/google/uuid v1.3.1 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.2.5 // indirect
github.com/googleapis/gax-go/v2 v2.11.0 // indirect
github.com/googleapis/gax-go/v2 v2.12.0 // indirect
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/hashicorp/go-hclog v1.5.0 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-retryablehttp v0.7.2 // indirect
github.com/hashicorp/go-retryablehttp v0.7.4 // indirect
github.com/hashicorp/go-rootcerts v1.0.2 // indirect
github.com/hashicorp/go-secure-stdlib/awsutil v0.2.3 // indirect
github.com/hashicorp/go-secure-stdlib/parseutil v0.1.7 // indirect
github.com/hashicorp/go-secure-stdlib/strutil v0.1.2 // indirect
github.com/hashicorp/go-sockaddr v1.0.2 // indirect
github.com/hashicorp/go-sockaddr v1.0.5 // indirect
github.com/hashicorp/go-uuid v1.0.3 // indirect
github.com/hashicorp/hcl v1.0.1-vault-5 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/leodido/go-urn v1.2.4 // indirect
Expand All @@ -98,6 +111,7 @@ require (
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
github.com/pelletier/go-toml/v2 v2.0.8 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/ryanuber/go-glob v1.0.0 // indirect
Expand All @@ -107,17 +121,18 @@ require (
github.com/subosito/gotenv v1.4.2 // indirect
github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
go.opencensus.io v0.24.0 // indirect
golang.org/x/net v0.11.0 // indirect
golang.org/x/oauth2 v0.9.0 // indirect
golang.org/x/sys v0.11.0 // indirect
golang.org/x/text v0.12.0 // indirect
golang.org/x/net v0.15.0 // indirect
golang.org/x/oauth2 v0.12.0 // indirect
golang.org/x/sync v0.3.0 // indirect
golang.org/x/sys v0.12.0 // indirect
golang.org/x/text v0.13.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230530153820-e85fd2cbaebc // indirect
google.golang.org/grpc v1.56.1 // indirect
google.golang.org/appengine v1.6.8 // indirect
google.golang.org/genproto v0.0.0-20230913181813-007df8e322eb // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20230913181813-007df8e322eb // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230913181813-007df8e322eb // indirect
google.golang.org/grpc v1.58.0 // indirect
google.golang.org/protobuf v1.31.0 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
Expand Down
Loading

0 comments on commit 38780f6

Please sign in to comment.