Add timeout support for vault client

[luanphantiki]

ref:

• Kubernetes #9
• [Feature] Timeout parameter support when snapshot a huge raft cluster #20

[luanphantiki]

@Lucretius Please review it, thanks!

[luanphantiki]

@devops-42 , please consider to test this feature, it's not harmful for your production, but you should test it at UAT if possible.

[devops-42]

@luanphantiki

unfortunately it seems not to work. Steps which I've done:

• cloned your repo
• switched into branch snapshot-timeout-support
• built the new Docker image using docker build -t imagename .
• pushed the new image into our registry
• added timeout setting in config file:

{
   "addr":"http://vault-leader:8200",
   "retain":72,
   "frequency":"3600s",
   "timeout":"300s",
   "role_id": "***",
   "secret_id":"***",
   "aws_storage":{
      "access_key_id":"***",
      "secret_access_key":"***",
      "s3_region":"us-east-1",
      "s3_bucket":"***,
      "s3_endpoint":***",
      "s3_force_path_style":true
   }
}

• performed pod re-deployment

The pod still exits after 60 secs with the well-known error:

2021/10/27 12:35:29 Reading configuration...
2021/10/27 12:36:30 Unable to generate snapshot context deadline exceeded (Client.Timeout or context cancellation while reading body)

I already logged into the pod and can confirm, that the timeout value is present in the mounted config file. I also grepped against the binary of the vault_raft_snapshot_agent to check whether the right image has been pull from our registry:

grep timeout vault_raft_snapshot_agent 
[...]
json:"timeout,ommitempty"
[...]

Maybe I missed something?

[luanphantiki]

@devops-42 : there was a typo on my code, fixed, can you please pull the latest and try again ?

[devops-42]

@luanphantiki

Tested again. The log file now shows that the timeout parameter could parsed correctly, but the task still aborts after 60s:

2021/10/28 08:13:05 Reading configuration...
2021/10/28 08:13:05 Vault client timeout has been set to 5m0s
2021/10/28 08:14:05 Unable to generate snapshot context deadline exceeded (Client.Timeout or context cancellation while reading body)

[luanphantiki]

@devops-42 thanks, I might be missed somethings, will update it soon.

[luanphantiki]

@devops-42: please test the latest commit once have a chance, thanks!

[devops-42]

Hey @luanphantiki

finally :) Thumbs up, good work!

2021/10/29 13:58:25 Reading configuration...
2021/10/29 13:58:25 Vault http client timeout has been set to 5m0s
2021/10/29 14:01:43 Successfully created aws snapshot to ....

Thanks

[luanphantiki]

Alright, thanks for the confirmation @devops-42 ;)

@Lucretius The new feature seems to work, please review it, thanks!

[kirrmann]

Can we merge this is? For me snapshot creation didn't work at all. Sometimes got infamous context deadline error, but other times the snapshots were created with only 0 bytes. I just tried this PR to increase the timeout and it does work for me.

[druesendieb]

@Lucretius The repository is stale since 2021 - are you planning on updating or enhancing it with PR's like this at all?

[Argelbargel]

HI @druesendieb: see #28 (comment)

This repo is not actively maintained any more. There are some forks with added features including the timeout for vault client (e.g. https://github.com/Boostport/vault_raft_snapshot_agent). I've created a fork too (https://github.com/Argelbargel/vault-raft-snapshot-agent) which includes most of the open prs and some further enhancements. Perhaps one of those might work for you?

[Lucretius]

Hello,

No, I built this years ago when I was working at a place that used Vault, which I no longer do. There are others who have forked this repo and have enhanced it. Feel free to do so as well. I will update this repo to reflect its status.