Email Retries and Log file inclusion in the attachment

[jayasimha-raghavan-unskript]

Description

Fixes EN-5684

This PR Brings in support for

• Email retrial (Max 5 times with exponential backoff support)
• Inclusion of unskript_ctl.log file in the email archive
• Improved compression with J option to help reduce the email attachment file size

Testing

Please describe the tests that you ran to verify your changes. Please summarize what did you test and what needs to be tested e.g. deployed and tested helm chart locally.

Positive Case

unskript@awesome-awesome-runbooks-0:~$ unskript-ctl.sh run check --type vault --report
Running: 100%|████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  3.09it/s]

╒══════════════════════════╤══════════╤════════════════╤═════════╕
│ Checks Name              │ Result   │   Failed Count │ Error   │
╞══════════════════════════╪══════════╪════════════════╪═════════╡
│ Get Vault service health │  PASS    │              0 │ N/A     │
╘══════════════════════════╧══════════╧════════════════╧═════════╛


Fetching logs...
Error: 'checks->diagnostics' section not found in '/etc/unskript/unskript_ctl_config.yaml'.
Skipping Diagnostics: No diagnostic command found. You can define them in the YAML configuration file
Uploading run artifacts to S3...
2024-09-10 19:37:02,776 - UnskriptCtlLogger - INFO - Slack Message was sent successfully!
2024-09-10 19:37:02,776 - UnskriptCtlLogger - INFO - Starting call to 'unskript_ctl_notification.Notification._send_email.<locals>._do_send_email', this is the 1st time calling it.
INFO:botocore.credentials:Found credentials in environment variables.
2024-09-10 19:37:03,700 - UnskriptCtlLogger - INFO - Email notification sent to [email protected]
2024-09-10 19:37:03,702 - UnskriptCtlLogger - INFO - Successfully sent Email notification via AWS SES.

Negative Case

unskript@awesome-awesome-runbooks-0:~$ unskript-ctl.sh run check --type vault --report
Running: 100%|████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  3.11it/s]

╒══════════════════════════╤══════════╤════════════════╤═════════╕
│ Checks Name              │ Result   │   Failed Count │ Error   │
╞══════════════════════════╪══════════╪════════════════╪═════════╡
│ Get Vault service health │  PASS    │              0 │ N/A     │
╘══════════════════════════╧══════════╧════════════════╧═════════╛


Fetching logs...
Error: 'checks->diagnostics' section not found in '/etc/unskript/unskript_ctl_config.yaml'.
Skipping Diagnostics: No diagnostic command found. You can define them in the YAML configuration file
Uploading run artifacts to S3...
2024-09-10 19:41:40,870 - UnskriptCtlLogger - INFO - Slack Message was sent successfully!
2024-09-10 19:41:40,871 - UnskriptCtlLogger - INFO - Starting call to 'unskript_ctl_notification.Notification._send_email.<locals>._do_send_email', this is the 1st time calling it.
2024-09-10 19:41:40,897 - UnskriptCtlLogger - INFO - Finished call to 'unskript_ctl_notification.Notification._send_email.<locals>._do_send_email' after 0.027(s), this was the 1st time calling it.
2024-09-10 19:41:44,898 - UnskriptCtlLogger - INFO - Starting call to 'unskript_ctl_notification.Notification._send_email.<locals>._do_send_email', this is the 2nd time calling it.
2024-09-10 19:41:44,922 - UnskriptCtlLogger - INFO - Finished call to 'unskript_ctl_notification.Notification._send_email.<locals>._do_send_email' after 4.051(s), this was the 2nd time calling it.
2024-09-10 19:41:48,923 - UnskriptCtlLogger - INFO - Starting call to 'unskript_ctl_notification.Notification._send_email.<locals>._do_send_email', this is the 3rd time calling it.
2024-09-10 19:41:48,951 - UnskriptCtlLogger - INFO - Finished call to 'unskript_ctl_notification.Notification._send_email.<locals>._do_send_email' after 8.080(s), this was the 3rd time calling it.
2024-09-10 19:41:52,951 - UnskriptCtlLogger - INFO - Starting call to 'unskript_ctl_notification.Notification._send_email.<locals>._do_send_email', this is the 4th time calling it.
2024-09-10 19:41:52,976 - UnskriptCtlLogger - INFO - Finished call to 'unskript_ctl_notification.Notification._send_email.<locals>._do_send_email' after 12.105(s), this was the 4th time calling it.
2024-09-10 19:42:00,977 - UnskriptCtlLogger - INFO - Starting call to 'unskript_ctl_notification.Notification._send_email.<locals>._do_send_email', this is the 5th time calling it.
2024-09-10 19:42:01,025 - UnskriptCtlLogger - INFO - Finished call to 'unskript_ctl_notification.Notification._send_email.<locals>._do_send_email' after 20.154(s), this was the 5th time calling it.
2024-09-10 19:42:01,025 - UnskriptCtlLogger - INFO - Error sending email: RetryError[<Future at 0x7f507df39290 state=finished raised TypeError>]
unskript@awesome-awesome-runbooks-0:~$ 

Checklist:

• My changes generate no new warnings.
• I have added tests that prove my fix is effective or that my feature works.
• Any dependent changes have been merged and published.

Documentation

Make sure that you have documented corresponding changes in this repository.