Refactoring output after merge function

[jayasimha-raghavan-unskript]

Description

Please include a summary of the change, motivation and context.

• Refactored output_after_merge function

Testing

Please describe the tests that you ran to verify your changes. Please summarize what did you test and what needs to be tested e.g. deployed and tested helm chart locally.

Before Refactoring

unskript@lb-unskript-0:~$ unskript-ctl.sh run check --type k8s --script "/usr/local/bin/lb_reports.sh" 
WARNING:ZODB.FileStorage:Ignoring index for /var/unskript/snippets.db
Running: 100%|██████████████████████████████████████████████████████| 83/83 [03:09<00:00,  2.28s/it]
Traceback (most recent call last):
  File "/usr/local/bin/./unskript_ctl_main.py", line 772, in <module>
    main()
  File "/usr/local/bin/./unskript_ctl_main.py", line 749, in main
    uc.run_main(args=args, parser=parser)
  File "/usr/local/bin/./unskript_ctl_main.py", line 141, in run_main
    status_of_run = self._check.run(checks_list=check_list)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/unskript_ctl_run.py", line 131, in run
    self.display_check_result(checks_output=outputs)
  File "/usr/local/bin/unskript_ctl_run.py", line 184, in display_check_result
    checks_output = self.output_after_merging_checks(checks_output, self.check_uuids)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/bin/unskript_ctl_run.py", line 318, in output_after_merging_checks
    current_output.setdefault('objects', []).extend(output.get('objects', []))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'dict' object has no attribute 'extend'

After Refactoring

unskript@lb-unskript-0:~$ unskript-ctl.sh run check --type k8s --script "/usr/local/bin/lb_reports.sh" 
Running: 100%|██████████████████████████████████████████████████████| 83/83 [03:12<00:00,  2.32s/it]

╒════════════════════════════════════════════════════════╤══════════╤════════════════╤═════════╕
│ Checks Name                                            │ Result   │   Failed Count │ Error   │
╞════════════════════════════════════════════════════════╪══════════╪════════════════╪═════════╡
│ Get K8S Cluster Health                                 │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get all K8s Pods in Terminating State                  │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Detect K8s service crashes                             │  FAIL    │              1 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get all K8s Pods in CrashLoopBackOff State             │  FAIL    │              1 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Check K8s worker CPU Utilization                       │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get K8s services exceeding memory utilization          │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get Kubernetes Unbound PVCs                            │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get K8s get pending pods                               │  FAIL    │              8 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get Kubernetes Failed Deployments                      │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get Kubernetes PODS with high restart                  │  FAIL    │              2 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get All Evicted PODS From Namespace                    │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get K8s nodes disk and memory pressure                 │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get Kubernetes PODs in not Running State               │  FAIL    │             35 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Check expiry of K8s cluster certificate                │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get Kubernetes Error PODs from All Jobs                │  FAIL    │              9 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get K8S Service with no associated endpoints           │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Check K8s service PVC utilization                      │  FAIL    │              5 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get K8s offline nodes                                  │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Check the status of K8s CronJob pods                   │  FAIL    │              3 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get expiring secret certificates                       │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get Kubernetes Nodes that have insufficient resources  │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Check K8s services endpoint and SSL certificate health │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get K8S OOMKilled Pods                                 │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get Deployment Status                                  │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Get all K8s Pods in ImagePullBackOff State             │  PASS    │              0 │ N/A     │
├────────────────────────────────────────────────────────┼──────────┼────────────────┼─────────┤
│ Inconsistent entities in MongoDB and ElasticSearch     │  PASS    │              0 │ N/A     │
╘════════════════════════════════════════════════════════╧══════════╧════════════════╧═════════╛

k8s:Detect K8s service crashes
Failed Objects:
- error: Exception
  namespace: lightbeam
  pod: lightbeam-elasticsearch-master
  service: lightbeam-elasticsearch-master
  timestamp: '2024-06-13T16:01:31'

 
k8s:Get all K8s Pods in CrashLoopBackOff State
Failed Objects:
- container: lb-aws-s3-consumer
  namespace: lightbeam
  pod: lb-aws-s3-consumer-64b0fe23a7b68bb048cbf859-884b85db6-bdbcm

 
k8s:Get K8s get pending pods
Failed Objects:
- - lb-gdrive-consumer-650dc339c69518063b498b83-ff74c9579-jzs5s
  - lightbeam
- - lb-gdrive-producer-hist-64b0fd872b9727c0f515f03c-28638244-vh7fg
  - lightbeam
- - lb-gdrive-producer-hist-659d5c1dd0e141588f1cb042-28638244-rkzqd
  - lightbeam
- - lb-gdrive-producer-live-659d5c1dd0e141588f1cb042-28638244-2jqcc
  - lightbeam
- - lb-onedrive-producer-64df0d050910d18920ea0363-28638244-mhbq5
  - lightbeam
- - lb-outlook-producer-64b104e8618f3f25807a9111-28638244-b98px
  - lightbeam
- - lb-sharepoint-producer-64df0c9f7c5e4444d31bb75a-28638244-22wx6
  - lightbeam
- - lb-zendesksupport-reconcile-64b188a22b9727c0f515f167-28623v8788
  - lightbeam

 
k8s:Get Kubernetes PODS with high restart
Failed Objects:
- interval_time_to_check: 2024-06-12 16:04:39 UTC
  namespace: lightbeam
  pod: lb-aws-s3-consumer-64b0fe23a7b68bb048cbf859-884b85db6-bdbcm
  termination_time: 2024-06-13 16:01:47 UTC
- interval_time_to_check: 2024-06-12 16:04:39 UTC
  namespace: lightbeam
  pod: lightbeam-policy-consumer-7db8db557c-pm7c4
  termination_time: 2024-06-13 15:55:41 UTC

 
k8s:Get Kubernetes PODs in not Running State
Failed Objects:
- name: delta-reporting-1717455600-workflow-status-2998737332
  namespace: lightbeam
  status: Failed
- name: document-labeling-notification-1717286400-document-labeling-notification-1077564594
  namespace: lightbeam
  status: Failed
- name: document-labeling-notification-1717286400-document-labeling-notification-1547190831
  namespace: lightbeam
  status: Failed
- name: document-labeling-notification-1717286400-document-labeling-notification-3091025969
  namespace: lightbeam
  status: Failed
- name: document-labeling-notification-1717286400-document-labeling-notification-3695167348
  namespace: lightbeam
  status: Failed
- name: document-labeling-notification-1717891200-document-labeling-notification-1405828689
  namespace: lightbeam
  status: Failed
- name: document-labeling-notification-1717891200-document-labeling-notification-1539902546
  namespace: lightbeam
  status: Failed
- name: document-labeling-notification-1717891200-document-labeling-notification-2009970068
  namespace: lightbeam
  status: Failed
- name: document-labeling-notification-1717891200-document-labeling-notification-4156960847
  namespace: lightbeam
  status: Failed
- name: document-labeling-notification-1717891200-workflow-status-3133917375
  namespace: lightbeam
  status: Failed
- name: generate-delta-report-1717457400-workflow-status-1546557041
  namespace: lightbeam
  status: Failed
- name: generate-delta-report-1717457400-workflow-status-2150698420
  namespace: lightbeam
  status: Failed
- name: generate-delta-report-1717457400-workflow-status-2721903
  namespace: lightbeam
  status: Failed
- name: generate-delta-report-1717457400-workflow-status-3828062962
  namespace: lightbeam
  status: Failed
- name: generate-delta-report-1717457400-workflow-status-629681776
  namespace: lightbeam
  status: Failed
- name: lb-gdrive-consumer-650dc339c69518063b498b83-ff74c9579-jzs5s
  namespace: lightbeam
  status: Pending
- name: lb-zendesksupport-reconcile-64b188a22b9727c0f515f167-28623v8788
  namespace: lightbeam
  status: Pending
- name: lightbeam-bootstrap-f2p6h
  namespace: lightbeam
  status: Failed
- name: scanner-64cd98e40910d18920e9f6fc-1717459200-split-scan-conditions-119489357
  namespace: lightbeam
  status: Failed
- name: scanner-64cd98e40910d18920e9f6fc-1717459200-split-scan-conditions-3743204798
  namespace: lightbeam
  status: Failed
- name: scanner-64cd98e40910d18920e9f6fc-1717459200-split-scan-conditions-4212831035
  namespace: lightbeam
  status: Failed
- name: scanner-64e30baea006cf2d80405f1d-1717459200-split-scan-conditions-2074548107
  namespace: lightbeam
  status: Failed
- name: scanner-64e30baea006cf2d80405f1d-1717459200-split-scan-conditions-2678689486
  namespace: lightbeam
  status: Failed
- name: send-alert-notifications-1717459200-policy-ops-1250506680
  namespace: lightbeam
  status: Failed
- name: send-alert-notifications-1717459200-policy-ops-3397497459
  namespace: lightbeam
  status: Failed
- name: send-alert-notifications-1717459200-policy-ops-3599123077
  namespace: lightbeam
  status: Failed
- name: send-alert-notifications-1717459200-policy-ops-4001638838
  namespace: lightbeam
  status: Failed
- name: unmute-alerts-1717437600-workflow-status-1895486181
  namespace: lightbeam
  status: Failed
- name: elasticsearch-curator-28637340-d9xsp
  namespace: logging
  status: Failed
- name: elasticsearch-curator-28637340-gk2vb
  namespace: logging
  status: Failed
- name: elasticsearch-curator-28637340-jndq9
  namespace: logging
  status: Failed
- name: elasticsearch-curator-28637340-kv6nf
  namespace: logging
  status: Failed
- name: elasticsearch-curator-28637340-pmmbk
  namespace: logging
  status: Failed
- name: elasticsearch-curator-28637340-sp9nd
  namespace: logging
  status: Failed
- name: elasticsearch-curator-28637340-tgglq
  namespace: logging
  status: Failed

 
k8s:Get Kubernetes Error PODs from All Jobs
Failed Objects:
- job_name: lb-zendesksupport-reconcile-64b188a22b9727c0f515f167-28623855
  namespace: lightbeam
  pod_name: lb-zendesksupport-reconcile-64b188a22b9727c0f515f167-28623v8788
- job_name: lightbeam-bootstrap
  namespace: lightbeam
  pod_name: lightbeam-bootstrap-f2p6h
- job_name: elasticsearch-curator-28637340
  namespace: logging
  pod_name: elasticsearch-curator-28637340-d9xsp
- job_name: elasticsearch-curator-28637340
  namespace: logging
  pod_name: elasticsearch-curator-28637340-gk2vb
- job_name: elasticsearch-curator-28637340
  namespace: logging
  pod_name: elasticsearch-curator-28637340-jndq9
- job_name: elasticsearch-curator-28637340
  namespace: logging
  pod_name: elasticsearch-curator-28637340-kv6nf
- job_name: elasticsearch-curator-28637340
  namespace: logging
  pod_name: elasticsearch-curator-28637340-pmmbk
- job_name: elasticsearch-curator-28637340
  namespace: logging
  pod_name: elasticsearch-curator-28637340-sp9nd
- job_name: elasticsearch-curator-28637340
  namespace: logging
  pod_name: elasticsearch-curator-28637340-tgglq

 
k8s:Check K8s service PVC utilization 
Failed Objects:
- capacity: 291G
  mount_path: /bitnami/kafka
  pvc_name: data-lightbeam-kafka-0
  used: 83
- capacity: 291G
  mount_path: /.keycloak
  pvc_name: lb-keycloakdir-lb-keycloak-0
  used: 83
- capacity: 291G
  mount_path: /bitnami/mongodb
  pvc_name: datadir-lightbeam-mongodb-0
  used: 83
- capacity: 291G
  mount_path: /usr/share/elasticsearch/data
  pvc_name: data-lightbeam-elasticsearch-master-0
  used: 83
- capacity: 290.6G
  mount_path: /data
  pvc_name: vault-claim-lb-vault-0
  used: 83

 
k8s:Check the status of K8s CronJob pods
Failed Objects:
- NotAssociated
- Pending
- UnexpectedState

 
Execution script /usr/local/bin/lb_reports.sh
OUTPUT FILE /unskript/data/execution/run_output-2024-06-13T16_03_26.153294/run_output.txt
/usr/local/bin/lb_reports.sh Timed out

Checklist:

• My changes generate no new warnings.
• I have added tests that prove my fix is effective or that my feature works.
• Any dependent changes have been merged and published.

Documentation

Make sure that you have documented corresponding changes in this repository.