Inconsistent database entries for TF

[sentry-io]

Sentry Issue: PCKT-002-PACKIT-SERVICE-8GN

AttributeError: 'NoneType' object has no attribute 'commit_sha'
(2 additional frame(s) were not displayed)
...
  File "packit_service/service/api/testing_farm.py", line 121, in get
    "ref": tf_result.commit_sha,
  File "packit_service/models.py", line 447, in commit_sha
    return self.group_of_targets.commit_sha

Public offender being:

packit=# SELECT * FROM tft_test_run_targets WHERE id=618858;
   id   | pipeline_id | status |         target         | web_url |                                data                                 |       submitted_time       |                  identifier                   | tft_test_run_group_id 
--------+-------------+--------+------------------------+---------+---------------------------------------------------------------------+----------------------------+-----------------------------------------------+-----------------------
 618858 |             | new    | centos-stream-8-x86_64 |         | {"base_project_url": "https://github.com/ComplianceAsCode/content"} | 2024-10-02 07:33:20.727411 | /hardening/host-os/ansible/cis_workstation_l2 |                      
(1 row)

Based on the short discussion with @lbarcziova there was a suspicion of race condition (possibly between Copr babysitting × Fedora Messaging).

However after a brief look through the database entries… it looks like a mess with no specific pattern :/

Filtered entries

id;pipeline_id;status;target;web_url;submitted_time;identifier;tft_test_run_group_id
618828;2f233fd3-5f17-4fdc-81c5-d4a76287f3e7;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/2f233fd3-5f17-4fdc-81c5-d4a76287f3e7;2024-10-02 07:33:05.572013;/hardening/host-os/ansible/anssi_bp28_high;144684
618855;c5c38261-36c7-4512-b0e1-eec74a2d3fe2;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/c5c38261-36c7-4512-b0e1-eec74a2d3fe2;2024-10-02 07:33:19.42574;/hardening/host-os/ansible/anssi_bp28_high;144684
618830;2fbb6ad7-ac95-440f-a2ed-9381d4e06cda;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/2fbb6ad7-ac95-440f-a2ed-9381d4e06cda;2024-10-02 07:33:06.984507;/hardening/host-os/ansible/cis;144684
618851;17acc7ad-563f-4faa-8587-6a40d51cc424;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/17acc7ad-563f-4faa-8587-6a40d51cc424;2024-10-02 07:33:16.880517;/hardening/host-os/ansible/cis;144684
618834;efbd1833-a728-41d9-83c8-5d1d767f1d95;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/efbd1833-a728-41d9-83c8-5d1d767f1d95;2024-10-02 07:33:08.725586;/hardening/host-os/ansible/cis_server_l1;144684
618857;a731cb6c-d66c-494b-9b08-4b80e07e529e;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/a731cb6c-d66c-494b-9b08-4b80e07e529e;2024-10-02 07:33:20.214558;/hardening/host-os/ansible/cis_server_l1;144684
618831;b1e3f21b-47eb-43f8-9847-41bd3cf884f6;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/b1e3f21b-47eb-43f8-9847-41bd3cf884f6;2024-10-02 07:33:07.173385;/hardening/host-os/ansible/cis_workstation_l1;144684
618866;4a0df6a2-2d37-4517-a61b-d97bfeabd907;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/4a0df6a2-2d37-4517-a61b-d97bfeabd907;2024-10-02 07:33:27.440763;/hardening/host-os/ansible/cis_workstation_l1;144684
618838;a15c0a06-17dc-4a9d-bf84-65c395fdf5da;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/a15c0a06-17dc-4a9d-bf84-65c395fdf5da;2024-10-02 07:33:11.4755;/hardening/host-os/ansible/cis_workstation_l2;144684
618858;;new;centos-stream-8-x86_64;;2024-10-02 07:33:20.727411;/hardening/host-os/ansible/cis_workstation_l2;
618870;e6eaecf3-dfb6-464a-8b57-3c157a8877bb;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/e6eaecf3-dfb6-464a-8b57-3c157a8877bb;2024-10-02 07:33:31.302185;/hardening/host-os/ansible/cis_workstation_l2;144684
618832;461016f3-df04-4a03-af55-16f0c31094c3;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/461016f3-df04-4a03-af55-16f0c31094c3;2024-10-02 07:33:08.485146;/hardening/host-os/ansible/cui;144684
618856;42654fb7-d75a-418d-a83d-bfef7807d534;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/42654fb7-d75a-418d-a83d-bfef7807d534;2024-10-02 07:33:20.182255;/hardening/host-os/ansible/cui;144684
618837;e2506816-0f72-498e-ac27-d8529ad57b91;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/e2506816-0f72-498e-ac27-d8529ad57b91;2024-10-02 07:33:11.337601;/hardening/host-os/ansible/e8;144684
618859;cbdee76f-8f51-4408-bce7-99f84fd68776;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/cbdee76f-8f51-4408-bce7-99f84fd68776;2024-10-02 07:33:21.988649;/hardening/host-os/ansible/e8;144684
618833;46bed0a8-0972-4780-b68f-1cd1d892edeb;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/46bed0a8-0972-4780-b68f-1cd1d892edeb;2024-10-02 07:33:08.686863;/hardening/host-os/ansible/hipaa;144684
618861;cb67e0cc-61e7-4232-8d67-7b5c345a4bd4;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/cb67e0cc-61e7-4232-8d67-7b5c345a4bd4;2024-10-02 07:33:23.121436;/hardening/host-os/ansible/hipaa;144684
618835;b7d645ec-b3a3-4289-8b75-533b40468241;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/b7d645ec-b3a3-4289-8b75-533b40468241;2024-10-02 07:33:10.197527;/hardening/host-os/ansible/ism_o;144684
618860;08559267-4142-4e15-8e29-9c0413a17b56;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/08559267-4142-4e15-8e29-9c0413a17b56;2024-10-02 07:33:22.978551;/hardening/host-os/ansible/ism_o;144684
618841;;new;centos-stream-8-x86_64;;2024-10-02 07:33:13.043939;/hardening/host-os/ansible/ospp;144684
618862;cd7aef83-4990-463b-92c3-c8fa643aac0d;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/cd7aef83-4990-463b-92c3-c8fa643aac0d;2024-10-02 07:33:23.797448;/hardening/host-os/ansible/ospp;144684
618836;86b21b70-fe78-4a01-ae6f-29f82df1cdd2;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/86b21b70-fe78-4a01-ae6f-29f82df1cdd2;2024-10-02 07:33:10.906165;/hardening/host-os/ansible/pci-dss;144684
618864;e0393514-860a-447c-aa1d-6deeec76acf7;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/e0393514-860a-447c-aa1d-6deeec76acf7;2024-10-02 07:33:25.813136;/hardening/host-os/ansible/pci-dss;144684
618842;dffac8c7-e11a-4523-8f46-f65aee35035c;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/dffac8c7-e11a-4523-8f46-f65aee35035c;2024-10-02 07:33:13.266415;/hardening/host-os/ansible/stig;144684
618863;20360b97-8c8b-4912-8cfb-aa7df3ada0b6;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/20360b97-8c8b-4912-8cfb-aa7df3ada0b6;2024-10-02 07:33:24.782195;/hardening/host-os/ansible/stig;144684
618845;763f9934-4cd2-4c3b-9f66-72693ae4833b;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/763f9934-4cd2-4c3b-9f66-72693ae4833b;2024-10-02 07:33:15.331668;/hardening/host-os/oscap/anssi_bp28_high;144684
618865;49388874-13cc-4037-94d3-437eaa938a8e;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/49388874-13cc-4037-94d3-437eaa938a8e;2024-10-02 07:33:26.434749;/hardening/host-os/oscap/anssi_bp28_high;144684
618839;c43a1865-bfef-414d-9d5b-46b4efb8d343;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/c43a1865-bfef-414d-9d5b-46b4efb8d343;2024-10-02 07:33:11.700199;/hardening/host-os/oscap/cis;144684
618867;6956b7c5-98a7-4fcd-a2e7-bea00757c724;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/6956b7c5-98a7-4fcd-a2e7-bea00757c724;2024-10-02 07:33:28.028449;/hardening/host-os/oscap/cis;144684
618840;95e72a52-72b3-4bad-b7ec-70ff8d636e5e;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/95e72a52-72b3-4bad-b7ec-70ff8d636e5e;2024-10-02 07:33:12.571092;/hardening/host-os/oscap/cis_server_l1;144684
618871;b4dafa70-0235-48fc-a47b-9084e5faa841;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/b4dafa70-0235-48fc-a47b-9084e5faa841;2024-10-02 07:33:32.307366;/hardening/host-os/oscap/cis_server_l1;144684
618848;65dffabf-ca70-4168-a7dd-ad1aa00acd4b;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/65dffabf-ca70-4168-a7dd-ad1aa00acd4b;2024-10-02 07:33:16.024607;/hardening/host-os/oscap/cis_workstation_l1;144684
618868;0f5df3ef-4a51-4cdd-8ae8-1dd5a8bde1e2;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/0f5df3ef-4a51-4cdd-8ae8-1dd5a8bde1e2;2024-10-02 07:33:29.915686;/hardening/host-os/oscap/cis_workstation_l1;144684
618843;0d760710-b6f4-4b67-a1dd-a85df20e8674;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/0d760710-b6f4-4b67-a1dd-a85df20e8674;2024-10-02 07:33:13.389087;/hardening/host-os/oscap/cis_workstation_l2;144684
618873;ad3d97b6-aaca-42b8-861c-b3b17d1a625c;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/ad3d97b6-aaca-42b8-861c-b3b17d1a625c;2024-10-02 07:33:33.510549;/hardening/host-os/oscap/cis_workstation_l2;144684
618844;abd23f8f-62ec-4c1a-aaba-2478cf8f13a0;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/abd23f8f-62ec-4c1a-aaba-2478cf8f13a0;2024-10-02 07:33:14.483607;/hardening/host-os/oscap/cui;144684
618869;63e64119-c14b-461e-b6fc-6878b80748ed;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/63e64119-c14b-461e-b6fc-6878b80748ed;2024-10-02 07:33:30.614234;/hardening/host-os/oscap/cui;144684
618847;1039ad6b-a64d-4c6f-aceb-3a39d3e6770c;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/1039ad6b-a64d-4c6f-aceb-3a39d3e6770c;2024-10-02 07:33:15.807111;/hardening/host-os/oscap/e8;144684
618872;b7e7b7ce-ff75-4486-91ee-339ebf3f4943;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/b7e7b7ce-ff75-4486-91ee-339ebf3f4943;2024-10-02 07:33:33.302449;/hardening/host-os/oscap/e8;144684
618852;ccc821ca-54b7-479f-aced-6fb0ed903200;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/ccc821ca-54b7-479f-aced-6fb0ed903200;2024-10-02 07:33:17.322681;/hardening/host-os/oscap/hipaa;144684
618876;1132d538-115f-4c6e-801f-d843af7afe69;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/1132d538-115f-4c6e-801f-d843af7afe69;2024-10-02 07:33:38.409593;/hardening/host-os/oscap/hipaa;144684
618853;c3a8ffe6-f758-4181-bbb5-488589151706;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/c3a8ffe6-f758-4181-bbb5-488589151706;2024-10-02 07:33:17.706154;/hardening/host-os/oscap/ism_o;144684
618874;a578b730-1c07-4890-8556-a73d4f0caeda;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/a578b730-1c07-4890-8556-a73d4f0caeda;2024-10-02 07:33:37.08899;/hardening/host-os/oscap/ism_o;144684
618846;f12d73a2-008e-4a70-a0e8-c6743fcd9bfa;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/f12d73a2-008e-4a70-a0e8-c6743fcd9bfa;2024-10-02 07:33:15.58427;/hardening/host-os/oscap/ospp;144684
618875;2df18364-f0df-4da9-8ff3-3b3174cbe9e9;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/2df18364-f0df-4da9-8ff3-3b3174cbe9e9;2024-10-02 07:33:37.541874;/hardening/host-os/oscap/ospp;144684
618854;01115614-cf7e-468c-87e3-584f93bdb04e;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/01115614-cf7e-468c-87e3-584f93bdb04e;2024-10-02 07:33:19.017987;/hardening/host-os/oscap/pci-dss;144684
618878;d1902d59-14b9-447e-b9c8-baf895d8f9bc;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/d1902d59-14b9-447e-b9c8-baf895d8f9bc;2024-10-02 07:33:46.141347;/hardening/host-os/oscap/pci-dss;144684
618849;12f5e951-bfea-4806-ae36-98f33e71668f;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/12f5e951-bfea-4806-ae36-98f33e71668f;2024-10-02 07:33:16.280889;/hardening/host-os/oscap/stig;144684
618877;c9b0d94a-ee06-40c4-a97f-4e259d917b5a;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/c9b0d94a-ee06-40c4-a97f-4e259d917b5a;2024-10-02 07:33:38.870747;/hardening/host-os/oscap/stig;144684
618829;34205986-029b-435c-8ea7-cda3824380b3;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/34205986-029b-435c-8ea7-cda3824380b3;2024-10-02 07:33:06.436006;/static-checks;144684
618850;da501259-d594-489d-a921-ee3b49c52ea8;passed;centos-stream-8-x86_64;https://artifacts.dev.testing-farm.io/da501259-d594-489d-a921-ee3b49c52ea8;2024-10-02 07:33:16.483964;/static-checks;144684

Notes:

• all of the the tests got triggered twice
• the only exception is 618858 which is actually three times in a list and in the most inconsistent state (missing TF group id)

Based on this, I would suspect that the aforementioned 618858 failed much more miserably than the other 618841 which is still in state=new, but has, at least, TFT group ID.

It almost looks like the failed request to TF got retried and triggered whole test suite again. Not sure what's up with the 618841 that appears to have quite similar fate as the other “run” that got propagated to Sentry.

This inconsistent state causes issues when exposing the runs via API, and thus causes outage when the runs are in the view.

@lbarcziova WDYT?