Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DISCOVERY-621: Mark slow tests #526

Merged
merged 1 commit into from
Jul 10, 2024
Merged

DISCOVERY-621: Mark slow tests #526

merged 1 commit into from
Jul 10, 2024

Conversation

mirekdlugosz
Copy link
Collaborator

Using data from Jenkins, add slow marker to all tests that on average take at least 30 seconds to complete. There are 11 tests like that. If we put threshold at 15 seconds, there would be 18 tests marked.

It's important to remember that data is obtained from complete test suite runs, and so it differs from times we would obtain when running tests individually. Good example is camayoc/tests/qpc/api/v1/reports/test_reports.py - only first two tests gets marked, because all subsequent tests would use results cached in scans fixture, and finish much faster. But if you were to run camayoc/tests/qpc/api/v1/reports/test_reports.py::test_installed_products_deployment_report individually, it would likely take long time to finish, as it would have to wait for scan to complete.

Another approach we might take is just adding slow marker automatically to tests that use scans fixture, leaving explicit mark to few tests that are slow due to other reasons.

@mirekdlugosz mirekdlugosz requested review from a team July 3, 2024 12:13
Copy link

codecov bot commented Jul 3, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.33%. Comparing base (4a03602) to head (200630b).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #526   +/-   ##
=======================================
  Coverage   78.33%   78.33%           
=======================================
  Files           7        7           
  Lines         683      683           
=======================================
  Hits          535      535           
  Misses        148      148           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Using data from Jenkins, add `slow` marker to all tests that on average
take at least 30 seconds to complete.

Relates to: DISCOVERY-621
Copy link
Member

@infinitewarp infinitewarp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These seem like reasonable choices to me. 👍

Copy link
Member

@ruda ruda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The marks are good, let's focus on 30 seconds at this time.
I ran the test_openshift suite (job/discovery-standalone/159/), they are taking at most 20 seconds to complete, so I don't think we need to mark it as slow.

@mirekdlugosz mirekdlugosz merged commit acb8a45 into main Jul 10, 2024
9 checks passed
@mirekdlugosz mirekdlugosz deleted the mark-slow-tests branch July 10, 2024 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants