-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support updating test durations when running for a single group #11
Comments
Interesting and good idea 👍 How would it actually work? Each |
I'm afraid that would indeed be a manual process for now. With the current format of storing execution times (JSON) it's a bit hard to combine the results since they are not easy to concatenate. However, by changing the format to CSV concatenation becomes trivial:
What do you think? |
I guess we could provide some simple command which would combine the reports, e.g. |
Yes I'm aware of In my opinion appending files is slightly preferred because the I'll you decide :) |
I don't think it's a biggie. For example, I just don't like that the user should know how the "internals" of |
I think just writing new results to the existing file makes the most sense. This would let users decide how they combine and/or manage their test-duration file. Here's what I would like it to look like: on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
# Load test durations file from cache
- uses: actions/cache@v2
with:
path: .test_durations
key: 0 # doesn't matter
restore-keys: 0 # doesn't matter
# Run tests
- run: pytest --splits 8 --group ${{ matrix.group }} --store-durations
# If .test_durations was found in the cache, then the plugin would write to the existing file
# If the cache was empty, it would create a new file with data from the tests run
# In both cases the cache will store the file for next time There are a few things to consider though:
If we wanted to do all three, you could also remove the on: push
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/cache@v2
with:
path: .pytest_cache
key:
restore-keys:
# pytest-split is enabled if `splits` and `group` is found,
# and writes to `.pytest_cache/v/cache/pytest_splits` automatically
- run: pytest --splits 8 --group ${{ matrix.group }} And then each run would get a little bit more accurate 🙂 Sorry for the block of text; hope some of it made sense. Looking forward to hearing your thoughts - I've probably definitely forgotten to consider some things 👍 |
Forgot to say I'd be happy to try and implement this if you want help @jerry-git; after we agree on what the implementation should look like 🙂 |
I agree with these suggestions. The test identifiers are considered unique anyway so using a dictionary to store the times makes sense. Writing it to the pytest cache and simply splitting into chunks if there is no cache yet available seems to be the right way to use existing functionality. I'm also happy to contribute further to this. |
Great discussion here! Regarding @sondrelg's points in this #11 (comment):
I'd only store the data in case Just to note, if the test durations data location would be configurable, one could still point it to .pytest_cache if it fits better their use case 🙂 |
Ok so to summarize:
I agree we probably don't need to involve the
Yeah I agree 👍 |
This was done in #14 |
It would be useful to add support for calculating test durations while using the parameters
--splits
andgroup
. To clarify, this is currently possible but it runs the full test suite, not just the tests for that particular group. Instead, users can calculate the test_durations for each group and then combine the results.Happy to send a merge request your way if you think this makes sense!
The text was updated successfully, but these errors were encountered: