Fix code/test correspondence; explicitly skip test empty cells/tests.

[ceball]

Fixes #97 and fixes #99. Same tests as in #111, just updated for 1-based counting.

Summary of new behavior

1. Test methods are only generated for code cells. If a non-code cell contains a test in its metadata, it's an error at generation time. Only code cells will be included in "cell coverage" calculation.

2. Test methods are named like test_code_cell_n (so you know it's only code cells), where n is the 1-based code cell number - i.e. they are numbered like execution count (if run from scratch).

Example: If there are 3 cells, and the 2nd cell is markdown, then there'll just be two test methods, test_code_cell_1 and test_code_cell_2.

3. Code cells that are empty or do not have a test still generate a test method, but decorated with @skip(reason) (where reason is "empty code cell" or "cell has no test").

4. Tests that do not contain a cell injection generate a skipped test method (reason is "no cell injection").

Changes in this PR

• Fix correspondence of code and test cells (e.g. markdown caused them to go out of sync)

• Change cell/test numbering to start from 1, and only include code cells (matching other notebook tools).

• If a non-code cell contains test in metadata, it's an error during test script generation.

• Create explicitly skipped tests for:

  • empty cell
  • empty test
  • code cell not injected into test

To do:

• Fix windows tests

Future work:

• Update coverage calculation (see Fix code/test correspondence; explicitly skip test empty cells/tests. #112 (comment))

EDIT: clarified that coverage will be addressed in a subsequent PR once behavior here is agreed.

[vidartf]

I see there is still an unchecked todo here. Is this ready for review?

[vidartf]

Also: Does this supersede #111 then?

[ceball]

#111 just demonstrates the same tests as are here failing on master. #111 contains only tests and is not for merging; it's for information only.

I see there is still an unchecked todo here. Is this ready for review?

@vidartf Yes, it's ready for review. I have removed the to-do. Since the coverage calculation* is not altered here, and is not tested, I should be allowed to address that in a future PR I think - after the behavior of testing here is agreed. The behavior changes here imply that the coverage calculation should change. It's currently:

num code cells with %cell in their tests / num code cells

This PR being accepted implies coverage should change to:

num non-empty code cells with %cell in their tests / num non-empty code cells

Note that the current definition of coverage can be gamed by adding empty code cells with tests :)

(*) Actually there are three separate calculations of coverage in the code. Two of them are the same, one is broken. Still, none of those three calculations or components thereof are touched by this PR.

[vidartf]

This PR being accepted implies coverage should change to: ...

I would say the final goal would be to get proper line/branch coverage reports :)

[ceball]

This PR being accepted implies coverage should change to: ...

I would say the final goal would be to get proper line/branch coverage reports :)

Ok, I should have said the "cell tests coverage" or similar, to distinguish from actual code coverage (see also #109).

[ceball]

Thanks, I'll follow up what we discussed here (e.g. creating an issue to track pytest in pytest skip handling) and then merge once I've done that.

[ceball]

Hmm, azure doesn't seem to be working here any more?