tests in files?

[jankatins]

This is more like a suggestion, if you expect that there should be multiple tests and not only 5-10 per kernel.

In knitpy, I found it easier to write testcases by putting each testcase into two files, one with the input, one with the output. The actually test then looks like this:

from knitpy.tests import AbstractOutputTestCase, _add_test_cases
class OutputTestCase(AbstractOutputTestCase):
    pass
_add_test_cases(OutputTestCase, "basics")
_add_test_cases(OutputTestCase, "chunk_options")

where 'basics' and 'chunk_options' are subdirs with files in it like 'test_name.pymd' and 'test_name.md' (pymd = input, md = output, *.md is generated on first test run, but must be manually checked and the file renamed to activate the test)

Implementation is here: https://github.com/JanSchulz/knitpy/blob/master/knitpy/tests/__init__.py

I think this nicely fits here too with names like 'tab_testname.input' and 'tab_testname.output' beeing one tab completition test named 'testname' which must produce the json structure in 'tab_testname.output'.

[takluyver]

For now, I'm happy with tests being defined in Python code. I think there are two many bits in the responses that can vary for a strict 'the response must be this' check to make sense. And once you start getting into defining wildcards, and ways to mark lists where the order doesn't matter, working out how to handle those things is going to be more effort than just writing bits of Python code.

Possibly we should build a framework like that later, but for now, I think it's best kept simple.