-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
YAML test loader #17
base: main
Are you sure you want to change the base?
YAML test loader #17
Conversation
Thanks for thinking about this. However, I don't think there is a major advantage to using yaml for this:
I also dislike YAML as a format - it's big and complex, and the Python bindings are big and complex, and they're not even safe by default: 'declarative' files can execute arbitrary code on load. If we do define a way to specify tests in a declarative format (which I'm not sold on, as described), I'd prefer to use TOML, which is simpler and lighter. |
My thought was that a declarative format looks more suitable to try and establish a standard that kernels would provide a I don't particularly mind changing the format; I don't think safe-parsing or the size of the bindings are particularly problematic, but the different string parsing rules and consequent ways newlines and escapes are handled can be a bit tricky, but I sense from the above that the choice of format isn't the main issue. |
That is a reasonable point - I've been thinking in terms of kernels setting up tests themselves, e.g. with Travis (we've done this with IRkernel, for instance). But for an external service that runs the tests, a declarative config file may be more palatable. Setting up the environment to install and run the kernel is still challenging, though. I'll keep thinking about this. |
knitpy has a similar thing, but using text files: https://github.com/JanSchulz/knitpy/tree/master/knitpy/tests As the the returned message is json, one could just dump the json content... The flow would be:
|
Just for comparison, I think this TOML document is equivalent to the YAML version. It's workable, but it also has some non-optimal features:
In any case, I suspect this PR as it currently exists is not particularly useful, and the details of how such a file might be consumed and how it should be formatted decided before an implementation. |
Copying this comment from here:
The point is that for some people like me, who like to write tests and use test data, this exact PR is potentially VERY valuable. |
(This one is perhaps a bit more speculative).
For the purposes of providing a cross-language test mechanism, this avoids the need to write the python/unittest around what is otherwise a declarative list of code samples to test against. Instead a loader script generates and runs the unittest from a YAML file.
The YAML file has equivalent structure to the unittest class variables that would otherwise be written. Indeed, all the loader does is load the top-level object as a dictionary and then construct a
KernelTest
class around it. Multiple YAML documents (defining one kernel to test each) can be bundled in one file with---
seperators.