[WIP] Refactor type annotation decorators to use click classes

[jbeezley]

There is still some work to do on this to handle edge cases related to how data is represented in item_tasks and to add better test coverage. I think the public API here is close enough to the final product that it would be worthwhile for others to look at.

Anyone familiar with click should find the API familiar, for example:

import click

from girder_worker_utils import types
from girder_worker_utils.decorators import task, task_input, task_output


@task()
@task_input('-n', type=types.Integer(min=1), required=True)
@task_output('value', type=types.Integer())
def fibonacci(n):
    """Compute the nth fibonacci number."""
    if n <= 2:
        return 1
    return fibonacci(n - 1) + fibonacci(n - 2)


if __name__ == '__main__':
    # This is just to print the return value for this example.  Ordinarily, you would just do `fibonacci.main()`
    click.echo(fibonacci.main(standalone_mode=False))

With this you get a CLI:

$ python fibonacci.py --help
Usage: fibonacci.py [OPTIONS]

  Compute the nth fibonacci number.

Options:
  -n INTEGER RANGE   [required]
  --item-tasks-json
  --help             Show this message and exit.

$ python fibonacci.py -n 10
55

And you get item_tasks annotations that works with docker:

$ python fibonacci.py --item-tasks-json 
{
  "inputs": [
    {
      "description": "", 
      "min": 1, 
      "max": null, 
      "step": 1, 
      "type": "number", 
      "id": "n", 
      "name": "n"
    }
  ], 
  "name": "fibonacci", 
  "outputs": [
    {
      "step": 1, 
      "description": "", 
      "min": null, 
      "max": null, 
      "type": "number", 
      "id": "value", 
      "name": "value"
    }
  ], 
  "description": "Compute the nth fibonacci number."
}

From the functional API that would be used by the item_task endpoints, it looks very similar to before:

fibonacci.describe() # returns the item_tasks json spec
fobonacci.call_item_task({'n': {'data': 10}}) # => 55

[kotfic]

i'm wondering if there is a way to provide this functionality through the girder_worker.app.app.task decorator? Otherwise functions will have to look like this:

from girder_worker.app import app

@task()
@task_input('-n', type=types.Integer(min=1), required=True)
@task_output('value', type=types.Integer())
@app.task(...)
def fibonacci(n):
    pass

[kotfic]

actually i take it back, does this task decorator belong in this PR at all? is it strictly necessary for the item_task functionality to work? See my comment below:

[jbeezley]

I haven't addressed the integration with girder_worker yet. We could probably combine the two @task decorators, but for click to work correctly we need a decorator on the top that serves the same purpose of the click.command() decorator. That's where click actually does all of it's magic.

[kotfic]

I think ideally we'd like to avoid introducing a CLI with this PR. That is on the road map, but i'd like to provide a single command that hooks into girder worker's task discovery mechanism - that would allow you list and run all task from the CLI that are installed in your environment. Using something like this

import click
from .gwcli import get_extensions, get_extension_tasks


class GWPluginToCommand(click.MultiCommand):

    def __init__(self, *args, **kwargs):
        super(GWPluginToCommand, self).__init__(*args, **kwargs)
        self._tasks = None

    def _uniquify_names(self, keys, tasks):
        keys = [k.split(".")[-1] for k in keys]
        return dict(zip(keys, tasks))

    @property
    def tasks(self):
        if self._tasks is None:
            _tasks = []
            for ext in get_extensions():
                _tasks.extend(get_extension_tasks(ext).items())

            self._tasks = self._uniquify_names(*zip(*_tasks))


        return self._tasks


    def list_commands(self, ctx):
        return self.tasks.keys()

    def get_command(self, ctx, name):
         # get the actual command object Here

[jbeezley]

I'm not sure I understand your comment. I had that kind of autogeneration of the multicommand in mind as well as future work. Are you saying this PR shouldn't provide even the single command CLI?

[kotfic]

yeah, so i think task_input and task_output should probably avoid returning click.option and click.argument objects directly and instead provide an internal representation (assigned to the user-defined-function) which will be used by a MultiCommand implemented in a different PR. I think we just want to use/extend the type system for now and have item_tasks_json be functions set on the celery task object (?).

[jbeezley]

I thought one of the goals was to avoid requiring celery. My goal here was to lean on click as much as possible. I suspect we need to hash this out in person.