Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adhoc enqueue params #579

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

paulgration
Copy link
Contributor

@paulgration paulgration commented Jan 31, 2018

This moves serializing/deserializing of enqueue_params property values for media packages to the setProperty and getProperty methods of mediapackage.py. It also ensures that wherever an enqueue_job or do_job call is made, the enqueue_params are passed along the chain until it reaches the point at which to ingest the media package.

In fixing this, it is possible to specify a target workflow and workflow_parameters per media package (rather than use the workflow specified in conf.ini) for manual (adhoc) recordings.

The code for ingesting and specifying the workflow is the _ingest method of galicaster/core/worker.py. Manual and scheduled recordings are handled differently:

  • Manual ingest uses the params parameter passed to the method (via jobs that are either queued or executed immediately)
  • Scheduled recordings can include a org.opencastproject.capture.agent.properties to specify a workflow definition

Currently (2.0.x), there are only two places that media package properties are set:

I had a requirement to do the same as the latter within a plugin, e.g. media_package.setProperty('enqueue_params', {'workflow': 'test'}).
Rather than import json and serialize/deserialize everywhere that the enqueue_params property needs to be set, this PR ensures that all enqueue_params properties are serialized/deserialized as part of the setProperty/getProperty methods of mediapackage.py. All other properties that are set remain as strings. So the only change here is to make sure that the enqueue_params property values are in a consistent format.

The managerui makes calls to the worker do_job and do_job_nightly methods but didn't pass the params parameter on to these methods so now the enqueue_params property is fetched and passed with these calls. Similarly the recorder service enqueue_ingest didn't pass any params down the line so the same change has been made here.

Generally, storing JSON representations of Python objects within a text node of XML doesn't seem like the best way to do this but for now I've gone for the path of least resistance to get this working - perhaps this file could be .json in future...?

Unfortunately, tests already appear to fail in 2.0.x (before this PR) when the manual config option of [ingest] is set to either nightly or immediately so testing this one isn't particularly easy. Tests pass for this but I've also been doing a mix of printing debug statements in _ingest, setProperty, getProperty, reviewing content written to the galicaster.xml file and using a test plugin to start the recording:

from gi.repository import Gtk
from galicaster.core import context
from galicaster.mediapackage.mediapackage import Mediapackage

dispatcher = context.get_dispatcher()
recorder = context.get_recorder()
repository = context.get_repository()

def init():
    dispatcher.connect('init', on_init)
    dispatcher.connect('recorder-stopped', on_recorder_stopped)

def on_init(_source):
    RecordDialog()

def on_recorder_stopped(_source, media_package_id):
    media_package = repository.get(media_package_id)
    print media_package.getProperty('enqueue_params')
    RecordDialog()

class RecordDialog(object):
    def __init__(self):
        parent = context.get_mainwindow().get_toplevel()
        self.dialog = Gtk.Dialog()
        self.dialog.set_title('Test')
        self.dialog.set_default_size(200, 100)
        self.dialog.set_transient_for(parent)
        self.dialog.set_modal(True)

        record = Gtk.Button('Record')
        record.set_hexpand(True)
        record.set_vexpand(True)
        record.connect('clicked', self.on_record_clicked)

        box = self.dialog.get_content_area()
        box.add(record)
        self.dialog.show_all()

    def on_record_clicked(self, _source):
        media_package = Mediapackage(title='Test passing params')
        media_package.setProperty('enqueue_params', {'workflow': 'test'})
        recorder.record(media_package)
        self.dialog.destroy()

@paulgration
Copy link
Contributor Author

Updated tests since 2071b23 so that they no longer fail with this PR

@andiempettJISC
Copy link
Collaborator

andiempettJISC commented Nov 13, 2018

this is definitely a piece of work that has value. i've found myself this division between manual properties and properties that come from opencast frustrating.

i'll take a look at this PR next

workflow_parameters = None
if isinstance(params, dict):
workflow = params.get('workflow')
workflow_parameters = params.get('workflow_parameters')

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think this is a good way to simplify the conditionals here as you can't have workflow without workflow_parameters i believe

@paulgration
Copy link
Contributor Author

I'm wondering whether this should be targeted at master now..? And if it is, whether anything needs to change about the json imports because since creating this PR there has been other commits that have changed the storing of data from galicaster.xml to galicaster.json so it may be that json.dumps isn't required any longer? I haven't look into it in detail though

@andiempettJISC
Copy link
Collaborator

andiempettJISC commented Nov 14, 2018

ah yeah i missed that. see https://github.com/teltek/Galicaster/blob/master/CONTRIBUTING.md. so use master

please feel free to change this to master as i don't think it could be accepted against 2.0.x

i think it is safe to still use json.dumps as i've tested this against latest master and its working so far.
its probably a larger piece of work to look at how Galicaster stores temporary data etc

@paulgration paulgration changed the base branch from 2.0.x to master November 14, 2018 12:59
@paulgration
Copy link
Contributor Author

ah yeah i missed that. see https://github.com/teltek/Galicaster/blob/master/CONTRIBUTING.md. so use master

please feel free to change this to master as i don't think it could be accepted against 2.0.x

i think it is safe to still use json.dumps as i've tested this against latest master and its working so far.
its probably a larger piece of work to look at how Galicaster stores temporary data etc

Ok, that sounds promising then. I just changed target branch to master

@andiempettJISC andiempettJISC added feature Use this label for new features or enhancements feature labels Nov 16, 2018
Copy link
Contributor

@Alfro Alfro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a really good idea. However, I think we should support these "dict" or "json" parameters on the serializer/deserializer classes, and the set/getProperty functions should not handle this kind of logic (doesn't make sense to keep a dictionary saved as a string within the mediapackage class).

self.properties[prop] = value
return True
property_value = value
if prop == "enqueue_params":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To allow for this kind of API:
media_package.setProperty('enqueue_params', {'workflow': 'test'})
What we need to do is to support serializing/deserializing objects on the serializer/deserializer, not here.

Also, this support should be added generically, not just for an "enqueue_params" property.

Essentially, to add this functionality what we should do is to support dictionary properties, at least. And serialize/deserialize them on the corresponding classes. The setProperty function should not be the one carrying this logic

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Alfro, I agree but went with the minimal changes to provide this functionality for enqueue_params as I wasn't sure how much else would be impacted by a bigger change to serialize/deserialize other properties as well.
The file where this data is stored has changed in master from xml to json (not part of a built release yet as far as I'm aware) so that should make things easier.
Due to some changes for us I'm not sure if/when I'll have time to work on this next and I'm unlikely to be able to devote much time to development so someone else may want to continue with this.

@Alfro Alfro removed the feature label Jun 7, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature Use this label for new features or enhancements
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants