Houdini PDG/TOPs connection

[RiLights]

Hi,

Is anybody tried to connect Houdini PDG/TOPs and Afanasy?

Thanks
Ostap

[RiLights]

I just remember that Mikhail Lesin (@mlesin) mentioned SLURM and Afanasy working together - #340.
And looks like, this kind of approach will work with TOPs but would be nice to hear some details about the connection or maybe some native Afanasy dynamic task submission.

Thanks
Ostap

[PaulSchweizer]

Hi guys,

We are currently evaluating Houdini's PDG in conjunction with Afanasy, here is what we found so far:

@alexwheezy already created an Afanasy Scheduler and it works great:
https://github.com/alexwheezy/python/tree/master/houdini/afanasy_scheduler

The problem however is, that it creates an individual Job for each work item as that is the only possibility at the moment. This is obiously not feasible in production.
To fix that, we would need a way to attach a new task to an existing job/block, as outlined in this issue #292 . The HQueue Scheduler currently works in that way, keeping a reference to a parent job and then attaching all related work items to it during the scheduling process.

If that would be possible, it would be a huge step towards making PDG usable with Afanasy in production!

If you need any further info let me know

[alexwheezy]

Hi Paul! That's right, and this is a problem, and unfortunately I haven’t yet come up with an opportunity to combine all the subtasks into one task. I don’t always have time to do this task, but I occasionally return to it. Perhaps Timur, something else will tell.

…

On Thu, 6 Jun 2019 at 16:37, Paul Schweizer ***@***.***> wrote: Hi guys, We are currently evaluating Houdini's PDG in conjunction with Afanasy, here is what we found so far: @alexwheezy <https://github.com/alexwheezy> already created an Afanasy Scheduler and it works great: https://github.com/alexwheezy/python/tree/master/houdini/afanasy_scheduler The problem however is, that it creates an individual Job for each work item as that is the only possibility at the moment. This is obiously not feasible in production. To fix that, we would need a way to attach a new task to an existing job/block, as outlined in this issue #292 <#292> . The HQueue Scheduler currently works in that way, keeping a reference to a parent job and then attaching all related work items to it during the scheduling process. If that would be possible, it would be a huge step towards making PDG usable with Afanasy in production! If you need any further info let me know — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#436?email_source=notifications&email_token=ADHC7QECISYR3DNTB3O7V4LPZEHHXA5CNFSM4GUK3CC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXC3OMI#issuecomment-499496753>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADHC7QCVGA2BVCL2O2SB3W3PZEHHXANCNFSM4GUK3CCQ> .

[timurhai]

Hi.
I have no experience with TOPs (yet).

[RiLights]

I also know @sergiy-nazarenko doing some work in this direction.

And cool that Alex already created Afanasy post in SideFX forum: https://www.sidefx.com/forum/topic/66730/

[yannci]

Hi,

After #451 has been merged and is now available in afanasy version 2.3.1 and above we try to push this further and develop a workflow that uses PDG in conjunction with Afanasy. I currently use the scheduler from @alexwheezy as a base and rewrite it.
Did anyone else has made any progress on this so far?

Best,
Yannic

[timurhai]

Hi!
Sorry i have not used PDG yet.
But we recently switched on a new Houdini version, so soon i will start to learn it.

[timurhai]

Hi!
Is there some news from anybody on this theme?
Have someone managed to enhance Afanasy Sheduler to push new blocks/tasks to the same Afanasy job?

May be there is a better way to construct job at once?
What do you think about:
https://www.sidefx.com/docs/houdini/tops/custom_scheduler.html#staticcook
There is Static Cook Hybrid Approach section (SideFX recomendation instead of a fully static job).
For this we can add some "pending" state to Afanasy tasks, and some API to make such tasks "ready".
May be someone heard that some people make this for other render managers, is it useful/handy?

[RiLights]

Hi Timur,

I'm wondering which way of implementation did you choose? Static Hybrid, dynamic task job/block expansion?

[timurhai]

Hi!
In the roadmap is a scheduler were you can choose any method.
May be even Full Static (no Houdini session needed) - but definitely with limitations.

[RiLights]

Is the Afanasy supports dynamic task appending to the block?

[timurhai]

Yes.
You can append and block(s) and task(s).
#451

[RiLights]

That is cool! Sorry, I missed it the posts above.

Would it make sense then to implement it in this way and if user wants to have independent PDG session then it creates one more block which is responsible to create all other blocks/tasks.

Almost the same what native HQueue is doing.

Referencing to @PaulSchweizer in the post above.

[RiLights]

Another option could be when Afanasy is responsible only for resource (machines/slots) allocation but not for the task creation because basicly PDG already contains task scheduler which needs only a farm manager. In other words - creating thinner layer between PDG and Afanasy.

[timurhai]

It is a dymanic approach, which is a common for PDG.
This is way will be implemented first, as it is the simples way.
Or i misundertand you?

[RiLights]

I see 2 ways for dynamic approach. HQueue and SLURM where you manipulating resources (not tasks). PDG Scheduler already contains dependencies, restart handling, logs, etc.

Actually Afanasy's tasks already are allocating resources and other things which is not necessary for PDG. Maybe it will be simpler to implement PDG scheduler without additional layer as "Task" (but I don't know how exactly do it for the Afanasy).

[timurhai]

HQueue uses tasks (at first i was digging HQueue to learn PDG), and as far as i understand other farm schedulers too.
And dependencies, restarts are performed by PDG.
Task needed to take resources and to run a command.
You are talking about some way to aquire resources w/o a task (in can be a task w/o a command), but let PDG to run a command remotely w/o render manager?
I noticed only the sinle way of a dynamic approach, and two static: full and hybrid.

[RiLights]

I want to notice that HQueue way is absolutely reasonable. But in my case, dynamic tasks (appending tasks/blocks) was't an option because it takes some time for the job rebase (huge amount submitted jobs) but maybe it is not related to Afanasy.
So task nature has been removed while invatigation how I can optimase PDG Scheduler.

[timurhai]

Close due #514