Run Julia, R and Python code in seperate Malt processes

[felixcremer]

We are trying to run Julia, Python and R code in Quarto without any interaction between the different language cells.
Unfortunately due to the usage of shared libraries in different versions this does not work, see JuliaPy/PythonCall.jl#397.

Could we run the code from different languages in separate Malt processes and thereby circumventing the interaction between the languages.?

@meggart who had the initial idea.

[MichaelHatherly]

So cells of different languages would happen to just be rendered in the same output document, but aside from that they would be evaluated in language-specific processes?

[felixcremer]

Yes that would help a lot. We thought we could have one QuartoNotebookWorker process per language and still reuse all of the machinery. Maybe this would have to be set by a flag in the notebook header.

[MichaelHatherly]

Seems potentially doable. I'm marking this as up for grabs since we won't be actively working on this ourselves (PumasAI) for the foreseeable future.

[felixcremer]

Could you give us some pointers where we best launch these processes? I might give it a try because this would be really helpful for multi language tutorials.

[jkrumbiegel]

You'd have to start making this a vector

QuartoNotebookRunner.jl/src/server.jl

Line 4 in bc028e0

worker::Malt.Worker

and go from there.

But this seems to be a lot of complication for little gain to me, as in principle the interop works, there's just conflicts between the different packages that they could potentially solve. QuartoNotebookRunner is not the only package where these conflicts would manifest so the larger ecosystem could benefit if this was solved somehow at the root instead.

I also think that interpolation from the Julia side to the R/Python side would get more complicated with separate processes. Everything is more complicated with multiple processes 😄

[MichaelHatherly]

I might give it a try because this would be really helpful for multi language tutorials.

There isn't even Python support in a release, it's only a very PoC PR, so it definitely should not be tackled until Python is a well supported feature of this package @felixcremer.

there's just conflicts between the different packages that they could potentially solve

Agreed, that would be the better solution here rather than adding another feature to this package just to get around problems in the ecosystem.