From 34324b0dbb8070952894128a7762cee6989bfb01 Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Tue, 12 Sep 2023 15:29:03 +0000 Subject: [PATCH] build based on 46731b4 --- dev/index.html | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/dev/index.html b/dev/index.html index e7d5b73..819c8b7 100644 --- a/dev/index.html +++ b/dev/index.html @@ -1,14 +1,14 @@ -Malt.jl · Malt
Edit on GitHub

Malt.jl

Malt is a multiprocessing package for Julia. You can use Malt to create Julia processes, and to perform computations in those processes. Unlike the standard library package Distributed.jl, Malt is focused on process sandboxing, not distributed computing.

Malt — Module

The Malt module doesn't export anything, use qualified names instead. Internal functions are marked with a leading underscore, these functions are not stable.

source

Malt workers

We call the Julia process that creates processes the manager, and the created processes are called workers. These workers communicate with the manager using the TCP protocol.

Workers are isolated from one another by default. There's no way for two workers to communicate with one another, unless you set up a communication mechanism between them explicitly.

Workers have separate memory, separate namespaces, and they can have separate project environments; meaning they can load separate packages, or different versions of the same package.

Since workers are separate Julia processes, the number of workers you can create, and whether worker execution is multi-threaded will depend on your operating system.

Malt.Worker — Type
Malt.Worker()

Create a new Worker. A Worker struct is a handle to a (separate) Julia process.

Examples

julia> w = Malt.worker()
-Malt.Worker(0x0000, Process(`…`, ProcessRunning))
source

Special workers

There are two special worker types that can be used for backwards-compatibility or other projects. You can also make your own worker type by extending the Malt.AbstractWorker type.

Malt.InProcessWorker — Type
Malt.InProcessWorker(mod::Module=Main)

This implements the same functions as Malt.Worker but runs in the same process as the caller.

source
Malt.DistributedStdlibWorker — Type
Malt.DistributedStdlibWorker()

This implements the same functions as Malt.Worker but it uses the Distributed stdlib as a backend. Can be used for backwards compatibility.

source

Calling Functions

The easiest way to execute code in a worker is with the remote_call* functions.

Depending on the computation you want to perform, you might want to get the result synchronously or asynchronously; you might want to store the result or throw it away. The following table lists each function according to its scheduling and return value:

FunctionSchedulingReturn value
Malt.remote_call_fetchBlocking<value>
Malt.remote_call_waitBlockingnothing
Malt.remote_callAsyncTask that resolves to <value>
Malt.remote_doAsyncnothing
Malt.remote_call_fetch — Function
Malt.remote_call_fetch(f, w::Worker, args...; kwargs...)

Shorthand for fetch(Malt.remote_call(…)). Blocks and then returns the result of the remote call.

source
Malt.remote_call_wait — Function
Malt.remote_call_wait(f, w::Worker, args...; kwargs...)

Shorthand for wait(Malt.remote_call(…)). Blocks and discards the resulting value.

source
Malt.remote_call — Function
Malt.remote_call(f, w::Worker, args...; kwargs...)

Evaluate f(args...; kwargs...) in worker w asynchronously. Returns a task that acts as a promise; the result value of the task is the result of the computation.

The function f must already be defined in the namespace of w.

Examples

julia> promise = Malt.remote_call(uppercase ∘ *, w, "I ", "declare ", "bankruptcy!");
+Malt.jl · Malt
Edit on GitHub
Malt.jl
Malt is a multiprocessing package for Julia. You can use Malt to create Julia processes, and to perform computations in those processes. Unlike the standard library package Distributed.jl, Malt is focused on process sandboxing, not distributed computing.
Malt — Module
The Malt module doesn't export anything, use qualified names instead. Internal functions are marked with a leading underscore, these functions are not stable.
source
Malt workers
We call the Julia process that creates processes the manager, and the created processes are called workers. These workers communicate with the manager using the TCP protocol.
Workers are isolated from one another by default. There's no way for two workers to communicate with one another, unless you set up a communication mechanism between them explicitly.
Workers have separate memory, separate namespaces, and they can have separate project environments; meaning they can load separate packages, or different versions of the same package.
Since workers are separate Julia processes, the number of workers you can create, and whether worker execution is multi-threaded will depend on your operating system.
Malt.Worker — Type
Malt.Worker()
Create a new Worker. A Worker struct is a handle to a (separate) Julia process.
Examples
julia> w = Malt.worker()
+Malt.Worker(0x0000, Process(`…`, ProcessRunning))
source
Special workers
There are two special worker types that can be used for backwards-compatibility or other projects. You can also make your own worker type by extending the Malt.AbstractWorker type.
Malt.InProcessWorker — Type
Malt.InProcessWorker(mod::Module=Main)
This implements the same functions as Malt.Worker but runs in the same process as the caller.
source
Malt.DistributedStdlibWorker — Type
Malt.DistributedStdlibWorker()
This implements the same functions as Malt.Worker but it uses the Distributed stdlib as a backend. Can be used for backwards compatibility.
source
Calling Functions
The easiest way to execute code in a worker is with the remote_call* functions.
Depending on the computation you want to perform, you might want to get the result synchronously or asynchronously; you might want to store the result or throw it away. The following table lists each function according to its scheduling and return value:
FunctionSchedulingReturn value
Malt.remote_call_fetchBlocking<value>
Malt.remote_call_waitBlockingnothing
Malt.remote_callAsyncTask that resolves to <value>
Malt.remote_doAsyncnothing
Malt.remote_call_fetch — Function
Malt.remote_call_fetch(f, w::Worker, args...; kwargs...)
Shorthand for fetch(Malt.remote_call(…)). Blocks and then returns the result of the remote call.
source
Malt.remote_call_wait — Function
Malt.remote_call_wait(f, w::Worker, args...; kwargs...)
Shorthand for wait(Malt.remote_call(…)). Blocks and discards the resulting value.
source
Malt.remote_call — Function
Malt.remote_call(f, w::Worker, args...; kwargs...)
Evaluate f(args...; kwargs...) in worker w asynchronously. Returns a task that acts as a promise; the result value of the task is the result of the computation.
The function f must already be defined in the namespace of w.
Examples
julia> promise = Malt.remote_call(uppercase ∘ *, w, "I ", "declare ", "bankruptcy!");
 
 julia> fetch(promise)
-"I DECLARE BANKRUPTCY!"
source
Malt.remote_do — Function
Malt.remote_do(f, w::Worker, args...; kwargs...)
Start evaluating f(args...; kwargs...) in worker w asynchronously, and return nothing.
Unlike remote_call, no reference to the remote call is available. This means:
  • You cannot wait for the call to complete on the worker.
  • The value returned by f is not available.
source
Evaluating expressions
In some cases, evaluating functions is not enough. For example, importing modules alters the global state of the worker and can only be performed in the top level scope. For situations like this, you can evaluate code using the remote_eval* functions.
Like the remote_call* functions, there's different a remote_eval* depending on the scheduling and return value.
FunctionSchedulingReturn value
Malt.remote_eval_fetchBlocking<value>
Malt.remote_eval_waitBlockingnothing
Malt.remote_evalAsyncTask that resolves to <value>
Malt.remote_eval — Function
Malt.remote_eval(mod::Module=Main, w::Worker, expr)
Evaluate expression expr under module mod on the worker w. Malt.remote_eval is asynchronous, like Malt.remote_call.
The module m and the type of the result of expr must be defined in both the main process and the worker.
Examples
julia> Malt.remote_eval(w, quote
+"I DECLARE BANKRUPTCY!"
source
Malt.remote_do — Function
Malt.remote_do(f, w::Worker, args...; kwargs...)
Start evaluating f(args...; kwargs...) in worker w asynchronously, and return nothing.
Unlike remote_call, no reference to the remote call is available. This means:
  • You cannot wait for the call to complete on the worker.
  • The value returned by f is not available.
source
Evaluating expressions
In some cases, evaluating functions is not enough. For example, importing modules alters the global state of the worker and can only be performed in the top level scope. For situations like this, you can evaluate code using the remote_eval* functions.
Like the remote_call* functions, there's different a remote_eval* depending on the scheduling and return value.
FunctionSchedulingReturn value
Malt.remote_eval_fetchBlocking<value>
Malt.remote_eval_waitBlockingnothing
Malt.remote_evalAsyncTask that resolves to <value>
Malt.remote_eval — Function
Malt.remote_eval(mod::Module=Main, w::Worker, expr)
Evaluate expression expr under module mod on the worker w. Malt.remote_eval is asynchronous, like Malt.remote_call.
The module m and the type of the result of expr must be defined in both the main process and the worker.
Examples
julia> Malt.remote_eval(w, quote
     x = "x is a global variable"
 end)
 
 julia> Malt.remote_eval_fetch(w, :x)
-"x is a global variable"
source
Malt.worker_channel — Function
Malt.worker_channel(w::AbstractWorker, expr)
Create a channel to communicate with worker w. expr must be an expression that evaluates to a Channel. expr should assign the Channel to a (global) variable so the worker has a handle that can be used to send messages back to the manager.
source
Exceptions
If an exception occurs on the worker while calling a function or evaluating an expression, this exception is rethrown to the host. For example:
julia> Malt.remote_call_fetch(m1, :(sqrt(-1)))
+"x is a global variable"
source
Malt.worker_channel — Function
Malt.worker_channel(w::AbstractWorker, expr)

Create a channel to communicate with worker w. expr must be an expression that evaluates to a Channel. expr should assign the Channel to a (global) variable so the worker has a handle that can be used to send messages back to the manager.

source

Exceptions

If an exception occurs on the worker while calling a function or evaluating an expression, this exception is rethrown to the host. For example:

julia> Malt.remote_call_fetch(m1, :(sqrt(-1)))
 ERROR: Remote exception from Malt.Worker on port 9115:
 
 DomainError with -1.0:
@@ -18,4 +18,4 @@
    @ Base.Math ./math.jl:33
  [2] sqrt
    @ ./math.jl:591 [inlined]
- ...

The thrown exception is of the type Malt.RemoteException, and contains two fields: worker and message::String. The original exception object (DomainError in the example above) is not availabale to the host.

Note

When using the async scheduling functions (remote_call, remote_eval), calling wait or fetch on the returned (failed) Task will throw a Base.TaskFailedException, not a Malt.RemoteException.

(The Malt.RemoteException is available with task_failed_exception.task.exception.)

Signals and Termination

Once you're done computing with a worker, or if you find yourself in an unrecoverable situation (like a worker executing a divergent function), you'll want to terminate the worker.

The ideal way to terminate a worker is to use the stop function, this will send a message to the worker requesting a graceful shutdown.

Note that the worker process runs in the same process group as the manager, so if you send a signal to a manager, the worker will also get a signal.

Malt.isrunning — Function
Malt.isrunning(w::Worker)::Bool

Check whether the worker process w is running.

source
Malt.stop — Function
Malt.stop(w::Worker; exit_timeout::Real=15.0, term_timeout::Real=15.0)::Bool

Terminate the worker process w in the nicest possible way. We first try using Base.exit, then SIGTERM, then SIGKILL. Waits for the worker process to be terminated.

If w is still alive, and now terinated, stop returns true. If w is already dead, stop returns false. If w failed to terminate, throw an exception.

source
Base.kill — Method
kill(w::Malt.Worker, signum=Base.SIGTERM)

Terminate the worker process w forcefully by sending a SIGTERM signal (unless otherwise specified).

This is not the recommended way to terminate the process. See Malt.stop.

source
Malt.interrupt — Function
Malt.interrupt(w::Worker)

Send an interrupt signal to the worker process. This will interrupt the latest request (remote_call* or remote_eval*) that was sent to the worker.

source
+ ...

The thrown exception is of the type Malt.RemoteException, and contains two fields: worker and message::String. The original exception object (DomainError in the example above) is not availabale to the host.

Note

When using the async scheduling functions (remote_call, remote_eval), calling wait or fetch on the returned (failed) Task will throw a Base.TaskFailedException, not a Malt.RemoteException.

(The Malt.RemoteException is available with task_failed_exception.task.exception.)

Signals and Termination

Once you're done computing with a worker, or if you find yourself in an unrecoverable situation (like a worker executing a divergent function), you'll want to terminate the worker.

The ideal way to terminate a worker is to use the stop function, this will send a message to the worker requesting a graceful shutdown.

Note that the worker process runs in the same process group as the manager, so if you send a signal to a manager, the worker will also get a signal.

Malt.isrunning — Function
Malt.isrunning(w::Worker)::Bool

Check whether the worker process w is running.

source
Malt.stop — Function
Malt.stop(w::Worker; exit_timeout::Real=15.0, term_timeout::Real=15.0)::Bool

Terminate the worker process w in the nicest possible way. We first try using Base.exit, then SIGTERM, then SIGKILL. Waits for the worker process to be terminated.

If w is still alive, and now terinated, stop returns true. If w is already dead, stop returns false. If w failed to terminate, throw an exception.

source
Base.kill — Method
kill(w::Malt.Worker, signum=Base.SIGTERM)

Terminate the worker process w forcefully by sending a SIGTERM signal (unless otherwise specified).

This is not the recommended way to terminate the process. See Malt.stop.

source
Malt.interrupt — Function
Malt.interrupt(w::Worker)

Send an interrupt signal to the worker process. This will interrupt the latest request (remote_call* or remote_eval*) that was sent to the worker.

source
Malt.TerminatedWorkerException — Type

Malt will raise a TerminatedWorkerException when a remote_call is made to a Worker that has already been terminated.

source