Multithread processing in python and segfault

[yg42]

Hello all,

I am new to diplib, so please excuse my naive comprehension of this fantastic tool.

I am trying to develop a multithread python code that processes multiple files. It appears that the reading of png files causes segfault. However, if the file type is tif, it does not segfaults. To reproduce the problem, put files in the Data folder and excute the following code.

Where is my mistake ?

Thank you in advance !
best regards
yann

import diplib as dip
from glob import glob
import concurrent.futures

def simulate_process(filename):
    # simulate
    img = dip.ImageRead(filename)

    return 0


def testParallelisation(REP0='Data', fmt='png'):
    # recursively list all png files in directory REP0
    listOfFiles = glob(f"{REP0}/**/*.{fmt}", recursive=True)

    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        futures = {executor.submit(simulate_process, filename): filename for filename in listOfFiles[:100]}

        for future in concurrent.futures.as_completed(futures):
              V = future.result()


testParallelisation(fmt='tif')    # ok 
testParallelisation(fmt='png') # segfault

[crisluengo]

Hi,

TIFF files are read through a built-in function, which is thread-safe. PNG files are read through the Bio-Formats Java library, which involves starting up a Java VM and so fourth. I had never really thought about this, but I'm not surprised that that process is not thread-safe.

@wcaarls you know much more about this than I do. Is there a way to make dipjavaio thread-safe? Or do we need to add a warning to the docs about using it only from one thread?

[wcaarls]

Currently, all calls to the JVM must be made on the same thread that first started the JVM. That is, that first called ImageReadJavaIO (see the GetEnv function in javio.cpp). I think it should be possible to make it multi-threaded using AttachCurrentThread, but it requires some looking into.

[yg42]

Hi

Thank you very much for your comments. My solution is to let OpenCV handle the reading, and convert the image to diplib:

cvimg = cv2.imread(filename)
img = dip.Image(cvimg)

It seems to be quite efficient. What do you think of this ?

yann

[crisluengo]

There is no data copy made when converting from a NumPy array (or compatible objects) to a dip.Image. In fact, you can give a NumPy array directly as input to any DIPlib function.

I tend to prefer using imageio for reading images when dipjavaio is not available to me. But OpenCV works fine too.

[crisluengo]

By the way, we implemented native PNG reading/writing in DIPlib in dae9724 (included in version 3.4.2). This works in a multi-threaded environment.