tif inferred as pickle

[robmarkcole]

🐛 Bug

Returning paths to tifs, but they are inferred as pickle:

# dataset yields paths 

test_dataset[0] == {'image_path': '/teamspace/studios/this_studio/dataset/test/T20MMT_20200505T142729_tile_17_11.tif',
 'mask_path': '/teamspace/studios/this_studio/dataset/test/T20MMT_20200505T142729_tile_17_11_mask.tif',
 'image_id': 'T20MMT_20200505T142729_tile_17_11'}

Logs

Rank 1 inferred the following `['str', 'pickle', 'pickle']` data format.

But expect:

Rank 1 inferred the following `['str', 'tif', 'tif']` data format. 

The script generates the same number of chunk files for both train and test, although train has significantly more tifs..

litdata==0.2.19

Reproducible example:

from litdata.streaming.serializers import _SERIALIZERS

image_path = '/teamspace/studios/this_studio/dataset/test/T10VDK_20200519T193911_tile_0_7.tif'

def evaluate_serializer(filename: str) -> str:
    """
    Evaluate which serializer would be used for a given filename.

    Args:
        filename: The name of the file to evaluate.

    Returns:
        The name of the selected serializer.
    """
    # Iterate through serializers in the order defined in _SERIALIZERS
    for serializer_name, serializer in _SERIALIZERS.items():
        if serializer.can_serialize(filename):
            return serializer_name

    # If no serializer can handle the file, raise an exception
    raise ValueError(f"No suitable serializer found for filename: {filename}")

assert evaluate_serializer(image_path) == 'pickle'

[robmarkcole]

OK, I see

# FileSerializer will be removed in the future.

With this going away, appears tif need a new serializer? Prototyped a solution using tifffile which supports multispectral data (pillow is limited)