Decompression error if the zso file is Block Size set to 2352

[GDX-X]

command used for compressing: ziso --cache-size 4 --replace --lz4hc -b 2352 -h -i "game.iso" -o "game.zso"

[Danixu]

Hello,

Thanks for the report. I'll take a look.

Best regards

[Danixu]

Hello,

Fixed in the latest release:
https://github.com/Danixu/ziso_compressor/releases/tag/v0.5.2

Best regards.

[Danixu]

I did no see the blocksize of your image, but is wrong considering the arguments. I have fixed the problem too in the above release.

Best regards

[GDX-X]

it's me again,😅 I haven't tried since the update but I still have this bug

command used:
ziso --cache-size 4 --replace --lz4hc --block-size 2352 --hdl-fix -i "game2.zso" -o "game1.iso"

[Danixu]

Hello, it's strange because there was a bug in the way the blocksize was managed, but I have fixed it.

• The compression command that you have used, is the same as the decompression command that you have posted?.
• What game are you trying to compress?

I'll give a try with several of my games to see if I can trigger the problem.

Greetings!

[malvarenga123]

Is the block size related to the sector size? If yes, you can't output a compliant ISO with a sector size of 2352. It has to be in bin/cue format.

[GDX-X]

Yes I use the same command I just changed the file format .zso > .iso

only bin/cue games converted to iso then converted to zso in sector size 2352 cannot be uncompressed

this does this with all games converted to sector size 2352 mainly with bin/cue as I don't use this size for DVD ISO

[Danixu]

Hello,

Is the block size related to the sector size? If yes, you can't output a compliant ISO with a sector size of 2352. It has to be in bin/cue format.

In this case is not relevant, because the compressor just read data and compress it in chunks (blocks) of the provided size. It doesn't change anything in the image, so you'll get the same image when decrompressed setting the blocksize to 2352, 23520 or anything else.
The only differences are:

• The compression ratio, that is higher when the blocksize is bigger (the compressor have more data to search similarities).
• The memory and cpu cycles required by the decompressor to decompress the block.

For this reason, what is recommended is to set the blocksize to the image sector size, then when the decompressor wants a sector doesn't have to decompress more than a block if the blocksize is smaller, or to decompress maybe non usefull data when the blocksize is bigger. The sector size of a CDROM is 2352, and for a DVD 2048.
Also, the PS2 memory is limited, so to create big blocks can lead to problems.

Yes I use the same command I just changed the file format .zso > .iso

only bin/cue games converted to iso then converted to zso in sector size 2352 cannot be uncompressed

this does this with all games converted to sector size 2352 mainly with bin/cue as I don't use this size for DVD ISO

I'll give a try with those settings to see if maybe is related with them. Which game are you trying to compress?, to check if maybe is related with that game too.

Greetings!!

[malvarenga123]

In this case is not relevant, because the compressor just read data and compress it in chunks (blocks) of the provided size. It doesn't change anything in the image, so you'll get the same image when decrompressed setting the blocksize to 2352, 23520 or anything else.

So it can't be an ISO file, which only has user data. It needs to be in a raw format (bin or img).

The sector size of a CDROM is 2352, and for a DVD 2048.

Not if it's in a compliant ISO file.

https://en.wikipedia.org/wiki/Optical_disc_image

ISO files store only the user data from each sector on an optical disc, ignoring the control headers and error correction data, and are therefore slightly smaller than a raw disc image of optical media. Since the size of the user-data portion of a sector (logical sector) in data optical discs is 2,048 bytes, the size of an ISO image will be a multiple of 2,048.

There may be plenty of PS2 CD games running around in actual ISO format, therefore using a sector of 2048 bytes. You can also have bin/cue files in Mode1/2048.
So some sort of check needs to be done in order to output (decompress) the files in the proper format. To be honest, I'm not really sure if the ZSO format was made with anything in mind besides ISO's with a sector of 2048 bytes as it was mainly made for PSP discs. Also that's probably the reason OPL has trouble with anything that's not a multiple of 2048.

[Danixu]

In this case is not relevant, because the compressor just read data and compress it in chunks (blocks) of the provided size. It doesn't change anything in the image, so you'll get the same image when decrompressed setting the blocksize to 2352, 23520 or anything else.

So it can't be an ISO file, which only has user data. It needs to be in a raw format (bin or img).

The sector size of a CDROM is 2352, and for a DVD 2048.

Not if it's in a compliant ISO file.

I have not seen a CD image without the headers and the correction code. Maybe an ISO is not compliant, but I have got no problem with it. Even I've converted bin images to ISO using programs like UltraISO and it keeps all. Of course is better a bin/cue.

https://en.wikipedia.org/wiki/Optical_disc_image

ISO files store only the user data from each sector on an optical disc, ignoring the control headers and error correction data, and are therefore slightly smaller than a raw disc image of optical media. Since the size of the user-data portion of a sector (logical sector) in data optical discs is 2,048 bytes, the size of an ISO image will be a multiple of 2,048.

There may be plenty of PS2 CD games running around in actual ISO format, therefore using a sector of 2048 bytes. You can also have bin/cue files in Mode1/2048. So some sort of check needs to be done in order to output (decompress) the files in the proper format. To be honest, I'm not really sure if the ZSO format was made with anything in mind besides ISO's with a sector of 2048 bytes as it was mainly made for PSP discs. Also that's probably the reason OPL has trouble with anything that's not a multiple of 2048.

I'm not sure about it, but maybe it was created mainly to be used on the PSP which uses UMD disks. Those disks are similar to DVD, and their sector size is 2048. Anyway, to create this program I've studied the ZSO format, and it just contains a header, an index, and the compressed data in blocks. The ZSO format and those blocks are data format independent and doesn't cares if you compress an ISO image, a bin image, or anything else. The only one that cares about the compressed data is the reader.
A compliant CD-ROM must contains the whole 2352 sector to works, and this is not different in the PS2 CD games and the images readed by the OPL. That is why a correct ZSO from a CD-ROM must contains the whole 2352 bytes of every sector. To improve the performance of the reader, the best is to sync the ZSO blocks size to the sector size to avoid an extra overhead when those sectors are readed.

The only format that I know that removes the CD-ROM sector headers and error correction data, is the ECM format. This is because the program has mathematical formulas to regenerate the correction code from the data and an index to recover the headers, and with this data it is able to recover the exact original 2352 sectors and generate a compliant CD-ROM again.

Best regards.

[malvarenga123]

According to ihaveanosebleed at fileforums, all Playstation (1 and 2) CD-ROMs are in Mode 2/2352, so there should not be any ISO files for them, if they were properly dumped.

The only format that I know that removes the CD-ROM sector headers and error correction data, is the ECM format.

As I've been saying, ISO files also do not have that data, as only user data is stored. If you have ISO files with 2352-byte sectors, it's just a renamed raw bin/img.
https://docs.fileformat.com/compression/iso/

It's easy to create an ISO from a PS2 CD-ROM. It probably won't run on real hardware, but it's a valid ISO nonetheless.
Take a look at Ico (USA) for example:

FILE "Ico (USA).bin" BINARY
  TRACK 01 MODE2/2352
    INDEX 01 00:00:00

Bin Size: 609 MB (638.640.912 bytes)
MD5: 12e5df728050e253bb455e49abf8ef24

If I create an ISO instead:
ISO Size: 530 MB (556.095.488 bytes)
MD5: f8a8b5036157c4aa4342489f7e608482

So if you can't determine the sector size directly from the ZSO file, then there should be an additional argument so that the decompressed file is on the correct format (ISO for 2048-byte sectors and BIN for 2352-byte sectors). Ideally the cuesheet should be recreated for the second case.

[Danixu]

According to ihaveanosebleed at fileforums, all Playstation (1 and 2) CD-ROMs are in Mode 2/2352, so there should not be any ISO files for them, if they were properly dumped.

The only format that I know that removes the CD-ROM sector headers and error correction data, is the ECM format.

As I've been saying, ISO files also do not have that data, as only user data is stored. If you have ISO files with 2352-byte sectors, it's just a renamed raw bin/img. https://docs.fileformat.com/compression/iso/

It's easy to create an ISO from a PS2 CD-ROM. It probably won't run on real hardware, but it's a valid ISO nonetheless. Take a look at Ico (USA) for example:

FILE "Ico (USA).bin" BINARY
  TRACK 01 MODE2/2352
    INDEX 01 00:00:00

Bin Size: 609 MB (638.640.912 bytes) MD5: 12e5df728050e253bb455e49abf8ef24

If I create an ISO instead: ISO Size: 530 MB (556.095.488 bytes) MD5: f8a8b5036157c4aa4342489f7e608482

So if you can't determine the sector size directly from the ZSO file, then there should be an additional argument so that the decompressed file is on the correct format (ISO for 2048-byte sectors and BIN for 2352-byte sectors). Ideally the cuesheet should be recreated for the second case.

A PS1/PS2 CD-ROM correct dump always have 2352 bytes per sector, and just reading the first 12 bytes of an image you're able to determine if the image contains a full sector or just the user data, using the sector sync header part which is always the same. I don't know how the OPL determines if the image is a CD-ROM or a DVD-ROM, and maybe even it doesn't cares because is the PS2 which determines if the image is a CD-ROM or a DVD-ROM, but it doesn't need a CUE file to do it. If you take a look to my source code, I am doing this detection in a function:

bool is_cdrom(std::fstream &fIn)
{
    // Store the current position
    uint64_t currentPos = fIn.tellg();

    // Seek to the start point
    fIn.seekg(0);

    // Read three sectors to ensure that the disk is a CDROM
    std::vector<char> buffer(12, 0);
    unsigned char cdSync[] = {0x00, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x00};
    for (uint8_t i = 0; i < 3; i++)
    {
        fIn.seekg(i * 2352);
        fIn.read(buffer.data(), buffer.size());

        if (
            std::strncmp(buffer.data(), (char *)cdSync, 12) != 0)
        {
            fIn.seekg(currentPos);
            return false;
        }
    }
    fIn.seekg(currentPos);
    return true;
}

I read three sectors to limit the possibility of a false detection if the user data starts with something similar to a sector sync data, but you can see how easy is. Of course a CDDA image will not be detected as CD-ROM, but If I'm not wrong, PSX/PS2 games never starts with a CDDA track.

EDIT: Even more, you can determine the CD-ROM sector type (mode), with just a few mathematical operations, The ISO to ECM converter uses that mathematical operation to determine which data to remove from every sector.

The CUE file is an index file created mainly for the PC programs, and it can even be recreated just reading the image.

Best regards.

[Danixu]

Hello, I forgot to answer this part:

So if you can't determine the sector size directly from the ZSO file, then there should be an additional argument so that the decompressed file is on the correct format (ISO for 2048-byte sectors and BIN for 2352-byte sectors). Ideally the cuesheet should be recreated for the second case.

The ZSO file is a RAW compressed file, which means that only compress bytes and no sectors. Is like any other compression format which doesn't cares the data you compress, and uses ZLIB like ZIP files. You don't need to know the sector size of the original image because it doesn't modify the sectors in any way. You compress a 2352 sector size image at 2048 block size?, doesn't cares, at the end of the decompression the file will be the same 2352 sector size image because is threated as RAW data.
The only difference with the ZIP files for example, is that the ZSO format was created to be seekable, so it is compressed in small independant blocks and contains an index file to know the exact point to where to go to decompress the required data.

The reader doesn't uses the ZSO info to determine the sector size in any way, It will does in any other way like for example the way I have provided above. It just read the index, and decompress the block or blocks that it requires to read a sector and send that sector to the hardware.
The reason to sync the block size to the sector size is just for performance. With smaller blocks or blocks wich doesn't matches the sector size, you will have to decompress more than one block to get the original sector, and with bigger blocks you will have to decompress several sectors in the block to get just the required sector. This will increase the CPU and Memory requirements and slowdown the reader. On a computer doesn't cares too much, but on a PS2 with a limited hardware it is important.

Best regards.

[Danixu]

Yes I use the same command I just changed the file format .zso > .iso

only bin/cue games converted to iso then converted to zso in sector size 2352 cannot be uncompressed

this does this with all games converted to sector size 2352 mainly with bin/cue as I don't use this size for DVD ISO

Hello,

Checking the releases I have seen that the one that I sent you is not the latest. Please, can you try the latest release?:

https://github.com/Danixu/ziso_compressor/releases

It includes a fix for a bug in the decompression:

Added more logging and fixed a bug decompressing (Danixu)

Maybe I sent you the wrong version by error.

Best regards.

[malvarenga123]

A PS1/PS2 CD-ROM correct dump always have 2352 bytes per sector, and just reading the first 12 bytes of an image you're able to determine if the image contains a full sector or just the user data, using the sector sync header part which is always the same.

That's the thing. You can't always assume the dump is correct, as I showed you with Ico. If the disc does not have audio, one can always create an ISO (2048-byte sector) from it.

The ZSO file is a RAW compressed file, which means that only compress bytes and no sectors.

If this is true, why does the program need to know if the disc is a CD or DVD? Why does it need the function is_cdrom()?

[Danixu]

The ZSO file is a RAW compressed file, which means that only compress bytes and no sectors.

If this is true, why does the program need to know if the disc is a CD or DVD? Why does it need the function is_cdrom()?

Just to select the blocksize according to the sector size if it's not provided by the user, to improve the performance on the reader. As I've explained above, the reader reads better if only have to decompress one block to read a sector, and I've added a function to do it automatically.

[GDX-X]

Yes I use the same command I just changed the file format .zso > .iso
only bin/cue games converted to iso then converted to zso in sector size 2352 cannot be uncompressed
this does this with all games converted to sector size 2352 mainly with bin/cue as I don't use this size for DVD ISO

Hello,

Checking the releases I have seen that the one that I sent you is not the latest. Please, can you try the latest release?:

https://github.com/Danixu/ziso_compressor/releases

It includes a fix for a bug in the decompression:

Added more logging and fixed a bug decompressing (Danixu)

Maybe I sent you the wrong version by error.

Best regards.

tested with this version I have the same problem

I try to do it with half-life but it does it for all CD type games

my steps

1 - Convert bin/cue to iso
2 - Convert iso to zso with 2352 sector size
3 - Decrompress zso to iso

[10522]

@GDX-X, @Danixu, @malvarenga123, check out this and this.

@JoseAaronLopezGarcia can probably help with this issue.

[JoseAaronLopezGarcia]

I've never tested anything other block size than 2048 or 8192. For OPL we only support 2048 anyways.

[Danixu]

@GDX-X, @Danixu, @malvarenga123, check out this and this.

@JoseAaronLopezGarcia can probably help with this issue.

I've never tested anything other block size than 2048 or 8192. For OPL we only support 2048 anyways.

I see, thank you both for the info.
So I can suppose that games with CDDA tracks or any protection that rely on the ECC and EDC data are not supported yet.

Anyway, this bug is related with my compressor, so I'd like to fix it. The problem here is that the compressed data should be damaged in any way when compressed, and then the decompression fails. Is independent of the OLP limit. I'm not sure about the reason because in the latest version I fixed a bug with a variable overflow that was causing an wrong block size and I remember to have tested it, but maybe there's something else related with the selected arguments.

I'll remove the automatic block size change to avoid problems with the OLP, and I'll add a warning about it when a CD-ROM is compressed.
Maybe even I'll add a function to remove the headers, the EDC and the ECC data from the original CD-ROM, keeping just the data. I can reuse some code that I did improving the ECM format to make it seeable 😁.

Best regards

[Danixu]

Yes I use the same command I just changed the file format .zso > .iso
only bin/cue games converted to iso then converted to zso in sector size 2352 cannot be uncompressed
this does this with all games converted to sector size 2352 mainly with bin/cue as I don't use this size for DVD ISO

Hello,
Checking the releases I have seen that the one that I sent you is not the latest. Please, can you try the latest release?:
https://github.com/Danixu/ziso_compressor/releases
It includes a fix for a bug in the decompression:

Added more logging and fixed a bug decompressing (Danixu)

Maybe I sent you the wrong version by error.
Best regards.

tested with this version I have the same problem

I try to do it with half-life but it does it for all CD type games

my steps

1 - Convert bin/cue to iso 2 - Convert iso to zso with 2352 sector size 3 - Decrompress zso to iso

Hello,

Looks like the problem is related with the Bin2ISO conversion, that make the compressor to fail. I still investigating the problem, because with the original BIN file works perfectly.

Best regards.

EDIT:

Found and almost fixed the bug. Was a missalign when reading the input file: the data was read before the start point. This part is fixed and working.

I have also fixed another bug that I have found that caused the last block to not be decompressed when its size doesn't fit the blocksize, but the LZ4 decompressor is not reporting the correct size when decompressing that block and then the output file is not an exact copy (contains extra data). Once I locate a way to make the LZ4 decompressor to report the correct size, I'll fix the bug. If not, I'll use the header size to do it.

[Danixu]

Yes I use the same command I just changed the file format .zso > .iso
only bin/cue games converted to iso then converted to zso in sector size 2352 cannot be uncompressed
this does this with all games converted to sector size 2352 mainly with bin/cue as I don't use this size for DVD ISO

Hello,
Checking the releases I have seen that the one that I sent you is not the latest. Please, can you try the latest release?:
https://github.com/Danixu/ziso_compressor/releases
It includes a fix for a bug in the decompression:

Added more logging and fixed a bug decompressing (Danixu)

Maybe I sent you the wrong version by error.
Best regards.

tested with this version I have the same problem

I try to do it with half-life but it does it for all CD type games

my steps

1 - Convert bin/cue to iso 2 - Convert iso to zso with 2352 sector size 3 - Decrompress zso to iso

Hello,

Check the latest release, it's fixed now. If the image that you are using is a converted ISO, then leave the blocksize to 2048. Also the above links says that the OPL is not compatible with blocks sizes other with 2048, so it's better to leave the default blocks size.

Best regards.