Rust/Python versions signature detection parity

[EXL]

Binwalk 2.3.3, Python:

$ binwalk boot.img 
DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             Android bootimg, kernel size: 3011816 bytes, 
2560          0xA00           Linux kernel ARM boot executable zImage (little-endian)
17484         0x444C          xz compressed data
17716         0x4534          xz compressed data
3015168       0x2E0200        gzip compressed data, from Unix, last modified: 1970-01-01 00:00:00 (null date)

Binwalk 3.1.0, Rust:

$ binwalk boot.img

                                                 /Users/madomr/Downloads/boot.img
-----------------------------------------------------------------------------------------------------------------------------------
DECIMAL                            HEXADECIMAL                        DESCRIPTION
-----------------------------------------------------------------------------------------------------------------------------------
17716                              0x4534                             XZ compressed data, total size: 2996084 bytes
3015168                            0x2E0200                           gzip compressed data, operating system: Unix, timestamp:
                                                                      1970-01-01 00:00:00, total size: 832955 bytes
-----------------------------------------------------------------------------------------------------------------------------------

Analyzed 1 file for 85 file signatures (187 magic patterns) in 12.0 milliseconds

boot.zip

[devttys0]

The first two signatures (Android bootimg and Linux kernel ARM boot) are easy to explain: these signatures have not (yet) been ported over to Binwalkv3.

The XZ compressed data at offset 0x444C reported by Binwalkv2 is a false postive, even though it seems to extract successfully. Looking at a hex dump of the boot.img file, the 7zXZ magic bytes at offset 0x444C are just a string, followed by a few NULL bytes and some more strings; it is clearly not a block of compressed data:

If you take all the data from offset 0x444C to the end of the boot.img file and pass it to the 7z utility (which is what Binwalkv2 effectively does), this is the output:

craig@laptop:~/Downloads$ 7z x test.xz 

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz (806EA),ASM,AES-NI)

Scanning the drive for archives:
1 file, 3830708 bytes (3741 KiB)

Extracting archive: test.xz
         
ERRORS:
There are data after the end of archive

WARNING:
test.xz
Can not open the file as [xz] archive
The file is open as [gzip] archive

--
Path = test.xz
Open WARNING: Can not open the file as [xz] archive
Type = gzip
ERRORS:
There are data after the end of archive
Offset = 2997684
Physical Size = 832955
Tail Size = 69
Headers Size = 10
Streams = 1

ERROR: There are some data after the end of the payload data : test

Sub items Errors: 1

Archives with Errors: 1

Open Errors: 1

Sub items Errors: 1

7zip is a great utility, but its results can be misleading sometimes. Here, 7z recognizes that the data provided to it is not a valid XZ file, so it scans the contents of the data it was provided and finds some gzip data, and decompresses that instead. This is the same gzip data that Binwalk identified inside of boot.img at offset 0x2E0200. This means that this gzip data is inadvertently extracted twice with Binwalkv2.

Binwalkv3 correctly recognizes that the XZ signature at offset 0x444C is not valid, and does not report it.

[EXL]

Thanks for the explanation! I'll wait for the new Android and Linux signatures.