espflash save-image always creates a binary which is the bootloader, partition table and factory image concatenated together

[ivmarkov]

I.e. invoking (via the command line, or via the espflash API):

espflash save-image --chip esp32s3 target/xtensa-esp32s3-espidf/debug/my-firmware my-firmware.bin

... would output a my-firmware.bin which - sequentially:

• Contains an optional empty page (4096 bytes of 0xff) for those chips where the bootloader starts at offset 0x1000
• Contains the bootloader itself - either the embedded one or whatever the user had supplied on the command line
• Contains the partition table padded to 4096 bytes
• Contains empty space (0xff) until the offset where the app factory image starts in the partition table
• Contains the app factory image itself

Now, I get it that this way it is maybe simpler for the user - in un-sophisticated scenarios - to flash the binary by just doing then espflash write-bin --chip esp32s3 0 my-firmware-.bin but this is making it difficult for that very same user to cover the scenario where she just wants her .ELF to be converted to .bin on the command-line without any bootloader and partition-table concatenations in-between.

Even when using the espflash API rather than the command line, it is a bit weird, as I still have to deal with the bootloader and the partition table, even though - as mentioned - I just want to convert the .ELF to bin no extra stuff included thank you very much :-)

Also the question is a bit - in those un-sophisticated scenarios - the user is likely not going through the two step process of "save-image" and "write-bin" anyway. She probably just wants to do espflash flash --chip esp32s3 target/xtensa-esp32s3-espidf/debug/my-firmware (optionally passing a custom bootloader image and part-table on the command line) and then the "concatenation" is invisible anyways as it happens internally to espflash. Which is fine.

So why not supporting a variant of "save-image" which is doing the "no-thrills" stuff of outputting a binary app image and that's it?
And then separate commands for converting and outputting the embedded bootloader to .bin and the embedded or externally-passed partition-table CSV to .bin?

[chris-subtlebytes]

By coincidence I was finally getting around to posting some of my patches I've had because I needed to sign and encrypt individual partitions and found this report posted on the same day. I'm not sure my patch addresses everything listed above though.

$ espflash save-image --help
Generate a binary application image and save it to a local disk

If the '--merge' option is used, then the bootloader, partition table, and all application segments
will be merged into a single binary file. Otherwise, each segment will be saved as individual
binaries, prefixed with their intended addresses in flash.

Actual:

$ espflash save-image --chip esp32c3 --bootloader target/riscv32imc-esp-espidf/release/bootloader.bin --partition-table partitions.csv --partition-table-offset 0x000c000 target/riscv32imc-esp-espidf/release/my-project /tmp/espflash/my-image.bin

$ tree /tmp/espflash
/tmp/espflash
└── my-image.bin # <-- only a single image is generated even though --merge is not specified

1 directory, 1 file

Expected each segment to be saved as individual binaries. This is critical for signing and encrypting partitions individually

With my proposed patch:

$ espflash save-image --chip esp32c3 --bootloader target/riscv32imc-esp-espidf/release/bootloader.bin --partition-table partitions.csv --partition-table-offset 0x000c000 target/riscv32imc-esp-espidf/release/my-project /tmp/espflash/my-image.bin

$ tree /tmp/espflash
/tmp/espflash
├── 0x0000000_my-image.bin # bootloader image
├── 0x000c000_my-image.bin # partition table
└── 0x0010000_my-image.bin # factory app image

1 directory, 3 files

NOTE: the default output now matches the documentation but if anyone has built any automation around the current behavior, they will break.

[ivmarkov]

UPDATED: I confused save-image with write-bin. Suggestion reworded

Thank you for chiming-in! Glad to see other folks are also looking into this.

An alternative might be to keep save-image as-is, and then introduce three new sub-commands:

• save-firmware [?? --chip CCC] [--secure-pad-v2] <input-elf> <output-bin-path> - Takes an ELF file and converts to .BIN format. Does not consult the partition table, does not check flash size etc. etc. If --secure-pad-v2 is present, pad the image as described in espflash save-image does not support secure padding #713. If save-firmware is not capable of secure-padding, it would be useless anyway for the purposes of Secure Boot V2 and signing images, so you'll have to use the esptools tool ... equivalent;
• save-bootloader --chip CCC [--part-table-offset XXX] [--flash-size YYY ] <output-bin-path> - Writes a default bootloader .BIN image, possibly patched to support partition table offset XXX and flash size YYY;
• save-part-table [?? --chip CCC] <part-table-csv> <output-bin-path> - Converts a .CSV partition table to a .BIN format.

BTW - and for the super-short-term (I'm really pressed by time) - there is now as mentioned esptools which has the esptool feature, so you can do (some) of the above stuff by just doing cargo install --git https://github.com/ivmarkov/esptools; esptools tool -h

You'll need esptools (or at least the espefuse binary or .py anyway) if you are looking into secure boot, as you need to burn efuse-es in the factory, and having an espefuse equivalent into espflash might be quite a bit of a lift-and-shift and a trash can full of wasted esp chips until we get it right...

[chris-subtlebytes]

I actually have a full working writeup on how to enable encryption and save images with espflash except for espsecure for efuses and one esptool command. I'm trying to get stuff ready as soon as I can. Maybe we can streamline it a bit with some tooling suggestions, especially for writing multiple partitions at once. All in good time.

No need for anyone to trash a bunch of ESPs anymore. The worst I did to mine was permanently enable Secure Download Mode and disable JTAG.

[ivmarkov]

I actually have a full working writeup on how to enable encryption and save images with espflash except for espsecure for efuses and one esptool command. I'm trying to get stuff ready as soon as I can. Maybe we can streamline it a bit with some tooling suggestions, especially for writing multiple partitions at once. All in good time.

I'm in the mid-to-end of my own journey in doing this. I'm using the following:

• esptool - for converting the firmware ELF to .BIN, as I do need secure padding, and even with the proposed fix here, espflash would still be useless for that task, as it cannot properly pad the image, as per espflash save-image does not support secure padding #713
• espsecure - for signing the images (actually, I do use my own pure-Rust espsign thing but I might switch temporarily to espsecure if I hit a roadblock / bugs I can't fix on time in my own espsign code)
• espefuse - for burning the efuse signing key RSA hash / for flipping various efuse bits => still to be tested
• espflash - for flashing, because it has better progress reporting than esptool

How do you deploy the whole circus at the factory premises?

I did over-design a bit and rather than a python/shell script that glues it all together, I have a WIP Rust tool - espfactory - that can pull from s3/http/filesystem something called a "bundle" (really, just a zip of the CSV partition table + all .BIN files + all e-fuses) and then just flash and burn those, while showing to the user an interactive UI rather than just a bunch of log lines. But this is still a bit WIP...

No need for anyone to trash a bunch of ESPs anymore. The worst I did to mine was permanently enable Secure Download Mode and disable JTAG.

No by trashing a bunch of ESPs what I mean is only if we (or somebody) tries to re-write espefuse.py into Rust and then merge this code into espflash, so that we have a pure-Rust tooling for burning efuses. Just using the Espressif Python tool (espefuse.py) is of course quite easy.