esptool.py fails with error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 (micropython build) (ESPTOOL-951)

[littlegreenbean33]

Operating System

ubutu 20.04

Esptool Version

4.7

Python Version

3.10

Full Esptool Command Line that Was Run

esptool.py -p /dev/ttyUSB0 -b 460800 --force --before default_reset --after no_reset --chip esp32s3 write_flash --flash_mode dio --flash_size 8MB --flash_freq 80m 0x0 tmp/bootloader-enc.bin 0x8000 tmp/partition-table-enc.bin 0x10000 tmp/micropython-enc.bin Traceback (most recent call last):

Esptool Output

Traceback (most recent call last):
  File "/home/julian/.espressif/python_env/idf5.1_py3.10_env/bin/esptool.py", line 37, in <module>
    esptool._main()
  File "/home/julian/.espressif/python_env/idf5.1_py3.10_env/lib/python3.10/site-packages/esptool/__init__.py", line 1139, in _main
    main()
  File "/home/julian/.espressif/python_env/idf5.1_py3.10_env/lib/python3.10/site-packages/esptool/__init__.py", line 669, in main
    args = parser.parse_args(argv)
  File "/usr/lib/python3.10/argparse.py", line 1845, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib/python3.10/argparse.py", line 1878, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python3.10/argparse.py", line 2073, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
  File "/usr/lib/python3.10/argparse.py", line 2050, in consume_positionals
    take_action(action, args)
  File "/usr/lib/python3.10/argparse.py", line 1955, in take_action
    action(self, namespace, argument_values, option_string)
  File "/usr/lib/python3.10/argparse.py", line 1233, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
  File "/usr/lib/python3.10/argparse.py", line 1878, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/usr/lib/python3.10/argparse.py", line 2094, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "/usr/lib/python3.10/argparse.py", line 2050, in consume_positionals
    take_action(action, args)
  File "/usr/lib/python3.10/argparse.py", line 1955, in take_action
    action(self, namespace, argument_values, option_string)
  File "/home/julian/.espressif/python_env/idf5.1_py3.10_env/lib/python3.10/site-packages/esptool/__init__.py", line 1113, in __call__
    argfile = intel_hex_to_bin(argfile, address)
  File "/home/julian/.espressif/python_env/idf5.1_py3.10_env/lib/python3.10/site-packages/esptool/bin_image.py", line 51, in intel_hex_to_bin
    ih.loadhex(file.name)
  File "/home/julian/.espressif/python_env/idf5.1_py3.10_env/lib/python3.10/site-packages/intelhex/__init__.py", line 210, in loadhex
    for s in fobj:
  File "/usr/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf0 in position 2: invalid continuation byte

What is the Expected Behaviour?

I ha externally generated key for flash encryption. In developer mode all goes smooth. In release mode bootloader and partition are fine ( after externally encrypted before sending to chip with esptool). However app.image ( micropython custom build) fails at flashing with esptool.py

More Information

No response

Other Steps to Reproduce

Crashes on micropython-enc.bin in all executions

[peterdragun]

Hi @littlegreenbean33, please try using the latest release of esptool 4.8.1. I believe that a2bdaa2 has already fixed this.

[littlegreenbean33]

@peterdragun Yes this works. Many thanks.

If only I get this issue

julian@portatilU:~/tmp/proyectos_python/firmware_micropython/micropython/ports/esp32$ idf.py monitor
The following Python requirements are not satisfied:
Requirement 'esptool~=4.7.0' was not met. Installed version: 4.8.1

Shall I go any further and reinstall all tools ? how should I go to get all necessary packages aligned ?

Many Thanks
JC

[dobairoland]

Crossref.: https://esp32.com/viewtopic.php?f=13&t=42543&p=139581#p139581

[littlegreenbean33]

apologies if I spammed the team.

[dobairoland]

Requirement 'esptool~=4.7.0' was not met. Installed version: 4.8.1

I assume that you are using ESP-IDF v5.2 or older. There is an outstanding issue there which prevents us from updating to the newest esptool version. It should be solved soon.

The workaround is to edit (or remove) the esptool requirement in ~/.espressif/espidf.constraints.v5.2.txt. The file name is dependent on you ESP-IDF version. You might need to do this every day (until the fix) because the file is downloaded every day. If you don't use pytest-embedded then there is no known issue for doing this hack.

[littlegreenbean33]

Thanks, my issue really is Micropython "recommends" certain versions of IDF.

Building up the binary firmware is the last step, so not so much an inconvenience. I work in an industrial environment and changes on devices can harm people. So we need to lock down changes, confidentiality is less of an issue. Hence, I follow this path freezing changes in the device via encryption.

[left-win]

Hi @littlegreenbean33, please try using the latest release of esptool 4.8.1. I believe that a2bdaa2 has already fixed this.
I have encountered the same problem, and it can be solved by upgrading the version of esptool. Thank you very much

[littlegreenbean33]

Hi @littlegreenbean33, please try using the latest release of esptool 4.8.1. I believe that a2bdaa2 has already fixed this.
I have encountered the same problem, and it can be solved by upgrading the version of esptool. Thank you very much

following @dobairoland recommendation, things worked fine.

I work with multiple version of IDF so it took me a while to set up script to dynamically alternate between environments and avoid all being locked at ~/.espressif