Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

generate_test_code.py does not support Unicode in .function files #64

Open
gilles-peskine-arm opened this issue Oct 30, 2024 · 0 comments · May be fixed by #66
Open

generate_test_code.py does not support Unicode in .function files #64

gilles-peskine-arm opened this issue Oct 30, 2024 · 0 comments · May be fixed by #66
Labels
bug Something isn't working size-xs Estimated task size: extra small (a few hours at most)

Comments

@gilles-peskine-arm
Copy link
Contributor

gilles-peskine-arm commented Oct 30, 2024 •
edited
Loading

generate_test_code.py supports non-ASCII characters in .data files, but not portably in .function files. Sample backtrace (from Mbed-TLS/mbedtls#9739 at Mbed-TLS/mbedtls@db226b4):

Traceback (most recent call last):
  File "../framework/scripts/generate_test_code.py", line 1294, in <module>
    main()
  File "../framework/scripts/generate_test_code.py", line 1289, in main
    c_file=out_c_file, out_data_file=out_data_file)
  File "../framework/scripts/generate_test_code.py", line 1217, in generate_code
    write_test_source_file(template_file, c_file, snippets)
  File "../framework/scripts/generate_test_code.py", line 1134, in write_test_source_file
    c_f.write(code)
UnicodeEncodeError: 'ascii' codec can't encode character '\xa7' in position 2012: ordinal not in range(128)

UTF-8 encoded characters work if the locale (LC_CTYPE) is based on UTF-8, or when using Python ≥3.7 in a non-Unicode locale. But on the CI, we try to use Python 3.6 and the locale is C, leading to the failure quoted above when I tried to use § in a .function file.
@gilles-peskine-arm gilles-peskine-arm added bug Something isn't working size-xs Estimated task size: extra small (a few hours at most) labels Oct 30, 2024
gilles-peskine-arm added a commit that referenced this issue Nov 5, 2024
@gilles-peskine-arm
Use utf-8 when opening text files
612ccb7 
All of our source files are encoded in UTF-8. We have other scripts that
make this assumption. We don't want to depend on the platform encoding.

This fixes a bug whereby non-ASCII characters in .function files led to an
error when writing the generated C file. Fix #64.

Signed-off-by: Gilles Peskine <[email protected]>
@gilles-peskine-arm gilles-peskine-arm linked a pull request Nov 5, 2024 that will close this issue
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working size-xs Estimated task size: extra small (a few hours at most)
Projects
Friction
Status: No status
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Support Unicode in .function files Mbed-TLS/mbedtls-framework
1 participant
@gilles-peskine-arm