Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] UnicodeEncodeError: 'charmap' codec can't encode character #67

Open
woctezuma opened this issue Nov 19, 2022 · 4 comments
Open

[BUG] UnicodeEncodeError: 'charmap' codec can't encode character #67

woctezuma opened this issue Nov 19, 2022 · 4 comments
Labels
bug Something isn't working needs-review

Comments

@woctezuma
Copy link

woctezuma commented Nov 19, 2022

Description
I want to export the list of apps to a text file using the Windows console, and there is an UnicodeEncodeError error.

Steps to Reproduce the behavior

Run:
steamctl apps list > apps_list.txt

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Wok\.local\bin\steamctl.exe\__main__.py", line 7, in <module>
  File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\__main__.py", line 59, in main
    rcode = cmd_func(args=args)
            ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\commands\apps\gcmds.py", line 97, in cmd_apps_list
    print(app_id, app_names.get(app_id, 'Unknown App {}'.format(app_id)))
  File "C:\Users\Wok\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u221e' in position 11: character maps to <undefined>

Expected behavior
The software should run without any error, and the text file should contain the lists of appIDs along the app names.

What actually happend

$ head apps_list.txt
5 Dedicated Server
7 Steam Client
8 winui2
10 Counter-Strike
20 Team Fortress Classic
30 Day of Defeat
40 Deathmatch Classic
50 Half-Life: Opposing Force
60 Ricochet
70 Half-Life
$ tail apps_list.txt
208090 Unknown App 208090
208140 ENDLESS▒ Space - Definitive Edition
208157 ENDLESS▒ Space - Amplitude Badge + GAMES2GETHER Points
208286 Train Simulator: Sherman Hill Route Add-On
208540 Dungeon Defenders Lucky Community Costume Pack
208546 Dungeon Denders - Jester Hero
208570 Dishonored: Dunwall City Trials DLC
208575 Dishonored: The Knife of Dunwall
208580 STAR WARS▒ Knights of the Old Republic▒ II: The Sith Lords▒
208610

The error happens for appID=208610 which is Skullgirls ∞Endless Beta∞. I believe the culprit is the character.

Logs

steamctl -l debug
[DEBUG] steamctl: Parsed args: {'versions_report': None, 'log_level': 'debug', 'anonymous': False, 'user': None, 'password': None, 'command': 'apps', '_cmd_func': 'steamctl.commands.apps.gcmds:cmd_apps_list', 'subcommand': 'list', 'all': False}
[DEBUG] SteamClient: Reading CM servers from 'C:\\Users\\Wok\\AppData\\Local\\steamctl\\steamctl\\client\\cm_servers.json'
[DEBUG] CMServerList: Added 80 new CM addresses.
[DEBUG] steamctl.utils.storage: Opening file (r): C:\Users\Wok\AppData\Local\steamctl\steamctl\client\lastuser
[INFO] SteamClient: Reusing previous username: XXX
[INFO] SteamClient: Hint: use 'steamctl --user <username> ...' to change
[DEBUG] steamctl.utils.storage: Opening file (r): C:\Users\Wok\AppData\Local\steamctl\steamctl\client\lastuser
[INFO] SteamClient: Attempting login with remembered credentials
[DEBUG] steamctl.utils.storage: Opening file (r): C:\Users\Wok\AppData\Local\steamctl\steamctl\client\XXX.key
[DEBUG] SteamClient: Attempting login
[DEBUG] SteamClient: Connect initiated.
[DEBUG] Connection: Attempting connection to ('162.254.192.75', 27017)
[DEBUG] Connection: Connected.
[DEBUG] SteamClient: Emit event: 'connected'
[DEBUG] SteamClient: Incoming: <Msg(<EMsg.ChannelEncryptRequest: 1303> | ChannelEncryptRequest)>
[DEBUG] SteamClient: Emit event: <EMsg.ChannelEncryptRequest: 1303>
[DEBUG] SteamClient: Securing channel
[DEBUG] SteamClient: Outgoing: <Msg(<EMsg.ChannelEncryptResponse: 1304> | ChannelEncryptResponse)>
[DEBUG] SteamClient: Incoming: <Msg(<EMsg.ChannelEncryptResult: 1305> | ChannelEncryptResult)>
[DEBUG] SteamClient: Emit event: <EMsg.ChannelEncryptResult: 1305>
[DEBUG] SteamClient: Channel secured
[DEBUG] SteamClient: Emit event: 'channel_secured'
[DEBUG] SteamClient: Outgoing: <MsgProto(<EMsg.ClientLogon: 5514> | CMsgClientLogon)>
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.Multi: 1> | CMsgMulti)>
[DEBUG] SteamClient: Emit event: <EMsg.Multi: 1>
[DEBUG] SteamClient: Multi: Unpacking
[DEBUG] SteamClient: Multi: Decompressing payload (655 -> 1207)
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.ClientServersAvailable: 5501> | not parsed)>
[DEBUG] SteamClient: Emit event: <EMsg.ClientServersAvailable: 5501>
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.ClientServersAvailable: 5501> | not parsed)>
[...]
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.Multi: 1> | CMsgMulti)>
[DEBUG] SteamClient: Emit event: <EMsg.Multi: 1>
[DEBUG] SteamClient: Multi: Unpacking
[DEBUG] SteamClient: Multi: Decompressing payload (20905 -> 68577)
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.ClientPICSProductInfoResponse: 8904> | not parsed)>
[DEBUG] SteamClient: Emit event: <EMsg.ClientPICSProductInfoResponse: 8904>
[DEBUG] SteamClient: Emit event: 'job_4'
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.Multi: 1> | CMsgMulti)>
[DEBUG] SteamClient: Emit event: <EMsg.Multi: 1>
[DEBUG] SteamClient: Multi: Unpacking
[DEBUG] SteamClient: Multi: Decompressing payload (24145 -> 72215)
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.ClientPICSProductInfoResponse: 8904> | not parsed)>
[DEBUG] SteamClient: Emit event: <EMsg.ClientPICSProductInfoResponse: 8904>
[DEBUG] SteamClient: Emit event: 'job_4'
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.Multi: 1> | CMsgMulti)>
[DEBUG] SteamClient: Emit event: <EMsg.Multi: 1>
[DEBUG] SteamClient: Multi: Unpacking
[DEBUG] SteamClient: Multi: Decompressing payload (3153 -> 9775)
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.ClientPICSProductInfoResponse: 8904> | not parsed)>
[DEBUG] SteamClient: Emit event: <EMsg.ClientPICSProductInfoResponse: 8904>
[DEBUG] SteamClient: Emit event: 'job_4'
[DEBUG] Connection: Disconnected.
[DEBUG] SteamClient: Emit event: 'disconnected'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Wok\.local\bin\steamctl.exe\__main__.py", line 7, in <module>
  File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\__main__.py", line 59, in main
    rcode = cmd_func(args=args)
            ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\commands\apps\gcmds.py", line 97, in cmd_apps_list
    print(app_id, app_names.get(app_id, 'Unknown App {}'.format(app_id)))
  File "C:\Users\Wok\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u221e' in position 11: character maps to <undefined>

Versions Report

steamctl --versions-report
steamctl: 0.9.5

Dependencies:
               steam: 1.4.3
             appdirs: 1.4.4
         argcomplete: 2.0.0
                tqdm: 4.64.1
               arrow: 1.2.3
            pyqrcode: 1.2.1
      beautifulsoup4: 4.11.1
                 vpk: 1.4.0
                 vdf: 3.4
 gevent-eventemitter: 2.1
              gevent: 22.10.2
            greenlet: 2.0.1
              pyyaml: Not Installed
       pycryptodomex: 3.15.0
            protobuf: 3.20.3

Python runtime:
          executable: C:\Users\Wok\.local\pipx\venvs\steamctl\Scripts\python.exe
             version: 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)]
            platform: win32

System info:
              system: Windows
             machine: AMD64
             release: 10
             version: 10.0.19044
@woctezuma woctezuma added bug Something isn't working needs-review labels Nov 19, 2022
@woctezuma
Copy link
Author

woctezuma commented Nov 19, 2022

There is no error if I run:

 steamctl apps list

I can even get the character to be rendered correctly:

208570 Dishonored: Dunwall City Trials DLC
208575 Dishonored: The Knife of Dunwall
208580 STAR WARS™ Knights of the Old Republic™ II: The Sith Lords™
208610 Skullgirls ∞Endless Beta∞
208620 Midnight Mysteries: Salem Witch Trials
208630 Midnight Mysteries 3: Devil on the Mississippi
208640 Midnight Mysteries 4: Haunted Houdini

However, the list is really long, so it gets truncated by the Windows console, and it would be handier to save it to a text file.

I remember encountering this issue in the past, without reporting it, and then editing the code to save the output to a file.
I wonder if there is a better fix.

On a side-note, I am not sure if the issue only happens on Windows.

@woctezuma
Copy link
Author

The error can be reproduced with a Python file scratch_1.py, unrelated to steamctl:

print('∞')

python scratch_1.py

Output:


python scratch_1.py > temp.txt

Output:

Traceback (most recent call last):
  File "C:\Users\Wok\AppData\Roaming\JetBrains\PyCharmCE2022.2\scratches\scratch_1.py", line 1, in <module>
    print('∞')
  File "C:\Users\Wok\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u221e' in position 0: character maps to <undefined>

@rossengeorgiev
Copy link
Member

rossengeorgiev commented Nov 20, 2022

Use a terminal that supports utf8 and is configured to use utf8

@woctezuma
Copy link
Author

woctezuma commented Nov 20, 2022

Use a terminal that supports utf8 and is configured to use utf8

That is the case. The issue is with the redirection to a file, which does not seem to work properly on Windows.

For instance, in the picture below, there is no error, but an inconsistency.

echoecho> temp.txt
type temp.txt

Windows terminal: echo ∞

I can fix the inconsistency with chcp 65001, but that does not fix the error when redirecting the output of a Python script.

chcp 65001
echoecho> temp.txt
type temp.txt

Windows terminal: chcp 65001

For the error with Python, I have a small workaround.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs-review
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants