Support for unicode utf-8 charmaps #97
Comments
|
also I would like to contribute towards this but i am confused since i have not worked on any open source stuff on github, especially in python 😅 |
|
It sounds to me like perhaps there is a problem with the CSV file. I'm hesitant to ignore errors because some users might genuinely have a problem with their CSV file. Perhaps another solution is produce a better error message. Could you post the smallest CSV file that creates this error? Maybe just one line? |
|
Sure, here's a raw CSV which does throw an error So anything like these might throw up the UnicodeDecodeError 😅 Yes a better error message would be ok and maybe just merge this with issue #96 to practically bypass and log errors for bad emails and/or parse errors Or can't we just use utf8 encodings to enable support for multi languages? |
|
Hmm, I'm having trouble replicating the problem using your database. Perhaps you could try attaching the csv file to this issue instead of copy-paste? $ mailmerge --version
mailmerge, version 2.1.0
$ cat mailmerge_database.csv
name,email
"大松 李",[email protected]
"Juán",[email protected]
"Jöse Felix",[email protected]
"Paveł",[email protected]
$ file mailmerge_database.csv
mailmerge_database.csv: UTF-8 Unicode text
$ mailmerge --no-limit
>>> message 1
TO: [email protected]
SUBJECT: Testing mailmerge
FROM: My Self <[email protected]>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Date: Tue, 14 Jul 2020 11:31:47 -0000
Hi 大松 李
>>> message 1 sent
>>> message 2
TO: [email protected]
SUBJECT: Testing mailmerge
FROM: My Self <[email protected]>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Date: Tue, 14 Jul 2020 11:31:47 -0000
Hi Juán
>>> message 2 sent
>>> message 3
TO: [email protected]
SUBJECT: Testing mailmerge
FROM: My Self <[email protected]>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Date: Tue, 14 Jul 2020 11:31:47 -0000
Hi Jöse Felix
>>> message 3 sent
>>> message 4
TO: [email protected]
SUBJECT: Testing mailmerge
FROM: My Self <[email protected]>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Date: Tue, 14 Jul 2020 11:31:47 -0000
Hi Paveł
>>> message 4 sent
>>> This was a dry run. To send messages, use the --no-dry-run option. |
|
Seems like it must be an issue with my file. |
|
GitHub doesn't support csv uploads so here's a txt file of the csv: edit: uploaded the mailmerge_database.csv here, just in case we have any issues with csv as a txt file also, I think that my python installation on windows is broken since the same file runs perfectly on linux, but gives the UnicodeDecodeError on Windows ---> Running mailmerge on Windows: ---> Running mailmerge on Linux with exact same database.csv: |
|
@awdeorio I wonder if this is related to the behavior of the
On my Mac, this returns UTF-8 but I think the Windows default is something else. In fact, it looks like Python is planning to output warnings if no encoding is specified to @thecodelearner do you want to try to verify if this fixes the issue for you? (It could make a great first-contribution! 😃 ) You'll need to "fork" the repository, clone your fork locally and setup your development environment as described in CONTRIBUTING.md. Then, find the spot where Let us know if you have questions about any of this. |
|
@seshrs awesome! I will try this. |
|
Seems like some dependent modules are unix only, I'll switch to Linux and try running pytest again, with the updated encoding ofcourse. |
|
@thecodelearner Were you able to dry-run (Could you also tell us what |
|
ah okay I'll post what |
|
Based on https://stackoverflow.com/a/30539963, I think you'll need to escape the square brackets on |
|
rookie mistake 😅 pytest report: test coverage reports: will get back to Windows and post what |
|
|
Yes, encoding="utf-8" solves the error on Windows: |
|
Closed by #98 |
Using mailmerge to send mass emails is better than using any gmail extensions, since i have the ability to use advanced markdown and html
however while working with a large (5k+) csv file of job applicants, i came across an issue that people had entered their names in a localized language (eg. mandarin, etc) that sometimes didn't fit the ascii charmaps.
It would be useful to open the csv file as
encoding=utf8witherrors=ignoreto avoid errors like this:'charmap' codec can't decode byte 0x9d in position 1710: character maps to <undefined>instead of just returning true if strings are ascii.
The text was updated successfully, but these errors were encountered: