Unsupported characters break CSV-import

[pheraph]

There are a lot of characters that break the CSV-import. I went through a long try and error and had to replace them to finally import a csv-file. These are the characters that are unsupported to my knowledge:

• ä, ö, ü, ß
• ”
• “
• ’
• …

[andygroo]

Hi all,

Is there a solution of the problem without replacing characters? I need to import descriptions and notes in Cyrillic.

[modsaid]

@pheraph Just exporting the files through excel or google docs to csv will ensure working csv even if the data included "'" or other characters. let me know if you hit a specific case that is not working

[modsaid]

@andygroo We are not pushing the development of the plugin at this point, however, any pull requests are welcomed.

on the other hand have you tried opening the data through excel or open office calc, and maybe save it as UTF8 csv?

[andygroo]

Excel opens the data file. I tried both Windows and UTF8 coding.

[modsaid]

@andygroo if u can see the data properly in excel, can you save AS CSV with UTF8 encoding?

[andygroo]

@modsaid ,
No, I can't - Excel just doesn't provide this feature - it has a feature "Export to CSV" saying nothing about encoding. But I did almost the same: I exported the file to CSV and converted the file into UTF8 manually (via notepad-plus-plus).

[modsaid]

@andygroo and did that finally work?

[andygroo]

No :-( The problem persists. Redmine displays an error window:
"
Internal error
An error occurred on the page you were trying to access.
If you continue to experience problems please contact your Redmine administrator for assistance.

If you are the Redmine administrator, check your log files for details about the error.

Back"

And log file is:

....................
Started GET "/projects/test-project-for-agpm/import_from_csv" for 192.168.21.182 at 2014-04-07 11:47:07 +0400
Processing by ImportFromCsvController#index as HTML
Parameters: {"name_prefix"=>"project", "project_id"=>"test-project-for-agpm"}
Current user: agpm (id=51)
Rendered plugins/redmine_import_from_csv/app/views/import_from_csv/index.html.erb within layouts/base (3.9ms)
Completed 200 OK in 39.4ms (Views: 25.0ms | ActiveRecord: 4.1ms)
Started POST "/projects/test-project-for-agpm/import_from_csv/csv_import" for 192.168.21.182 at 2014-04-07 11:47:20 +0400
Processing by ImportFromCsvController#csv_import as HTML
Parameters: {"utf8"=>"вњ“", "authenticity_token"=>"T9JPH0mfo4lxITf4N2hUSo/uDFIbNSwJdyrw9PQMF1g=", "project_id"=>"test-project-for-agpm", "dump"=>{"file"=>#<ActionDispatch::Http::UploadedFile:0x007f18e1089390 @original_filename="Sample5rus_fromexcel.csv", @content_type="application/vnd.ms-excel", @headers="Content-Disposition: form-data; name="dump[file]"; filename="C:\Users\Andrey.Grachev\Documents\Projects\RedMine DoIT\2014 03 20 \xD0\xA2\xD0\xB5\xD1\x81\xD1\x82\xD0\xB8\xD1\x80\xD0\xBE\xD0\xB2\xD0\xB0\xD0\xBD\xD0\xB8\xD0\xB5 \xD0\xBF\xD0\xBB\xD0\xB0\xD0\xB3\xD0\xB8\xD0\xBD\xD0\xB0 IssueImporter\Sample5rus_fromexcel.csv"\r\nContent-Type: application/vnd.ms-excel\r\n", @tempfile=#File:/tmp/RackMultipart20140407-30907-10cgeit>, "tracker_id"=>"1", "daily_working_hrs"=>"7"}, "commit"=>"Import", "name_prefix"=>"project"}
Current user: agpm (id=51)
Redirected to http://192.168.32.16/projects/test-project-for-agpm/issues
Completed 302 Found in 53.1ms (ActiveRecord: 3.7ms)
Started GET "/projects/test-project-for-agpm/issues" for 192.168.21.182 at 2014-04-07 11:47:20 +0400
Processing by IssuesController#index as HTML
Parameters: {"project_id"=>"test-project-for-agpm"}
Current user: agpm (id=51)
Rendered queries/_filters.html.erb (15.8ms)
Rendered queries/_columns.html.erb (1.7ms)
Rendered issues/_list.html.erb (69.1ms)
Rendered plugins/redmine_import_from_csv/app/views/issues/_import_issues_from_csv.html.erb (0.3ms)
Rendered issues/_sidebar.html.erb (7.7ms)
Rendered issues/index.html.erb within layouts/base (102.2ms)
Completed 200 OK in 169.8ms (Views: 115.2ms | ActiveRecord: 9.0ms)
Started GET "/projects/test-project-for-agpm/import_from_csv" for 192.168.21.182 at 2014-04-07 11:48:42 +0400
Processing by ImportFromCsvController#index as HTML
Parameters: {"name_prefix"=>"project", "project_id"=>"test-project-for-agpm"}
Current user: agpm (id=51)
Rendered plugins/redmine_import_from_csv/app/views/import_from_csv/index.html.erb within layouts/base (4.0ms)
Completed 200 OK in 41.1ms (Views: 25.2ms | ActiveRecord: 4.6ms)
Started POST "/projects/test-project-for-agpm/import_from_csv/csv_import" for 192.168.21.182 at 2014-04-07 11:48:55 +0400
Processing by ImportFromCsvController#csv_import as HTML
Parameters: {"utf8"=>"вњ“", "authenticity_token"=>"T9JPH0mfo4lxITf4N2hUSo/uDFIbNSwJdyrw9PQMF1g=", "project_id"=>"test-project-for-agpm", "dump"=>{"file"=>#<ActionDispatch::Http::UploadedFile:0x007f18e1253ce8 @original_filename="Sample5rus_fromexcel.csv", @content_type="application/vnd.ms-excel", @headers="Content-Disposition: form-data; name="dump[file]"; filename="C:\Users\Andrey.Grachev\Documents\Projects\RedMine DoIT\2014 03 20 \xD0\xA2\xD0\xB5\xD1\x81\xD1\x82\xD0\xB8\xD1\x80\xD0\xBE\xD0\xB2\xD0\xB0\xD0\xBD\xD0\xB8\xD0\xB5 \xD0\xBF\xD0\xBB\xD0\xB0\xD0\xB3\xD0\xB8\xD0\xBD\xD0\xB0 IssueImporter\Sample5rus_fromexcel.csv"\r\nContent-Type: application/vnd.ms-excel\r\n", @tempfile=#File:/tmp/RackMultipart20140407-30907-7ht439>, "tracker_id"=>"1", "daily_working_hrs"=>"7"}, "commit"=>"Import", "name_prefix"=>"project"}
Current user: agpm (id=51)
Redirected to http://192.168.32.16/projects/test-project-for-agpm/issues
Completed 302 Found in 51.5ms (ActiveRecord: 3.7ms)
Started GET "/projects/test-project-for-agpm/issues" for 192.168.21.182 at 2014-04-07 11:48:55 +0400
Processing by IssuesController#index as HTML
Parameters: {"project_id"=>"test-project-for-agpm"}
Current user: agpm (id=51)
Rendered queries/_filters.html.erb (15.6ms)
Rendered queries/_columns.html.erb (1.6ms)
Rendered issues/_list.html.erb (10.2ms)
Rendered plugins/redmine_import_from_csv/app/views/issues/_import_issues_from_csv.html.erb (0.3ms)
Rendered issues/_sidebar.html.erb (7.5ms)
Rendered issues/index.html.erb within layouts/base (42.8ms)
Completed 200 OK in 108.9ms (Views: 56.3ms | ActiveRecord: 8.8ms)
Started GET "/projects/test-project-for-agpm/import_from_csv" for 192.168.21.182 at 2014-04-07 11:49:54 +0400
Processing by ImportFromCsvController#index as HTML
Parameters: {"name_prefix"=>"project", "project_id"=>"test-project-for-agpm"}
Current user: agpm (id=51)
Rendered plugins/redmine_import_from_csv/app/views/import_from_csv/index.html.erb within layouts/base (3.9ms)
Completed 200 OK in 41.5ms (Views: 24.8ms | ActiveRecord: 4.5ms)
Started POST "/projects/test-project-for-agpm/import_from_csv/csv_import" for 192.168.21.182 at 2014-04-07 11:50:07 +0400
Processing by ImportFromCsvController#csv_import as HTML
Parameters: {"utf8"=>"вњ“", "authenticity_token"=>"T9JPH0mfo4lxITf4N2hUSo/uDFIbNSwJdyrw9PQMF1g=", "project_id"=>"test-project-for-agpm", "dump"=>{"file"=>#<ActionDispatch::Http::UploadedFile:0x007f18e169f590 @original_filename="Sample5rus_fromexcel.csv", @content_type="application/vnd.ms-excel", @headers="Content-Disposition: form-data; name="dump[file]"; filename="C:\Users\Andrey.Grachev\Documents\Projects\RedMine DoIT\2014 03 20 \xD0\xA2\xD0\xB5\xD1\x81\xD1\x82\xD0\xB8\xD1\x80\xD0\xBE\xD0\xB2\xD0\xB0\xD0\xBD\xD0\xB8\xD0\xB5 \xD0\xBF\xD0\xBB\xD0\xB0\xD0\xB3\xD0\xB8\xD0\xBD\xD0\xB0 IssueImporter\Sample5rus_fromexcel.csv"\r\nContent-Type: application/vnd.ms-excel\r\n", @tempfile=#File:/tmp/RackMultipart20140407-30907-19idy6e>, "tracker_id"=>"1", "daily_working_hrs"=>"7"}, "commit"=>"Import", "name_prefix"=>"project"}
Current user: agpm (id=51)
Completed 500 Internal Server Error in 58.7ms

Encoding::CompatibilityError (incompatible character encodings: UTF-8 and ASCII-8BIT):
app/models/issue.rb:165:in `create_or_update'