Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Excon::Errors::Forbidden with files containing Umlauts #6

Open
tvdeyen opened this issue Feb 6, 2014 · 5 comments
Open

Excon::Errors::Forbidden with files containing Umlauts #6

tvdeyen opened this issue Feb 6, 2014 · 5 comments

Comments

@tvdeyen
Copy link

tvdeyen commented Feb 6, 2014

Uploading files to S3 fails if the file has an umlaut in its name.

Excon::Errors::Forbidden - Expected(200) <=> Actual(403 Forbidden)
  response => #<Excon::Response:0x007f90aecc91b0 @data={:body=>"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>50 55 54 0a 0a 69 6d 61 67 65 2f 70 6e 67 0a 54 68 75 2c 20 30 36 20 46 65 62 20 32 30 31 34 20 31 32 3a 30 36 3a 30 30 20 2b 30 30 30 30 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 78 2d 61 6d 7a 2d 6d 65 74 61 2d 6a 73 6f 6e 3a 7b 22 6e 61 6d 65 22 3a 22 70 61 73 73 66 6f 74 6f 2d 6d 69 74 2d 6c c3 83 c2 a4 63 68 65 6c 6e 2e 70 6e 67 22 2c 22 6d 6f 64 65 6c 5f 63 6c 61 73 73 22 3a 22 50 69 63 74 75 72 65 22 2c 22 6d 6f 64 65 6c 5f 61 74 74 61 63 68 6d 65 6e 74 22 3a 22 69 6d 61 67 65 22 2c 22 61 6e 61 6c 79 73 65 72 5f 63 61 63 68 65 22 3a 7b 22 69 6d 61 67 65 5f 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 66 6f 72 6d 61 74 22 3a 22 70 6e 67 22 2c 22 77 69 64 74 68 22 3a 39 32 2c 22 68 65 69 67 68 74 22 3a 31 32 38 7d 2c 22 77 69 64 74 68 22 3a 39 32 2c 22 68 65 69 67 68 74 22 3a 31 32 38 7d 7d 0a 2f 66 61 76 69 63 73 2d 74 65 73 74 69 6e 67 2f 32 30 31 34 2f 30 32 2f 30 36 2f 31 33 2f 30 35 2f 35 39 2f 39 38 39 2f 70 61 73 73 66 6f 74 6f 5f 6d 69 74 5f 6c 5f 63 68 65 6c 6e 2e 70 6e 67</StringToSignBytes><RequestId>BF4085C899FB04BC</RequestId><HostId>BgGDAsOaAeSg3UtcYa1VsjXW5wImkyj55S+yPcYBE5SaSjC3xcVGWdH2a12X872f</HostId><SignatureProvided>wmJyDEg8Djr9E/Jm1huYjpZmK8A=</SignatureProvided><StringToSign>PUT\n\nimage/png\nThu, 06 Feb 2014 12:06:00 +0000\nx-amz-acl:public-read\nx-amz-meta-json:{\"name\":\"passfoto-mit-l\xC3\x83\xC2\xA4cheln.png\",\"model_class\":\"Picture\",\"model_attachment\":\"image\",\"analyser_cache\":{\"image_properties\":{\"format\":\"png\",\"width\":92,\"height\":128},\"width\":92,\"height\":128}}\n/favics-testing/2014/02/06/13/05/59/989/passfoto_mit_l_cheln.png</StringToSign><AWSAccessKeyId>AKIAJWH23ATSHYAAVEKA</AWSAccessKeyId></Error>", :headers=>{"x-amz-request-id"=>"BF4085C899FB04BC", "x-amz-id-2"=>"BgGDAsOaAeSg3UtcYa1VsjXW5wImkyj55S+yPcYBE5SaSjC3xcVGWdH2a12X872f", "Content-Type"=>"application/xml", "Transfer-Encoding"=>"", "Date"=>"Thu, 06 Feb 2014 12:06:01 GMT", "Connection"=>"close", "Server"=>"AmazonS3"}, :status=>403, :remote_ip=>"178.236.4.121"}, @body="<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><StringToSignBytes>50 55 54 0a 0a 69 6d 61 67 65 2f 70 6e 67 0a 54 68 75 2c 20 30 36 20 46 65 62 20 32 30 31 34 20 31 32 3a 30 36 3a 30 30 20 2b 30 30 30 30 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 78 2d 61 6d 7a 2d 6d 65 74 61 2d 6a 73 6f 6e 3a 7b 22 6e 61 6d 65 22 3a 22 70 61 73 73 66 6f 74 6f 2d 6d 69 74 2d 6c c3 83 c2 a4 63 68 65 6c 6e 2e 70 6e 67 22 2c 22 6d 6f 64 65 6c 5f 63 6c 61 73 73 22 3a 22 50 69 63 74 75 72 65 22 2c 22 6d 6f 64 65 6c 5f 61 74 74 61 63 68 6d 65 6e 74 22 3a 22 69 6d 61 67 65 22 2c 22 61 6e 61 6c 79 73 65 72 5f 63 61 63 68 65 22 3a 7b 22 69 6d 61 67 65 5f 70 72 6f 70 65 72 74 69 65 73 22 3a 7b 22 66 6f 72 6d 61 74 22 3a 22 70 6e 67 22 2c 22 77 69 64 74 68 22 3a 39 32 2c 22 68 65 69 67 68 74 22 3a 31 32 38 7d 2c 22 77 69 64 74 68 22 3a 39 32 2c 22 68 65 69 67 68 74 22 3a 31 32 38 7d 7d 0a 2f 66 61 76 69 63 73 2d 74 65 73 74 69 6e 67 2f 32 30 31 34 2f 30 32 2f 30 36 2f 31 33 2f 30 35 2f 35 39 2f 39 38 39 2f 70 61 73 73 66 6f 74 6f 5f 6d 69 74 5f 6c 5f 63 68 65 6c 6e 2e 70 6e 67</StringToSignBytes><RequestId>BF4085C899FB04BC</RequestId><HostId>BgGDAsOaAeSg3UtcYa1VsjXW5wImkyj55S+yPcYBE5SaSjC3xcVGWdH2a12X872f</HostId><SignatureProvided>wmJyDEg8Djr9E/Jm1huYjpZmK8A=</SignatureProvided><StringToSign>PUT\n\nimage/png\nThu, 06 Feb 2014 12:06:00 +0000\nx-amz-acl:public-read\nx-amz-meta-json:{\"name\":\"passfoto-mit-l\xC3\x83\xC2\xA4cheln.png\",\"model_class\":\"Picture\",\"model_attachment\":\"image\",\"analyser_cache\":{\"image_properties\":{\"format\":\"png\",\"width\":92,\"height\":128},\"width\":92,\"height\":128}}\n/favics-testing/2014/02/06/13/05/59/989/passfoto_mit_l_cheln.png</StringToSign><AWSAccessKeyId>AKIAJWH23ATSHYAAVEKA</AWSAccessKeyId></Error>", @headers={"x-amz-request-id"=>"BF4085C899FB04BC", "x-amz-id-2"=>"BgGDAsOaAeSg3UtcYa1VsjXW5wImkyj55S+yPcYBE5SaSjC3xcVGWdH2a12X872f", "Content-Type"=>"application/xml", "Transfer-Encoding"=>"", "Date"=>"Thu, 06 Feb 2014 12:06:01 GMT", "Connection"=>"close", "Server"=>"AmazonS3"}, @status=403, @remote_ip="178.236.4.121">:

uri encode the filename before storing works:

dragonfly_accessor :image do
  after_assign { |a| a.name = URI.escape(a.name) }
end
@felixbuenemann
Copy link

For now I've monkey-patched dragonfly-s3_data_store to escape utf8 in json, which does the trick:

module Dragonfly
  class S3DataStore
    def meta_to_headers(meta)
      {'x-amz-meta-json' => JSON.generate(meta, ascii_only: true)}
    end
  end
end

The default implementation uses MultiJson.encode(meta), which does not escape utf8.

I've opened an issue for fog at fog/fog#2942.

@janraasch
Copy link
Contributor

Hi guys, just chiming in, because I ran into this problem as well. IMHO this really needs to be fixed in fog (see fog/fog#2942), but just for reference I want to post our solution here, since JSON.generate(meta, ascii_only: true) did not work for us.

We decided to monkey patch this in our rails v3 app (running on ruby v2.1.2) with a module

module Dragonfly
  module S3DataStoreUmlauts
    def meta_to_headers(meta)
      { 'x-amz-meta-json' => ActiveSupport::JSON.encode(meta) }
    end
  end
end

which we then use to extend our datastore like so

Dragonfly.app.datastore.extend Dragonfly::S3DataStoreUmlauts

PS:
This won't work with rails v4, see rails/rails@8f8397e and rails/rails#3727

@felixbuenemann
Copy link

Due to another signing error, when the filename contains multiple spaces I'm now using this:

module Dragonfly
  class S3DataStore
    def meta_to_headers(meta)
      {'x-amz-meta-json' => JSON.generate(meta, ascii_only: true).gsub(' ', '\u0020')}
    end
  end
end

@felixbuenemann
Copy link

@markevans Maybe dragonfly should simply base64 encode the json? There are so many cases where the generated headers can break either RFC2616 Section 4.2 or S3's buggy interpretation of those rules.

@felixbuenemann
Copy link

Oops, deleted last comment which should've landed on a fog/fog-aws#160 ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants