diff --git a/README.md b/README.md index cd103e9..2c5e5ef 100644 --- a/README.md +++ b/README.md @@ -1,177 +1,185 @@ -Version 3.21.4 - -New Features: - -1. If an error occurs during API access and the server returns an error message, all error information will be displayed in the response body. - -Resolved Issues: - -1. Fixed the bug that can not resume upload task without headers at uploadFile API - -------------------------------------------------------------------------------------------------- -Version 3.20.11 - -New Features: - -Documentation & Demo: - -Resolved Issues: -1. Fixed the issue that the uploadFile and downloadFile APIs do not support server-side encryption header. - -------------------------------------------------------------------------------------------------- -Version 3.20.9.1 - -New Features: - -Documentation & Demo: - -Resolved Issues: -1. Fixed the issue that an exception is thrown if both the checkSum and enableCheckpoint parameters are specified as True when calling the uploadFile API. - -------------------------------------------------------------------------------------------------- - -Version 3.20.5 - -New Features: -1. Added APIs related to asynchronous fetch policies, including ObsClient.setBucketFetchPolicy, ObsClient.getBucketFetchPolicy, and ObsClient.deleteBucketFetchPolicy. -2. Added APIs related to asynchronous fetch tasks, including ObsClient.setBucketFetchJob and ObsClient.getBucketFetchJob. -3. Added service orchestration APIs. For details, see obs/workflow.py. - -Documentation & Demo: -1. Added sections of asynchronous fetch and service orchestration in OBS Python SDK Developer Guide. -2. Added sections of asynchronous fetch and service orchestration APIs in OBS Python SDK API Reference. -3. Added the topic of asynchronous fetch policy status to section "Pre-defined Constants" and the topics of the response results of the asynchronous fetch APIs and service orchestration APIs to section "Data Types" in OBS Python SDK API Reference. - -Resolved Issues: - -------------------------------------------------------------------------------------------------- - -Version 3.20.1 - -New Features: -1. Added the ObsClient.headObject API for determining whether an object exists. -2. Added the ObsClient.setBucketRequestPayment and ObsClient.getBucketRequestPayment APIs respectively for configuring the Requester Pays function and obtaining related configuration. -3. Supports the Requester Pays header by configuring the extensionHeaders parameter when calling an API. - -Documentation & Demo: -1. Added the topic of checking whether an object exists to section "Object Management" in OBS Python SDK Developer Guide; added the API for checking whether an object exists to section "Bucket-Related APIs" in OBS Python SDK API Reference. -2. Added the topic of Requester Pays to section "Bucket Management" in OBS Python SDK Developer Guide; added the APIs for configuring the Requester Pays function and obtaining related configuration to section "Bucket-Related APIs" in OBS Python SDK API Reference; added the response result of obtaining Requester Pays configuration and extended additional header to section "Data Types" in OBS Python SDK API Reference. -3. Added the topic of Requester Pays configuration to section "Pre-defined Constants" in OBS Python SDK API Reference. -4. Added the description of extended additional headers to the API method definitions in OBS Python SDK API Reference. - -Resolved Issues: - -------------------------------------------------------------------------------------------------- - -Version 3.19.11 - -Documentation & Demo: - -Resolved Issues: -1. Fixed the issue that the authentication information header is added when redirection is performed upon a 302 response returned for a GET request. -2. Fixed the issue that the content-type cannot be obtained based on the file name extension if the extension is in uppercase. -3. Fixed the issue that the sequence of request parameters is incorrect in Authentication make_canonicalstring for calculating the authentication value. -4. Fixed the issue that the sample code examples/concurrent_copy_part_sample.py does not process failed requests. -5. Fixed the issue that the sample code examples/concurrent_download_object_sample.py does not process failed requests. -6. Fixed the issue that the sample code examples/concurrent_upload_part_sample.py does not process failed requests. -7. Fixed the issue that some response fields are empty in anonymous access. - -------------------------------------------------------------------------------------------------- - -Version 3.19.7.1 - -New Features: -1. Supports obtaining access keys in customized mode to create an instance of ObsClient. Currently, users can obtain access keys from environment variables or obtain temporary access keys from the ECS server. Or users can customize other obtaining methods. Multiple methods can be combined to obtain access keys. In this case, users can obtain access keys by using the methods in sequence or by specifying which method goes first. - -Documentation & Demo: -1. Added the security_providers and security_provider_policy parameters to section "Initializing an Instance of ObsClient" in OBS Python SDK API Reference. -2. Added the code example for obtaining access keys to create an instance of ObsClient in predefined mode to section "Initializing an Instance of ObsClient" in OBS Python SDK API Reference. -3. Added the code examples for obtaining access keys in predefined mode and in combination mode to section "Creating an Instance of ObsClient" in OBS Python SDK Developer Guide. -4. Added the security_providers and security_provider_policy parameters to section "Configuring an Instance of ObsClient" in OBS Python SDK Developer Guide. - -Resolved issues: - -------------------------------------------------------------------------------------------------- - -Version 3.19.5.2 - -Documentation & Demo - -Resolved issues: -1. Fixed the issue that an error occurs indicating no attribute when the broken pipe exception occurs during the API calling. -------------------------------------------------------------------------------------------------- - -Version 3.19.5.1 - -Documentation & Demo -1. Added the description of the max_redirect_count parameter in the OBS client initialization section of the API Reference. -2. Added the description of the max_redirect_count parameter in the OBS client initialization section of the Developer Guide. - -Resolved issues: -1. Fixed the issue that infinite number of redirects, resulting in an infinite loop. - -------------------------------------------------------------------------------------------------- - -Version 3.19.5 -Updated the version ID format. The new version ID is named in the following format: Main version ID.Year ID.Month ID. - -New features: -1. Additional header field is added to the resumable upload API uploadFile. You can set parameters such as acl, storageClass in the header. -2. By default, error-level logs are added to the non-2xx response statuses of OBS. - -Documentation & Demo -1. Added the description of the additional header fields for resumable upload in the section about data types in the API Reference. -2. Added the section about predefined constant in the API Reference. The value of BUCKET_OWNER_FULL_CONTROL is added to the predefined access control policies. -3. Added the creation and usage of temporary access keys in the "Getting Started" section of the Developer Guide. - -Resolved issues: -1. When the GET request is initiated and the server returns 302 redirection, the issue that new request does not extract the complete location information is fixed. -2. Fixed the issue that the process exits abnormally due to logrotate in special scenarios. -3. Optimized the resumable upload API uploadFile. The default part size is changed to 9 MB. -4. Fixed the issue that bucketClient does not support bucket encryption APIs. - -------------------------------------------------------------------------------------------------- - -Version 3.1.4 -New features: - -Documentation & Demo - -Resolved issues: -1. Fixed the issue that uploading objects in chunk mode does not comply with HTTP regulations. -2. Fixed the issue that the progress is not cleared immediately in an error scenario if the progress bar is enabled for the following APIs: ObsClient.putContent, ObsClient.putObject, ObsClient.putFile, ObsClient.appendObject, and ObsClient.uploadPart. -3. Fixed the issue that ObsClient.initLog may cause log conflicts by modifying that the logging of ObsClient does not inherit the parent configuration. -4. Fixed the issue that ObsClient.close fails to close the log file handle correctly. - -------------------------------------------------------------------------------------------------- - -Version 3.1.2.1 -New features: -1. Added bucket encryption APIs: ObsClient.setBucketEncryption, ObsClient.getBucketEncryption, and ObsClient.deleteBucketEncryption. Currently, only the SSE-KMS encryption is supported. -2. Added the identifier indicating whether to automatically close the input stream in the following APIs: ObsClient.putContent, ObsClient.putObject, ObsClient.appendObject, and ObsClient.uploadPart. The default value is true. - -Documentation & Demo - -Resolved issues: -1. Fixed the issue that multiple subprocesses are forked when ObsClient is initialized for multiple times in the Linux OS. - - -------------------------------------------------------------------------------------------------- -Version 3.1.2 -New features: -1. FunctionGraph configuration and query are supported in the bucket event notification APIs: ObsClient.setBucketNotification and ObsClient.getBucketNotification. -2. Added the image processing parameters to the resumable download API ObsClient.downloadFile. -3. Added the batch download API ObsClient.downloadFiles to support download of objects by specified prefix, progress return, automatic multipart download of large files, and resumable download. - -Documentation & Demo -1. Added the description of FunctionGraph configuration in the section about event notification in the Developer Guide. -2. Added the parameter description of FunctionGraph configuration in sections related to configuring and obtaining bucket notification in the API Reference. -3. Modified the sample code for enabling the bucket logging in the section related to access logs in the Developer Guide. - -Resolved issues: -1. Fixed the issue that the error information reported by the bucket creation API ObsClient.createBucket is incorrect due to protocol negotiation. -2. Rectified the error of the SetBucketLogging function in the sample code examples/obs_python_sample.py. -3. Fixed the issue that the contentDisposition parameter is incorrectly processed by the object upload API ObsClient.setObjectMetadata in the SDK. -4. Modified the coding policy of special characters in the object temporary authentication access API ObsClient.createSignedUrl. Tilde (~) is used as the reserved character of the URL encoding to solve the problem that results are inconsistent in the Python2.x and 3.x environments. -5. Optimized the bottom-layer code to improve the SDK performance in uploading and downloading small files in the Python2.x environment. -6. Fixed the issue that the process is forked when the OBS package is imported to the Linux OS. +Version 3.21.8 + +New Features: + +1. Add crypto client, could encrypt object at client side. + +------------------------------------------------------------------------------------------------- + +Version 3.21.4 + +New Features: + +1. If an error occurs during API access and the server returns an error message, all error information will be displayed in the response body. + +Resolved Issues: + +1. Fixed the bug that can not resume upload task without headers at uploadFile API + +------------------------------------------------------------------------------------------------- +Version 3.20.11 + +New Features: + +Documentation & Demo: + +Resolved Issues: +1. Fixed the issue that the uploadFile and downloadFile APIs do not support server-side encryption header. + +------------------------------------------------------------------------------------------------- +Version 3.20.9.1 + +New Features: + +Documentation & Demo: + +Resolved Issues: +1. Fixed the issue that an exception is thrown if both the checkSum and enableCheckpoint parameters are specified as True when calling the uploadFile API. + +------------------------------------------------------------------------------------------------- + +Version 3.20.5 + +New Features: +1. Added APIs related to asynchronous fetch policies, including ObsClient.setBucketFetchPolicy, ObsClient.getBucketFetchPolicy, and ObsClient.deleteBucketFetchPolicy. +2. Added APIs related to asynchronous fetch tasks, including ObsClient.setBucketFetchJob and ObsClient.getBucketFetchJob. +3. Added service orchestration APIs. For details, see obs/workflow.py. + +Documentation & Demo: +1. Added sections of asynchronous fetch and service orchestration in OBS Python SDK Developer Guide. +2. Added sections of asynchronous fetch and service orchestration APIs in OBS Python SDK API Reference. +3. Added the topic of asynchronous fetch policy status to section "Pre-defined Constants" and the topics of the response results of the asynchronous fetch APIs and service orchestration APIs to section "Data Types" in OBS Python SDK API Reference. + +Resolved Issues: + +------------------------------------------------------------------------------------------------- + +Version 3.20.1 + +New Features: +1. Added the ObsClient.headObject API for determining whether an object exists. +2. Added the ObsClient.setBucketRequestPayment and ObsClient.getBucketRequestPayment APIs respectively for configuring the Requester Pays function and obtaining related configuration. +3. Supports the Requester Pays header by configuring the extensionHeaders parameter when calling an API. + +Documentation & Demo: +1. Added the topic of checking whether an object exists to section "Object Management" in OBS Python SDK Developer Guide; added the API for checking whether an object exists to section "Bucket-Related APIs" in OBS Python SDK API Reference. +2. Added the topic of Requester Pays to section "Bucket Management" in OBS Python SDK Developer Guide; added the APIs for configuring the Requester Pays function and obtaining related configuration to section "Bucket-Related APIs" in OBS Python SDK API Reference; added the response result of obtaining Requester Pays configuration and extended additional header to section "Data Types" in OBS Python SDK API Reference. +3. Added the topic of Requester Pays configuration to section "Pre-defined Constants" in OBS Python SDK API Reference. +4. Added the description of extended additional headers to the API method definitions in OBS Python SDK API Reference. + +Resolved Issues: + +------------------------------------------------------------------------------------------------- + +Version 3.19.11 + +Documentation & Demo: + +Resolved Issues: +1. Fixed the issue that the authentication information header is added when redirection is performed upon a 302 response returned for a GET request. +2. Fixed the issue that the content-type cannot be obtained based on the file name extension if the extension is in uppercase. +3. Fixed the issue that the sequence of request parameters is incorrect in Authentication make_canonicalstring for calculating the authentication value. +4. Fixed the issue that the sample code examples/concurrent_copy_part_sample.py does not process failed requests. +5. Fixed the issue that the sample code examples/concurrent_download_object_sample.py does not process failed requests. +6. Fixed the issue that the sample code examples/concurrent_upload_part_sample.py does not process failed requests. +7. Fixed the issue that some response fields are empty in anonymous access. + +------------------------------------------------------------------------------------------------- + +Version 3.19.7.1 + +New Features: +1. Supports obtaining access keys in customized mode to create an instance of ObsClient. Currently, users can obtain access keys from environment variables or obtain temporary access keys from the ECS server. Or users can customize other obtaining methods. Multiple methods can be combined to obtain access keys. In this case, users can obtain access keys by using the methods in sequence or by specifying which method goes first. + +Documentation & Demo: +1. Added the security_providers and security_provider_policy parameters to section "Initializing an Instance of ObsClient" in OBS Python SDK API Reference. +2. Added the code example for obtaining access keys to create an instance of ObsClient in predefined mode to section "Initializing an Instance of ObsClient" in OBS Python SDK API Reference. +3. Added the code examples for obtaining access keys in predefined mode and in combination mode to section "Creating an Instance of ObsClient" in OBS Python SDK Developer Guide. +4. Added the security_providers and security_provider_policy parameters to section "Configuring an Instance of ObsClient" in OBS Python SDK Developer Guide. + +Resolved issues: + +------------------------------------------------------------------------------------------------- + +Version 3.19.5.2 + +Documentation & Demo + +Resolved issues: +1. Fixed the issue that an error occurs indicating no attribute when the broken pipe exception occurs during the API calling. +------------------------------------------------------------------------------------------------- + +Version 3.19.5.1 + +Documentation & Demo +1. Added the description of the max_redirect_count parameter in the OBS client initialization section of the API Reference. +2. Added the description of the max_redirect_count parameter in the OBS client initialization section of the Developer Guide. + +Resolved issues: +1. Fixed the issue that infinite number of redirects, resulting in an infinite loop. + +------------------------------------------------------------------------------------------------- + +Version 3.19.5 +Updated the version ID format. The new version ID is named in the following format: Main version ID.Year ID.Month ID. + +New features: +1. Additional header field is added to the resumable upload API uploadFile. You can set parameters such as acl, storageClass in the header. +2. By default, error-level logs are added to the non-2xx response statuses of OBS. + +Documentation & Demo +1. Added the description of the additional header fields for resumable upload in the section about data types in the API Reference. +2. Added the section about predefined constant in the API Reference. The value of BUCKET_OWNER_FULL_CONTROL is added to the predefined access control policies. +3. Added the creation and usage of temporary access keys in the "Getting Started" section of the Developer Guide. + +Resolved issues: +1. When the GET request is initiated and the server returns 302 redirection, the issue that new request does not extract the complete location information is fixed. +2. Fixed the issue that the process exits abnormally due to logrotate in special scenarios. +3. Optimized the resumable upload API uploadFile. The default part size is changed to 9 MB. +4. Fixed the issue that bucketClient does not support bucket encryption APIs. + +------------------------------------------------------------------------------------------------- + +Version 3.1.4 +New features: + +Documentation & Demo + +Resolved issues: +1. Fixed the issue that uploading objects in chunk mode does not comply with HTTP regulations. +2. Fixed the issue that the progress is not cleared immediately in an error scenario if the progress bar is enabled for the following APIs: ObsClient.putContent, ObsClient.putObject, ObsClient.putFile, ObsClient.appendObject, and ObsClient.uploadPart. +3. Fixed the issue that ObsClient.initLog may cause log conflicts by modifying that the logging of ObsClient does not inherit the parent configuration. +4. Fixed the issue that ObsClient.close fails to close the log file handle correctly. + +------------------------------------------------------------------------------------------------- + +Version 3.1.2.1 +New features: +1. Added bucket encryption APIs: ObsClient.setBucketEncryption, ObsClient.getBucketEncryption, and ObsClient.deleteBucketEncryption. Currently, only the SSE-KMS encryption is supported. +2. Added the identifier indicating whether to automatically close the input stream in the following APIs: ObsClient.putContent, ObsClient.putObject, ObsClient.appendObject, and ObsClient.uploadPart. The default value is true. + +Documentation & Demo + +Resolved issues: +1. Fixed the issue that multiple subprocesses are forked when ObsClient is initialized for multiple times in the Linux OS. + + +------------------------------------------------------------------------------------------------- +Version 3.1.2 +New features: +1. FunctionGraph configuration and query are supported in the bucket event notification APIs: ObsClient.setBucketNotification and ObsClient.getBucketNotification. +2. Added the image processing parameters to the resumable download API ObsClient.downloadFile. +3. Added the batch download API ObsClient.downloadFiles to support download of objects by specified prefix, progress return, automatic multipart download of large files, and resumable download. + +Documentation & Demo +1. Added the description of FunctionGraph configuration in the section about event notification in the Developer Guide. +2. Added the parameter description of FunctionGraph configuration in sections related to configuring and obtaining bucket notification in the API Reference. +3. Modified the sample code for enabling the bucket logging in the section related to access logs in the Developer Guide. + +Resolved issues: +1. Fixed the issue that the error information reported by the bucket creation API ObsClient.createBucket is incorrect due to protocol negotiation. +2. Rectified the error of the SetBucketLogging function in the sample code examples/obs_python_sample.py. +3. Fixed the issue that the contentDisposition parameter is incorrectly processed by the object upload API ObsClient.setObjectMetadata in the SDK. +4. Modified the coding policy of special characters in the object temporary authentication access API ObsClient.createSignedUrl. Tilde (~) is used as the reserved character of the URL encoding to solve the problem that results are inconsistent in the Python2.x and 3.x environments. +5. Optimized the bottom-layer code to improve the SDK performance in uploading and downloading small files in the Python2.x environment. +6. Fixed the issue that the process is forked when the OBS package is imported to the Linux OS. diff --git a/README_CN.md b/README_CN.md index e47ffe9..8d37d32 100644 --- a/README_CN.md +++ b/README_CN.md @@ -1,4 +1,12 @@ -Version 3.21.4 +Version 3.21.8 + +新特性： + +1. 新增客户端加密特性。 + +------------------------------------------------------------------------------------------------- + +Version 3.21.4 新特性： diff --git a/examples/crypto_client_sample.py b/examples/crypto_client_sample.py new file mode 100644 index 0000000..091248a --- /dev/null +++ b/examples/crypto_client_sample.py @@ -0,0 +1,83 @@ +#!/usr/bin/python +# -*- coding:utf-8 -*- +# Copyright 2019 Huawei Technologies Co.,Ltd. +# Licensed under the Apache License, Version 2.0 (the "License"); you may not use +# this file except in compliance with the License. You may obtain a copy of the +# License at + +# http://www.apache.org/licenses/LICENSE-2.0 + +# Unless required by applicable law or agreed to in writing, software distributed +# under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR +# CONDITIONS OF ANY KIND, either express or implied. See the License for the +# specific language governing permissions and limitations under the License. + +""" + This sample demonstrates how to using crypto client in OBS Python SDK. +""" +from obs import CompleteMultipartUploadRequest, CompletePart, CryptoObsClient +from obs.obs_cipher_suite import CTRCipherGenerator, CtrRSACipherGenerator + +AK = '*** Provide your Access Key ***' +SK = '*** Provide your Secret Key ***' +server = 'https://your-endpoint' + +bucketName = 'my-obs-bucket-demo' +test_file = "path/to/your/test/file" + +# Construct a crypto obs client with CtrRSACipherGenerator +# CtrRSACipherGenerator using public key +public_g = CtrRSACipherGenerator("/path/to/public_key.pem", master_key_info="Test_Key22") +public_client = CryptoObsClient(access_key_id=AK, secret_access_key=SK, server=server, cipher_generator=public_g) + +# CtrRSACipherGenerator using private key +private_g = CtrRSACipherGenerator("/path/to/private_key.pem", master_key_info="Test_Key22") +private_client = CryptoObsClient(access_key_id=AK, secret_access_key=SK, server=server, cipher_generator=private_g) + +# Construct a crypto obs client with CTRCipherGenerator +# The byte length of master key mast equal 32 +ctr_g = CTRCipherGenerator("your-master-key") +ctr_client = CryptoObsClient(access_key_id=AK, secret_access_key=SK, server=server, cipher_generator=ctr_g) + +# Create bucket +bucketClient = ctr_client.bucketClient(bucketName) + +# Upload file +# Uploading file in crypto obsClient is same as in normal obsClient +upload_object_key = "upload_test_file_with_ctr_client" +ctr_client_result = ctr_client.putFile(bucketName, upload_object_key, test_file) +if ctr_client_result.status < 300: + print('Upload finished\n') + +# Multipart upload File + +object_key = "Multipart_upload_File" + +# Step 1: Generate a cipher using empty string +print('Step 1: Generate a cipher using empty string \n') +cipher = ctr_client.cipher_generator.new("") + +# Step 2: initiate multipart upload +print('Step 2: initiate multipart upload \n') +init_result = ctr_client.initiateEncryptedMultipartUpload(bucketName, object_key, cipher) +uploadId = init_result.body.uploadId + +# Step 3: upload a part +print('Step 3: upload a part\n') +partNum = 1 +resp = ctr_client.uploadEncryptedPart(bucketName, object_key, partNumber=partNum, uploadId=uploadId, + crypto_cipher=cipher, content='Hello OBS') +etag = dict(resp.header).get('etag') + +# Step 4: complete multipart upload +print('Step 4: complete multipart upload\n') +resp = ctr_client.completeMultipartUpload(bucketName, object_key, uploadId, + CompleteMultipartUploadRequest([CompletePart(partNum=partNum, etag=etag)])) +if resp.status < 300: + print('Complete finished\n') + +# Download file +# Downloading file in crypto obsClient is same as in normal obsClient +download_result = ctr_client.getObject(bucketName, upload_object_key, downloadPath="/path/to/save") +if download_result.status < 300: + print('Download finished\n') diff --git a/src/obs/__init__.py b/src/obs/__init__.py index 5eb14fb..0e7237c 100644 --- a/src/obs/__init__.py +++ b/src/obs/__init__.py @@ -25,6 +25,9 @@ from obs.model import ListMultipartUploadsRequest, GetObjectRequest, UploadFileHeader, Payer from obs.model import ExtensionHeader, FetchStatus from obs.workflow import WorkflowClient +from obs.crypto_client import CryptoObsClient +from obs.obs_cipher_suite import CTRCipherGenerator +from obs.obs_cipher_suite import CtrRSACipherGenerator __all__ = [ 'LogConf', @@ -85,5 +88,8 @@ 'Payer', 'ExtensionHeader', 'FetchStatus', - 'WorkflowClient' + 'WorkflowClient', + 'CryptoObsClient', + 'CTRCipherGenerator', + 'CtrRSACipherGenerator' ] diff --git a/src/obs/auth.py b/src/obs/auth.py index 97f609f..c20da5e 100644 --- a/src/obs/auth.py +++ b/src/obs/auth.py @@ -37,7 +37,7 @@ def doAuth(self, method, bucket, key, path_args, headers, expires=None): } def getSignature(self, method, bucket, key, path_args, headers, expires=None): - canonical_string = self.__make_canonicalstring(method, bucket, key, path_args, headers, expires) + canonical_string = self.__make_canonical_string(method, bucket, key, path_args, headers, expires) return { 'Signature': self.hmacSha128(canonical_string), const.CANONICAL_STRING: canonical_string @@ -53,10 +53,10 @@ def hmacSha128(self, canonical_string): return encode_canonical - def __make_canonicalstring(self, method, bucket_name, key, path_args, headers, expires=None): + def __make_canonical_string(self, method, bucket_name, key, path_args, headers, expires=None): interesting_headers = self.__make_canonicalstring_interesting_headers(headers, expires) - keylist = sorted(interesting_headers.keys()) - str_list = self.__make_canonicalstring_str_list(keylist, method, interesting_headers) + key_list = sorted(interesting_headers.keys()) + str_list = self.__make_canonicalstring_str_list(key_list, method, interesting_headers) URI = '' _bucket_name = self.server if self.is_cname else bucket_name if _bucket_name: @@ -100,18 +100,18 @@ def __make_canonicalstring_interesting_headers(self, headers, expires): s = headers.get(hash_key) interesting_headers[lk] = ''.join(s) - keylist = interesting_headers.keys() + key_list = interesting_headers.keys() - if self.ha.date_header() in keylist: + if self.ha.date_header() in key_list: interesting_headers[const.DATE_HEADER.lower()] = '' if expires: interesting_headers[const.DATE_HEADER.lower()] = expires - if const.CONTENT_TYPE_HEADER.lower() not in keylist: + if const.CONTENT_TYPE_HEADER.lower() not in key_list: interesting_headers[const.CONTENT_TYPE_HEADER.lower()] = '' - if const.CONTENT_MD5_HEADER.lower() not in keylist: + if const.CONTENT_MD5_HEADER.lower() not in key_list: interesting_headers[const.CONTENT_MD5_HEADER.lower()] = '' return interesting_headers @@ -148,24 +148,25 @@ def doAuth(self, method, bucket, key, args_path, headers): headers = headers if isinstance(headers, dict) else {} headers[self.ha.content_sha256_header()] = self.CONTENT_SHA256 - credenttial = self.getCredenttial() + credential = self.getCredential() headMap = self.setMapKeyLower(headers) signedHeaders = self.getSignedHeaders(headMap) ret = self.getSignature(method, bucket, key, args_path, headMap, signedHeaders) auth = 'AWS4-HMAC-SHA256 Credential=%s,SignedHeaders=%s,Signature=%s' % ( - credenttial, signedHeaders, ret['Signature']) + credential, signedHeaders, ret['Signature']) return { const.AUTHORIZATION_HEADER: auth, const.CANONICAL_REQUEST: ret[const.CANONICAL_REQUEST] } - def getCredenttial(self): + def getCredential(self): return '%s/%s/%s/s3/aws4_request' % (self.ak, self.shortDate, self.region) def getScope(self): return '%s/%s/s3/aws4_request' % (self.shortDate, self.region) - def getSignedHeaders(self, headMap): + @staticmethod + def getSignedHeaders(headMap): headList = sorted(headMap.items(), key=lambda d: d[0]) signedHeaders = '' i = 0 @@ -194,7 +195,8 @@ def getSignature(self, method, bucket, key, args_path, headMap, signedHeaders, p const.CANONICAL_REQUEST: cannonicalRequest } - def hmacSha256(self, signingKey, stringToSign): + @staticmethod + def hmacSha256(signingKey, stringToSign): return hmac.new(signingKey, stringToSign, hashlib.sha256).hexdigest() def getSigningKey_python2(self): @@ -222,10 +224,12 @@ def getCanonicalRequest(self, method, bucket, key, args_path, headMap, signedHea output.append(self.CONTENT_SHA256 if payload is None else payload) return '\n'.join(output) - def __shaCannonicalRequest_python2(self, cannonicalRequest): + @staticmethod + def __shaCannonicalRequest_python2(cannonicalRequest): return hashlib.sha256(cannonicalRequest).hexdigest() - def __shaCannonicalRequest_python3(self, cannonicalRequest): + @staticmethod + def __shaCannonicalRequest_python3(cannonicalRequest): return hashlib.sha256(cannonicalRequest.encode('UTF-8')).hexdigest() def getCanonicalURI(self, bucket=None, key=None): @@ -238,7 +242,8 @@ def getCanonicalURI(self, bucket=None, key=None): URI = '/' return util.encode_object_key(URI) - def getCanonicalQueryString(self, args_path): + @staticmethod + def getCanonicalQueryString(args_path): canonMap = {} for key, value in args_path.items(): canonMap[key] = value @@ -252,7 +257,8 @@ def getCanonicalQueryString(self, args_path): i = 1 return queryStr - def getCanonicalHeaders(self, headMap): + @staticmethod + def getCanonicalHeaders(headMap): headList = sorted(headMap.items(), key=lambda d: d[0]) canonicalHeaderStr = '' for val in headList: @@ -264,7 +270,8 @@ def getCanonicalHeaders(self, headMap): canonicalHeaderStr += val[0] + ':' + str(val[1]) + '\n' return canonicalHeaderStr - def setMapKeyLower(self, inputMap): + @staticmethod + def setMapKeyLower(inputMap): outputMap = {} for key in inputMap.keys(): outputMap[key.lower()] = inputMap[key] diff --git a/src/obs/bucket.py b/src/obs/bucket.py index e6e16c1..e4372f5 100644 --- a/src/obs/bucket.py +++ b/src/obs/bucket.py @@ -91,8 +91,8 @@ def __init__(self, obsClient, bucketName): def __getattr__(self, key): if key in self.allowedMethod and hasattr(self.__obsClient, key): - orignalMethod = getattr(self.__obsClient, key) - if callable(orignalMethod): + original_method = getattr(self.__obsClient, key) + if callable(original_method): def delegate(*args, **kwargs): _args = list(args) if key == 'copyObject': @@ -104,7 +104,7 @@ def delegate(*args, **kwargs): else: if 'bucketName' not in kwargs: _args.insert(0, self.__bucketName) - return orignalMethod(*_args, **kwargs) + return original_method(*_args, **kwargs) return delegate return super(BucketClient, self).__getattribute__(key) diff --git a/src/obs/client.py b/src/obs/client.py index 44af208..0b74156 100644 --- a/src/obs/client.py +++ b/src/obs/client.py @@ -13,40 +13,26 @@ # specific language governing permissions and limitations under the License. from __future__ import print_function -import time + import functools -import threading +import math import os +import random import re +import threading +import time import traceback -import math -import random -from obs import const, convertor, util, auth, locks, progress +from inspect import isfunction + +from obs import auth, const, convertor, loadtoken, locks, progress, util +from obs.bucket import BucketClient from obs.cache import LocalCache -from obs.ilog import NoneLogClient, INFO, WARNING, ERROR, DEBUG, LogClient -from obs.transfer import _resumer_upload, _resumer_download from obs.extension import _download_files -from obs.model import Logging -from obs.model import AppendObjectHeader -from obs.model import AppendObjectContent -from obs.model import Notification -from obs.model import ListMultipartUploadsRequest -from obs.model import PutObjectHeader -from obs.model import BaseModel -from obs.model import GetResult -from obs.model import ObjectStream -from obs.model import ResponseWrapper -from obs.model import CreateBucketHeader -from obs.model import ACL -from obs.model import Versions -from obs.model import GetObjectRequest -from obs.model import GetObjectHeader -from obs.model import CopyObjectHeader -from obs.model import SetObjectMetadataHeader -from obs.bucket import BucketClient -from obs import loadtoken -from inspect import isfunction -from obs.model import FetchPolicy, _FetchJob +from obs.ilog import DEBUG, ERROR, INFO, LogClient, NoneLogClient, WARNING +from obs.model import ACL, AppendObjectContent, AppendObjectHeader, BaseModel, CopyObjectHeader, CreateBucketHeader, \ + FetchPolicy, GetObjectHeader, GetObjectRequest, GetResult, ListMultipartUploadsRequest, Logging, Notification, \ + ObjectStream, PutObjectHeader, ResponseWrapper, SetObjectMetadataHeader, Versions, _FetchJob +from obs.transfer import _resume_download, _resume_upload if const.IS_PYTHON2: from urlparse import urlparse @@ -202,7 +188,7 @@ def __getattr__(self, item): class _BasicClient(object): def __init__(self, access_key_id='', secret_access_key='', is_secure=True, server=None, signature='obs', region='region', path_style=False, ssl_verify=False, - port=None, max_retry_count=3, timeout=60, chunk_size=65536, + port=None, max_retry_count=3, timeout=60, chunk_size=const.READ_ONCE_LENGTH, long_conn_mode=False, proxy_host=None, proxy_port=None, proxy_username=None, proxy_password=None, security_token=None, custom_ciphers=None, use_http2=False, is_signature_negotiation=True, is_cname=False, @@ -276,13 +262,15 @@ def __init__(self, access_key_id='', secret_access_key='', is_secure=True, serve self.ha = convertor.Adapter(self.signature) self.convertor = convertor.Convertor(self.signature, self.ha) - def _parse_server_hostname(self, server): + @staticmethod + def _parse_server_hostname(server): hostname = server.netloc if util.is_valid(server.netloc) else server.path if not util.is_valid(hostname): raise Exception('server is not set correctly') return hostname - def _check_server_secure(self, server, is_secure): + @staticmethod + def _check_server_secure(server, is_secure): if util.is_valid(server.scheme): if server.scheme == 'https': is_secure = True @@ -306,7 +294,8 @@ def _parse_security_providers(self, security_providers): self.security_provider_policy = None print(traceback.format_exc()) - def _split_host_port(self, hostname, port): + @staticmethod + def _split_host_port(hostname, port): host_port = hostname.split(':') if len(host_port) == 2: port = util.to_int(host_port[1]) @@ -315,7 +304,8 @@ def _split_host_port(self, hostname, port): def _check_path_style(self, path_style): return True if util.is_ipaddress(self.server) else path_style - def _parse_port(self, port, is_secure): + @staticmethod + def _parse_port(port, is_secure): if port is None: if is_secure: port = const.DEFAULT_SECURE_PORT @@ -325,9 +315,9 @@ def _parse_port(self, port, is_secure): def _parse_calling_format(self): if self.path_style: - return util.RequestFormat.get_pathformat() + return util.RequestFormat.get_path_format() else: - return util.RequestFormat.get_subdomainformat() + return util.RequestFormat.get_sub_domain_format() def _get_token(self): from obs.searchmethod import get_token @@ -402,7 +392,8 @@ def initLog(self, log_config=None, log_name='OBS_LOGGER'): msg.append('Access Mode=' + ('Path' if self.path_style else 'Virtual Hosting') + ']') self.log_client.log(WARNING, '];['.join(msg)) - def _assert_not_null(self, param, msg): + @staticmethod + def _assert_not_null(param, msg): param = util.safe_encode(param) if param is None or util.to_string(param).strip() == '': raise Exception(msg) @@ -492,7 +483,6 @@ def _make_request_with_retry(self, methodType, bucketName, objectKey=None, pathA time.sleep(math.pow(2, flag) * 0.05) self.log_client.log(WARNING, 'request again, time:%d' % int(flag)) continue - break def _make_request_internal(self, method, bucketName='', objectKey=None, pathArgs=None, headers=None, entity=None, chunkedMode=False, redirectLocation=None, skipAuthentication=False, redirectFlag=False, @@ -546,7 +536,8 @@ def _make_request_internal(self, method, bucketName='', objectKey=None, pathArgs chunkedMode) return conn - def _parse_extension_headers(self, headers, extension_headers): + @staticmethod + def _parse_extension_headers(headers, extension_headers): if len(extension_headers) > 0: if headers is None or not isinstance(headers, dict): headers = {} @@ -556,7 +547,8 @@ def _parse_extension_headers(self, headers, extension_headers): headers[key] = value return headers - def _parse_entity(self, entity, headers): + @staticmethod + def _parse_entity(entity, headers): if entity is not None and not callable(entity): entity = util.safe_encode(entity) if not isinstance(entity, str) and not isinstance(entity, bytes): @@ -740,7 +732,8 @@ def _parse_request_connection(self, server, port, scheme, connection_key, redire header[const.CONNECTION_HEADER] = const.CONNECTION_CLOSE_VALUE return conn, header - def _parse_connection_chunked_mode(self, conn, chunkedMode, method, path, header): + @staticmethod + def _parse_connection_chunked_mode(conn, chunkedMode, method, path, header): if chunkedMode: conn.putrequest(method, path, skip_host=1) for k, v in header.items(): @@ -772,93 +765,46 @@ def _parse_xml(self, conn, methodName=None, readable=False): finally: util.do_close(result, conn, self.connHolder, self.log_client) - def _parse_content_with_notifier(self, conn, objectKey, chuckSize=65536, downloadPath=None, notifier=None): - if not conn: - return self._getNoneResult('connection is none') - result = None - close_conn_flag = True - try: - result = conn.getresponse(True) if const.IS_PYTHON2 else conn.getresponse() - if not result: - return self._getNoneResult('response is none') - - if not util.to_int(result.status) < 300: - return self._parse_xml_internal(result) - - headers = {} - for k, v in result.getheaders(): - headers[k.lower()] = v - - content_length = headers.get('content-length') - content_length = util.to_long(content_length) if content_length is not None else None - resultWrapper = ResponseWrapper(conn, result, self.connHolder, content_length, notifier) - if downloadPath is None: - self.log_client.log(DEBUG, 'DownloadPath is none, return conn directly') - close_conn_flag = False - body = ObjectStream(response=resultWrapper) - else: - objectKey = util.safe_encode(objectKey) - downloadPath = util.safe_encode(downloadPath) - file_path, _ = self._get_data(resultWrapper, downloadPath, chuckSize) - body = ObjectStream(url=util.to_string(file_path)) - self.log_client.log(DEBUG, 'DownloadPath is ' + util.to_string(file_path)) - - status = util.to_int(result.status) - reason = result.reason - self.convertor.parseGetObject(headers, body) - header = self._rename_response_headers(headers) - requestId = dict(header).get('request-id') - return GetResult(status=status, reason=reason, header=header, body=body, requestId=requestId) - except _RedirectException as ex: - raise ex - except Exception as e: - self.log_client.log(ERROR, traceback.format_exc()) - raise e - finally: - if close_conn_flag: - util.do_close(result, conn, self.connHolder, self.log_client) - - def _parse_content(self, conn, objectKey, downloadPath=None, chuckSize=65536, loadStreamInMemory=False, - progressCallback=None): + def _parse_content(self, objectKey, conn, response, download_start='', + downloadPath=None, chuckSize=const.READ_ONCE_LENGTH, loadStreamInMemory=False, + progressCallback=None, notifier=None): if not conn: return self._getNoneResult('connection is none') close_conn_flag = True - result = None - resultWrapper = None + result_wrapper = None try: - result = conn.getresponse(True) if const.IS_PYTHON2 else conn.getresponse() - if not result: + if not response: return self._getNoneResult('response is none') - if not util.to_int(result.status) < 300: - return self._parse_xml_internal(result) + if not util.to_int(response.status) < 300: + return self._parse_xml_internal(response) headers = {} - for k, v in result.getheaders(): + for k, v in response.getheaders(): headers[k.lower()] = v content_length = headers.get('content-length') content_length = util.to_long(content_length) if content_length is not None else None - notifier = self._get_notifier(content_length, progressCallback) - notifier.start() - resultWrapper = ResponseWrapper(conn, result, self.connHolder, content_length, notifier) + if not notifier: + notifier = self._get_notifier(content_length, progressCallback) + notifier.start() + result_wrapper = ResponseWrapper(conn, response, self.connHolder, content_length, notifier) if loadStreamInMemory: self.log_client.log(DEBUG, 'loadStreamInMemory is True, read stream into memory') - buf = self._get_buffer_data(resultWrapper, chuckSize) + buf = self._get_buffer_data(result_wrapper, chuckSize) body = ObjectStream(buffer=buf, size=util.to_long(len(buf)) if buf is not None else 0) elif downloadPath is None: self.log_client.log(DEBUG, 'DownloadPath is none, return conn directly') close_conn_flag = False - body = ObjectStream(response=resultWrapper) + body = ObjectStream(response=result_wrapper) else: - objectKey = util.safe_encode(objectKey) downloadPath = util.safe_encode(downloadPath) - file_path, _ = self._get_data(resultWrapper, downloadPath, chuckSize) + file_path, _ = self._get_data(result_wrapper, downloadPath, chuckSize) body = ObjectStream(url=util.to_string(file_path)) self.log_client.log(DEBUG, 'DownloadPath is ' + util.to_string(file_path)) - status = util.to_int(result.status) - reason = result.reason + status = util.to_int(response.status) + reason = response.reason self.convertor.parseGetObject(headers, body) header = self._rename_response_headers(headers) requestId = dict(header).get('request-id') @@ -870,12 +816,13 @@ def _parse_content(self, conn, objectKey, downloadPath=None, chuckSize=65536, lo raise e finally: if close_conn_flag: - if resultWrapper is not None: - resultWrapper.close() + if result_wrapper is not None: + result_wrapper.close() else: - util.do_close(result, conn, self.connHolder, self.log_client) + util.do_close(response, conn, self.connHolder, self.log_client) - def _get_buffer_data(self, resultWrapper, chuckSize): + @staticmethod + def _get_buffer_data(resultWrapper, chuckSize): buf = None appendList = [] while True: @@ -891,14 +838,16 @@ def _get_buffer_data(self, resultWrapper, chuckSize): appendList.append(chunk) return buf - def _get_notifier(self, content_length, progressCallback): + @staticmethod + def _get_notifier(content_length, progressCallback): return progress.ProgressNotifier(progressCallback, content_length) if content_length is not None and content_length > 0 \ and progressCallback is not None else progress.NONE_NOTIFIER - def _get_data(self, resultWrapper, downloadPath, chuckSize): + @staticmethod + def _get_data(resultWrapper, downloadPath, chuckSize): origin_file_path = downloadPath - readed_count = 0 + read_count = 0 if const.IS_WINDOWS: downloadPath = util.safe_trans_to_gb2312(downloadPath) pathDir = os.path.dirname(downloadPath) @@ -910,10 +859,11 @@ def _get_data(self, resultWrapper, downloadPath, chuckSize): if not chunk: break f.write(chunk) - readed_count += len(chunk) - return origin_file_path, readed_count + read_count += len(chunk) + return origin_file_path, read_count - def _rename_key(self, k, v): + @staticmethod + def _rename_key(k, v): flag = 0 if k.startswith(const.V2_META_HEADER_PREFIX): k = k[k.index(const.V2_META_HEADER_PREFIX) + len(const.V2_META_HEADER_PREFIX):] @@ -947,7 +897,8 @@ def _rename_response_headers(self, headers): header.append((k, v)) return header - def _prepare_response_data(self, result, chuckSize): + @staticmethod + def _prepare_response_data(result, chuckSize): responseData = None while True: chunk = result.read(chuckSize) @@ -964,7 +915,7 @@ def _prepare_body(self, methodName, responseData, isJson, headers): try: if responseData: responseData = responseData if const.IS_PYTHON2 else responseData.decode('UTF-8') - self.log_client.log(DEBUG, 'recv Msg:%s', responseData) + self.log_client.log(DEBUG, 'receive Msg:%s', responseData) if not isJson: search = self.pattern.search(responseData) responseData = responseData if search is None else responseData.replace(search.group(), @@ -977,17 +928,19 @@ def _prepare_body(self, methodName, responseData, isJson, headers): self.log_client.log(ERROR, traceback.format_exc()) return responseData, body - def _prepare_request_id(self, requestId, headers): + @staticmethod + def _prepare_request_id(requestId, headers): if requestId is None: requestId = headers.get('x-obs-request-id') if requestId is None: requestId = headers.get('x-amz-request-id') return requestId - def _is_redirect_exception(self, status, headers): - return status >= 300 and status < 400 and status != 304 and const.LOCATION_HEADER.lower() in headers + @staticmethod + def _is_redirect_exception(status, headers): + return 300 <= status < 400 and status != 304 and const.LOCATION_HEADER.lower() in headers - def _parse_xml_internal(self, result, methodName=None, chuckSize=65536, readable=False): + def _parse_xml_internal(self, result, methodName=None, chuckSize=const.READ_ONCE_LENGTH, readable=False): status = util.to_int(result.status) reason = result.reason code = None @@ -1125,8 +1078,8 @@ def _createV2SignedUrl(self, method, bucketName=None, objectKey=None, specialPar v2Auth = auth.Authentication(securityProvider.access_key_id, securityProvider.secret_access_key, self.path_style, self.ha, self.server, self.is_cname) - signature = v2Auth.getSignature(method, bucketName, objectKey, queryParams, headers, util.to_string(expires))[ - 'Signature'] + signature = v2Auth.getSignature(method, bucketName, objectKey, queryParams, + headers, util.to_string(expires))['Signature'] queryParams['Expires'] = expires queryParams['AccessKeyId' if self.signature == 'obs' else 'AWSAccessKeyId'] = securityProvider.access_key_id @@ -1135,11 +1088,10 @@ def _createV2SignedUrl(self, method, bucketName=None, objectKey=None, specialPar if self.is_cname: bucketName = None - result = { - 'signedUrl': calling_format.get_full_url(self.is_secure, self.server, self.port, bucketName, objectKey, - queryParams), - 'actualSignedRequestHeaders': headers - } + result = {'signedUrl': calling_format.get_full_url(self.is_secure, self.server, + self.port, bucketName, objectKey, + queryParams), + 'actualSignedRequestHeaders': headers} return _CreateSignedUrlResponse(**result) @@ -1173,7 +1125,7 @@ def _createV4SignedUrl(self, method, bucketName=None, objectKey=None, specialPar shortDate, longDate, self.path_style, self.ha) queryParams['X-Amz-Algorithm'] = 'AWS4-HMAC-SHA256' - queryParams['X-Amz-Credential'] = v4Auth.getCredenttial() + queryParams['X-Amz-Credential'] = v4Auth.getCredential() queryParams['X-Amz-Date'] = longDate queryParams['X-Amz-Expires'] = expires @@ -1182,9 +1134,8 @@ def _createV4SignedUrl(self, method, bucketName=None, objectKey=None, specialPar queryParams['X-Amz-SignedHeaders'] = signedHeaders - signature = \ - v4Auth.getSignature(method, bucketName, objectKey, queryParams, headMap, signedHeaders, 'UNSIGNED-PAYLOAD')[ - 'Signature'] + signature = v4Auth.getSignature(method, bucketName, objectKey, queryParams, + headMap, signedHeaders, 'UNSIGNED-PAYLOAD')['Signature'] queryParams['X-Amz-Signature'] = signature @@ -1323,14 +1274,16 @@ def _getApiVersion(self, bucketName=''): @funcCache def listBuckets(self, isQueryLocation=True, extensionHeaders=None): if self.is_cname: - raise Exception('listBuckets is not allowed in customdomain mode') + raise Exception('listBuckets is not allowed in custom domain mode') return self._make_get_request(methodName='listBuckets', extensionHeaders=extensionHeaders, **self.convertor.trans_list_buckets(isQueryLocation=isQueryLocation)) @funcCache - def createBucket(self, bucketName, header=CreateBucketHeader(), location=None, extensionHeaders=None): + def createBucket(self, bucketName, header=None, location=None, extensionHeaders=None): + if header is None: + header = CreateBucketHeader() if self.is_cname: - raise Exception('createBucket is not allowed in customdomain mode') + raise Exception('createBucket is not allowed in custom domain mode') res = self._make_put_request(bucketName, extensionHeaders=extensionHeaders, **self.convertor.trans_create_bucket(header=header, location=location)) try: @@ -1344,10 +1297,12 @@ def createBucket(self, bucketName, header=CreateBucketHeader(), location=None, e return res @funcCache - def listObjects(self, bucketName, prefix=None, marker=None, max_keys=None, delimiter=None, extensionHeaders=None): + def listObjects(self, bucketName, prefix=None, marker=None, max_keys=None, delimiter=None, + extensionHeaders=None, encoding_type=None): return self._make_get_request(bucketName, methodName='listObjects', extensionHeaders=extensionHeaders, **self.convertor.trans_list_objects(prefix=prefix, marker=marker, - max_keys=max_keys, delimiter=delimiter)) + max_keys=max_keys, delimiter=delimiter, + encoding_type=encoding_type)) @funcCache def headBucket(self, bucketName, extensionHeaders=None): @@ -1392,7 +1347,9 @@ def getBucketStorageInfo(self, bucketName, extensionHeaders=None): extensionHeaders=extensionHeaders) @funcCache - def setBucketAcl(self, bucketName, acl=ACL(), aclControl=None, extensionHeaders=None): + def setBucketAcl(self, bucketName, acl=None, aclControl=None, extensionHeaders=None): + if acl is None: + acl = ACL() if acl is not None and len(acl) > 0 and aclControl is not None: raise Exception('Both acl and aclControl are set') if not acl and not aclControl: @@ -1433,14 +1390,20 @@ def getBucketVersioning(self, bucketName, extensionHeaders=None): extensionHeaders=extensionHeaders) @funcCache - def listVersions(self, bucketName, version=Versions(), extensionHeaders=None): + def listVersions(self, bucketName, version=None, extensionHeaders=None): + if version is None: + version = Versions() return self._make_get_request(bucketName, methodName='listVersions', extensionHeaders=extensionHeaders, **self.convertor.trans_list_versions(version=version)) @funcCache - def listMultipartUploads(self, bucketName, multipart=ListMultipartUploadsRequest(), extensionHeaders=None): + def listMultipartUploads(self, bucketName, multipart=None, extensionHeaders=None, + encoding_type=None): + if multipart is None: + multipart = ListMultipartUploadsRequest() return self._make_get_request(bucketName, methodName='listMultipartUploads', extensionHeaders=extensionHeaders, - **self.convertor.trans_list_multipart_uploads(multipart=multipart)) + **self.convertor.trans_list_multipart_uploads(multipart=multipart, + encoding_type=encoding_type)) @funcCache def deleteBucketLifecycle(self, bucketName, extensionHeaders=None): @@ -1473,7 +1436,7 @@ def getBucketWebsite(self, bucketName, extensionHeaders=None): extensionHeaders=extensionHeaders) @funcCache - def setBucketLogging(self, bucketName, logstatus=Logging(), extensionHeaders=None): + def setBucketLogging(self, bucketName, logstatus=None, extensionHeaders=None): if logstatus is None: logstatus = Logging() return self._make_put_request(bucketName, pathArgs={'logging': None}, @@ -1519,7 +1482,9 @@ def optionsBucket(self, bucketName, option, extensionHeaders=None): return self.optionsObject(bucketName, None, option=option, extensionHeaders=extensionHeaders) @funcCache - def setBucketNotification(self, bucketName, notification=Notification(), extensionHeaders=None): + def setBucketNotification(self, bucketName, notification=None, extensionHeaders=None): + if notification is None: + notification = Notification() if notification is None: notification = Notification() return self._make_put_request(bucketName, pathArgs={'notification': None}, @@ -1574,36 +1539,29 @@ def setObjectMetadata(self, bucketName, objectKey, metadata=None, headers=None, versionId=versionId)) @funcCache - def getObject(self, bucketName, objectKey, downloadPath=None, getObjectRequest=GetObjectRequest(), - headers=GetObjectHeader(), loadStreamInMemory=False, progressCallback=None, extensionHeaders=None): + def getObject(self, bucketName, objectKey, downloadPath=None, getObjectRequest=None, + headers=None, loadStreamInMemory=False, progressCallback=None, extensionHeaders=None, notifier=None): + if getObjectRequest is None: + getObjectRequest = GetObjectRequest() + if headers is None: + headers = GetObjectHeader() _parse_content = self._parse_content - CHUNKSIZE = self.chunk_size + CHUNK_SIZE = self.chunk_size readable = False if progressCallback is None else True def parseMethod(conn): - return _parse_content(conn, objectKey, downloadPath, CHUNKSIZE, loadStreamInMemory, progressCallback) + result = conn.getresponse() + return _parse_content(objectKey, conn, result, download_start=headers.range, downloadPath=downloadPath, + chuckSize=CHUNK_SIZE, loadStreamInMemory=loadStreamInMemory, notifier=notifier, + progressCallback=progressCallback) return self._make_get_request(bucketName, objectKey, parseMethod=parseMethod, readable=readable, extensionHeaders=extensionHeaders, **self.convertor.trans_get_object(getObjectRequest=getObjectRequest, headers=headers)) - @funcCache - def _getObjectWithNotifier(self, bucketName, objectKey, getObjectRequest=GetObjectRequest(), - headers=GetObjectHeader(), downloadPath=None, notifier=None, extensionHeaders=None): - _parse_content_with_notifier = self._parse_content_with_notifier - CHUNKSIZE = self.chunk_size - readable = False if notifier is None else True - - def parseMethod(conn): - return _parse_content_with_notifier(conn, objectKey, CHUNKSIZE, downloadPath, notifier) - - return self._make_get_request(bucketName, objectKey, parseMethod=parseMethod, readable=readable, - extensionHeaders=extensionHeaders, - **self.convertor.trans_get_object(getObjectRequest=getObjectRequest, - headers=headers)) - - def _preapare_append_object_input(self, objectKey, headers, content): + @staticmethod + def _prepare_append_object_input(objectKey, headers, content): objectKey = util.safe_encode(objectKey) if objectKey is None: objectKey = '' @@ -1619,7 +1577,7 @@ def _preapare_append_object_input(self, objectKey, headers, content): return objectKey, headers, content - def _prepare_file_notifier_and_entiy(self, offset, file_size, headers, progressCallback, file_path, readable): + def _prepare_file_notifier_and_entity(self, offset, file_size, headers, progressCallback, file_path, readable): if offset is not None and 0 < offset < file_size: headers['contentLength'] = headers['contentLength'] if 0 < headers['contentLength'] <= ( file_size - offset) else file_size - offset @@ -1629,7 +1587,9 @@ def _prepare_file_notifier_and_entiy(self, offset, file_size, headers, progressC notifier = progress.ProgressNotifier(progressCallback, totalCount) else: notifier = progress.NONE_NOTIFIER - entity = util.get_file_entity_by_offset_partsize(file_path, offset, totalCount, self.chunk_size, notifier) + readable_object = self.gen_readable_object_from_file(file_path) + readable_object.seek(offset) + entity = util.get_entity_for_send_with_total_count(readable_object, totalCount, self.chunk_size, notifier) else: totalCount = headers['contentLength'] if totalCount > 0 and progressCallback is not None: @@ -1637,12 +1597,13 @@ def _prepare_file_notifier_and_entiy(self, offset, file_size, headers, progressC notifier = progress.ProgressNotifier(progressCallback, totalCount) else: notifier = progress.NONE_NOTIFIER - entity = util.get_file_entity_by_totalcount(file_path, totalCount, self.chunk_size, notifier) + readable_object = self.gen_readable_object_from_file(file_path) + entity = util.get_entity_for_send_with_total_count(readable_object, totalCount, self.chunk_size, notifier) return headers, readable, notifier, entity - def _prepare_content_notifier_and_entiy(self, entity, headers, progressCallback, autoClose, readable, chunkedMode, - notifier): + def _prepare_content_notifier_and_entity(self, entity, headers, progressCallback, autoClose, readable, chunkedMode, + notifier): if entity is None: entity = '' elif hasattr(entity, 'read') and callable(entity.read): @@ -1657,25 +1618,21 @@ def _prepare_content_notifier_and_entiy(self, entity, headers, progressCallback, notifier = progress.ProgressNotifier(progressCallback, totalCount) if totalCount > 0 and progressCallback is not None \ else progress.NONE_NOTIFIER - entity = util.get_readable_entity_by_totalcount(entity, totalCount, self.chunk_size, notifier, - autoClose) + entity = util.get_entity_for_send_with_total_count(entity, totalCount, self.chunk_size, notifier, + autoClose) return entity, readable, chunkedMode, notifier @funcCache def appendObject(self, bucketName, objectKey, content=None, metadata=None, headers=None, progressCallback=None, autoClose=True, extensionHeaders=None): - objectKey, headers, content = self._preapare_append_object_input(objectKey, headers, content) + objectKey, headers, content = self._prepare_append_object_input(objectKey, headers, content) chunkedMode = False readable = False notifier = None if content.get('isFile'): - file_path = util.safe_encode(content.get('content')) - if not os.path.exists(file_path): - file_path = util.safe_trans_to_gb2312(file_path) - if not os.path.exists(file_path): - raise Exception('file [%s] does not exist' % file_path) + file_path = self.check_file_path(content.get('content')) if headers.get('contentType') is None: headers['contentType'] = const.MIME_TYPES.get(file_path[file_path.rfind('.') + 1:].lower()) @@ -1685,17 +1642,17 @@ def appendObject(self, bucketName, objectKey, content=None, metadata=None, heade headers['contentLength'] = headers['contentLength'] if headers.get('contentLength') is not None and headers[ 'contentLength'] <= file_size else file_size offset = util.to_long(content.get('offset')) - headers, readable, notifier, entity = self._prepare_file_notifier_and_entiy(offset, file_size, headers, - progressCallback, file_path, - readable) + headers, readable, notifier, entity = self._prepare_file_notifier_and_entity(offset, file_size, headers, + progressCallback, file_path, + readable) headers = self.convertor.trans_put_object(metadata=metadata, headers=headers) self.log_client.log(DEBUG, 'send Path:%s' % file_path) else: entity = content.get('content') - entity, readable, chunkedMode, notifier = self._prepare_content_notifier_and_entiy(entity, headers, - progressCallback, - autoClose, readable, - chunkedMode, notifier) + entity, readable, chunkedMode, notifier = self._prepare_content_notifier_and_entity(entity, headers, + progressCallback, + autoClose, readable, + chunkedMode, notifier) headers = self.convertor.trans_put_object(metadata=metadata, headers=headers) @@ -1727,10 +1684,10 @@ def putContent(self, bucketName, objectKey, content=None, metadata=None, headers readable = False chunkedMode = False + notifier = None try: entity = content - notifier = None if entity is None: entity = '' elif hasattr(entity, 'read') and callable(entity.read): @@ -1745,8 +1702,8 @@ def putContent(self, bucketName, objectKey, content=None, metadata=None, headers notifier = progress.ProgressNotifier(progressCallback, totalCount) if totalCount > 0 and progressCallback \ is not None else progress.NONE_NOTIFIER - entity = util.get_readable_entity_by_totalcount(entity, totalCount, self.chunk_size, notifier, - autoClose) + entity = util.get_entity_for_send_with_total_count(entity, totalCount, self.chunk_size, notifier, + autoClose) notifier.start() ret = self._make_put_request(bucketName, objectKey, headers=_headers, entity=entity, @@ -1766,16 +1723,13 @@ def putObject(self, bucketName, objectKey, content, metadata=None, headers=None, @funcCache def putFile(self, bucketName, objectKey, file_path, metadata=None, headers=None, progressCallback=None, extensionHeaders=None): - file_path = util.safe_encode(file_path) - if not os.path.exists(file_path): - file_path = util.safe_trans_to_gb2312(file_path) - if not os.path.exists(file_path): - raise Exception('file [{0}] doesnot exist'.format(file_path)) - + file_path = self.check_file_path(file_path) _flag = os.path.isdir(file_path) if headers is None: headers = PutObjectHeader() + if metadata is None: + metadata = dict() if _flag: headers['contentLength'] = None @@ -1802,6 +1756,8 @@ def putFile(self, bucketName, objectKey, file_path, metadata=None, headers=None, headers = self._putFileHandleHeader(headers, size, objectKey, file_path) + readable_object = self.gen_readable_object_from_file(file_path) + metadata = self.add_metadata_from_content(metadata, headers, readable_object) _headers = self.convertor.trans_put_object(metadata=metadata, headers=headers) if const.CONTENT_LENGTH_HEADER not in _headers: _headers[const.CONTENT_LENGTH_HEADER] = util.to_string(size) @@ -1815,7 +1771,8 @@ def putFile(self, bucketName, objectKey, file_path, metadata=None, headers=None, else: notifier = progress.NONE_NOTIFIER readable = False - entity = util.get_file_entity_by_totalcount(file_path, totalCount, self.chunk_size, notifier) + + entity = util.get_entity_for_send_with_total_count(readable_object, totalCount, self.chunk_size, notifier) try: notifier.start() ret = self._make_put_request(bucketName, objectKey, headers=_headers, entity=entity, @@ -1825,7 +1782,15 @@ def putFile(self, bucketName, objectKey, file_path, metadata=None, headers=None, self._generate_object_url(ret, bucketName, objectKey) return ret - def _putFileHandleHeader(self, headers, size, objectKey, file_path): + @staticmethod + def add_metadata_from_content(metadata, headers, content): + return metadata + + def gen_readable_object_from_file(self, file_path): + return open(file_path, "rb") + + @staticmethod + def _putFileHandleHeader(headers, size, objectKey, file_path): headers['contentLength'] = util.to_long(headers.get('contentLength')) if headers.get('contentLength') is not None: headers['contentLength'] = size if headers['contentLength'] > size else headers['contentLength'] @@ -1837,11 +1802,13 @@ def _putFileHandleHeader(self, headers, size, objectKey, file_path): headers['contentType'] = const.MIME_TYPES.get(file_path[file_path.rfind('.') + 1:].lower()) return headers - def _get_offset(self, offset, file_size): + @staticmethod + def _get_offset(offset, file_size): offset = offset if offset is not None and 0 <= offset < file_size else 0 return offset - def _get_partsize(self, partSize, file_size, offset): + @staticmethod + def _get_part_size(partSize, file_size, offset): partSize = partSize if partSize is not None and 0 < partSize <= (file_size - offset) else file_size - offset return partSize @@ -1857,7 +1824,8 @@ def _prepare_headers(self, md5, isAttachMd5, file_path, partSize, offset, sseHea return headers - def _prepare_uploadpart_notifier(self, partSize, progressCallback, readable): + @staticmethod + def _prepare_upload_part_notifier(partSize, progressCallback, readable): if partSize > 0 and progressCallback is not None: readable = True notifier = progress.ProgressNotifier(progressCallback, partSize) @@ -1874,15 +1842,27 @@ def _get_headers(self, md5, sseHeader, headers): return headers - def _get_notifier_without_size(self, progressCallback): + @staticmethod + def _get_notifier_without_size(progressCallback): return progress.ProgressNotifier(progressCallback, -1) if progressCallback is not None else progress.NONE_NOTIFIER - def _get_notifier_with_size(self, progressCallback, totalCount): + @staticmethod + def _get_notifier_with_size(progressCallback, totalCount): return progress.ProgressNotifier(progressCallback, totalCount) if totalCount > 0 and progressCallback is not None \ else progress.NONE_NOTIFIER + def _check_file_part_info(self, file_path, offset, partSize): + file_part_info = dict() + file_part_info["file_path"] = self.check_file_path(file_path) + file_size = util.to_long(os.path.getsize(file_path)) + offset = util.to_long(offset) + file_part_info["offset"] = self._get_offset(offset, file_size) + partSize = util.to_long(partSize) + file_part_info["partSize"] = self._get_part_size(partSize, file_size, offset) + return file_part_info + @funcCache def uploadPart(self, bucketName, objectKey, partNumber, uploadId, object=None, isFile=False, partSize=None, offset=0, sseHeader=None, isAttachMd5=False, md5=None, content=None, progressCallback=None, @@ -1890,29 +1870,27 @@ def uploadPart(self, bucketName, objectKey, partNumber, uploadId, object=None, i self._assert_not_null(partNumber, 'partNumber is empty') self._assert_not_null(uploadId, 'uploadId is empty') + chunkedMode = False + readable = False + if content is None: content = object - chunkedMode = False - readable = False notifier = None if isFile: - file_path = util.safe_encode(content) - if not os.path.exists(file_path): - file_path = util.safe_trans_to_gb2312(file_path) - if not os.path.exists(file_path): - raise Exception('file [%s] does not exist' % file_path) - file_size = util.to_long(os.path.getsize(file_path)) - offset = util.to_long(offset) - offset = self._get_offset(offset, file_size) - partSize = util.to_long(partSize) - partSize = self._get_partsize(partSize, file_size, offset) - - headers = {const.CONTENT_LENGTH_HEADER: util.to_string(partSize)} - headers = self._prepare_headers(md5, isAttachMd5, file_path, partSize, offset, sseHeader, headers) - - readable, notifier = self._prepare_uploadpart_notifier(partSize, progressCallback, readable) - entity = util.get_file_entity_by_offset_partsize(file_path, offset, partSize, self.chunk_size, notifier) + checked_file_part_info = self._check_file_part_info(content, offset, partSize) + + headers = {const.CONTENT_LENGTH_HEADER: util.to_string(checked_file_part_info["partSize"])} + headers = self._prepare_headers(md5, isAttachMd5, checked_file_part_info["file_path"], + checked_file_part_info["partSize"], checked_file_part_info["offset"], + sseHeader, headers) + + readable, notifier = self._prepare_upload_part_notifier(checked_file_part_info["partSize"], + progressCallback, readable) + readable_object = open(checked_file_part_info["file_path"], "rb") + readable_object.seek(checked_file_part_info["offset"]) + entity = util.get_entity_for_send_with_total_count(readable_object, checked_file_part_info["partSize"], + self.chunk_size, notifier) else: headers = {} if content is not None and hasattr(content, 'read') and callable(content.read): @@ -1928,8 +1906,8 @@ def uploadPart(self, bucketName, objectKey, partNumber, uploadId, object=None, i headers[const.CONTENT_LENGTH_HEADER] = util.to_string(partSize) totalCount = util.to_long(partSize) notifier = self._get_notifier_with_size(progressCallback, totalCount) - entity = util.get_readable_entity_by_totalcount(content, totalCount, self.chunk_size, notifier, - autoClose) + entity = util.get_entity_for_send_with_total_count(content, totalCount, self.chunk_size, notifier, + autoClose) else: entity = content if entity is None: @@ -1948,36 +1926,40 @@ def uploadPart(self, bucketName, objectKey, partNumber, uploadId, object=None, i notifier.end() return ret + @staticmethod + def check_file_path(file_path): + file_path = util.safe_encode(file_path) + if not os.path.exists(file_path): + file_path = util.safe_trans_to_gb2312(file_path) + if not os.path.exists(file_path): + raise Exception('file [%s] does not exist' % file_path) + return file_path + @funcCache def _uploadPartWithNotifier(self, bucketName, objectKey, partNumber, uploadId, content=None, isFile=False, - partSize=None, - offset=0, sseHeader=None, isAttachMd5=False, md5=None, notifier=None, - extensionHeaders=None): + partSize=None, offset=0, sseHeader=None, isAttachMd5=False, md5=None, notifier=None, + extensionHeaders=None, headers=None): self._assert_not_null(partNumber, 'partNumber is empty') self._assert_not_null(uploadId, 'uploadId is empty') chunkedMode = False readable = False + if headers is None: + headers = dict() if isFile: - file_path = util.safe_encode(content) - if not os.path.exists(file_path): - file_path = util.safe_trans_to_gb2312(file_path) - if not os.path.exists(file_path): - raise Exception('file [%s] does not exist' % file_path) - file_size = util.to_long(os.path.getsize(file_path)) - offset = util.to_long(offset) - offset = self._get_offset(offset, file_size) - partSize = util.to_long(partSize) - partSize = self._get_partsize(partSize, file_size, offset) + checked_file_part_info = self._check_file_part_info(content, offset, partSize) - headers = {const.CONTENT_LENGTH_HEADER: util.to_string(partSize)} - headers = self._prepare_headers(md5, isAttachMd5, file_path, partSize, offset, sseHeader, headers) + headers[const.CONTENT_LENGTH_HEADER] = util.to_string(checked_file_part_info["partSize"]) + headers = self._prepare_headers(md5, isAttachMd5, checked_file_part_info["file_path"], + checked_file_part_info["partSize"], checked_file_part_info["offset"], + sseHeader, headers) if notifier is not None and not isinstance(notifier, progress.NoneNotifier): readable = True - entity = util.get_file_entity_by_offset_partsize(file_path, offset, partSize, self.chunk_size, notifier) + readable_object = open(checked_file_part_info["file_path"], "rb") + readable_object.seek(checked_file_part_info["offset"]) + entity = util.get_entity_for_send_with_total_count(readable_object, partSize, self.chunk_size, notifier) else: - headers = {} if content is not None and hasattr(content, 'read') and callable(content.read): readable = True headers = self._get_headers(md5, sseHeader, headers) @@ -1987,8 +1969,8 @@ def _uploadPartWithNotifier(self, bucketName, objectKey, partNumber, uploadId, c entity = util.get_readable_entity(content, self.chunk_size, notifier) else: headers[const.CONTENT_LENGTH_HEADER] = util.to_string(partSize) - entity = util.get_readable_entity_by_totalcount(content, util.to_long(partSize), self.chunk_size, - notifier) + entity = util.get_entity_for_send_with_total_count(content, util.to_long(partSize), self.chunk_size, + notifier) else: entity = content if entity is None: @@ -2022,7 +2004,9 @@ def copyObject(self, sourceBucketName, sourceObjectKey, destBucketName, destObje sourceObjectKey=sourceObjectKey)) @funcCache - def setObjectAcl(self, bucketName, objectKey, acl=ACL(), versionId=None, aclControl=None, extensionHeaders=None): + def setObjectAcl(self, bucketName, objectKey, acl=None, versionId=None, aclControl=None, extensionHeaders=None): + if acl is None: + acl = ACL() if acl is not None and len(acl) > 0 and aclControl is not None: raise Exception('Both acl and aclControl are set') if not acl and not aclControl: @@ -2063,7 +2047,7 @@ def restoreObject(self, bucketName, objectKey, days, tier=None, versionId=None, @funcCache def initiateMultipartUpload(self, bucketName, objectKey, acl=None, storageClass=None, metadata=None, websiteRedirectLocation=None, contentType=None, sseHeader=None, - expires=None, extensionGrants=None, extensionHeaders=None): + expires=None, extensionGrants=None, extensionHeaders=None, encoding_type=None): objectKey = util.safe_encode(objectKey) if objectKey is None: objectKey = '' @@ -2081,7 +2065,8 @@ def initiateMultipartUpload(self, bucketName, objectKey, acl=None, storageClass= contentType=contentType, sseHeader=sseHeader, expires=expires, - extensionGrants=extensionGrants) + extensionGrants=extensionGrants, + encoding_type=encoding_type) ) @funcCache @@ -2100,11 +2085,14 @@ def copyPart(self, bucketName, objectKey, partNumber, uploadId, copySource, copy @funcCache def completeMultipartUpload(self, bucketName, objectKey, uploadId, completeMultipartUploadRequest, - extensionHeaders=None): + extensionHeaders=None, encoding_type=None): self._assert_not_null(uploadId, 'uploadId is empty') self._assert_not_null(completeMultipartUploadRequest, 'completeMultipartUploadRequest is empty') - - ret = self._make_post_request(bucketName, objectKey, pathArgs={'uploadId': uploadId}, + pathArgs = {'uploadId': uploadId} + if encoding_type is not None: + pathArgs["encoding-type"] = encoding_type + ret = self._make_post_request(bucketName, objectKey, + pathArgs=pathArgs, entity=self.convertor.trans_complete_multipart_upload_request( completeMultipartUploadRequest), methodName='completeMultipartUpload', extensionHeaders=extensionHeaders) @@ -2118,13 +2106,16 @@ def abortMultipartUpload(self, bucketName, objectKey, uploadId, extensionHeaders extensionHeaders=extensionHeaders) @funcCache - def listParts(self, bucketName, objectKey, uploadId, maxParts=None, partNumberMarker=None, extensionHeaders=None): + def listParts(self, bucketName, objectKey, uploadId, maxParts=None, partNumberMarker=None, extensionHeaders=None, + encoding_type=None): self._assert_not_null(uploadId, 'uploadId is empty') pathArgs = {'uploadId': uploadId} if maxParts is not None: pathArgs['max-parts'] = maxParts if partNumberMarker is not None: pathArgs['part-number-marker'] = partNumberMarker + if encoding_type is not None: + pathArgs['encoding-type'] = encoding_type return self._make_get_request(bucketName, objectKey, pathArgs=pathArgs, methodName='listParts', extensionHeaders=extensionHeaders) @@ -2186,26 +2177,16 @@ def getBucketRequestPayment(self, bucketName, extensionHeaders=None): @funcCache def uploadFile(self, bucketName, objectKey, uploadFile, partSize=9 * 1024 * 1024, taskNum=1, enableCheckpoint=False, checkpointFile=None, - checkSum=False, metadata=None, progressCallback=None, headers=None, extensionHeaders=None): + checkSum=False, metadata=None, progressCallback=None, headers=None, + extensionHeaders=None, encoding_type=None): self.log_client.log(INFO, 'enter resume upload file...') self._assert_not_null(bucketName, 'bucketName is empty') self._assert_not_null(objectKey, 'objectKey is empty') self._assert_not_null(uploadFile, 'uploadFile is empty') - if enableCheckpoint and checkpointFile is None: - checkpointFile = uploadFile + '.upload_record' - if partSize < const.DEFAULT_MINIMUM_SIZE: - partSize = const.DEFAULT_MINIMUM_SIZE - elif partSize > const.DEFAULT_MAXIMUM_SIZE: - partSize = const.DEFAULT_MAXIMUM_SIZE - else: - partSize = util.to_int(partSize) - if taskNum <= 0: - taskNum = 1 - else: - taskNum = int(math.ceil(taskNum)) - return _resumer_upload(bucketName, objectKey, uploadFile, partSize, taskNum, enableCheckpoint, checkpointFile, - checkSum, metadata, progressCallback, self, headers, extensionHeaders=extensionHeaders) + return _resume_upload(bucketName, objectKey, uploadFile, partSize, taskNum, enableCheckpoint, checkpointFile, + checkSum, metadata, progressCallback, self, headers, + extensionHeaders=extensionHeaders, encoding_type=encoding_type) @funcCache def _downloadFileWithNotifier(self, bucketName, objectKey, downloadFile=None, partSize=5 * 1024 * 1024, taskNum=1, @@ -2219,22 +2200,10 @@ def _downloadFileWithNotifier(self, bucketName, objectKey, downloadFile=None, pa header = GetObjectHeader() if downloadFile is None: downloadFile = objectKey - if enableCheckpoint and checkpointFile is None: - checkpointFile = downloadFile + '.download_record' - if partSize < const.DEFAULT_MINIMUM_SIZE: - partSize = const.DEFAULT_MINIMUM_SIZE - elif partSize > const.DEFAULT_MAXIMUM_SIZE: - partSize = const.DEFAULT_MAXIMUM_SIZE - else: - partSize = util.to_int(partSize) - if taskNum <= 0: - taskNum = 1 - else: - taskNum = int(math.ceil(taskNum)) - return _resumer_download(bucketName, objectKey, downloadFile, partSize, taskNum, enableCheckpoint, - checkpointFile, header, versionId, progressCallback, self, - imageProcess, notifier, extensionHeaders=extensionHeaders) + return _resume_download(bucketName, objectKey, downloadFile, partSize, taskNum, enableCheckpoint, + checkpointFile, header, versionId, progressCallback, self, + imageProcess, notifier, extensionHeaders=extensionHeaders) def downloadFile(self, bucketName, objectKey, downloadFile=None, partSize=5 * 1024 * 1024, taskNum=1, enableCheckpoint=False, @@ -2246,10 +2215,12 @@ def downloadFile(self, bucketName, objectKey, downloadFile=None, partSize=5 * 10 def downloadFiles(self, bucketName, prefix, downloadFolder=None, taskNum=const.DEFAULT_TASK_NUM, taskQueueSize=const.DEFAULT_TASK_QUEUE_SIZE, - headers=GetObjectHeader(), imageProcess=None, interval=const.DEFAULT_BYTE_INTTERVAL, + headers=None, imageProcess=None, interval=const.DEFAULT_BYTE_INTTERVAL, taskCallback=None, progressCallback=None, threshold=const.DEFAULT_MAXIMUM_SIZE, partSize=5 * 1024 * 1024, subTaskNum=1, enableCheckpoint=False, checkpointFile=None, extensionHeaders=None): + if headers is None: + headers = GetObjectHeader() return _download_files(self, bucketName, prefix, downloadFolder, taskNum, taskQueueSize, headers, imageProcess, interval, taskCallback, progressCallback, threshold, partSize, subTaskNum, enableCheckpoint, checkpointFile, extensionHeaders=extensionHeaders) diff --git a/src/obs/const.py b/src/obs/const.py index 0a077e5..1e193f4 100644 --- a/src/obs/const.py +++ b/src/obs/const.py @@ -16,6 +16,8 @@ import platform import os +READ_ONCE_LENGTH = 65536 + CONTENT_LENGTH_HEADER = 'Content-Length' CONTENT_TYPE_HEADER = 'Content-Type' CONTENT_MD5_HEADER = 'Content-MD5' @@ -85,7 +87,7 @@ DEFAULT_TASK_NUM = 8 DEFAULT_TASK_QUEUE_SIZE = 20000 -OBS_SDK_VERSION = '3.21.4' +OBS_SDK_VERSION = '3.21.8' V2_META_HEADER_PREFIX = 'x-amz-meta-' V2_HEADER_PREFIX = 'x-amz-' diff --git a/src/obs/convertor.py b/src/obs/convertor.py index ecbb2cd..a8eb4aa 100644 --- a/src/obs/convertor.py +++ b/src/obs/convertor.py @@ -38,6 +38,11 @@ Redirect, FilterRule, FunctionGraphConfiguration, Upload, CompleteMultipartUploadResponse, ListPartsResponse, \ Grant, ReplicationRule, Transition, Grantee +if const.IS_PYTHON2: + from urllib import unquote_plus, quote_plus +else: + from urllib.parse import unquote_plus, quote_plus + class Adapter(object): OBS_ALLOWED_ACL_CONTROL = ['private', 'public-read', 'public-read-write', 'public-read-delivered', @@ -94,7 +99,8 @@ def content_sha256_header(self): def default_storage_class_header(self): return self._get_header_prefix() + 'storage-class' if self.is_obs else 'x-default-storage-class' - def az_redundancy_header(self): + @staticmethod + def az_redundancy_header(): return 'x-obs-az-redundancy' def storage_class_header(self): @@ -103,7 +109,8 @@ def storage_class_header(self): def request_id_header(self): return self._get_header_prefix() + 'request-id' - def indicator_header(self): + @staticmethod + def indicator_header(): return 'x-reserved-indicator' def location_header(self): @@ -113,7 +120,8 @@ def bucket_region_header(self): return self._get_header_prefix() + 'bucket-location' if self.is_obs \ else self._get_header_prefix() + 'bucket-region' - def server_version_header(self): + @staticmethod + def server_version_header(): return 'x-obs-version' def version_id_header(self): @@ -153,13 +161,15 @@ def sse_c_key_md5_header(self): def website_redirect_location_header(self): return self._get_header_prefix() + 'website-redirect-location' - def success_action_redirect_header(self): + @staticmethod + def success_action_redirect_header(): return 'success-action-redirect' def restore_header(self): return self._get_header_prefix() + 'restore' - def expires_header(self): + @staticmethod + def expires_header(): return 'x-obs-expires' def expiration_header(self): @@ -186,10 +196,12 @@ def copy_source_if_modified_since_header(self): def copy_source_if_unmodified_since_header(self): return self._get_header_prefix() + 'copy-source-if-unmodified-since' - def next_position_header(self): + @staticmethod + def next_position_header(): return 'x-obs-next-append-position' - def object_type_header(self): + @staticmethod + def object_type_header(): return 'x-obs-object-type' def request_payer_header(self): @@ -210,7 +222,7 @@ def _adapt_group_is_obs(self, group): return group if group in self.OBS_ALLOWED_GROUP else 'Everyone' \ if group in ('http://acs.amazonaws.com/groups/global/AllUsers', 'AllUsers') else None - def adapt_retore_tier(self, tier): + def adapt_restore_tier(self, tier): if self.is_obs: return tier if tier in self.OBS_ALLOWED_RESTORE_TIER else None @@ -266,7 +278,17 @@ def __init__(self, signature, ha=None): self.is_obs = signature.lower() == 'obs' self.ha = ha - def _put_key_value(self, headers, key, value): + @staticmethod + def url_encode(value, encoding_type): + if encoding_type and encoding_type.lower() == "url": + if const.IS_PYTHON2 and isinstance(value, unicode): + value = quote_plus(util.safe_encode(value)) + return value + value = quote_plus(value) + return value + + @staticmethod + def _put_key_value(headers, key, value): if value is not None: if const.IS_PYTHON2: value = util.safe_encode(value) @@ -317,6 +339,7 @@ def trans_list_objects(self, **kwargs): self._put_key_value(pathArgs, 'marker', kwargs.get('marker')) self._put_key_value(pathArgs, 'delimiter', kwargs.get('delimiter')) self._put_key_value(pathArgs, 'max-keys', kwargs.get('max_keys')) + self._put_key_value(pathArgs, 'encoding-type', kwargs.get('encoding_type')) return {'pathArgs': pathArgs} def trans_list_versions(self, **kwargs): @@ -328,6 +351,7 @@ def trans_list_versions(self, **kwargs): self._put_key_value(pathArgs, 'max-keys', version.get('max_keys')) self._put_key_value(pathArgs, 'delimiter', version.get('delimiter')) self._put_key_value(pathArgs, 'version-id-marker', version.get('version_id_marker')) + self._put_key_value(pathArgs, 'encoding-type', version.get('encoding_type')) return {'pathArgs': pathArgs} def trans_get_bucket_metadata(self, **kwargs): @@ -372,7 +396,8 @@ def trans_encryption(self, encryption, key=None): ET.SubElement(sse, 'KMSMasterKeyID').text = util.to_string(key) return ET.tostring(root, 'UTF-8') - def trans_quota(self, quota): + @staticmethod + def trans_quota(quota): root = ET.Element('Quota') ET.SubElement(root, 'StorageQuota').text = util.to_string(quota) return ET.tostring(root, 'UTF-8') @@ -385,7 +410,8 @@ def trans_set_bucket_tagging(self, **kwargs): 'entity': entity } - def trans_tag_info(self, tagInfo): + @staticmethod + def trans_tag_info(tagInfo): root = ET.Element('Tagging') tagSetEle = ET.SubElement(root, 'TagSet') if tagInfo.get('tagSet') is not None and len(tagInfo['tagSet']) > 0: @@ -401,7 +427,8 @@ def trans_set_bucket_cors(self, **kwargs): headers = {const.CONTENT_MD5_HEADER: util.base64_encode(util.md5_encode(entity))} return {'pathArgs': {'cors': None}, 'headers': headers, 'entity': entity} - def trans_cors_rules(self, corsRuleList): + @staticmethod + def trans_cors_rules(corsRuleList): root = ET.Element('CORSConfiguration') for cors in corsRuleList: corsRuleEle = ET.SubElement(root, 'CORSRule') @@ -430,19 +457,25 @@ def trans_delete_objects(self, **kwargs): def trans_delete_objects_request(self, deleteObjectsRequest): root = ET.Element('Delete') + encoding_type = None if deleteObjectsRequest is not None: if deleteObjectsRequest.get('quiet') is not None: ET.SubElement(root, 'Quiet').text = util.to_string(deleteObjectsRequest['quiet']).lower() + if deleteObjectsRequest.get('encoding_type') is not None: + ET.SubElement(root, 'EncodingType').text = util.to_string(deleteObjectsRequest['encoding_type']) + encoding_type = util.to_string(deleteObjectsRequest['encoding_type']) if isinstance(deleteObjectsRequest.get('objects'), list) and len(deleteObjectsRequest['objects']) > 0: for obj in deleteObjectsRequest['objects']: if obj.get('key') is not None: objectEle = ET.SubElement(root, 'Object') - ET.SubElement(objectEle, 'Key').text = util.safe_decode(obj['key']) + key_text = self.url_encode(obj['key'], encoding_type) + ET.SubElement(objectEle, 'Key').text = util.safe_decode(key_text) if obj.get('versionId') is not None: ET.SubElement(objectEle, 'VersionId').text = util.safe_decode(obj['versionId']) return ET.tostring(root, 'UTF-8') - def trans_version_status(self, status): + @staticmethod + def trans_version_status(status): root = ET.Element('VersioningConfiguration') ET.SubElement(root, 'Status').text = util.to_string(status) return ET.tostring(root, 'UTF-8') @@ -540,7 +573,8 @@ def trans_website(self, website): root = self._trans_website_routingRules(root, website) return ET.tostring(root, 'UTF-8') - def _trans_website_routingRules(self, root, website): + @staticmethod + def _trans_website_routingRules(root, website): if isinstance(website.get('routingRules'), list) and bool(website['routingRules']): routingRulesEle = ET.SubElement(root, 'RoutingRules') for routingRule in website['routingRules']: @@ -613,7 +647,8 @@ def _set_configuration(config_type, urn_type): return ET.tostring(root, 'UTF-8') - def trans_complete_multipart_upload_request(self, completeMultipartUploadRequest): + @staticmethod + def trans_complete_multipart_upload_request(completeMultipartUploadRequest): root = ET.Element('CompleteMultipartUpload') parts = [] if completeMultipartUploadRequest.get('parts') is None else ( sorted(completeMultipartUploadRequest['parts'], key=lambda d: d.partNum)) @@ -724,7 +759,7 @@ def trans_logging(self, logging): def trans_restore(self, days, tier): root = ET.Element('RestoreRequest') ET.SubElement(root, 'Days').text = util.to_string(days) - tier = self.ha.adapt_retore_tier(tier) + tier = self.ha.adapt_restore_tier(tier) if tier is not None: glacierJobEle = ET.SubElement(root, 'RestoreJob') if self.is_obs else ET.SubElement(root, 'GlacierJobParameters') @@ -742,6 +777,7 @@ def trans_put_object(self, **kwargs): self._put_key_value(_headers, k, v) if headers is not None and len(headers) > 0: self._put_key_value(_headers, const.CONTENT_MD5_HEADER, headers.get('md5')) + self._put_key_value(_headers, self.ha.content_sha256_header(), headers.get('sha256')) self._put_key_value(_headers, self.ha.acl_header(), self.ha.adapt_acl_control(headers.get('acl'))) self._put_key_value(_headers, self.ha.website_redirect_location_header(), headers.get('location')) self._put_key_value(_headers, const.CONTENT_TYPE_HEADER, headers.get('contentType')) @@ -771,6 +807,8 @@ def trans_put_object(self, **kwargs): return _headers def trans_initiate_multipart_upload(self, **kwargs): + pathArgs = {'uploads': None} + self._put_key_value(pathArgs, "encoding-type", kwargs.get('encoding_type')) headers = {} self._put_key_value(headers, self.ha.acl_header(), self.ha.adapt_acl_control(kwargs.get('acl'))) self._put_key_value(headers, self.ha.storage_class_header(), @@ -800,7 +838,7 @@ def trans_initiate_multipart_upload(self, **kwargs): for key, value in grantDict.items(): self._put_key_value(headers, key, ','.join(value)) - return {'pathArgs': {'uploads': None}, 'headers': headers} + return {'pathArgs': pathArgs, 'headers': headers} def trans_set_object_metadata(self, **kwargs): versionId = kwargs.get('versionId') @@ -955,6 +993,7 @@ def trans_get_object(self, **kwargs): def trans_list_multipart_uploads(self, **kwargs): pathArgs = {'uploads': None} + self._put_key_value(pathArgs, 'encoding-type', kwargs.get('encoding_type')) multipart = kwargs.get('multipart') if multipart is not None: self._put_key_value(pathArgs, 'delimiter', multipart.get('delimiter')) @@ -1022,7 +1061,8 @@ def trans_replication(self, replication): replicationRule['storageClass']) return ET.tostring(root, 'UTF-8') - def trans_bucket_request_payment(self, payer): + @staticmethod + def trans_bucket_request_payment(payer): root = ET.Element('RequestPaymentConfiguration') ET.SubElement(root, 'Payer').text = util.to_string(payer) return ET.tostring(root, 'UTF-8') @@ -1059,13 +1099,18 @@ def trans_set_bucket_fetch_job(self, fetchJob): entity = json.dumps(fetchJob, ensure_ascii=False) return {'headers': headers, 'entity': entity} - def _find_item(self, root, itemname): - result = root.find(itemname) + @staticmethod + def _find_item(root, item_name, encoding_type=None): + result = root.find(item_name) if result is None: return None result = result.text + if result is None: + return None if const.IS_PYTHON2: result = util.safe_encode(result) + if encoding_type == "url": + return util.to_string(unquote_plus(result)) return util.to_string(result) def parseListBuckets(self, xml, headers=None): @@ -1100,20 +1145,21 @@ def parseErrorResult(self, xml, headers=None): def parseListObjects(self, xml, headers=None): root = ET.fromstring(xml) + encoding_type = self._find_item(root, 'EncodingType') name = self._find_item(root, 'Name') - prefix = self._find_item(root, 'Prefix') - marker = self._find_item(root, 'Marker') - delimiter = self._find_item(root, 'Delimiter') + prefix = self._find_item(root, 'Prefix', encoding_type) + marker = self._find_item(root, 'Marker', encoding_type) + delimiter = self._find_item(root, 'Delimiter', encoding_type) max_keys = self._find_item(root, 'MaxKeys') is_truncated = self._find_item(root, 'IsTruncated') - next_marker = self._find_item(root, 'NextMarker') + next_marker = self._find_item(root, 'NextMarker', encoding_type) key_entries = [] contents = root.findall('Contents') if contents is not None: for node in contents: - key = self._find_item(node, 'Key') + key = self._find_item(node, 'Key', encoding_type) lastmodified = self._find_item(node, 'LastModified') etag = self._find_item(node, 'ETag') size = self._find_item(node, 'Size') @@ -1130,19 +1176,19 @@ def parseListObjects(self, xml, headers=None): isAppendable=isAppendable == 'Appendable') key_entries.append(key_entry) - commonprefixs = [] + common_prefixes = [] prefixes = root.findall('CommonPrefixes') if prefixes is not None: for p in prefixes: - pre = self._find_item(p, 'Prefix') + pre = self._find_item(p, 'Prefix', encoding_type) commonprefix = CommonPrefix(prefix=pre) - commonprefixs.append(commonprefix) + common_prefixes.append(commonprefix) location = headers.get(self.ha.bucket_region_header()) return ListObjectsResponse(name=name, location=location, prefix=prefix, marker=marker, delimiter=delimiter, max_keys=util.to_int(max_keys), is_truncated=util.to_bool(is_truncated), next_marker=next_marker, - contents=key_entries, commonPrefixs=commonprefixs) + contents=key_entries, commonPrefixs=common_prefixes, encoding_type=encoding_type) def parseGetBucketMetadata(self, headers): option = GetBucketMetadataResponse() @@ -1169,7 +1215,8 @@ def parseGetBucketStorageInfo(self, xml, headers=None): objectNumber = self._find_item(root, 'ObjectNumber') return GetBucketStorageInfoResponse(size=util.to_long(size), objectNumber=util.to_int(objectNumber)) - def parseGetBucketPolicy(self, json_str, headers=None): + @staticmethod + def parseGetBucketPolicy(json_str, headers=None): return Policy(policyJSON=json_str) def parseGetBucketStoragePolicy(self, xml, headers=None): @@ -1193,7 +1240,8 @@ def parseGetBucketEncryption(self, xml, headers=None): return result - def parseGetBucketTagging(self, xml, headers=None): + @staticmethod + def parseGetBucketTagging(xml, headers=None): result = TagInfo() root = ET.fromstring(xml) tags = root.findall('TagSet/Tag') @@ -1244,12 +1292,14 @@ def parseGetBucketCors(self, xml, headers=None): def parseListVersions(self, xml, headers=None): root = ET.fromstring(xml) + encoding_type = self._find_item(root, 'EncodingType') + Name = self._find_item(root, 'Name') - Prefix = self._find_item(root, 'Prefix') - Delimiter = self._find_item(root, 'Delimiter') - KeyMarker = self._find_item(root, 'KeyMarker') + Prefix = self._find_item(root, 'Prefix', encoding_type) + Delimiter = self._find_item(root, 'Delimiter', encoding_type) + KeyMarker = self._find_item(root, 'KeyMarker', encoding_type) VersionIdMarker = self._find_item(root, 'VersionIdMarker') - NextKeyMarker = self._find_item(root, 'NextKeyMarker') + NextKeyMarker = self._find_item(root, 'NextKeyMarker', encoding_type) NextVersionIdMarker = self._find_item(root, 'NextVersionIdMarker') MaxKeys = self._find_item(root, 'MaxKeys') IsTruncated = self._find_item(root, 'IsTruncated') @@ -1258,12 +1308,12 @@ def parseListVersions(self, xml, headers=None): versionIdMarker=VersionIdMarker, nextKeyMarker=NextKeyMarker, nextVersionIdMarker=NextVersionIdMarker, maxKeys=util.to_int(MaxKeys), - isTruncated=util.to_bool(IsTruncated)) + isTruncated=util.to_bool(IsTruncated), encoding_type=encoding_type) version_list = [] versions = root.findall('Version') for version in versions: - Key = self._find_item(version, 'Key') + Key = self._find_item(version, 'Key', encoding_type) VersionId = self._find_item(version, 'VersionId') IsLatest = self._find_item(version, 'IsLatest') LastModified = self._find_item(version, 'LastModified') @@ -1286,7 +1336,7 @@ def parseListVersions(self, xml, headers=None): marker_list = [] markers = root.findall('DeleteMarker') for marker in markers: - Key = self._find_item(marker, 'Key') + Key = self._find_item(marker, 'Key', encoding_type) VersionId = self._find_item(marker, 'VersionId') IsLatest = self._find_item(marker, 'IsLatest') LastModified = self._find_item(marker, 'LastModified') @@ -1300,15 +1350,16 @@ def parseListVersions(self, xml, headers=None): lastModified=DateTime.UTCToLocal(LastModified), owner=Owners) marker_list.append(Marker) - prefixs = root.findall('CommonPrefixes') + prefixes = root.findall('CommonPrefixes') prefix_list = [] - for prefix in prefixs: - Prefix = self._find_item(prefix, 'Prefix') + for prefix in prefixes: + Prefix = self._find_item(prefix, 'Prefix', encoding_type) Pre = CommonPrefix(prefix=Prefix) prefix_list.append(Pre) return ObjectVersions(head=head, markers=marker_list, commonPrefixs=prefix_list, versions=version_list) - def parseOptionsBucket(self, headers): + @staticmethod + def parseOptionsBucket(headers): option = OptionsResponse() option.accessContorlAllowOrigin = headers.get('access-control-allow-origin') option.accessContorlAllowHeaders = headers.get('access-control-allow-headers') @@ -1322,9 +1373,11 @@ def parseDeleteObjects(self, xml, headers=None): deleted_list = [] error_list = [] deleteds = root.findall('Deleted') + encoding_type = self._find_item(root, 'EncodingType') + if deleteds: for d in deleteds: - key = self._find_item(d, 'Key') + key = self._find_item(d, 'Key', encoding_type) versionId = self._find_item(d, 'VersionId') deleteMarker = d.find('DeleteMarker') deleteMarker = util.to_bool(deleteMarker.text) if deleteMarker is not None else None @@ -1334,12 +1387,12 @@ def parseDeleteObjects(self, xml, headers=None): errors = root.findall('Error') if errors: for e in errors: - _key = self._find_item(e, 'Key') + _key = self._find_item(e, 'Key', encoding_type) _versionId = self._find_item(e, 'VersionId') _code = self._find_item(e, 'Code') _message = self._find_item(e, 'Message') error_list.append(ErrorResult(key=_key, versionId=_versionId, code=_code, message=_message)) - return DeleteObjectsResponse(deleted=deleted_list, error=error_list) + return DeleteObjectsResponse(deleted=deleted_list, error=error_list, encoding_type=encoding_type) def parseDeleteObject(self, headers): deleteObjectResponse = DeleteObjectResponse() @@ -1360,18 +1413,18 @@ def parseGetBucketLifecycle(self, xml, headers=None): _id = self._find_item(rule, 'ID') prefix = self._find_item(rule, 'Prefix') status = self._find_item(rule, 'Status') - expira = rule.find('Expiration') + expired = rule.find('Expiration') expiration = None - if expira is not None: - d = expira.find('Date') + if expired is not None: + d = expired.find('Date') date = DateTime.UTCToLocalMid(d.text) if d is not None else None - day = expira.find('Days') + day = expired.find('Days') days = util.to_int(day.text) if day is not None else None expiration = Expiration(date=date, days=days) - nocurrentExpira = rule.find('NoncurrentVersionExpiration') + nocurrent_expire = rule.find('NoncurrentVersionExpiration') noncurrentVersionExpiration = NoncurrentVersionExpiration(noncurrentDays=util.to_int( - nocurrentExpira.find('NoncurrentDays').text)) if nocurrentExpira is not None else None + nocurrent_expire.find('NoncurrentDays').text)) if nocurrent_expire is not None else None transis = rule.findall('Transition') @@ -1500,10 +1553,12 @@ def _get_configuration(config_class, config_type, urn_type): def parseListMultipartUploads(self, xml, headers=None): root = ET.fromstring(xml) + encoding_type = self._find_item(root, 'EncodingType') + bucket = self._find_item(root, 'Bucket') - KeyMarker = self._find_item(root, 'KeyMarker') + KeyMarker = self._find_item(root, 'KeyMarker', encoding_type) UploadIdMarker = self._find_item(root, 'UploadIdMarker') - NextKeyMarker = self._find_item(root, 'NextKeyMarker') + NextKeyMarker = self._find_item(root, 'NextKeyMarker', encoding_type) NextUploadIdMarker = self._find_item(root, 'NextUploadIdMarker') MaxUploads = root.find('MaxUploads') @@ -1512,14 +1567,14 @@ def parseListMultipartUploads(self, xml, headers=None): IsTruncated = root.find('IsTruncated') IsTruncated = util.to_bool(IsTruncated.text) if IsTruncated is not None else None - prefix = self._find_item(root, 'Prefix') - delimiter = self._find_item(root, 'Delimiter') + prefix = self._find_item(root, 'Prefix', encoding_type) + delimiter = self._find_item(root, 'Delimiter', encoding_type) rules = root.findall('Upload') uploadlist = [] if rules: for rule in rules: - Key = self._find_item(rule, 'Key') + Key = self._find_item(rule, 'Key', encoding_type) UploadId = self._find_item(rule, 'UploadId') ID = self._find_item(rule, 'Initiator/ID') @@ -1529,36 +1584,38 @@ def parseListMultipartUploads(self, xml, headers=None): owner_id = self._find_item(rule, 'Owner/ID') owner_name = None if self.is_obs else self._find_item(rule, 'Owner/DisplayName') - ower = Owner(owner_id=owner_id, owner_name=owner_name) + owner = Owner(owner_id=owner_id, owner_name=owner_name) StorageClass = self._find_item(rule, 'StorageClass') Initiated = rule.find('Initiated') Initiated = DateTime.UTCToLocal(Initiated.text) if Initiated is not None else None - upload = Upload(key=Key, uploadId=UploadId, initiator=initiator, owner=ower, storageClass=StorageClass, + upload = Upload(key=Key, uploadId=UploadId, initiator=initiator, owner=owner, storageClass=StorageClass, initiated=Initiated) uploadlist.append(upload) common = root.findall('CommonPrefixes') commonlist = [] if common: for comm in common: - comm_prefix = self._find_item(comm, 'Prefix') + comm_prefix = self._find_item(comm, 'Prefix', encoding_type) Comm_Prefix = CommonPrefix(prefix=comm_prefix) commonlist.append(Comm_Prefix) return ListMultipartUploadsResponse(bucket=bucket, keyMarker=KeyMarker, uploadIdMarker=UploadIdMarker, nextKeyMarker=NextKeyMarker, nextUploadIdMarker=NextUploadIdMarker, maxUploads=MaxUploads, isTruncated=IsTruncated, prefix=prefix, delimiter=delimiter, - upload=uploadlist, commonPrefixs=commonlist) + upload=uploadlist, commonPrefixs=commonlist, encoding_type=encoding_type) def parseCompleteMultipartUpload(self, xml, headers=None): root = ET.fromstring(xml) + encoding_type = self._find_item(root, 'EncodingType') + location = self._find_item(root, 'Location') bucket = self._find_item(root, 'Bucket') - key = self._find_item(root, 'Key') + key = self._find_item(root, 'Key', encoding_type) eTag = self._find_item(root, 'ETag') completeMultipartUploadResponse = CompleteMultipartUploadResponse(location=location, bucket=bucket, key=key, - etag=eTag) + etag=eTag, encoding_type=encoding_type) completeMultipartUploadResponse.versionId = headers.get(self.ha.version_id_header()) completeMultipartUploadResponse.sseKms = headers.get(self.ha.sse_kms_header()) completeMultipartUploadResponse.sseKmsKey = headers.get(self.ha.sse_kms_key_header()) @@ -1569,8 +1626,10 @@ def parseCompleteMultipartUpload(self, xml, headers=None): def parseListParts(self, xml, headers=None): root = ET.fromstring(xml) + encoding_type = self._find_item(root, 'EncodingType') + bucketName = self._find_item(root, 'Bucket') - objectKey = self._find_item(root, 'Key') + objectKey = self._find_item(root, 'Key', encoding_type) uploadId = self._find_item(root, 'UploadId') storageClass = self._find_item(root, 'StorageClass') @@ -1588,30 +1647,30 @@ def parseListParts(self, xml, headers=None): initiator = Initiator(id=initiatorid, name=displayname) - ownerid = self._find_item(root, 'Owner/ID') - ownername = self._find_item(root, 'Owner/DisplayName') - owner = Owner(owner_id=ownerid, owner_name=ownername) + owner_id = self._find_item(root, 'Owner/ID') + owner_name = self._find_item(root, 'Owner/DisplayName') + owner = Owner(owner_id=owner_id, owner_name=owner_name) parts = self._parseListPartsHandleParts(root) return ListPartsResponse(bucketName=bucketName, objectKey=objectKey, uploadId=uploadId, initiator=initiator, owner=owner, storageClass=storageClass, partNumberMarker=partNumbermarker, nextPartNumberMarker=nextPartNumberMarker, - maxParts=maxParts, isTruncated=isTruncated, parts=parts) + maxParts=maxParts, isTruncated=isTruncated, parts=parts, encoding_type=encoding_type) def _parseListPartsHandleParts(self, root): part_list = root.findall('Part') parts = [] if part_list: for part in part_list: - partnumber = part.find('PartNumber') - partnumber = util.to_int(partnumber.text) if partnumber is not None else None - modifieddate = part.find('LastModified') - modifieddate = DateTime.UTCToLocal(modifieddate.text) if modifieddate is not None else None + part_number = part.find('PartNumber') + part_number = util.to_int(part_number.text) if part_number is not None else None + modified_date = part.find('LastModified') + modified_date = DateTime.UTCToLocal(modified_date.text) if modified_date is not None else None etag = self._find_item(part, 'ETag') size = part.find('Size') size = util.to_long(size.text) if size is not None else None - parts.append(Part(partNumber=partnumber, lastModified=modifieddate, etag=etag, size=size)) + parts.append(Part(partNumber=part_number, lastModified=modified_date, etag=etag, size=size)) return parts def parseGetBucketAcl(self, xml, headers=None): @@ -1706,10 +1765,13 @@ def parseAppendObject(self, headers): def parseInitiateMultipartUpload(self, xml, headers=None): root = ET.fromstring(xml) + encoding_type = self._find_item(root, 'EncodingType') + bucketName = self._find_item(root, 'Bucket') - objectKey = self._find_item(root, 'Key') + objectKey = self._find_item(root, 'Key', encoding_type) uploadId = self._find_item(root, 'UploadId') - response = InitiateMultipartUploadResponse(bucketName=bucketName, objectKey=objectKey, uploadId=uploadId) + response = InitiateMultipartUploadResponse(bucketName=bucketName, objectKey=objectKey, uploadId=uploadId, + encoding_type=encoding_type) response.sseKms = headers.get(self.ha.sse_kms_header()) response.sseKmsKey = headers.get(self.ha.sse_kms_key_header()) response.sseC = headers.get(self.ha.sse_c_header()) @@ -1822,8 +1884,9 @@ def parseGetBucketRequestPayment(self, xml, headers=None): payment = GetBucketRequestPaymentResponse(payer=payer) return payment - def _find_json_item(self, value, itemname): - result = value.get(itemname) + @staticmethod + def _find_json_item(value, item_name): + result = value.get(item_name) if result is None: return None if const.IS_PYTHON2: @@ -1890,15 +1953,18 @@ def parseGetBucketFetchJob(self, jsons, header=None): # begin workflow related # begin workflow related - def parseGetJsonResponse(self, jsons, header=None): + @staticmethod + def parseGetJsonResponse(jsons, header=None): return jsons - def parseCreateWorkflowTemplateResponse(self, jsons, header=None): + @staticmethod + def parseCreateWorkflowTemplateResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) templateName = result.get('template_name') return CreateWorkflowTemplateResponse(templateName=templateName) - def parseGetWorkflowTemplateResponse(self, jsons, header=None): + @staticmethod + def parseGetWorkflowTemplateResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) templateName = result.get('template_name') description = result.get('description') @@ -1911,7 +1977,8 @@ def parseGetWorkflowTemplateResponse(self, jsons, header=None): inputs=inputs, tags=tags, createTime=createTime, lastModifyTime=lastModifyTime) - def parseListWorkflowTemplateResponse(self, jsons, header=None): + @staticmethod + def parseListWorkflowTemplateResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) count = result.get('count') templates = result.get('templates') @@ -1920,14 +1987,16 @@ def parseListWorkflowTemplateResponse(self, jsons, header=None): return ListWorkflowTemplateResponse(templates=templates, count=count, nextStart=nextStart, isTruncated=isTruncated) - def parseCreateWorkflowResponse(self, jsons, header=None): + @staticmethod + def parseCreateWorkflowResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) graphName = result.get('graph_name') graphUrn = result.get('graph_urn') createdAt = result.get('created_at') return CreateWorkflowResponse(graphName=graphName, graphUrn=graphUrn, createdAt=createdAt) - def parseGetWorkflowResponse(self, jsons, header=None): + @staticmethod + def parseGetWorkflowResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) name = result.get('name') createdAt = result.get('created_at') @@ -1937,14 +2006,16 @@ def parseGetWorkflowResponse(self, jsons, header=None): return GetWorkflowResponse(name=name, createdAt=createdAt, definition=definition, graphUrn=graphUrn, description=description) - def parseUpdateWorkflowResponse(self, jsons, header=None): + @staticmethod + def parseUpdateWorkflowResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) graphName = result.get('graph_name') graphUrn = result.get('graph_urn') lastModified = result.get('last_modified') return UpdateWorkflowResponse(graphName=graphName, graphUrn=graphUrn, lastModified=lastModified) - def parseListWorkflowResponse(self, jsons, header=None): + @staticmethod + def parseListWorkflowResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) count = result.get('count') graphs = result.get('graphs') @@ -1952,7 +2023,8 @@ def parseListWorkflowResponse(self, jsons, header=None): isTruncated = result.get('is_truncated') return ListWorkflowResponse(graphs=graphs, count=count, nextStart=nextStart, isTruncated=isTruncated) - def parseAsyncAPIStartWorkflowResponse(self, jsons, header=None): + @staticmethod + def parseAsyncAPIStartWorkflowResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) executionUrn = result.get('execution_urn') startedAt = result.get('started_at') @@ -1960,7 +2032,8 @@ def parseAsyncAPIStartWorkflowResponse(self, jsons, header=None): return AsyncAPIStartWorkflowResponse(executionUrn=executionUrn, startedAt=startedAt, executionName=executionName) - def parseListWorkflowExecutionResponse(self, jsons, header=None): + @staticmethod + def parseListWorkflowExecutionResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) count = result.get('count') nextMarker = result.get('next_marker') @@ -1969,12 +2042,14 @@ def parseListWorkflowExecutionResponse(self, jsons, header=None): return ListWorkflowExecutionResponse(count=count, nextMarker=nextMarker, isTruncated=isTruncated, executions=executions) - def parseGetWorkflowExecutionResponse(self, jsons, header=None): + @staticmethod + def parseGetWorkflowExecutionResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) executionInfo = result.get('execution_info') return GetWorkflowExecutionResponse(executionInfo=executionInfo) - def parseRestoreFailedWorkflowExecutionResponse(self, jsons, header=None): + @staticmethod + def parseRestoreFailedWorkflowExecutionResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) executionUrn = result.get('execution_urn') restoredAt = result.get('restored_at') @@ -1982,7 +2057,8 @@ def parseRestoreFailedWorkflowExecutionResponse(self, jsons, header=None): return RestoreFailedWorkflowExecutionResponse(executionUrn=executionUrn, restoredAt=restoredAt, executionName=executionName) - def parseGetTriggerPolicyResponse(self, jsons, header=None): + @staticmethod + def parseGetTriggerPolicyResponse(jsons, header=None): result = util.jsonLoadsForPy2(jsons) if const.IS_PYTHON2 else json.loads(jsons) rules = result.get('rules') return GetTriggerPolicyResponse(rules=rules) diff --git a/src/obs/crypto_client.py b/src/obs/crypto_client.py new file mode 100644 index 0000000..89f5a47 --- /dev/null +++ b/src/obs/crypto_client.py @@ -0,0 +1,267 @@ +# coding:utf-8 + +import io + +from obs import GetObjectHeader, ObsClient, PutObjectHeader, UploadFileHeader, const, progress, util +from obs.ilog import INFO +from obs.transfer import _resume_download_with_operation, downloadOperation, uploadOperation + + +class CryptoObsClient(ObsClient): + def __init__(self, cipher_generator, *args, **kwargs): + self.cipher_generator = cipher_generator + super(CryptoObsClient, self).__init__(*args, **kwargs) + + def appendObject(self, bucketName, objectKey, content=None, metadata=None, headers=None, progressCallback=None, + autoClose=True, extensionHeaders=None): + raise Exception("AppendObject is not supported in CryptoObsClient") + + def copyPart(self, bucketName, objectKey, partNumber, uploadId, copySource, copySourceRange=None, + destSseHeader=None, sourceSseHeader=None, extensionHeaders=None): + raise Exception("CopyPart is not supported in CryptoObsClient") + + def initiateMultipartUpload(self, bucketName, objectKey, acl=None, storageClass=None, + metadata=None, websiteRedirectLocation=None, contentType=None, sseHeader=None, + expires=None, extensionGrants=None, extensionHeaders=None, encoding_type=None): + raise Exception("InitiateMultipartUpload is not supported in CryptoObsClient") + + def uploadPart(self, bucketName, objectKey, partNumber, uploadId, object=None, isFile=False, partSize=None, + offset=0, sseHeader=None, isAttachMd5=False, md5=None, content=None, progressCallback=None, + autoClose=True, extensionHeaders=None): + raise Exception("UploadPart is not supported in CryptoObsClient") + + def initiateEncryptedMultipartUpload(self, bucketName, objectKey, crypto_cipher, acl=None, storageClass=None, + metadata=None, websiteRedirectLocation=None, contentType=None, sseHeader=None, + expires=None, extensionGrants=None, extensionHeaders=None, encoding_type=None): + if self.cipher_generator.need_sha256: + raise Exception("Could not calculate sha256 for initiateMultipartUpload") + if metadata is None: + metadata = dict() + content = crypto_cipher + metadata = content.gen_need_metadata_and_headers(metadata, UploadFileHeader()) + resp = super(CryptoObsClient, self).initiateMultipartUpload(bucketName, objectKey, acl, storageClass, metadata, + websiteRedirectLocation, contentType, sseHeader, + expires, extensionGrants, + extensionHeaders, encoding_type) + resp.body["crypto_info"] = content.safe_crypto_info() + content.close() + return resp + + def putContent(self, bucketName, objectKey, content=None, metadata=None, headers=None, + progressCallback=None, autoClose=True, extensionHeaders=None): + if headers is None: + headers = PutObjectHeader() + if (const.IS_PYTHON2 and isinstance(content, unicode)) or isinstance(content, str): + content = self._covert_string_to_bytes_io(content) + headers.contentLength = content.seek(0, 2) + content.seek(0) + elif self.cipher_generator.need_sha256: + # 如果不是字符串，不允许计算 sha256 + raise Exception("Could not calculate sha256 for a stream object") + content = self.cipher_generator.new(content) + if metadata is None: + metadata = dict() + metadata = content.gen_need_metadata_and_headers(metadata, headers) + put_result = super(CryptoObsClient, self).putContent(bucketName, objectKey, content, metadata, headers=headers, + progressCallback=progressCallback, autoClose=autoClose, + extensionHeaders=extensionHeaders) + return put_result + + def _gen_readable_object_from_file(self, file_path): + return self.cipher_generator.new(open(file_path, "rb")) + + def putObject(self, bucketName, objectKey, content, metadata=None, headers=None, progressCallback=None, + autoClose=True, extensionHeaders=None): + raise Exception("putObject is not supported in CryptoObsClient") + + def uploadEncryptedPart(self, bucketName, objectKey, partNumber, uploadId, crypto_cipher, object=None, + isFile=False, partSize=None, offset=0, sseHeader=None, isAttachMd5=False, md5=None, + content=None, progressCallback=None, autoClose=True, extensionHeaders=None): + if isAttachMd5: + raise Exception("Could not calculate md5 for uploadEncryptedPart") + if content is None: + content = object + if isFile: + checked_file_part_info = self._check_file_part_info(content, offset, partSize) + content = crypto_cipher + content._file = open(checked_file_part_info["file_path"], "rb") + content._file.seek(checked_file_part_info["offset"]) + partSize = checked_file_part_info["partSize"] + else: + if content is not None and hasattr(content, 'read') and callable(content.read): + crypto_cipher._file = content + else: + crypto_cipher._file = self._covert_string_to_bytes_io(content) + content = crypto_cipher + return super(CryptoObsClient, self).uploadPart(bucketName, objectKey, partNumber, uploadId, + object=None, isFile=False, partSize=partSize, + offset=0, sseHeader=sseHeader, + isAttachMd5=isAttachMd5, md5=md5, content=content, + progressCallback=progressCallback, + autoClose=autoClose, extensionHeaders=extensionHeaders) + + def uploadFile(self, bucketName, objectKey, uploadFile, partSize=9 * 1024 * 1024, + taskNum=1, enableCheckpoint=False, checkpointFile=None, + checkSum=False, metadata=None, progressCallback=None, headers=None, + extensionHeaders=None, encoding_type=None): + self.log_client.log(INFO, 'enter resume upload file...') + self._assert_not_null(bucketName, 'bucketName is empty') + self._assert_not_null(objectKey, 'objectKey is empty') + self._assert_not_null(uploadFile, 'uploadFile is empty') + upload_operation = EncryptedUploadOperation(self.cipher_generator, util.to_string(bucketName), + util.to_string(objectKey), + util.to_string(uploadFile), partSize, taskNum, enableCheckpoint, + util.to_string(checkpointFile), checkSum, metadata, + progressCallback, self, headers, extensionHeaders=extensionHeaders, + encoding_type=encoding_type) + return upload_operation._upload() + + def downloadFile(self, bucketName, objectKey, downloadFile=None, partSize=5 * 1024 * 1024, taskNum=1, + enableCheckpoint=False, checkpointFile=None, header=None, versionId=None, + progressCallback=None, imageProcess=None, extensionHeaders=None): + if header is None: + header = GetObjectHeader() + if downloadFile is None: + downloadFile = objectKey + + down_operation = DecryptedDownloadOperation(self.cipher_generator, util.to_string(bucketName), + util.to_string(objectKey), util.to_string(downloadFile), + partSize, taskNum, enableCheckpoint, util.to_string(checkpointFile), + header, versionId, progressCallback, self, imageProcess, + progress.NONE_NOTIFIER, extensionHeaders=extensionHeaders) + return _resume_download_with_operation(down_operation) + + def _parse_content(self, objectKey, conn, readable, result_wrapper=None, download_start=None, + downloadPath=None, chuckSize=const.READ_ONCE_LENGTH, loadStreamInMemory=False, + progressCallback=None, notifier=None): + if readable.status >= 300: + return super(CryptoObsClient, self)._parse_content(objectKey, conn, readable, + download_start=download_start, downloadPath=downloadPath, + chuckSize=chuckSize, + loadStreamInMemory=loadStreamInMemory, + progressCallback=progressCallback, notifier=notifier) + crypto_info = self.cipher_generator.get_crypto_info_from_headers(dict(readable.getheaders())) + try: + iv_offset = int(download_start.split("-")[0]) + except (AttributeError, ValueError): + iv_offset = 0 + decryptedObject = self.cipher_generator.new(readable, is_decrypt=True, crypto_info=crypto_info) + decryptedObject.seek_iv(iv_offset) + return super(CryptoObsClient, self)._parse_content(objectKey, conn, decryptedObject, + download_start=download_start, downloadPath=downloadPath, + chuckSize=chuckSize, loadStreamInMemory=loadStreamInMemory, + progressCallback=progressCallback, notifier=notifier) + + def _encrypted_upload_part(self, bucketName, objectKey, partNumber, uploadId, crypto_info, + content=None, partSize=None, offset=0, sseHeader=None, isAttachMd5=False, + md5=None, notifier=None, extensionHeaders=None): + checked_file_part_info = self._check_file_part_info(content, offset, partSize) + content = self.cipher_generator.new(open(checked_file_part_info["file_path"], "rb"), crypto_info=crypto_info) + content.seek(checked_file_part_info["offset"]) + headers = dict() + if self.cipher_generator.need_sha256: + headers[self.ha.content_sha256_header()] = content.calculate_sha256(partSize)[1] + return super(CryptoObsClient, self)._uploadPartWithNotifier(bucketName, objectKey, partNumber, uploadId, + content, False, checked_file_part_info["partSize"], + checked_file_part_info["offset"], sseHeader, + isAttachMd5, md5, notifier, extensionHeaders, + headers) + + def gen_readable_object_from_file(self, file_path): + content = self.cipher_generator.new(open(file_path, "rb")) + return content + + @staticmethod + def add_metadata_from_content(metadata, headers, content): + return content.gen_need_metadata_and_headers(metadata, headers) + + @staticmethod + def _covert_string_to_bytes_io(str_object): + if const.IS_PYTHON2 and isinstance(str_object, unicode) \ + or (not const.IS_PYTHON2 and isinstance(str_object, str)): + return io.BytesIO(str_object.encode("UTF-8")) + return io.BytesIO(str_object) + + +class EncryptedUploadOperation(uploadOperation): + def __init__(self, cipher_generator, bucketName, objectKey, uploadFile, partSize, taskNum, enableCheckPoint, + checkPointFile, checkSum, metadata, progressCallback, obsClient, headers, extensionHeaders=None, + encoding_type=None): + self.cipher_generator = cipher_generator + self.encrypted_content = cipher_generator.new(open(uploadFile, "rb")) + self.crypto_info = self.encrypted_content.safe_crypto_info() + if metadata is None: + metadata = dict() + if headers is None: + headers = UploadFileHeader() + metadata = self.encrypted_content.gen_need_metadata_and_headers(metadata, headers) + super(EncryptedUploadOperation, self).__init__(bucketName, objectKey, uploadFile, partSize, taskNum, + enableCheckPoint, checkPointFile, checkSum, metadata, + progressCallback, obsClient, headers, extensionHeaders, + encoding_type) + self._record = self.encrypted_content.gen_need_record(self._record) + + def _check_upload_record(self, record): + self._record = self._get_record() + if not self.cipher_generator.check_upload_record(self._record, self.encrypted_content.safe_crypto_info()): + self.obsClient.log_client.log(INFO, 'The crypto_info was changed. clear the record') + return False + return super(EncryptedUploadOperation, self)._check_upload_record(record) + + def _load(self): + super(EncryptedUploadOperation, self)._load() + # 如果 record 通过校验，使用 record 里的信息初始化新 cipher, 否则使用当前的 cipher 补全 record 信息 + if "crypto_mod" in self._record: + self.encrypted_content = self.cipher_generator.new("", crypto_info=self._record) + else: + self._record = self.encrypted_content.gen_need_record(self._record) + + def _upload(self): + try: + return super(EncryptedUploadOperation, self)._upload() + finally: + self.encrypted_content.close() + + def get_upload_part_resp(self, part): + return self.obsClient._encrypted_upload_part(self.bucketName, self.objectKey, part['partNumber'], + self._record['uploadId'], + self.encrypted_content.crypto_info(), + self.fileName, partSize=part['length'], + offset=part['offset'], notifier=self.notifier, + extensionHeaders=self.extensionHeaders, + sseHeader=self.headers.sseHeader) + + def get_init_upload_result(self): + return super(self.obsClient.__class__, self.obsClient).initiateMultipartUpload( + self.bucketName, self.objectKey, metadata=self.metadata, acl=self.headers.acl, + storageClass=self.headers.storageClass, websiteRedirectLocation=self.headers.websiteRedirectLocation, + contentType=self.headers.contentType, sseHeader=self.headers.sseHeader, + expires=self.headers.expires, extensionGrants=self.headers.extensionGrants, + extensionHeaders=self.extensionHeaders, encoding_type=self.encoding_type) + + +class DecryptedDownloadOperation(downloadOperation): + def __init__(self, cipher_generator, *args, **kwargs): + super(DecryptedDownloadOperation, self).__init__(*args, **kwargs) + self.cipher_generator = cipher_generator + header_dict = dict(self._metadata_resp.header) + crypto_info = self.cipher_generator.get_crypto_info_from_headers(header_dict) + # 用空字符串生成个临时加密下载对象，用以获取加密信息 + self.decrypted_content = cipher_generator.new("", crypto_info=crypto_info) + + def _check_download_record(self, record): + self._record = self._get_record() + if not self.cipher_generator.check_download_record(self._record, self.decrypted_content.safe_crypto_info()): + self.obsClient.log_client.log(INFO, 'The crypto_info was changed. clear the record') + return False + return super(DecryptedDownloadOperation, self)._check_download_record(record) + + def _load(self): + super(DecryptedDownloadOperation, self)._load() + self._record = self.decrypted_content.gen_need_record(self._record) + + def _download(self): + try: + return super(DecryptedDownloadOperation, self)._download() + finally: + self.decrypted_content.close() diff --git a/src/obs/extension.py b/src/obs/extension.py index 04a8f4d..bf3189c 100644 --- a/src/obs/extension.py +++ b/src/obs/extension.py @@ -22,10 +22,12 @@ def _download_files(obsClient, bucketName, prefix, downloadFolder=None, taskNum=const.DEFAULT_TASK_NUM, taskQueueSize=const.DEFAULT_TASK_QUEUE_SIZE, - headers=GetObjectHeader(), imageProcess=None, interval=const.DEFAULT_BYTE_INTTERVAL, + headers=None, imageProcess=None, interval=const.DEFAULT_BYTE_INTTERVAL, taskCallback=None, progressCallback=None, threshold=const.DEFAULT_MAXIMUM_SIZE, partSize=5 * 1024 * 1024, subTaskNum=1, enableCheckpoint=False, checkpointFile=None, extensionHeaders=None): + if headers is None: + headers = GetObjectHeader() try: executor = None notifier = None @@ -75,7 +77,7 @@ def _download_files(obsClient, bucketName, prefix, downloadFolder=None, taskNum= if objectKey.endswith('/'): state._successful_increment() elif content.size < threshold: - executor.execute(_task_wrap, obsClient, obsClient._getObjectWithNotifier, key=objectKey, + executor.execute(_task_wrap, obsClient, obsClient.getObject, key=objectKey, taskCallback=taskCallback, state=state, bucketName=bucketName, objectKey=objectKey, getObjectRequest=query, headers=headers, downloadPath=downloadPath, notifier=notifier, extensionHeaders=extensionHeaders) diff --git a/src/obs/ilog.py b/src/obs/ilog.py index d0a65a7..fdd37ad 100644 --- a/src/obs/ilog.py +++ b/src/obs/ilog.py @@ -128,12 +128,12 @@ def close(self): def log(self, level, msg, *args, **kwargs): base_back = sys._getframe().f_back - funcname = base_back.f_code.co_name - while funcname.lower() == 'log': + func_name = base_back.f_code.co_name + while func_name.lower() == 'log': base_back = base_back.f_back - funcname = base_back.f_code.co_name + func_name = base_back.f_code.co_name line = base_back.f_lineno - msg = '%(logger)s|%(name)s,%(lineno)d|' % {'logger': self.display_name, 'name': funcname, + msg = '%(logger)s|%(name)s,%(lineno)d|' % {'logger': self.display_name, 'name': func_name, 'lineno': int(line)} + str(msg) if level == CRITICAL: diff --git a/src/obs/loadtoken.py b/src/obs/loadtoken.py index 08670c5..7ef83cb 100644 --- a/src/obs/loadtoken.py +++ b/src/obs/loadtoken.py @@ -152,8 +152,8 @@ def search(): @staticmethod def _search_handle_expires(datetime, timedelta): if ECS.expires is not None: - token_datenow = datetime.utcnow() - if token_datenow < (ECS.expires - timedelta(minutes=10)): + token_date_now = datetime.utcnow() + if token_date_now < (ECS.expires - timedelta(minutes=10)): return { 'accessKey': ECS.ak, 'secretKey': ECS.sk, @@ -177,10 +177,7 @@ def _search_handle_response_body(responseBody): @staticmethod def _search_judge(datetime): - if ECS.expires is not None and datetime.utcnow() < ECS.expires: - return True - else: - return False + return ECS.expires is not None and datetime.utcnow() < ECS.expires @staticmethod def _search_get_result(conn): diff --git a/src/obs/model.py b/src/obs/model.py index 59755d3..687e23f 100644 --- a/src/obs/model.py +++ b/src/obs/model.py @@ -345,7 +345,7 @@ def __init__(self, year, month, day, hour=0, min=0, sec=0): def ToUTTime(self): strTime = '%04d-%02d-%02dT%02d:%02d:%02d.000Z' % ( - self.year, self.month, self.day, self.hour, self.min, self.sec) + self.year, self.month, self.day, self.hour, self.min, self.sec) return strTime def ToGMTTime(self): @@ -620,10 +620,10 @@ class ObjectVersionHead(BaseModel): allowedAttr = {'name': BASESTRING, 'location': BASESTRING, 'prefix': BASESTRING, 'delimiter': BASESTRING, 'keyMarker': BASESTRING, 'versionIdMarker': BASESTRING, 'nextKeyMarker': BASESTRING, 'nextVersionIdMarker': BASESTRING, - 'maxKeys': int, 'isTruncated': bool} + 'maxKeys': int, 'isTruncated': bool, "encoding_type": BASESTRING} - def __init__(self, name=None, location=None, prefix=None, delimiter=None, keyMarker=None, - versionIdMarker=None, nextKeyMarker=None, nextVersionIdMarker=None, maxKeys=None, isTruncated=None): + def __init__(self, name=None, location=None, prefix=None, delimiter=None, keyMarker=None, versionIdMarker=None, + nextKeyMarker=None, nextVersionIdMarker=None, maxKeys=None, isTruncated=None, encoding_type=None): self.name = name self.location = location self.prefix = prefix @@ -634,6 +634,7 @@ def __init__(self, name=None, location=None, prefix=None, delimiter=None, keyMar self.nextVersionIdMarker = nextVersionIdMarker self.maxKeys = maxKeys self.isTruncated = isTruncated + self.encoding_type = encoding_type class ObjectVersion(BaseModel): @@ -673,11 +674,12 @@ class PutObjectHeader(BaseModel): allowedAttr = {'md5': BASESTRING, 'acl': BASESTRING, 'location': BASESTRING, 'contentType': BASESTRING, 'sseHeader': SseHeader, 'contentLength': [int, LONG, BASESTRING], 'storageClass': BASESTRING, 'successActionRedirect': BASESTRING, 'expires': int, - 'extensionGrants': list} + 'extensionGrants': list, "sha256": BASESTRING} def __init__(self, md5=None, acl=None, location=None, contentType=None, sseHeader=None, contentLength=None, - storageClass=None, successActionRedirect=None, expires=None, extensionGrants=None): + storageClass=None, successActionRedirect=None, expires=None, extensionGrants=None, sha256=None): self.md5 = md5 + self.sha256 = sha256 self.acl = acl self.location = location self.contentType = contentType @@ -694,9 +696,8 @@ def __init__(self, md5=None, acl=None, location=None, contentType=None, sseHeade class UploadFileHeader(BaseModel): allowedAttr = {'acl': BASESTRING, 'websiteRedirectLocation': BASESTRING, 'contentType': BASESTRING, - 'sseHeader': SseHeader, - 'storageClass': BASESTRING, 'successActionRedirect': BASESTRING, 'expires': int, - 'extensionGrants': list} + 'sseHeader': SseHeader, 'storageClass': BASESTRING, + 'successActionRedirect': BASESTRING, 'expires': int, 'extensionGrants': list} def __init__(self, acl=None, websiteRedirectLocation=None, contentType=None, sseHeader=None, storageClass=None, successActionRedirect=None, expires=None, extensionGrants=None): @@ -829,14 +830,16 @@ def __init__(self, key=None, uploadId=None, initiator=None, owner=None, storageC class Versions(BaseModel): allowedAttr = {'prefix': BASESTRING, 'key_marker': BASESTRING, 'max_keys': [int, BASESTRING], - 'delimiter': BASESTRING, 'version_id_marker': BASESTRING} + 'delimiter': BASESTRING, 'version_id_marker': BASESTRING, 'encoding_type': BASESTRING} - def __init__(self, prefix=None, key_marker=None, max_keys=None, delimiter=None, version_id_marker=None): + def __init__(self, prefix=None, key_marker=None, max_keys=None, delimiter=None, version_id_marker=None, + encoding_type=None): self.prefix = prefix self.key_marker = key_marker self.max_keys = max_keys self.delimiter = delimiter self.version_id_marker = version_id_marker + self.encoding_type = encoding_type class Object(BaseModel): @@ -901,13 +904,13 @@ def add_part(self, part): class CompleteMultipartUploadResponse(BaseModel): - allowedAttr = {'location': BASESTRING, 'bucket': BASESTRING, + allowedAttr = {'location': BASESTRING, 'bucket': BASESTRING, "encoding_type": BASESTRING, 'key': BASESTRING, 'etag': BASESTRING, 'versionId': BASESTRING, 'sseKms': BASESTRING, 'sseKmsKey': BASESTRING, 'sseC': BASESTRING, 'sseCKeyMd5': BASESTRING, 'objectUrl': BASESTRING} def __init__(self, location=None, bucket=None, key=None, etag=None, versionId=None, sseKms=None, sseKmsKey=None, sseC=None, - sseCKeyMd5=None, objectUrl=None): + sseCKeyMd5=None, objectUrl=None, encoding_type=None): self.location = location self.bucket = bucket self.key = key @@ -918,6 +921,7 @@ def __init__(self, location=None, bucket=None, key=None, etag=None, self.sseC = sseC self.sseCKeyMd5 = sseCKeyMd5 self.objectUrl = objectUrl + self.encoding_type = encoding_type class CopyObjectResponse(BaseModel): @@ -961,11 +965,12 @@ def __init__(self, deleteMarker=None, versionId=None): class DeleteObjectsRequest(BaseModel): - allowedAttr = {'quiet': bool, 'objects': list} + allowedAttr = {'quiet': bool, 'objects': list, "encoding_type": BASESTRING} - def __init__(self, quiet=None, objects=None): + def __init__(self, quiet=None, objects=None, encoding_type=None): self.quiet = quiet self.objects = objects + self.encoding_type = encoding_type def add_object(self, object): if self.objects is None: @@ -975,11 +980,12 @@ def add_object(self, object): class DeleteObjectsResponse(BaseModel): - allowedAttr = {'deleted': list, 'error': list} + allowedAttr = {'deleted': list, 'error': list, "encoding_type": BASESTRING} - def __init__(self, deleted=None, error=None): + def __init__(self, deleted=None, error=None, encoding_type=None): self.deleted = deleted self.error = error + self.encoding_type = encoding_type class ErrorResult(BaseModel): @@ -1018,12 +1024,12 @@ def __init__(self, delimiter=None, prefix=None, max_uploads=None, key_marker=Non class ListPartsResponse(BaseModel): allowedAttr = {'bucketName': BASESTRING, 'objectKey': BASESTRING, 'uploadId': BASESTRING, 'initiator': Initiator, 'owner': Owner, 'storageClass': BASESTRING, 'partNumberMarker': int, 'nextPartNumberMarker': int, - 'maxParts': int, + 'maxParts': int, "encoding_type": BASESTRING, 'isTruncated': bool, 'parts': list} def __init__(self, bucketName=None, objectKey=None, uploadId=None, initiator=None, owner=None, storageClass=None, partNumberMarker=None, nextPartNumberMarker=None, maxParts=None, isTruncated=None, - parts=None): + parts=None, encoding_type=None): self.bucketName = bucketName self.objectKey = objectKey self.uploadId = uploadId @@ -1035,6 +1041,7 @@ def __init__(self, bucketName=None, objectKey=None, uploadId=None, initiator=Non self.maxParts = maxParts self.isTruncated = isTruncated self.parts = parts + self.encoding_type = encoding_type class GetBucketMetadataResponse(BaseModel): @@ -1151,13 +1158,14 @@ def __init__(self, content_type=None, content_language=None, expires=None, cache class InitiateMultipartUploadResponse(BaseModel): - allowedAttr = {'bucketName': BASESTRING, 'objectKey': BASESTRING, 'uploadId': BASESTRING, - 'sseKms': BASESTRING, 'sseKmsKey': BASESTRING, 'sseC': BASESTRING, 'sseCKeyMd5': BASESTRING} + allowedAttr = {'bucketName': BASESTRING, 'objectKey': BASESTRING, 'uploadId': BASESTRING, 'sseKms': BASESTRING, + 'sseKmsKey': BASESTRING, 'sseC': BASESTRING, 'sseCKeyMd5': BASESTRING, "encoding_type": BASESTRING} - def __init__(self, bucketName=None, objectKey=None, uploadId=None): + def __init__(self, bucketName=None, objectKey=None, uploadId=None, encoding_type=None): self.bucketName = bucketName self.objectKey = objectKey self.uploadId = uploadId + self.encoding_type = encoding_type class LifecycleResponse(BaseModel): @@ -1179,10 +1187,11 @@ class ListMultipartUploadsResponse(BaseModel): allowedAttr = {'bucket': BASESTRING, 'keyMarker': BASESTRING, 'uploadIdMarker': BASESTRING, 'nextKeyMarker': BASESTRING, 'nextUploadIdMarker': BASESTRING, 'maxUploads': int, 'isTruncated': bool, 'prefix': BASESTRING, 'delimiter': BASESTRING, 'upload': list, - 'commonPrefixs': list} + 'commonPrefixs': list, "encoding_type": BASESTRING} def __init__(self, bucket=None, keyMarker=None, uploadIdMarker=None, nextKeyMarker=None, nextUploadIdMarker=None, - maxUploads=None, isTruncated=None, prefix=None, delimiter=None, upload=None, commonPrefixs=None): + maxUploads=None, isTruncated=None, prefix=None, delimiter=None, upload=None, commonPrefixs=None, + encoding_type=None): self.bucket = bucket self.keyMarker = keyMarker self.uploadIdMarker = uploadIdMarker @@ -1195,16 +1204,16 @@ def __init__(self, bucket=None, keyMarker=None, uploadIdMarker=None, nextKeyMark self.delimiter = delimiter self.upload = upload self.commonPrefixs = commonPrefixs + self.encoding_type = encoding_type class ListObjectsResponse(BaseModel): allowedAttr = {'name': BASESTRING, 'location': BASESTRING, 'prefix': BASESTRING, 'marker': BASESTRING, - 'delimiter': BASESTRING, - 'max_keys': int, 'is_truncated': bool, 'next_marker': BASESTRING, 'contents': list, - 'commonPrefixs': list} + 'delimiter': BASESTRING, 'commonPrefixs': list, 'encoding_type': BASESTRING, + 'max_keys': int, 'is_truncated': bool, 'next_marker': BASESTRING, 'contents': list} - def __init__(self, name=None, location=None, prefix=None, marker=None, delimiter=None, - max_keys=None, is_truncated=None, next_marker=None, contents=None, commonPrefixs=None): + def __init__(self, name=None, location=None, prefix=None, marker=None, delimiter=None, max_keys=None, + is_truncated=None, next_marker=None, contents=None, commonPrefixs=None, encoding_type=None): self.name = name self.location = location self.prefix = prefix @@ -1215,6 +1224,7 @@ def __init__(self, name=None, location=None, prefix=None, marker=None, delimiter self.next_marker = next_marker self.contents = contents self.commonPrefixs = commonPrefixs + self.encoding_type = encoding_type class LocationResponse(BaseModel): @@ -1303,7 +1313,7 @@ def __init__(self, conn, result, connHolder, contentLength=None, notifier=None): self.result = result self.connHolder = connHolder self.contentLength = contentLength - self.readedCount = 0 + self.read_count = 0 self.notifier = notifier if self.notifier is None: self.notifier = progress.NONE_NOTIFIER @@ -1313,15 +1323,15 @@ def __getattr__(self, name): def _read(*args, **kwargs): chunk = self.result.read(*args, **kwargs) if not chunk: - if self.contentLength is not None and self.contentLength != self.readedCount: + if self.contentLength is not None and self.contentLength != self.read_count: raise Exception( 'premature end of Content-Length delimiter message body (expected:' + util.to_string( - self.contentLength) + '; received:' + util.to_string(self.readedCount) + ')') + self.contentLength) + '; received:' + util.to_string(self.read_count) + ')') else: newReadCount = len(chunk) if newReadCount > 0: self.notifier.send(newReadCount) - self.readedCount += newReadCount + self.read_count += newReadCount return chunk return _read diff --git a/src/obs/obs_cipher_suite.py b/src/obs/obs_cipher_suite.py new file mode 100644 index 0000000..a680449 --- /dev/null +++ b/src/obs/obs_cipher_suite.py @@ -0,0 +1,344 @@ +# coding:utf-8 +import binascii +import hashlib +import os + +from Crypto.Cipher import AES, PKCS1_v1_5 +from Crypto.PublicKey import RSA +from Crypto.Util import Counter +from Crypto.Util.number import bytes_to_long + +from obs import const, util + + +class CipherGenerator(object): + def __init__(self, need_sha256=False): + self.need_sha256 = need_sha256 + self.crypto_mod = "CipherGenerator" + self.master_key_sha256 = "" + + def new(self, readable): + pass + + @staticmethod + def gen_random_key(key_length): + return os.urandom(key_length) + + def get_crypto_info_from_headers(self, header_dict): + key_list = [i for i in header_dict.keys()] + for key in key_list: + if key.startswith("x-obs-meta"): + header_dict[key.replace("x-obs-meta-", "")] = header_dict.pop(key) + key_list = [i for i in header_dict.keys()] + for key in key_list: + if key.startswith("x-amz-meta"): + header_dict[key.replace("x-amz-meta-", "")] = header_dict.pop(key) + if "encrypted-algorithm" not in header_dict: + raise Exception("Crypto mod is not in object's metadata") + header_dict["crypto_mod"] = header_dict.pop("encrypted-algorithm") + if header_dict["crypto_mod"] != self.crypto_mod: + raise Exception("Object's crypto mod is not equals cipher-generator's, " + "please change a different cipher-generator") + return header_dict + + def check_record(self, record, crypto_info): + return record["crypto_mod"] == crypto_info["crypto_mod"] \ + and record["master_key_sha256"] == self.master_key_sha256 + + +class OBSCipher(object): + def __init__(self, readable, is_decrypt=False, need_sha256=False): + self._file = readable + self.sha256 = hashlib.sha256() + self.encrypted_sha256 = hashlib.sha256() + self.crypto_mod = "EncryptedObject" + self.is_decrypt = is_decrypt + self.read_count = 0 + self.need_sha256 = need_sha256 + if self.is_decrypt: + self.read = self.decrypt + self.original_response = self._file + self.status = self.original_response.status + self.reason = self.original_response.reason + else: + self.read = self.encrypt + self.original_response = None + self.status = None + self.reason = None + + def decrypt(self, n=const.READ_ONCE_LENGTH): + pass + + def encrypt(self, n=const.READ_ONCE_LENGTH): + pass + + def gen_need_metadata_and_headers(self, metadata, headers=None): + if self.need_sha256: + metadata["plaintext-sha256"], metadata["encrypted-sha256"], metadata[ + "plaintext-content-length"] = self.calculate_sha256() + if headers is not None: + headers.sha256 = metadata["encrypted-sha256"] + metadata["encrypted-algorithm"] = self.crypto_mod + return metadata + + def calculate_sha256(self, read_length=None): + return self.sha256.hexdigest(), self.encrypted_sha256.hexdigest(), self.read_count + + def get_content_length(self): + current_pointer = self._file.tell() + self._file.seek(0, 2) + total_length = self._file.tell() + self._file.seek(current_pointer) + return total_length + + def getheader(self, key, default_value=None): + if not self.is_decrypt: + return None + return self.original_response.getheader(key, default_value) + + def getheaders(self): + if not self.is_decrypt: + return None + return self.original_response.getheaders() + + def gen_need_record(self, record): + record["crypto_mod"] = self.crypto_mod + return record + + def crypto_info(self): + return self.safe_crypto_info() + + def safe_crypto_info(self): + return {"crypto_mod": self.crypto_mod} + + def close(self): + if hasattr(self._file, 'close') and callable(self._file.close): + self._file.close() + + def __str__(self): + return "EncryptedObject" + + +class CTRCipherGenerator(CipherGenerator): + def __init__(self, crypto_key, master_key_info=None, crypto_iv=None, *args, **kwargs): + super(CTRCipherGenerator, self).__init__(*args, **kwargs) + self.crypto_key = util.covert_string_to_bytes(crypto_key) + self.crypto_iv = util.covert_string_to_bytes(crypto_iv) + self.crypto_mod = "AES256-Ctr/iv_base64/NoPadding" + self.master_key_sha256 = hashlib.sha256(self.crypto_key).hexdigest() + self.master_key_info = "" if master_key_info is None else master_key_info + + def new(self, readable, is_decrypt=False, crypto_info=None): + if crypto_info is not None: + iv = binascii.a2b_base64(crypto_info["crypto_iv"]) + return OBSCtrCipher(readable, self.crypto_key, self.master_key_info, self.master_key_sha256, + iv, is_decrypt, self.need_sha256) + if self.crypto_iv is None: + return OBSCtrCipher(readable, self.crypto_key, self.master_key_info, self.master_key_sha256, + self.gen_random_key(16), is_decrypt, self.need_sha256) + return OBSCtrCipher(readable, self.crypto_key, self.master_key_info, self.master_key_sha256, + self.crypto_iv, is_decrypt, self.need_sha256) + + def get_crypto_info_from_headers(self, header_dict): + header_dict = super(CTRCipherGenerator, self).get_crypto_info_from_headers(header_dict) + if "encrypted-start" not in header_dict: + raise Exception("Encryption info is not in metadata") + header_dict["crypto_iv"] = header_dict.pop("encrypted-start") + if self.crypto_iv is not None and header_dict["crypto_iv"] != self.crypto_iv: + raise Exception("Crypto_iv is different between local and server") + if "master-key-info" in header_dict: + header_dict["master_key_info"] = header_dict.pop("master-key-info") + return header_dict + + def check_download_record(self, record, crypto_info): + return super(CTRCipherGenerator, self).check_record(record, crypto_info) \ + and record["master_key_sha256"] == crypto_info["master_key_sha256"] \ + and record["crypto_iv"] == crypto_info["crypto_iv"] + + def check_upload_record(self, record, crypto_info): + is_iv_match = binascii.a2b_base64(record["crypto_iv"]) == self.crypto_iv if self.crypto_iv else True + return super(CTRCipherGenerator, self).check_record(record, crypto_info) \ + and is_iv_match \ + and record["master_key_info"] == crypto_info["master_key_info"] \ + and record["master_key_sha256"] == crypto_info["master_key_sha256"] + + +class OBSCtrCipher(OBSCipher): + def __init__(self, readable, crypto_key, master_key_info, master_key_sha256, + crypto_iv=None, is_decrypt=False, need_sha256=False): + super(OBSCtrCipher, self).__init__(readable, is_decrypt, need_sha256) + self.master_key_sha256 = master_key_sha256 + ctr = Counter.new(128, initial_value=bytes_to_long(crypto_iv)) + self.crypto_iv = crypto_iv + self.crypto_mod = "AES256-Ctr/iv_base64/NoPadding" + if (const.IS_PYTHON2 and isinstance(crypto_key, unicode)) \ + or (not const.IS_PYTHON2 and isinstance(crypto_key, str)): + crypto_key = crypto_key.encode("UTF-8") + self.crypto_key = crypto_key + self.master_key_info = master_key_info + self.aes = AES.new(crypto_key, mode=AES.MODE_CTR, counter=ctr) + + def encrypt(self, n=const.READ_ONCE_LENGTH): + chunk = self._file.read(n) + if not isinstance(chunk, bytes): + # todo 这个说明是否合适 + raise Exception("Only support bytes for encrypt, please open your stream with 'rb' mode") + encrypted_chunk = self.aes.encrypt(chunk) + return encrypted_chunk + + def decrypt(self, n=const.READ_ONCE_LENGTH): + return self.aes.decrypt(self.original_response.read(n)) + + def gen_need_metadata_and_headers(self, metadata, headers=None): + metadata["encrypted-start"] = binascii.b2a_base64(self.crypto_iv).strip().decode("UTF-8") + metadata["master-key-info"] = self.master_key_info + return super(OBSCtrCipher, self).gen_need_metadata_and_headers(metadata, headers) + + def calculate_sha256(self, total_read_length=None): + current_pointer = self._file.tell() + current_read_length = 0 + while True: + if total_read_length is not None and total_read_length - current_read_length > const.READ_ONCE_LENGTH: + read_size = total_read_length - current_read_length + else: + read_size = const.READ_ONCE_LENGTH + chunk = self._file.read(read_size) + if not chunk or (total_read_length is not None and current_read_length == total_read_length): + self.seek(current_pointer) + return self.sha256.hexdigest(), self.encrypted_sha256.hexdigest(), self.read_count + if not isinstance(chunk, bytes): + # todo 这个说明是否合适 + raise Exception("Only support bytes for encrypt, please open your stream with 'rb' mode") + self.sha256.update(chunk) + encrypted_chunk = self.aes.encrypt(chunk) + self.encrypted_sha256.update(encrypted_chunk) + self.read_count += len(chunk) + current_read_length += read_size + + def seek(self, offset, whence=0): + if whence == 1: + current_pointer = self._file.tell() + elif whence == 2: + self._file.seek(0, 2) + current_pointer = self._file.tell() + else: + current_pointer = 0 + self.seek_iv(offset + current_pointer) + self._file.seek(offset + current_pointer) + + def seek_iv(self, offset): + now_iv = bytes_to_long(self.crypto_iv) + int(offset / 16) + new_ctr = Counter.new(128, initial_value=now_iv) + self.aes = AES.new(self.crypto_key, mode=AES.MODE_CTR, counter=new_ctr) + if self.is_decrypt: + self.aes.decrypt(b"1" * (offset % 16)) + else: + self.aes.encrypt(b"1" * (offset % 16)) + + def gen_need_record(self, record): + record["crypto_iv"] = binascii.b2a_base64(self.crypto_iv).strip().decode("UTF-8") + record["master_key_info"] = self.master_key_info + record["master_key_sha256"] = self.master_key_sha256 + return super(OBSCtrCipher, self).gen_need_record(record) + + def safe_crypto_info(self): + crypto_info = super(OBSCtrCipher, self).safe_crypto_info() + crypto_info["crypto_iv"] = binascii.b2a_base64(self.crypto_iv).strip().decode("UTF-8") + crypto_info["master_key_info"] = self.master_key_info + crypto_info["master_key_sha256"] = self.master_key_sha256 + return crypto_info + + def __str__(self): + return "OBSCtrCipher Encrypted Object start at " + binascii.b2a_base64(self.crypto_iv).strip().decode("UTF-8") + + +class CtrRSACipherGenerator(CipherGenerator): + def __init__(self, master_crypto_key_path, master_key_info=None, *args, **kwargs): + super(CtrRSACipherGenerator, self).__init__(*args, **kwargs) + with open(master_crypto_key_path, "rb") as f: + key = f.read() + self.master_key_sha256 = hashlib.sha256(key).hexdigest() + self.master_crypto_key = RSA.importKey(key) + self.rsa = PKCS1_v1_5.new(self.master_crypto_key) + self.crypto_mod = "AES256-Ctr/RSA-Object-Key/NoPadding" + self.master_key_info = "" if master_key_info is None else master_key_info + + def new(self, readable, is_decrypt=False, crypto_info=None): + if crypto_info is not None: + iv = binascii.a2b_base64(crypto_info["crypto_iv"]) + if "object_encryption_key" in crypto_info: + object_encryption_key = binascii.a2b_base64(crypto_info["object_encryption_key"]) + else: + object_encryption_key = self.decrypt_object_encryption_key(crypto_info["encrypted_object_key"]) + if object_encryption_key == 0: + raise Exception("Wrong private key, could not decrypt object encryption key") + return OBSCtrRSACipher(readable, object_encryption_key, crypto_info["encrypted_object_key"], + self.master_key_info, self.master_key_sha256, iv, is_decrypt, self.need_sha256) + random_key = self.gen_random_key(32) + random_iv = self.gen_random_key(16) + return OBSCtrRSACipher(readable, random_key, self.encrypt_object_encryption_key(random_key), + self.master_key_info, + self.master_key_sha256, random_iv, is_decrypt, self.need_sha256) + + def encrypt_object_encryption_key(self, key_str): + return binascii.b2a_base64(self.rsa.encrypt(key_str)).strip().decode("UTF-8") + + def decrypt_object_encryption_key(self, key_str): + return self.rsa.decrypt(binascii.a2b_base64(key_str), 0) + + def get_crypto_info_from_headers(self, header_dict): + header_dict = super(CtrRSACipherGenerator, self).get_crypto_info_from_headers(header_dict) + if "encrypted-object-key" not in header_dict: + raise Exception("Encryption info is not in metadata") + header_dict["encrypted_object_key"] = header_dict.pop("encrypted-object-key") + header_dict["crypto_iv"] = header_dict.pop("encrypted-start") + if "master-key-info" in header_dict: + header_dict["master_key_info"] = header_dict.pop("master-key-info") + return header_dict + + def check_download_record(self, record, crypto_info): + return super(CtrRSACipherGenerator, self).check_record(record, crypto_info) \ + and record["master_key_sha256"] == crypto_info["master_key_sha256"] \ + and record["crypto_iv"] == crypto_info["crypto_iv"] \ + and record["encrypted_object_key"] == crypto_info["encrypted_object_key"] + + def check_upload_record(self, record, crypto_info): + return super(CtrRSACipherGenerator, self).check_record(record, crypto_info) \ + and record["master_key_info"] == crypto_info["master_key_info"] \ + and record["master_key_sha256"] == crypto_info["master_key_sha256"] + + +class OBSCtrRSACipher(OBSCtrCipher): + def __init__(self, readable, crypto_key, encrypted_object_key, master_key_info, master_key_sha256, + crypto_iv=None, is_decrypt=False, need_sha256=False): + super(OBSCtrRSACipher, self).__init__(readable, crypto_key, master_key_info, master_key_sha256, crypto_iv, + is_decrypt, need_sha256) + self.encrypted_object_key = encrypted_object_key + self.crypto_mod = "AES256-Ctr/RSA-Object-Key/NoPadding" + self.master_key_info = master_key_info + + def gen_need_metadata_and_headers(self, metadata, headers=None): + metadata["encrypted-object-key"] = self.encrypted_object_key + return super(OBSCtrRSACipher, self).gen_need_metadata_and_headers(metadata, headers) + + def gen_need_record(self, record): + record["encrypted_object_key"] = self.encrypted_object_key + record["master_key_sha256"] = self.master_key_sha256 + record["object_encryption_key"] = binascii.b2a_base64(self.crypto_key).strip().decode("UTF-8") + return super(OBSCtrRSACipher, self).gen_need_record(record) + + def crypto_info(self): + crypto_info = self.safe_crypto_info() + crypto_info["object_encryption_key"] = binascii.b2a_base64(self.crypto_key).strip().decode("UTF-8") + return crypto_info + + def safe_crypto_info(self): + crypto_info = super(OBSCtrRSACipher, self).safe_crypto_info() + crypto_info["encrypted_object_key"] = self.encrypted_object_key + crypto_info["master_key_info"] = self.master_key_info + crypto_info["master_key_sha256"] = self.master_key_sha256 + return crypto_info + + def __str__(self): + return "OBSCtrRSACipher Encrypted Object start at " \ + + binascii.b2a_base64(self.crypto_iv).strip().decode("UTF-8") diff --git a/src/obs/progress.py b/src/obs/progress.py index c70c572..0ca6ade 100644 --- a/src/obs/progress.py +++ b/src/obs/progress.py @@ -46,7 +46,7 @@ def _run(self): if self._newlyTransferredAmount >= self.interval and ( self._transferredAmount < self.totalAmount or self.totalAmount <= 0): self._newlyTransferredAmount = 0 - self.callback(*self._caculate()) + self.callback(*self._calculate()) def start(self): now = time.time() @@ -55,7 +55,7 @@ def start(self): t.daemon = True t.start() - def _caculate(self): + def _calculate(self): totalSeconds = time.time() - self._startCheckpoint return self._transferredAmount, self.totalAmount, totalSeconds if totalSeconds > 0 else 0.001 @@ -65,7 +65,7 @@ def send(self, data): def end(self): self._queue.put(None) - self.callback(*self._caculate()) + self.callback(*self._calculate()) self.callback = None diff --git a/src/obs/transfer.py b/src/obs/transfer.py index f451623..85c6cab 100644 --- a/src/obs/transfer.py +++ b/src/obs/transfer.py @@ -7,28 +7,23 @@ # http://www.apache.org/licenses/LICENSE-2.0 +import functools +import json # Unless required by applicable law or agreed to in writing, software distributed # under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR # CONDITIONS OF ANY KIND, either express or implied. See the License for the # specific language governing permissions and limitations under the License. - +import math +import operator import os -import json -import threading import sys +import threading import traceback -import functools -import operator -from obs import util -from obs import const -from obs import progress -from obs.model import BaseModel -from obs.model import CompletePart -from obs.model import CompleteMultipartUploadRequest -from obs.model import GetObjectRequest -from obs.model import GetObjectHeader -from obs.model import UploadFileHeader -from obs.ilog import INFO, ERROR, DEBUG + +from obs import const, progress, util +from obs.ilog import DEBUG, ERROR, INFO +from obs.model import BaseModel, CompleteMultipartUploadRequest, CompletePart, GetObjectHeader, GetObjectRequest, \ + UploadFileHeader if const.IS_PYTHON2: import Queue as queue @@ -36,23 +31,28 @@ import queue -def _resumer_upload(bucketName, objectKey, uploadFile, partSize, taskNum, enableCheckPoint, checkPointFile, checkSum, - metadata, progressCallback, obsClient, headers, extensionHeaders=None): +def _resume_upload(bucketName, objectKey, uploadFile, partSize, taskNum, enableCheckPoint, checkPointFile, checkSum, + metadata, progressCallback, obsClient, headers, extensionHeaders=None, encoding_type=None): upload_operation = uploadOperation(util.to_string(bucketName), util.to_string(objectKey), util.to_string(uploadFile), partSize, taskNum, enableCheckPoint, util.to_string(checkPointFile), checkSum, metadata, progressCallback, obsClient, - headers, extensionHeaders=extensionHeaders) + headers, extensionHeaders=extensionHeaders, encoding_type=encoding_type) return upload_operation._upload() -def _resumer_download(bucketName, objectKey, downloadFile, partSize, taskNum, enableCheckPoint, checkPointFile, - header, versionId, progressCallback, obsClient, imageProcess=None, - notifier=progress.NONE_NOTIFIER, extensionHeaders=None): +def _resume_download(bucketName, objectKey, downloadFile, partSize, taskNum, enableCheckPoint, checkPointFile, + header, versionId, progressCallback, obsClient, imageProcess=None, + notifier=progress.NONE_NOTIFIER, extensionHeaders=None): down_operation = downloadOperation(util.to_string(bucketName), util.to_string(objectKey), util.to_string(downloadFile), partSize, taskNum, enableCheckPoint, util.to_string(checkPointFile), header, versionId, progressCallback, obsClient, imageProcess, notifier, extensionHeaders=extensionHeaders) + + return _resume_download_with_operation(down_operation) + + +def _resume_download_with_operation(down_operation): if down_operation.size == 0: down_operation._delete_record() down_operation._delete_tmp_file() @@ -60,7 +60,7 @@ def _resumer_download(bucketName, objectKey, downloadFile, partSize, taskNum, en pass if down_operation.progressCallback is not None and callable(down_operation.progressCallback): down_operation.progressCallback(0, 0, 0) - return down_operation._metedata_resp + return down_operation._metadata_resp return down_operation._download() @@ -119,13 +119,16 @@ def _write_record(self, record): class uploadOperation(Operation): def __init__(self, bucketName, objectKey, uploadFile, partSize, taskNum, enableCheckPoint, checkPointFile, - checkSum, metadata, progressCallback, obsClient, headers, extensionHeaders=None): + checkSum, metadata, progressCallback, obsClient, headers, extensionHeaders=None, encoding_type=None): + if enableCheckPoint and not checkPointFile: + checkPointFile = uploadFile + '.upload_record' super(uploadOperation, self).__init__(bucketName, objectKey, uploadFile, partSize, taskNum, enableCheckPoint, checkPointFile, progressCallback, obsClient) self.checkSum = checkSum self.metadata = metadata self.headers = headers self.extensionHeaders = extensionHeaders + self.encoding_type = encoding_type try: self.size = os.path.getsize(self.fileName) @@ -135,10 +138,32 @@ def __init__(self, bucketName, objectKey, uploadFile, partSize, taskNum, enableC self.obsClient.log_client.log(ERROR, 'something is happened when obtain uploadFile information. Please check') raise e + self.total_parts = 0 + self.reset_partSize() + self._record = {'bucketName': self.bucketName, 'objectKey': self.objectKey, 'uploadFile': self.fileName, + 'partEtags': []} resp = self.obsClient.headBucket(self.bucketName, extensionHeaders=extensionHeaders) if resp.status > 300: raise Exception('head bucket {0} failed. Please check. Status:{1}.'.format(self.bucketName, resp.status)) + def reset_partSize(self): + if self.partSize < const.DEFAULT_MINIMUM_SIZE: + self.partSize = const.DEFAULT_MINIMUM_SIZE + elif self.partSize > const.DEFAULT_MAXIMUM_SIZE: + self.partSize = const.DEFAULT_MAXIMUM_SIZE + else: + self.partSize = util.to_int(self.partSize) + if self.taskNum <= 0: + self.taskNum = 1 + else: + self.taskNum = int(math.ceil(self.taskNum)) + self.total_parts = int(self.size / self.partSize) + if self.total_parts >= 10000: + self.partSize = int(self.size / 10000) if self.size % 10000 == 0 else int(self.size / 10000) + 1 + self.total_parts = int(self.size / self.partSize) + if self.size % self.partSize != 0: + self.total_parts += 1 + def _upload(self): if self.headers is None: self.headers = UploadFileHeader() @@ -146,7 +171,7 @@ def _upload(self): if self.enableCheckPoint: self._load() - if self._record is None: + if "uploadId" not in self._record: self._prepare() unfinished_upload_parts = [] @@ -176,7 +201,7 @@ def _upload(self): extensionHeaders=self.extensionHeaders) self.obsClient.log_client.log( ERROR, - 'the code from server is 4**, please check space、persimission and so on.' + 'the code from server is 4**, please check space, permission and so on.' ) self._delete_record() if self._exception is not None: @@ -193,10 +218,11 @@ def _upload(self): part_Etags = [] for part in self._record['partEtags']: part_Etags.append(CompletePart(partNum=part['partNum'], etag=part['etag'])) - self.obsClient.log_client.log(INFO, 'Completing to upload multiparts') + self.obsClient.log_client.log(INFO, 'Completing to upload multi_parts') resp = self.obsClient.completeMultipartUpload(self.bucketName, self.objectKey, self._record['uploadId'], CompleteMultipartUploadRequest(part_Etags), - extensionHeaders=self.extensionHeaders) + extensionHeaders=self.extensionHeaders, + encoding_type=self.encoding_type) self._upload_handle_response(resp) return resp finally: @@ -233,6 +259,9 @@ def _load(self): self.obsClient.log_client.log(ERROR, 'checkpointFile is invalid') self._delete_record() self._record = None + if self._record is None: + self._record = {'bucketName': self.bucketName, 'objectKey': self.objectKey, 'uploadFile': self.fileName, + 'partEtags': []} def _type_check(self, record): try: @@ -284,53 +313,53 @@ def _check_upload_record(self, record): def _slice_file(self): uploadParts = [] - num_counts = int(self.size / self.partSize) - if num_counts >= 10000: - self.partSize = int(self.size / 10000) if self.size % 10000 == 0 else int(self.size / 10000) + 1 - num_counts = int(self.size / self.partSize) - if self.size % self.partSize != 0: - num_counts += 1 - if num_counts == 0: + if self.total_parts == 0: part = Part(util.to_long(1), util.to_long(0), util.to_long(0), False) uploadParts.append(part) else: offset = 0 - for i in range(1, num_counts + 1, 1): + for i in range(1, self.total_parts + 1, 1): part = Part(util.to_long(i), util.to_long(offset), util.to_long(self.partSize), False) offset += self.partSize uploadParts.append(part) if self.size % self.partSize != 0: - uploadParts[num_counts - 1].length = util.to_long(self.size % self.partSize) + uploadParts[self.total_parts - 1].length = util.to_long(self.size % self.partSize) return uploadParts - def _prepare(self): - fileStatus = [self.size, self.lastModified] - if self.checkSum: - fileStatus.append(util.base64_encode(util.md5_file_encode_by_size_offset(self.fileName, self.size, 0))) - - resp = self.obsClient.initiateMultipartUpload(self.bucketName, self.objectKey, metadata=self.metadata, + def get_init_upload_result(self): + return self.obsClient.initiateMultipartUpload(self.bucketName, self.objectKey, metadata=self.metadata, acl=self.headers.acl, storageClass=self.headers.storageClass, websiteRedirectLocation=self.headers.websiteRedirectLocation, contentType=self.headers.contentType, sseHeader=self.headers.sseHeader, expires=self.headers.expires, extensionGrants=self.headers.extensionGrants, - extensionHeaders=self.extensionHeaders) + extensionHeaders=self.extensionHeaders, + encoding_type=self.encoding_type) + + def _prepare(self): + fileStatus = [self.size, self.lastModified] + if self.checkSum: + fileStatus.append(util.base64_encode(util.md5_file_encode_by_size_offset(self.fileName, self.size, 0))) + + resp = self.get_init_upload_result() + if resp.status > 300: raise Exception('initiateMultipartUpload failed. ErrorCode:{0}. ErrorMessage:{1}'.format(resp.errorCode, resp.errorMessage)) uploadId = resp.body.uploadId - self._record = {'bucketName': self.bucketName, 'objectKey': self.objectKey, 'uploadId': uploadId, - 'uploadFile': self.fileName, 'fileStatus': fileStatus, 'uploadParts': self._slice_file(), - 'partEtags': []} + self._record["uploadParts"] = self._slice_file() + self._record["uploadId"] = uploadId + self._record["fileStatus"] = fileStatus self.obsClient.log_client.log(INFO, 'prepare new upload task success. uploadId = {0}'.format(uploadId)) if self.enableCheckPoint: self._write_record(self._record) - def _produce(self, ThreadPool, upload_parts): + @staticmethod + def _produce(ThreadPool, upload_parts): for part in upload_parts: ThreadPool.put(part) @@ -344,12 +373,7 @@ def _consume(self, ThreadPool): def _upload_part(self, part): if not self._is_abort(): try: - resp = self.obsClient._uploadPartWithNotifier(self.bucketName, self.objectKey, part['partNumber'], - self._record['uploadId'], self.fileName, - isFile=True, partSize=part['length'], - offset=part['offset'], notifier=self.notifier, - extensionHeaders=self.extensionHeaders, - sseHeader=self.headers.sseHeader) + resp = self.get_upload_part_resp(part) if resp.status < 300: self._record['uploadParts'][part['partNumber'] - 1]['isCompleted'] = True self._record['partEtags'].append(CompletePart(util.to_int(part['partNumber']), resp.body.etag)) @@ -367,11 +391,21 @@ def _upload_part(self, part): self.obsClient.log_client.log(DEBUG, 'upload part %s error, %s' % (part['partNumber'], e)) self.obsClient.log_client.log(ERROR, traceback.format_exc()) + def get_upload_part_resp(self, part): + return self.obsClient._uploadPartWithNotifier(self.bucketName, self.objectKey, part['partNumber'], + self._record['uploadId'], self.fileName, + isFile=True, partSize=part['length'], + offset=part['offset'], notifier=self.notifier, + extensionHeaders=self.extensionHeaders, + sseHeader=self.headers.sseHeader) + class downloadOperation(Operation): def __init__(self, bucketName, objectKey, downloadFile, partSize, taskNum, enableCheckPoint, checkPointFile, header, versionId, progressCallback, obsClient, imageProcess=None, notifier=progress.NONE_NOTIFIER, extensionHeaders=None): + if enableCheckPoint and not checkPointFile: + checkPointFile = downloadFile + '.download_record' super(downloadOperation, self).__init__(bucketName, objectKey, downloadFile, partSize, taskNum, enableCheckPoint, checkPointFile, progressCallback, obsClient, notifier) @@ -379,32 +413,34 @@ def __init__(self, bucketName, objectKey, downloadFile, partSize, taskNum, enabl self.versionId = versionId self.imageProcess = imageProcess self.extensionHeaders = extensionHeaders + self._record = {'bucketName': self.bucketName, 'objectKey': self.objectKey, 'versionId': self.versionId, + 'downloadFile': self.fileName, 'imageProcess': self.imageProcess} parent_dir = os.path.dirname(self.fileName) if not os.path.exists(parent_dir): os.makedirs(parent_dir) self._tmp_file = self.fileName + '.tmp' - metedata_resp = self.obsClient.getObjectMetadata(self.bucketName, self.objectKey, self.versionId, + metadata_resp = self.obsClient.getObjectMetadata(self.bucketName, self.objectKey, self.versionId, extensionHeaders=self.extensionHeaders, sseHeader=self.header.sseHeader, origin=self.header.origin, requestHeaders=self.header.requestHeaders) - if metedata_resp.status < 300: - self.lastModified = metedata_resp.body.lastModified - self.size = metedata_resp.body.contentLength \ - if metedata_resp.body.contentLength is not None and metedata_resp.body.contentLength >= 0 else 0 + if metadata_resp.status < 300: + self.lastModified = metadata_resp.body.lastModified + self.size = metadata_resp.body.contentLength \ + if metadata_resp.body.contentLength is not None and metadata_resp.body.contentLength >= 0 else 0 else: - if 400 <= metedata_resp.status < 500: + if 400 <= metadata_resp.status < 500: self._delete_record() self._delete_tmp_file() self.obsClient.log_client.log( ERROR, 'there are something wrong when touch the object {0}. ErrorCode:{1}, ErrorMessage:{2}'.format( - self.objectKey, metedata_resp.errorCode, metedata_resp.errorMessage)) + self.objectKey, metadata_resp.errorCode, metadata_resp.errorMessage)) raise Exception( 'there are something wrong when touch the object {0}. ErrorCode:{1}, ErrorMessage:{2}'.format( - self.objectKey, metedata_resp.status, metedata_resp.errorMessage)) - self._metedata_resp = metedata_resp + self.objectKey, metadata_resp.status, metadata_resp.errorMessage)) + self._metadata_resp = metadata_resp def _delete_tmp_file(self): if os.path.exists(self._tmp_file): @@ -415,7 +451,7 @@ def _do_rename(self): with open(self.fileName, 'wb') as wf: with open(self._tmp_file, 'rb') as rf: while True: - chunk = rf.read(65536) + chunk = rf.read(const.READ_ONCE_LENGTH) if not chunk: break wf.write(chunk) @@ -432,7 +468,7 @@ def _download(self): if self.enableCheckPoint: self._load() - if not self._record: + if "downloadParts" not in self._record: self._prepare() sent_bytes, unfinished_down_parts = self._download_prepare() @@ -465,11 +501,11 @@ def _download(self): if self.enableCheckPoint: self._delete_record() self.obsClient.log_client.log(INFO, 'download success.') - return self._metedata_resp + return self._metadata_resp except Exception as e: if self._do_rename(): self.obsClient.log_client.log(INFO, 'download success.') - return self._metedata_resp + return self._metadata_resp if not self.enableCheckPoint: self._delete_tmp_file() self.obsClient.log_client.log( @@ -497,19 +533,21 @@ def _load(self): self._delete_record() self._delete_tmp_file() self._record = None + if self._record is None: + self._record = {'bucketName': self.bucketName, 'objectKey': self.objectKey, 'versionId': self.versionId, + 'downloadFile': self.fileName, 'imageProcess': self.imageProcess} def _prepare(self): - object_staus = [self.objectKey, self.size, self.lastModified, self.versionId, self.imageProcess] + object_status = [self.objectKey, self.size, self.lastModified, self.versionId, self.imageProcess] with open(_to_unicode(self._tmp_file), 'wb') as f: if self.size > 0: f.seek(self.size - 1, 0) f.write('b' if const.IS_PYTHON2 else 'b'.encode('UTF-8')) tmp_file_status = [os.path.getsize(self._tmp_file), os.path.getmtime(self._tmp_file)] - self._record = {'bucketName': self.bucketName, 'objectKey': self.objectKey, 'versionId': self.versionId, - 'downloadFile': self.fileName, 'downloadParts': self._split_object(), - 'objectStatus': object_staus, - 'tmpFileStatus': tmp_file_status, 'imageProcess': self.imageProcess} + self._record["downloadParts"] = self._split_object() + self._record["objectStatus"] = object_status + self._record["tmpFileStatus"] = tmp_file_status self.obsClient.log_client.log(INFO, 'prepare new download task success.') if self.enableCheckPoint: self._write_record(self._record) @@ -572,7 +610,8 @@ def _split_object(self): downloadParts.append(part) return downloadParts - def _produce(self, ThreadPool, download_parts): + @staticmethod + def _produce(ThreadPool, download_parts): for part in download_parts: ThreadPool.put(part) @@ -583,7 +622,8 @@ def _consume(self, ThreadPool): break self._download_part(part) - def _copy_get_object_header(self, src_header): + @staticmethod + def _copy_get_object_header(src_header): get_object_header = GetObjectHeader() get_object_header.sseHeader = src_header.sseHeader get_object_header.if_match = src_header.if_match @@ -597,15 +637,16 @@ def _download_part(self, part): get_object_header = self._copy_get_object_header(self.header) get_object_header.range = util.to_string(part['offset']) + '-' + util.to_string(part['length']) if not self._is_abort(): + # todo 检视 response 的作用与 part_response 的重构 response = None try: - resp = self.obsClient._getObjectWithNotifier(bucketName=self.bucketName, objectKey=self.objectKey, - getObjectRequest=get_object_request, - headers=get_object_header, notifier=self.notifier, - extensionHeaders=self.extensionHeaders) + resp = self.obsClient.getObject(bucketName=self.bucketName, objectKey=self.objectKey, + getObjectRequest=get_object_request, + headers=get_object_header, notifier=self.notifier, + extensionHeaders=self.extensionHeaders) if resp.status < 300: - respone = resp.body.response - self._download_part_write(respone, part) + part_response = resp.body.response + self._download_part_write(part_response, part) self._record['downloadParts'][part['partNumber'] - 1]['isCompleted'] = True if self.enableCheckPoint: with self._lock: @@ -625,15 +666,15 @@ def _download_part(self, part): self.obsClient.log_client.log(ERROR, traceback.format_exc()) finally: if response is not None: - respone.close() + part_response.close() - def _download_part_write(self, respone, part): - chunk_size = 65536 - if respone is not None: + def _download_part_write(self, response, part): + chunk_size = const.READ_ONCE_LENGTH + if response is not None: with open(_to_unicode(self._tmp_file), 'rb+') as fs: fs.seek(part['offset'], 0) while True: - chunk = respone.read(chunk_size) + chunk = response.read(chunk_size) if not chunk: break fs.write(chunk) @@ -704,7 +745,8 @@ def ok(self): with self._lock: return self._exc_info is None - def _add_and_run(self, thread, pool): + @staticmethod + def _add_and_run(thread, pool): thread.daemon = True thread.start() pool.append(thread) diff --git a/src/obs/util.py b/src/obs/util.py index 1f1dbeb..b113e58 100644 --- a/src/obs/util.py +++ b/src/obs/util.py @@ -12,13 +12,12 @@ # CONDITIONS OF ANY KIND, either express or implied. See the License for the # specific language governing permissions and limitations under the License. -import re import base64 import hashlib -import os import json -from obs import const -from obs import progress +import re + +from obs import const, progress if const.IS_PYTHON2: import urllib @@ -69,11 +68,11 @@ def is_valid(item): class RequestFormat(object): @staticmethod - def get_pathformat(): + def get_path_format(): return PathFormat() @staticmethod - def get_subdomainformat(): + def get_sub_domain_format(): return SubdomainFormat() @classmethod @@ -98,7 +97,7 @@ def convert_path_string(cls, path_args, allowdNames=None): def get_endpoint(self, server, port, bucket): return - def get_pathbase(self, bucket, key): + def get_path_base(self, bucket, key): return def get_url(self, bucket, key, path_args): @@ -107,10 +106,11 @@ def get_url(self, bucket, key, path_args): class PathFormat(RequestFormat): - def get_server(self, server, bucket): + @staticmethod + def get_server(server, bucket): return server - def get_pathbase(self, bucket, key): + def get_path_base(self, bucket, key): if bucket: return '/' + bucket + '/' + encode_object_key(key) if key else '/' + bucket return '/' + encode_object_key(key) if key else '/' @@ -121,7 +121,7 @@ def get_endpoint(self, server, port, bucket): return server + ':' + str(port) def get_url(self, bucket, key, path_args): - path_base = self.get_pathbase(bucket, key) + path_base = self.get_path_base(bucket, key) path_arguments = self.convert_path_string(path_args) return path_base + path_arguments @@ -134,10 +134,11 @@ def get_full_url(self, is_secure, server, port, bucket, key, path_args): class SubdomainFormat(RequestFormat): - def get_server(self, server, bucket): + @staticmethod + def get_server(server, bucket): return bucket + '.' + server if bucket else server - def get_pathbase(self, bucket, key): + def get_path_base(self, bucket, key): if key is None: return '/' return '/' + encode_object_key(key) @@ -149,7 +150,7 @@ def get_endpoint(self, server, port, bucket): def get_url(self, bucket, key, path_args): url = self.convert_path_string(path_args) - return self.get_pathbase(bucket, key) + url + return self.get_path_base(bucket, key) + url def get_full_url(self, is_secure, server, port, bucket, key, path_args): url = 'https://' if is_secure else 'http://' @@ -170,7 +171,7 @@ def conn_delegate(conn): return delegate(conn) -def get_readable_entity(readable, chunk_size=65536, notifier=None, auto_close=True): +def get_readable_entity(readable, chunk_size=const.READ_ONCE_LENGTH, notifier=None, auto_close=True): if notifier is None: notifier = progress.NONE_NOTIFIER @@ -196,96 +197,49 @@ def entity(conn): return entity -def get_readable_entity_by_totalcount(readable, totalCount, chunk_size=65536, notifier=None, auto_close=True): - if notifier is None: - notifier = progress.NONE_NOTIFIER +def get_readable_entity_by_total_count(readable, totalCount, chunk_size=const.READ_ONCE_LENGTH, notifier=None, + auto_close=True): + return get_entity_for_send_with_total_count(readable, totalCount, chunk_size, notifier, auto_close) - def entity(conn): - try: - readCount = 0 - while True: - readCountOnce = chunk_size if totalCount - readCount >= chunk_size else totalCount - readCount - chunk = readable.read(readCountOnce) - newReadCount = len(chunk) - readCount += newReadCount - if newReadCount > 0: - notifier.send(newReadCount) - if readCount >= totalCount: - conn.send(chunk, final=True) - break - conn.send(chunk) - finally: - if hasattr(readable, 'close') and callable(readable.close) and auto_close: - readable.close() - - return entity - - -def get_file_entity(file_path, chunk_size=65536, notifier=None): - if notifier is None: - notifier = progress.NONE_NOTIFIER - - def entity(conn): - fileSize = os.path.getsize(file_path) - readCount = 0 - with open(file_path, 'rb') as f: - while True: - chunk = f.read(chunk_size) - newReadCount = len(chunk) - if newReadCount > 0: - notifier.send(newReadCount) - readCount += newReadCount - if readCount >= fileSize: - conn.send(chunk, final=True) - break - conn.send(chunk) - return entity +def get_file_entity_by_total_count(file_path, totalCount, chunk_size=const.READ_ONCE_LENGTH, notifier=None): + f = open(file_path, "rb") + return get_entity_for_send_with_total_count(f, totalCount, chunk_size, notifier) -def get_file_entity_by_totalcount(file_path, totalCount, chunk_size=65536, notifier=None): +def get_entity_for_send_with_total_count(readable, totalCount=None, chunk_size=const.READ_ONCE_LENGTH, notifier=None, + auto_close=True): if notifier is None: notifier = progress.NONE_NOTIFIER def entity(conn): readCount = 0 - with open(file_path, 'rb') as f: + try: while True: - readCountOnce = chunk_size if totalCount - readCount >= chunk_size else totalCount - readCount - chunk = f.read(readCountOnce) + if totalCount is None or totalCount - readCount >= chunk_size: + readCountOnce = chunk_size + else: + readCountOnce = totalCount - readCount + chunk = readable.read(readCountOnce) newReadCount = len(chunk) if newReadCount > 0: notifier.send(newReadCount) readCount += newReadCount - if readCount >= totalCount: + if (totalCount is not None and readCount >= totalCount) or (totalCount is not None and not chunk): conn.send(chunk, final=True) break conn.send(chunk) + finally: + if hasattr(readable, 'close') and callable(readable.close) and auto_close: + readable.close() return entity -def get_file_entity_by_offset_partsize(file_path, offset, partSize, chunk_size=65536, notifier=None): - if notifier is None: - notifier = progress.NONE_NOTIFIER - - def entity(conn): - readCount = 0 - with open(file_path, 'rb') as f: - f.seek(offset) - while True: - readCountOnce = chunk_size if partSize - readCount >= chunk_size else partSize - readCount - chunk = f.read(readCountOnce) - newReadCount = len(chunk) - if newReadCount > 0: - notifier.send(newReadCount) - readCount += newReadCount - if readCount >= partSize: - conn.send(chunk, final=True) - break - conn.send(chunk) - - return entity +def get_file_entity_by_offset_partsize(file_path, offset, totalCount, chunk_size=const.READ_ONCE_LENGTH, notifier=None): + f = open(file_path, "rb") + f.seek(offset) + return get_entity_for_send_with_total_count(f, totalCount, chunk_size, notifier) def is_ipaddress(item): @@ -300,11 +254,18 @@ def md5_encode(unencoded): return m.digest() +def covert_string_to_bytes(str_object): + if not const.IS_PYTHON2: + if isinstance(str_object, str): + return str_object.encode("UTF-8") + return str_object + + def base64_encode(unencoded): unencoded = unencoded if const.IS_PYTHON2 else ( unencoded.encode('UTF-8') if not isinstance(unencoded, bytes) else unencoded) - encodeestr = base64.b64encode(unencoded, altchars=None) - return encodeestr if const.IS_PYTHON2 else encodeestr.decode('UTF-8') + encode_str = base64.b64encode(unencoded, altchars=None) + return encode_str if const.IS_PYTHON2 else encode_str.decode('UTF-8') def encode_object_key(key): @@ -375,11 +336,11 @@ def md5_file_encode_by_size_offset(file_path=None, size=None, offset=None, chuck if file_path is not None and size is not None and offset is not None: m = hashlib.md5() with open(file_path, 'rb') as fp: - CHUNKSIZE = 65536 if chuckSize is None else chuckSize + CHUNK_SIZE = const.READ_ONCE_LENGTH if chuckSize is None else chuckSize fp.seek(offset) read_count = 0 while read_count < size: - read_size = CHUNKSIZE if size - read_count >= CHUNKSIZE else size - read_count + read_size = CHUNK_SIZE if size - read_count >= CHUNK_SIZE else size - read_count data = fp.read(read_size) read_count_once = len(data) if read_count_once <= 0: diff --git a/src/obs/workflow.py b/src/obs/workflow.py index 5e436a0..d9d1867 100644 --- a/src/obs/workflow.py +++ b/src/obs/workflow.py @@ -52,8 +52,8 @@ def _resultFilter(result, executionState): return result -def _listWorkflowExecutionMethodName(isJsonResult, defualtMethodName): - return 'ListWorkflowExecutionResponse' if not isJsonResult else defualtMethodName +def _listWorkflowExecutionMethodName(isJsonResult, defaultMethodName): + return 'ListWorkflowExecutionResponse' if not isJsonResult else defaultMethodName def _listWorkflowExecutionCount(isJsonResult, body): @@ -88,7 +88,7 @@ def _listWorkflowExecutionPathArgs(graphName, nextMarker, limit, executionType): pathArgs['x-workflow-next-marker'] = nextMarker if limit: if limit > 1000: - raise Exception('Invaild parameter: limit') + raise Exception('Invalid parameter: limit') pathArgs['x-workflow-limit'] = limit if executionType: pathArgs['x-workflow-execution-type'] = executionType @@ -157,7 +157,7 @@ def __init__(self, isJsonResult=False, *args, **kwargs): super(WorkflowClient, self).__init__(client_mode='workflow', *args, **kwargs) self.__resource = 'v2/' self.__isJsonResult = isJsonResult - self.__defualtMethodName = 'GetJsonResponse' + self.__defaultMethodName = 'GetJsonResponse' # begin workflow api # begin workflow api @@ -170,7 +170,7 @@ def createWorkflowTemplate(self, templateName, description=None, states=None, in bucketName=None, objectKey=self.__resource + combine(const.WORKFLOW_TEMPLATES, templateName), headers=prepareHeader(), - methodName='CreateWorkflowTemplateResponse' if not self.__isJsonResult else self.__defualtMethodName + methodName='CreateWorkflowTemplateResponse' if not self.__isJsonResult else self.__defaultMethodName ) @entrance @@ -179,7 +179,7 @@ def getWorkflowTemplate(self, templateName): bucketName=None, objectKey=self.__resource + combine(const.WORKFLOW_TEMPLATES, templateName), headers=prepareHeader(), - methodName='GetWorkflowTemplateResponse' if not self.__isJsonResult else self.__defualtMethodName + methodName='GetWorkflowTemplateResponse' if not self.__isJsonResult else self.__defaultMethodName ) @entrance @@ -188,7 +188,7 @@ def deleteWorkflowTemplate(self, templateName): bucketName=None, objectKey=self.__resource + combine(const.WORKFLOW_TEMPLATES, templateName), headers=prepareHeader(), - methodName=self.__defualtMethodName + methodName=self.__defaultMethodName ) @entrance @@ -205,7 +205,7 @@ def listWorkflowTemplate(self, templateNamePrefix=None, start=None, limit=None): templateNamePrefix if templateNamePrefix is not None else ''), pathArgs=pathArgs, headers=prepareHeader(), - methodName='ListWorkflowTemplateResponse' if not self.__isJsonResult else self.__defualtMethodName + methodName='ListWorkflowTemplateResponse' if not self.__isJsonResult else self.__defaultMethodName ) @entrance @@ -216,7 +216,7 @@ def createWorkflow(self, templateName, graphName, agency, description=None, para objectKey=self.__resource + combine(const.WORKFLOWS, graphName), pathArgs={'x-workflow-template-name': templateName}, headers=prepareHeader(), - methodName='CreateWorkflowResponse' if not self.__isJsonResult else self.__defualtMethodName + methodName='CreateWorkflowResponse' if not self.__isJsonResult else self.__defaultMethodName ) @entrance @@ -225,7 +225,7 @@ def getWorkflow(self, graphName): bucketName=None, objectKey=self.__resource + combine(const.WORKFLOWS, graphName), headers=prepareHeader(), - methodName='GetWorkflowResponse' if not self.__isJsonResult else self.__defualtMethodName + methodName='GetWorkflowResponse' if not self.__isJsonResult else self.__defaultMethodName ) @entrance @@ -234,7 +234,7 @@ def deleteWorkflow(self, graphName): bucketName=None, objectKey=self.__resource + combine(const.WORKFLOWS, graphName), headers=prepareHeader(), - methodName=self.__defualtMethodName + methodName=self.__defaultMethodName ) @entrance @@ -244,7 +244,7 @@ def updateWorkflow(self, graphName, parameters=None, description=None): bucketName=None, objectKey=self.__resource + combine(const.WORKFLOWS, graphName), headers=prepareHeader(), - methodName='UpdateWorkflowResponse' if not self.__isJsonResult else self.__defualtMethodName + methodName='UpdateWorkflowResponse' if not self.__isJsonResult else self.__defaultMethodName ) @entrance @@ -261,7 +261,7 @@ def listWorkflow(self, graphNamePrefix=None, start=None, limit=None): graphNamePrefix if graphNamePrefix is not None else ''), pathArgs=pathArgs, headers=prepareHeader(), - methodName='ListWorkflowResponse' if not self.__isJsonResult else self.__defualtMethodName + methodName='ListWorkflowResponse' if not self.__isJsonResult else self.__defaultMethodName ) @entrance @@ -271,7 +271,7 @@ def asyncAPIStartWorkflow(self, graphName, bucket, object, inputs=None): bucketName=None, objectKey=self.__resource + combine(const.WORKFLOWS, graphName), headers=prepareHeader(), - methodName='AsyncAPIStartWorkflowResponse' if not self.__isJsonResult else self.__defualtMethodName + methodName='AsyncAPIStartWorkflowResponse' if not self.__isJsonResult else self.__defaultMethodName ) @entrance @@ -279,14 +279,14 @@ def listWorkflowExecution(self, graphName, executionType=None, nextMarker=None, pathArgs = _listWorkflowExecutionPathArgs(graphName, nextMarker, limit, executionType) if executionState: if executionState not in ['RUNNING', 'SUCCESS', 'FAILED']: - raise Exception('Invaild parameter: execution state') + raise Exception('Invalid parameter: execution state') resp = self._make_get_request( bucketName=None, objectKey=self.__resource + const.WORKFLOW_EXECUTIONS, pathArgs=pathArgs, headers=prepareHeader(), - methodName=_listWorkflowExecutionMethodName(self.__isJsonResult, self.__defualtMethodName) + methodName=_listWorkflowExecutionMethodName(self.__isJsonResult, self.__defaultMethodName) ) if resp.status > 300: return resp @@ -314,7 +314,7 @@ def listWorkflowExecution(self, graphName, executionType=None, nextMarker=None, objectKey=self.__resource + const.WORKFLOW_EXECUTIONS, pathArgs=pathArgs, headers=prepareHeader(), - methodName=_listWorkflowExecutionMethodName(self.__isJsonResult, self.__defualtMethodName) + methodName=_listWorkflowExecutionMethodName(self.__isJsonResult, self.__defaultMethodName) ) if tempResp.status > 300: return tempResp @@ -344,7 +344,7 @@ def listWorkflowExecution(self, graphName, executionType=None, nextMarker=None, objectKey=self.__resource + const.WORKFLOW_EXECUTIONS, pathArgs=pathArgs, headers=prepareHeader(), - methodName=_listWorkflowExecutionMethodName(self.__isJsonResult, self.__defualtMethodName) + methodName=_listWorkflowExecutionMethodName(self.__isJsonResult, self.__defaultMethodName) ) @entrance @@ -358,7 +358,7 @@ def getWorkflowExecution(self, executionName, graphName): objectKey=self.__resource + combine(const.WORKFLOW_EXECUTIONS, executionName), pathArgs=pathArgs, headers=prepareHeader(), - methodName='GetWorkflowExecutionResponse' if not self.__isJsonResult else self.__defualtMethodName + methodName='GetWorkflowExecutionResponse' if not self.__isJsonResult else self.__defaultMethodName ) @entrance @@ -372,7 +372,7 @@ def restoreFailedWorkflowExecution(self, executionName, graphName): objectKey=self.__resource + combine(const.WORKFLOW_EXECUTIONS, executionName), pathArgs=pathArgs, headers=prepareHeader(), - methodName='RestoreFailedWorkflowExecutionResponse' if not self.__isJsonResult else self.__defualtMethodName + methodName='RestoreFailedWorkflowExecutionResponse' if not self.__isJsonResult else self.__defaultMethodName ) @entrance @@ -387,7 +387,7 @@ def putTriggerPolicy(self, bucketName, rules): objectKey=None, pathArgs=pathArgs, headers=prepareHeader(), - methodName=self.__defualtMethodName + methodName=self.__defaultMethodName ) @entrance @@ -401,7 +401,7 @@ def getTriggerPolicy(self, bucketName): objectKey=None, pathArgs=pathArgs, headers=prepareHeader(), - methodName='GetTriggerPolicyResponse' if not self.__isJsonResult else self.__defualtMethodName + methodName='GetTriggerPolicyResponse' if not self.__isJsonResult else self.__defaultMethodName ) @entrance @@ -415,7 +415,7 @@ def deleteTriggerPolicy(self, bucketName): objectKey=None, pathArgs=pathArgs, headers=prepareHeader(), - methodName=self.__defualtMethodName + methodName=self.__defaultMethodName ) # end workflow api diff --git a/src/tests/__init__.py b/src/tests/__init__.py new file mode 100644 index 0000000..f512dea --- /dev/null +++ b/src/tests/__init__.py @@ -0,0 +1 @@ +# coding:utf-8 diff --git a/src/tests/conftest.py b/src/tests/conftest.py new file mode 100644 index 0000000..7977e87 --- /dev/null +++ b/src/tests/conftest.py @@ -0,0 +1,76 @@ +# coding:utf-8 +import hashlib +import json +import os +import random +import sys + +import pytest + +sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '../'))) + +from obs.const import IS_PYTHON2 + +if IS_PYTHON2: + chr = unichr + + +def read_env_info(): + with open(os.path.join(os.getcwd(), "test_config.json"), "r") as f: + return json.loads(f.read()) + + +def compare_sha256(original_file, target_file, is_file=True): + original_sha256 = hashlib.sha256() + target_sha256 = hashlib.sha256() + with open(original_file, "rb") as f: + while True: + chunk = f.read(65536) + if not chunk: + break + original_sha256.update(chunk) + if is_file: + with open(target_file, "rb") as f: + while True: + chunk = f.read(65536) + if not chunk: + break + target_sha256.update(chunk) + else: + while True: + chunk = target_file.read(65536) + if not chunk: + break + target_sha256.update(chunk) + return target_sha256.hexdigest() == original_sha256.hexdigest() + + +def download_and_check(obsClient, bucket_name, object_name, download_path, source_file): + try: + obsClient.downloadFile(bucket_name, object_name, downloadFile=download_path, taskNum=10) + compare_result = compare_sha256(source_file, download_path) + return compare_result + except Exception: + return False + finally: + os.remove(download_path) + obsClient.deleteObject(bucket_name, object_name) + + +test_config = read_env_info() + + +@pytest.fixture(params=test_config["test_files"].items()) +def gen_test_file(request): + file_name = request.keywords.node.name + "_" + request.param[0] + gen_random_file(file_name, request.param[1]) + yield file_name + os.remove(test_config["path_prefix"] + file_name) + + +def gen_random_file(file_name, file_size): + tmp_1024 = "".join(chr(random.randint(10000, 40000)) for _ in range(341)).encode("UTF-8") + tmp_1024 += b"m" + with open(test_config["path_prefix"] + file_name, "wb") as f: + for _ in range(file_size): + f.write(tmp_1024) diff --git a/src/tests/test_config.json b/src/tests/test_config.json new file mode 100644 index 0000000..17e8000 --- /dev/null +++ b/src/tests/test_config.json @@ -0,0 +1,16 @@ +{ + "ak": "", + "sk": "", + "endpoint": "", + "bucketName": "", + "path_prefix": "", + "public_key": "", + "private_key": "", + "auth_type": "obs", + "test_files": { + "1k": 1, + "99k": 99, + "1M": 1024, + "100M": 102400 + } +} \ No newline at end of file diff --git a/src/tests/test_crypto_obs_client.py b/src/tests/test_crypto_obs_client.py new file mode 100644 index 0000000..24971c3 --- /dev/null +++ b/src/tests/test_crypto_obs_client.py @@ -0,0 +1,257 @@ +# coding:utf-8 +import hashlib +import io +import os + +import pytest + +from conftest import gen_random_file, test_config +from obs import CompleteMultipartUploadRequest, CompletePart, CryptoObsClient +from test_obs_client import TestOBSClient + + +class TestCryptoOBSClient(TestOBSClient): + def get_client(self): + client_type = "CTRCryptoClient" + path_style = True if test_config["auth_type"] == "v2" else False + + uploadClient = CryptoObsClient(access_key_id=test_config["ak"], secret_access_key=test_config["sk"], + server=test_config["endpoint"], cipher_generator="", + is_signature_negotiation=False, path_style=path_style) + downloadClient = CryptoObsClient(access_key_id=test_config["ak"], + secret_access_key=test_config["sk"], + server=test_config["endpoint"], + cipher_generator="", + is_signature_negotiation=False, path_style=path_style) + return client_type, uploadClient, downloadClient + + def get_encrypted_content(self): + pass + + def test_initiateEncryptedMultipartUpload_and_uploadEncryptedPart(self): + client_type, uploadClient, downloadClient = self.get_client() + object_name = client_type + "test_initiateEncryptedMultipartUpload" + test_bytes = io.BytesIO(b"test_initiateEncryptedMultipartUpload") + cipher = uploadClient.cipher_generator.new(test_bytes) + init_result = uploadClient.initiateEncryptedMultipartUpload(test_config["bucketName"], object_name, cipher) + assert init_result.status == 200 + assert "uploadId" in init_result.body + gen_random_file(object_name, 10240) + uploaded_parts = [] + upload_result = uploadClient.uploadEncryptedPart(test_config["bucketName"], object_name, 1, + init_result.body["uploadId"], crypto_cipher=cipher, + object=test_config["path_prefix"] + object_name, isFile=True) + assert upload_result.status == 200 + uploaded_parts.append(CompletePart(partNum=1, etag=dict(upload_result.header).get('etag'))) + upload_result = uploadClient.uploadEncryptedPart(test_config["bucketName"], object_name, 2, + init_result.body["uploadId"], crypto_cipher=cipher, + object=open(test_config["path_prefix"] + object_name, "rb")) + assert upload_result.status == 200 + + uploaded_parts.append(CompletePart(partNum=2, etag=dict(upload_result.header).get('etag'))) + upload_result = uploadClient.uploadEncryptedPart(test_config["bucketName"], object_name, 3, + init_result.body["uploadId"], crypto_cipher=cipher, + object="test obs") + assert upload_result.status == 200 + + uploaded_parts.append(CompletePart(partNum=3, etag=dict(upload_result.header).get('etag'))) + complete_result = uploadClient.completeMultipartUpload(test_config["bucketName"], object_name, + init_result.body["uploadId"], + CompleteMultipartUploadRequest(uploaded_parts)) + assert complete_result.status == 200 + + download_result = downloadClient.getObject(test_config["bucketName"], object_name, loadStreamInMemory=True) + assert download_result.status == 200 + download_sha256 = hashlib.sha256() + download_sha256.update(download_result.body.buffer) + local_sha256 = hashlib.sha256() + with open(test_config["path_prefix"] + object_name, "rb") as f: + local_sha256.update(f.read()) + f.seek(0) + local_sha256.update(f.read()) + local_sha256.update("test obs".encode("UTF-8")) + assert local_sha256.hexdigest() == download_sha256.hexdigest() + + uploadClient.deleteObject(test_config["bucketName"], object_name) + os.remove(test_config["path_prefix"] + object_name) + + def test_appendObject(self): + client_type, uploadClient, downloadClient = self.get_client() + has_exception = False + try: + uploadClient.appendObject(test_config["bucketName"], "test_append_object") + except Exception as e: + has_exception = True + assert e.message == 'AppendObject is not supported in CryptoObsClient' + assert has_exception + + def test_copyPart(self): + client_type, uploadClient, downloadClient = self.get_client() + has_exception = False + try: + uploadClient.copyPart(test_config["bucketName"], "test_copyPart", 1, + "test_copyPart_ID", "test_copyPart_source") + except Exception as e: + has_exception = True + assert e.message == 'CopyPart is not supported in CryptoObsClient' + assert has_exception + + def test_initiateMultipartUpload(self): + client_type, uploadClient, downloadClient = self.get_client() + has_exception = False + try: + uploadClient.initiateMultipartUpload(test_config["bucketName"], "test_initiateMultipartUpload") + except Exception as e: + has_exception = True + assert e.message == 'InitiateMultipartUpload is not supported in CryptoObsClient' + assert has_exception + + def test_uploadPart(self): + client_type, uploadClient, downloadClient = self.get_client() + has_exception = False + try: + uploadClient.uploadPart(test_config["bucketName"], "test_uploadPart", 1, "test_uploadPart") + except Exception as e: + has_exception = True + assert e.message == 'UploadPart is not supported in CryptoObsClient' + assert has_exception + + def test_getObject_with_no_metadata(self): + has_exception = False + object_key = "test_downloadFile_with_no_metadata" + client_type, uploadClient, downloadClient = self.get_client() + upload_result = uploadClient.putContent(test_config["bucketName"], object_key, + "test obs") + assert upload_result.status == 200 + + set_result = uploadClient.setObjectMetadata(test_config["bucketName"], object_key, + metadata={"aaa": "bbb"}) + assert set_result.status == 200 + try: + downloadClient.getObject(test_config["bucketName"], object_key) + except Exception as e: + has_exception = True + assert e.message == "Crypto mod is not in object's metadata" + assert has_exception + uploadClient.deleteObject(test_config["bucketName"], object_key) + + def test_getObject_with_wrong_crypto_mod(self): + has_exception = False + object_key = "test_downloadFile_with_no_metadata" + client_type, uploadClient, downloadClient = self.get_client() + upload_result = uploadClient.putContent(test_config["bucketName"], object_key, + "test obs") + assert upload_result.status == 200 + + set_result = uploadClient.setObjectMetadata(test_config["bucketName"], object_key, + metadata={"encrypted-algorithm": "wrong encrypted-algorithm"}) + assert set_result.status == 200 + try: + downloadClient.getObject(test_config["bucketName"], object_key) + except Exception as e: + has_exception = True + assert e.message == "Object's crypto mod is not equals cipher-generator's, " \ + "please change a different cipher-generator" + assert has_exception + uploadClient.deleteObject(test_config["bucketName"], object_key) + + def test_initiateMultipartUpload_and_uploadPart_and_copyPart(self): + has_exception = False + try: + super(TestCryptoOBSClient, self).test_initiateMultipartUpload_and_uploadPart_and_copyPart() + except Exception as e: + has_exception = True + assert e.message == 'InitiateMultipartUpload is not supported in CryptoObsClient' + assert has_exception + + +class TestCryptoOBSClientWithSha256(TestCryptoOBSClient): + def test_putContent_with_file_stream(self, gen_test_file): + has_exception = False + try: + super(TestCryptoOBSClientWithSha256, self).test_putContent_with_file_stream(gen_test_file) + except Exception as e: + has_exception = True + assert e.message == "Could not calculate sha256 for a stream object" + assert has_exception + + def test_putContent_with_network_stream(self, gen_test_file): + has_exception = False + try: + super(TestCryptoOBSClientWithSha256, self).test_putContent_with_network_stream(gen_test_file) + except Exception as e: + has_exception = True + assert e.message == "Could not calculate sha256 for a stream object" + assert has_exception + + def test_putContent_has_sha256(self): + client_type, uploadClient, downloadClient = self.get_client() + object_key = "test_putContent_has_sha256" + upload_result = uploadClient.putContent(test_config["bucketName"], object_key, "test obs") + assert upload_result.status == 200 + + metadata = uploadClient.getObjectMetadata(test_config["bucketName"], object_key) + assert metadata.status == 200 + metadata_dict = dict(metadata.header) + assert metadata_dict["plaintext-sha256"] == "4c3600ae5c56b61a2b0d15681a5f3945ad92f671891d7ec3db366743d09ade08" + assert "encrypted-start" in metadata_dict + uploadClient.deleteObject(test_config["bucketName"], object_key) + + def test_uploadFile_has_sha256(self): + client_type, uploadClient, downloadClient = self.get_client() + object_key = "test_uploadFile_has_sha256" + gen_random_file(object_key, 10240) + upload_result = uploadClient.uploadFile(test_config["bucketName"], object_key, + test_config["path_prefix"] + object_key) + assert upload_result.status == 200 + sha256 = hashlib.sha256() + with open(test_config["path_prefix"] + object_key) as f: + while True: + chunk = f.read(65536) + if not chunk: + break + sha256.update(chunk) + + metadata = uploadClient.getObjectMetadata(test_config["bucketName"], object_key) + assert metadata.status == 200 + metadata_dict = dict(metadata.header) + assert metadata_dict["plaintext-sha256"] == sha256.hexdigest() + assert "encrypted-start" in metadata_dict + uploadClient.deleteObject(test_config["bucketName"], object_key) + os.remove(test_config["path_prefix"] + object_key) + + def test_putFile_has_sha256(self): + client_type, uploadClient, downloadClient = self.get_client() + object_key = "test_putFile_has_sha256" + gen_random_file(object_key, 10240) + upload_result = uploadClient.putFile(test_config["bucketName"], object_key, + test_config["path_prefix"] + object_key) + assert upload_result.status == 200 + sha256 = hashlib.sha256() + with open(test_config["path_prefix"] + object_key) as f: + while True: + chunk = f.read(65536) + if not chunk: + break + sha256.update(chunk) + + metadata = uploadClient.getObjectMetadata(test_config["bucketName"], object_key) + assert metadata.status == 200 + metadata_dict = dict(metadata.header) + assert metadata_dict["plaintext-sha256"] == sha256.hexdigest() + assert "encrypted-start" in metadata_dict + uploadClient.deleteObject(test_config["bucketName"], object_key) + os.remove(test_config["path_prefix"] + object_key) + + def test_initiateEncryptedMultipartUpload_and_uploadEncryptedPart(self): + has_exception = False + try: + super(TestCryptoOBSClientWithSha256, self).test_initiateEncryptedMultipartUpload_and_uploadEncryptedPart() + except Exception as e: + has_exception = True + assert e.message == 'Could not calculate sha256 for initiateMultipartUpload' + assert has_exception + + +if __name__ == "__main__": + pytest.main(["-v", 'test_ctr_crypto_client.py::TestCryptoOBSClient']) diff --git a/src/tests/test_ctr_crypto_client.py b/src/tests/test_ctr_crypto_client.py new file mode 100644 index 0000000..530e849 --- /dev/null +++ b/src/tests/test_ctr_crypto_client.py @@ -0,0 +1,72 @@ +# coding:utf-8 + +import pytest + +from conftest import test_config +from obs import CTRCipherGenerator, CryptoObsClient, const +from test_crypto_obs_client import TestCryptoOBSClient, TestCryptoOBSClientWithSha256 + +if const.IS_PYTHON2: + chr = unichr + + +class TestCTRCryptoClient(TestCryptoOBSClient): + def get_client(self): + client_type = "CTRCryptoClient" + path_style = True if test_config["auth_type"] == "v2" else False + ctr_cipher_generator = CTRCipherGenerator("0123456789abcdef0123456789abcdef", need_sha256=False) + uploadClient = CryptoObsClient(access_key_id=test_config["ak"], secret_access_key=test_config["sk"], + server=test_config["endpoint"], cipher_generator=ctr_cipher_generator, + is_signature_negotiation=False, path_style=path_style) + downloadClient = CryptoObsClient(access_key_id=test_config["ak"], + secret_access_key=test_config["sk"], + server=test_config["endpoint"], + cipher_generator=ctr_cipher_generator, + is_signature_negotiation=False, path_style=path_style) + return client_type, uploadClient, downloadClient + + def test_downloadFile_with_wrong_iv(self): + has_exception = False + object_key = "downloadFile_with_wrong_iv" + upload_cipher_generator = CTRCipherGenerator("0123456789abcdef0123456789abcdef", crypto_iv="12345678", + need_sha256=False) + uploadClient = CryptoObsClient(access_key_id=test_config["ak"], + secret_access_key=test_config["sk"], + server=test_config["endpoint"], + cipher_generator=upload_cipher_generator) + download_cipher_generator = CTRCipherGenerator("0123456789abcdef0123456789abcdef", crypto_iv="12345678", + need_sha256=False) + downloadClient = CryptoObsClient(access_key_id=test_config["ak"], + secret_access_key=test_config["sk"], + server=test_config["endpoint"], + cipher_generator=download_cipher_generator) + upload_result = uploadClient.putContent(test_config["bucketName"], object_key, "test OBS") + assert upload_result.status == 200 + try: + downloadClient.getObject(test_config["bucketName"], object_key) + except Exception as e: + has_exception = True + assert e.message == "Crypto_iv is different between local and server" + assert has_exception + uploadClient.deleteObject(test_config["bucketName"], object_key) + + +class TestCTRCryptoClientWithSha256(TestCryptoOBSClientWithSha256): + def get_client(self): + client_type = "CTRCryptoClient" + path_style = True if test_config["auth_type"] == "v2" else False + ctr_cipher_generator = CTRCipherGenerator("0123456789abcdef0123456789abcdef", need_sha256=True) + uploadClient = CryptoObsClient(access_key_id=test_config["ak"], secret_access_key=test_config["sk"], + server=test_config["endpoint"], cipher_generator=ctr_cipher_generator, + is_signature_negotiation=False, path_style=path_style) + downloadClient = CryptoObsClient(access_key_id=test_config["ak"], + secret_access_key=test_config["sk"], + server=test_config["endpoint"], + cipher_generator=ctr_cipher_generator, + is_signature_negotiation=False, path_style=path_style) + return client_type, uploadClient, downloadClient + + +if __name__ == "__main__": + pytest.main(["-v", 'test_ctr_crypto_client.py::TestCTRCryptoClient', + 'test_ctr_crypto_client.py::TestCTRCryptoClientWithSha256']) diff --git a/src/tests/test_obs_client.py b/src/tests/test_obs_client.py new file mode 100644 index 0000000..55062d9 --- /dev/null +++ b/src/tests/test_obs_client.py @@ -0,0 +1,302 @@ +# coding:utf-8 +import io +import os +import random +import time +from datetime import datetime + +import pytest + +import conftest +from obs import GetObjectHeader, ObsClient, UploadFileHeader +from conftest import test_config + +from obs.const import IS_PYTHON2 + +if IS_PYTHON2: + chr = unichr + + +class TestOBSClient(object): + def get_client(self): + client_type = "OBSClient" + path_style = True if test_config["auth_type"] == "v2" else False + uploadClient = ObsClient(access_key_id=test_config["ak"], secret_access_key=test_config["sk"], + server=test_config["endpoint"], + is_signature_negotiation=False, path_style=path_style) + downloadClient = ObsClient(access_key_id=test_config["ak"], secret_access_key=test_config["sk"], + server=test_config["endpoint"], + is_signature_negotiation=False, path_style=path_style) + return client_type, uploadClient, downloadClient + + def test_uploadFile_and_getObject_to_file(self, gen_test_file): + client_type, uploadClient, downloadClient = self.get_client() + object_name = client_type + "test_uploadFile_and_getObject_to_file_" + gen_test_file + upload_result = uploadClient.uploadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + gen_test_file, taskNum=10) + assert upload_result.status == 200 + download_result = downloadClient.getObject(test_config["bucketName"], object_name, + downloadPath=test_config["path_prefix"] + object_name) + assert download_result.status == 200 + assert conftest.compare_sha256(test_config["path_prefix"] + gen_test_file, + test_config["path_prefix"] + object_name) + os.remove(test_config["path_prefix"] + object_name) + uploadClient.deleteObject(test_config["bucketName"], object_name) + + def test_putFile_and_downloadFile(self, gen_test_file): + client_type, uploadClient, downloadClient = self.get_client() + object_name = client_type + "test_putFile_and_downloadFile_" + gen_test_file + upload_result = uploadClient.putFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + gen_test_file) + assert upload_result.status == 200 + assert conftest.download_and_check(downloadClient, test_config["bucketName"], object_name, + test_config["path_prefix"] + object_name, + test_config["path_prefix"] + gen_test_file) + + def test_putContent_with_file_stream(self, gen_test_file): + client_type, uploadClient, downloadClient = self.get_client() + object_name = client_type + "test_putContent_with_file_stream" + gen_test_file + upload_result = uploadClient.putContent(test_config["bucketName"], object_name, + open(test_config["path_prefix"] + gen_test_file, "rb")) + assert upload_result.status == 200 + assert conftest.download_and_check(downloadClient, test_config["bucketName"], object_name, + test_config["path_prefix"] + object_name, + test_config["path_prefix"] + gen_test_file) + + def test_putContent_with_network_stream(self, gen_test_file): + client_type, uploadClient, downloadClient = self.get_client() + object_for_upload = "" + try: + object_for_upload = client_type + "test_putContent_with_stream_from_getObject_upload_" + gen_test_file + object_for_verify = client_type + "test_putContent_with_stream_from_getObject_verify_" + gen_test_file + upload_result = uploadClient.uploadFile(test_config["bucketName"], object_for_upload, + test_config["path_prefix"] + gen_test_file, taskNum=10) + assert upload_result.status == 200 + resp = downloadClient.getObject(test_config["bucketName"], object_for_upload, loadStreamInMemory=False) + upload_result2 = uploadClient.putContent(test_config["bucketName"], object_for_verify, resp.body.response) + + assert upload_result2.status == 200 + assert conftest.download_and_check(downloadClient, test_config["bucketName"], object_for_verify, + test_config["path_prefix"] + object_for_verify, + test_config["path_prefix"] + gen_test_file) + finally: + uploadClient.deleteObject(test_config["bucketName"], object_for_upload) + + def test_getObject_to_memory(self, gen_test_file): + client_type, uploadClient, downloadClient = self.get_client() + object_name = client_type + "test_uploadFile_and_getObject_to_file_" + gen_test_file + upload_result = uploadClient.uploadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + gen_test_file, taskNum=10) + assert upload_result.status == 200 + resp = downloadClient.getObject(test_config["bucketName"], object_name, loadStreamInMemory=True) + bytes_io = io.BytesIO(resp.body.buffer) + assert conftest.compare_sha256(test_config["path_prefix"] + gen_test_file, + bytes_io, False) + uploadClient.deleteObject(test_config["bucketName"], object_name) + + def test_putContent_with_string_and_range_get_object(self): + client_type, uploadClient, downloadClient = self.get_client() + test_string = "".join(chr(random.randint(10000, 40000)) for _ in range(341)) + object_name = client_type + "test_range_get_object" + header = GetObjectHeader() + header.range = "60-119" + upload_result = uploadClient.putContent(test_config["bucketName"], object_name, test_string) + assert upload_result.status == 200 + resp = downloadClient.getObject(test_config["bucketName"], object_name, headers=header, loadStreamInMemory=True) + assert resp.body.buffer == test_string[20:40].encode("UTF-8") + uploadClient.deleteObject(test_config["bucketName"], object_name) + + def test_appendObject(self): + pass + + def test_initiateMultipartUpload_and_uploadPart_and_copyPart(self): + client_type, uploadClient, downloadClient = self.get_client() + object_name = client_type + "test_uploadPart" + init_result = uploadClient.initiateMultipartUpload(test_config["bucketName"], object_name) + assert init_result.status == 200 + assert "uploadId" in init_result.body + upload_result = uploadClient.uploadPart(test_config["bucketName"], object_name, 1, + init_result.body["uploadId"], + object="test_initiateMultipartUpload_and_uploadPart") + assert upload_result.status == 200 + # 生成一个大于 100k 的对象 + test_string = "".join(chr(random.randint(10000, 40000)) for _ in range(342)) + test_100k = "".join([test_string] * 100) + uploadClient.putContent(test_config["bucketName"], "test_copyPart", test_100k) + uploadClient.copyPart(test_config["bucketName"], object_name, 2, init_result.body["uploadId"], "test_copyPart") + uploadClient.abortMultipartUpload(test_config["bucketName"], object_name, init_result.body["uploadId"]) + + def test_uploadFile_part_size_less_than_100k(self): + pass + + def test_uploadFile_part_num_more_than_10000(self): + conftest.gen_random_file("test_uploadFile_part_num_more_than_10000", 1024 * 1024) + client_type, uploadClient, downloadClient = self.get_client() + object_name = "test_uploadFile_part_num_more_than_10000" + upload_result = uploadClient.uploadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + object_name, + partSize=100 * 1024, taskNum=50) + assert upload_result.status == 200 + object_metadata = uploadClient.getObjectMetadata(test_config["bucketName"], object_name) + assert dict(object_metadata.header)["etag"].endswith("-10000\"") + uploadClient.deleteObject(test_config["bucketName"], object_name) + os.remove(test_config["path_prefix"] + object_name) + + def test_downloadFile_part_num_more_than_10000(self): + conftest.gen_random_file("test_downloadFile_part_num_more_than_10000", 1024 * 1024) + client_type, uploadClient, downloadClient = self.get_client() + object_name = "test_downloadFile_part_num_more_than_10000" + upload_result = uploadClient.uploadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + object_name, taskNum=50) + assert upload_result.status == 200 + + download_result = downloadClient.downloadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + + "test_downloadFile_part_num_more_than_10000_download", + taskNum=50) + assert download_result.status == 200 + assert conftest.compare_sha256(test_config["path_prefix"] + object_name, + test_config["path_prefix"] + + "test_downloadFile_part_num_more_than_10000_download") + uploadClient.deleteObject(test_config["bucketName"], object_name) + os.remove(test_config["path_prefix"] + object_name) + os.remove(test_config["path_prefix"] + "test_downloadFile_part_num_more_than_10000_download") + + def test_downloadFile_with_lastModify(self): + has_exception = False + client_type, uploadClient, downloadClient = self.get_client() + object_name = "test_downloadFile_with_lastModify" + upload_result = uploadClient.putContent(test_config["bucketName"], object_name, "test OBS") + assert upload_result.status == 200 + time.sleep(30) + download_headers = GetObjectHeader() + download_headers.if_modified_since = datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT') + try: + downloadClient.downloadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + + "test_downloadFile_with_lastModify", header=download_headers, + taskNum=50) + except Exception as e: + has_exception = True + assert "response from server is something wrong." in e.message + assert has_exception + + def test_uploadFile_with_checksum(self): + conftest.gen_random_file("test_uploadFile_with_checksum", 1024 * 1024) + client_type, uploadClient, downloadClient = self.get_client() + object_name = "test_uploadFile_with_checksum" + upload_result = uploadClient.uploadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + object_name, + checkSum=True, taskNum=50, partSize=1024 * 1024) + assert upload_result.status == 200 + object_metadata = uploadClient.getObjectMetadata(test_config["bucketName"], object_name) + assert dict(object_metadata.header)["etag"].endswith("-1024\"") + uploadClient.deleteObject(test_config["bucketName"], object_name) + os.remove(test_config["path_prefix"] + object_name) + + def test_uploadFile_with_storage_type(self): + client_type, uploadClient, downloadClient = self.get_client() + object_name = "test_uploadFile_with_storage_type" + conftest.gen_random_file(object_name, 1024) + for i in ["WARM", "COLD"]: + upload_header = UploadFileHeader() + upload_header.storageClass = i + upload_result = uploadClient.uploadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + object_name, headers=upload_header, + checkSum=True, taskNum=10) + assert upload_result.status == 200 + object_metadata = uploadClient.getObjectMetadata(test_config["bucketName"], object_name) + assert dict(object_metadata.header)["storage-class"] == i + uploadClient.deleteObject(test_config["bucketName"], object_name) + os.remove(test_config["path_prefix"] + object_name) + + def test_downloadFile_with_if_match(self): + has_exception = False + client_type, uploadClient, downloadClient = self.get_client() + object_name = "test_downloadFile_with_if_match" + upload_result = uploadClient.putContent(test_config["bucketName"], object_name, "test OBS") + assert upload_result.status == 200 + download_headers = GetObjectHeader() + download_headers.if_match = "Wrong etag" + try: + downloadClient.downloadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + + "test_downloadFile_with_lastModify", header=download_headers, + taskNum=50) + except Exception as e: + has_exception = True + assert "PreconditionFailed" in e.message + assert has_exception + + def test_downloadFile_with_if_none_match(self): + has_exception = False + client_type, uploadClient, downloadClient = self.get_client() + object_name = "test_downloadFile_with_if_none_match" + upload_result = uploadClient.putContent(test_config["bucketName"], object_name, "test OBS") + assert upload_result.status == 200 + download_headers = GetObjectHeader() + download_headers.if_none_match = '64a58230c3d4db2fe8fcb83c0f45c50b' + try: + downloadClient.downloadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + + "test_downloadFile_with_lastModify", header=download_headers, + taskNum=50) + except Exception: + has_exception = True + assert has_exception + + def test_downloadFile_with_wrong_version_id(self): + has_exception = False + client_type, uploadClient, downloadClient = self.get_client() + object_name = "test_downloadFile_with_wrong_version_id" + upload_result = uploadClient.putContent(test_config["bucketName"], object_name, "test OBS") + assert upload_result.status == 200 + try: + downloadClient.downloadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + + "test_downloadFile_with_lastModify", versionId="Wrong Version ID", + taskNum=50) + except Exception: + has_exception = True + assert has_exception + + def test_downloadFile_with_unmodified_since(self): + has_exception = False + client_type, uploadClient, downloadClient = self.get_client() + object_name = "test_downloadFile_with_if_match" + upload_result = uploadClient.putContent(test_config["bucketName"], object_name, "test OBS") + assert upload_result.status == 200 + download_headers = GetObjectHeader() + download_headers.if_unmodified_since = "Tue, 02 Jul 2018 08:28:00 GMT" + try: + downloadClient.downloadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + + "test_downloadFile_with_lastModify", header=download_headers, + taskNum=50) + except Exception as e: + has_exception = True + assert "PreconditionFailed" in e.message + assert has_exception + + def test_uploadFile_with_metadata(self): + client_type, uploadClient, downloadClient = self.get_client() + object_name = "test_uploadFile_with_metadata" + conftest.gen_random_file(object_name, 1024) + metadata = {"content_type": "text/plain", + "expires": 1, + "meta_key1": "value1", + "meta_key-2": "value-2"} + upload_result = uploadClient.uploadFile(test_config["bucketName"], object_name, + test_config["path_prefix"] + object_name, metadata=metadata, + checkSum=True, taskNum=10) + assert upload_result.status == 200 + object_metadata = uploadClient.getObjectMetadata(test_config["bucketName"], object_name) + meta_dict = dict(object_metadata.header) + assert meta_dict["content_type"] == "text/plain" + assert meta_dict["expires"] == '1' + assert meta_dict["meta_key1"] == "value1" + assert meta_dict["meta_key-2"] == "value-2" + + +if __name__ == "__main__": + pytest.main(["-v", 'test_obs_client.py::TestOBSClient::test_uploadFile_with_metadata']) diff --git a/src/tests/test_rsa_crypto_client.py b/src/tests/test_rsa_crypto_client.py new file mode 100644 index 0000000..04c2fb7 --- /dev/null +++ b/src/tests/test_rsa_crypto_client.py @@ -0,0 +1,45 @@ +# coding:utf-8 +import pytest + +from obs import CryptoObsClient, CtrRSACipherGenerator +from conftest import test_config +from test_crypto_obs_client import TestCryptoOBSClient, TestCryptoOBSClientWithSha256 + + +class TestRSACTRCryptoClient(TestCryptoOBSClient): + def get_client(self): + client_type = "RSACTRCryptoClient" + public_cipher_generator = CtrRSACipherGenerator(test_config["public_key"], need_sha256=False) + path_style = True if test_config["auth_type"] == "v2" else False + uploadClient = CryptoObsClient(access_key_id=test_config["ak"], secret_access_key=test_config["sk"], + server=test_config["endpoint"], cipher_generator=public_cipher_generator, + is_signature_negotiation=False, path_style=path_style) + private_cipher_generator = CtrRSACipherGenerator(test_config["private_key"], need_sha256=False) + downloadClient = CryptoObsClient(access_key_id=test_config["ak"], + secret_access_key=test_config["sk"], + server=test_config["endpoint"], + cipher_generator=private_cipher_generator, + is_signature_negotiation=False, path_style=path_style) + return client_type, uploadClient, downloadClient + + +class TestRSACTRCryptoClientWithSha256(TestCryptoOBSClientWithSha256): + def get_client(self): + client_type = "RSACTRCryptoClient" + public_cipher_generator = CtrRSACipherGenerator(test_config["public_key"], need_sha256=True) + path_style = True if test_config["auth_type"] == "v2" else False + uploadClient = CryptoObsClient(access_key_id=test_config["ak"], secret_access_key=test_config["sk"], + server=test_config["endpoint"], cipher_generator=public_cipher_generator, + is_signature_negotiation=False, path_style=path_style) + private_cipher_generator = CtrRSACipherGenerator(test_config["private_key"], need_sha256=True) + downloadClient = CryptoObsClient(access_key_id=test_config["ak"], + secret_access_key=test_config["sk"], + server=test_config["endpoint"], + cipher_generator=private_cipher_generator, + is_signature_negotiation=False, path_style=path_style) + return client_type, uploadClient, downloadClient + + +if __name__ == "__main__": + pytest.main(["-v", 'test_rsa_crypto_client.py::TestRSACTRCryptoClient', + 'test_rsa_crypto_client.py::TestRSACTRCryptoClientWithSha256']) diff --git a/src/tests/test_util.py b/src/tests/test_util.py new file mode 100644 index 0000000..52afec5 --- /dev/null +++ b/src/tests/test_util.py @@ -0,0 +1,120 @@ +# coding:utf-8 +import os +import random +import unittest + +import obs.util as util + + +class MockConn(object): + def __init__(self): + self.send_length = 0 + self.end = False + self.not_data_times = 0 + self.read_memory_list = [] + self.read_memory = b"" + + def send(self, chunk, final=False): + self.end = final + self.read_memory_list.append(chunk) + if len(chunk) == 0: + self.not_data_times += 1 + if final: + self.read_memory = b"".join(self.read_memory_list) + self.send_length += len(chunk) + + +class TestUtilGetEntity(unittest.TestCase): + def setUp(self): + self.file_path_list = ["3k", "300k", "3M", "300M", "3G"] + self.path_prefix = "D:/workspace/mpb_tools/" + + def test_send_file_entity_by_total_count(self): + for i in self.file_path_list: + new_conn = MockConn() + file_path = self.path_prefix + i + file_total_count = os.path.getsize(file_path) + entity = util.get_file_entity_by_total_count(file_path, file_total_count) + while True: + entity(new_conn) + if new_conn.end or new_conn.not_data_times > 10: + break + self.assertEqual(new_conn.send_length, file_total_count) + with open(file_path, "rb") as f: + self.assertEqual(new_conn.read_memory, f.read()) + + def test_send_file_entity_by_offset_and_partsize(self): + for i in self.file_path_list: + new_conn = MockConn() + file_path = self.path_prefix + i + file_total_count = os.path.getsize(file_path) + partSize = int(file_total_count / 3) + offset = partSize + entity = util.get_file_entity_by_offset_partsize(file_path, offset, partSize) + while True: + entity(new_conn) + if new_conn.end or new_conn.not_data_times > 10: + break + self.assertEqual(new_conn.send_length, partSize) + with open(file_path, "rb") as f: + f.seek(offset) + self.assertEqual(new_conn.read_memory, f.read(partSize)) + + def test_send_entity_by_file(self): + # 验证重构后读文件结果和原本一致 + for i in self.file_path_list: + new_conn = MockConn() + file_path = self.path_prefix + i + file_total_count = os.path.getsize(file_path) + readable_object = open(file_path, "rb") + entity = util.get_entity_for_send_with_total_count(readable_object, file_total_count) + while True: + entity(new_conn) + if new_conn.end or new_conn.not_data_times > 10: + break + self.assertEqual(new_conn.send_length, file_total_count) + with open(file_path, "rb") as f: + self.assertEqual(new_conn.read_memory, f.read()) + + def test_send_entity_by_file_with_offset_and_partsize(self): + for i in self.file_path_list: + new_conn = MockConn() + file_path = self.path_prefix + i + file_total_count = os.path.getsize(file_path) + partSize = int(file_total_count / 3) + offset = partSize + readable_object = open(file_path, "rb") + readable_object.seek(offset) + entity = util.get_entity_for_send_with_total_count(readable_object, partSize) + while True: + entity(new_conn) + if new_conn.end or new_conn.not_data_times > 10: + break + self.assertEqual(new_conn.send_length, partSize) + with open(file_path, "rb") as f: + f.seek(offset) + self.assertEqual(new_conn.read_memory, f.read(partSize)) + + def test_send_readable_entity_by_total_count(self): + for n in range(5): + offset_percent = random.random() + for i in self.file_path_list: + new_conn = MockConn() + new_conn2 = MockConn() + file_path = self.path_prefix + i + file_total_count = os.path.getsize(file_path) + readable_object = open(file_path, "rb") + readable_object2 = open(file_path, "rb") + readable_object.seek(int(file_total_count * offset_percent)) + readable_object2.seek(int(file_total_count * offset_percent)) + entity = util.get_entity_for_send_with_total_count(readable_object, file_total_count + - int(file_total_count * offset_percent)) + entity2 = util.get_readable_entity_by_total_count(readable_object2, file_total_count + - int(file_total_count * offset_percent)) + while True: + entity(new_conn) + entity2(new_conn2) + if new_conn.end or new_conn.not_data_times > 10: + break + self.assertEqual(new_conn.send_length, new_conn2.send_length) + self.assertEqual(new_conn.read_memory, new_conn2.read_memory)