introduce Google Cloud Bucket support #52
Conversation
|
We typically try to avoid transferring through CKAN when possible, because CKAN installs are typically not configured for a bunch of long-running processes dishing out downloads - it's not async. Not to mention the bandwith. Having a fallback is an a-okay idea, but the way you really want to solve this for google cloud is the same way we solve it for AWS S3 - Signed URLs. See https://cloud.google.com/storage/docs/access-control/signed-urls. This allows you to keep your bucket private by default, do permission checks with CKAN, return a temporary URL with a cryptographic value attached to it, and allow the user to download the file directly from GCP without proxying. |
This implements: Upgrade libcloud to the latest release supporting python TkTech#53
|
Thanks @TkTech yes I know but having a quick alternative is better than nothing ;) Thank you also for that link, I've seen als that approach but, if possible, I really don't want to implement it for a specific driver only (f.e.: google). be sure, we need this driver working in production soon over google so, I'll try soon to add also that support in another pull request on this plugin and possibly directly into apache-libcloud. Thanks |
ccancellieri
force-pushed
the
google-cloud-support
branch
from
43f8f17 to
cbbbef3
Compare
ccancellieri
changed the title
|
@TkTech Just splitted (as you may have seen) and added the patch to the specific google support branch to provide signed private url from google and installation procedure. Tested it works well. How it looks now? Thanks for your comments |
Fix/safe url
update requirements
Co-authored-by: boxdev <[email protected]>
* update generate_signed_url * update requirements.txt * Add Function for Generating Signed URLs with User Impersonation in Google Cloud Storage * Refactor and Extend Cloud Storage Integration for Enhanced Azure, Google Cloud, and Error Handling * Implement Bucket Management and IAM Permission Features for Google Cloud Storage Integration * Implement Command Pattern for Group Management and GCP Bucket Creation * Add Exception Handling for GCP Group Creation, Deletion, and Member Management * Add Functionality for Creating Authorized Sessions with GCP ID Tokens * Implement Authorization Checks for GCP Group and Member Management * Integrate GCP Group Workspace Management with CKAN Organization Actions * Extend CloudStoragePlugin with Organization and Member Management Actions * update bucket path * Add etl feature * Update readme file * Add check_resource_directories * Add Storage dir constant --------- Co-authored-by: boxdev <[email protected]>
* update generate_signed_url * update requirements.txt * Add Function for Generating Signed URLs with User Impersonation in Google Cloud Storage * Refactor and Extend Cloud Storage Integration for Enhanced Azure, Google Cloud, and Error Handling * Implement Bucket Management and IAM Permission Features for Google Cloud Storage Integration * Implement Command Pattern for Group Management and GCP Bucket Creation * Add Exception Handling for GCP Group Creation, Deletion, and Member Management * Add Functionality for Creating Authorized Sessions with GCP ID Tokens * Implement Authorization Checks for GCP Group and Member Management * Integrate GCP Group Workspace Management with CKAN Organization Actions * Extend CloudStoragePlugin with Organization and Member Management Actions * update bucket path * Add etl feature * Update readme file * Add check_resource_directories * Add Storage dir constant * remove ckan api key form constants * Add ckan root path constant * update readme file --------- Co-authored-by: boxdev <[email protected]>
* update readme file * REfactor etl
* retrieve organization members and organization description for single organization * Refactor entire codebase * Update readme file * Add gcp service api * remove hardcoded values
* add better exceptions handling * Refactor group service to manage and retrieve group members
* crete bucket if it does not exit * add iam permission to bucket * update readme file
* crete bucket if it does not exit * add iam permission to bucket * update readme file * fix typo
may fix:
Apologizing mixing both but it 'solve' all the above.
Comments? :)