Skip to content
This repository has been archived by the owner on Aug 27, 2023. It is now read-only.

GCS error 400. Can't upload packages or create bucket #228

Open
actual-nsnow opened this issue Jan 29, 2020 · 1 comment
Open

GCS error 400. Can't upload packages or create bucket #228

actual-nsnow opened this issue Jan 29, 2020 · 1 comment

Comments

@actual-nsnow
Copy link

actual-nsnow commented Jan 29, 2020 •
edited
Loading

I'm getting error 400s when attempting to create a bucket or, after creating the bucket manually, attempting to upload a python package using python or twine. Using pypicloud-docker in kubernetes behind a nginx proxy. SSL is enabled and valid; I can log into the pypicloud web interface and confirm functionality.

I don't believe this to be an issue with credentials or permissions. To test, I have execed into the container, cat the gcs-sa.json file and saved it to my system, performed a gcloud auth login activate-service-account, and confirmed I can list/create buckets and bucket objects. Also, I get a different error if the bucket exists vs when the bucket doesn't exist. This leads me to believe that the credentials are working and pypicloud can list the buckets and move on if it already exists.

If the bucket does not exist I receive the following error:

google.api_core.exceptions.BadRequest: 400 POST https://storage.googleapis.com/storage/v1/b?project=myorg-engineering: Invalid Value

If the bucket already exists I receive the following error when attempting to upload the package:

google.api_core.exceptions.BadRequest: 400 PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&predefinedAcl=projectPrivate&upload_id=AEnB2UoaxtswaTtbm-WkM6HOAk74BR8o50v0Kn49ZSUuuy4uf1ExI5awrMBvTH35r25Gd31S221Ycs3totMPeITdGrqMgQSpCXKQBSCjAnEE2-dgRdkkr6o: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)

Startup Logs:
These are the logs when the bucket exists. If the bucket doesn't exists, it fails with the above error right after the log entry: Creating GCS client from service account JSON file /etc/pypicloud/gcs-sa.json

[uWSGI] getting INI configuration from /etc/pypicloud/config.ini
*** Starting uWSGI 2.0.18 (64bit) on [Wed Jan 29 17:34:42 2020] ***
compiled with version: 6.4.0 on 02 January 2020 01:20:39
os: Linux-4.14.138+ #1 SMP Tue Sep 3 02:58:08 PDT 2019
nodename: pypicloud-prod-78f9c7f5f4-5txct
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /app
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on 0.0.0.0:8080 fd 4
uwsgi socket 0 bound to TCP address 127.0.0.1:41069 (port auto-assigned) fd 3
Python version: 3.6.8 (default, Jan 30 2019, 23:54:38)  [GCC 6.4.0]
Python main interpreter initialized at 0x55f037928860
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 15 seconds
mapped 1531320 bytes (1495 KB) for 20 cores
*** Operational MODE: preforking ***
Loading paste environment: config:/etc/pypicloud/config.ini
INFO 2020-01-29 17:34:44,610 [pypicloud.storage.gcs] Using GCP project id `myorg-engineering`
INFO 2020-01-29 17:34:44,610 [pypicloud.storage.gcs] Creating GCS client from service account JSON file /etc/pypicloud/gcs-sa.json
INFO 2020-01-29 17:34:44,774 [pypicloud.cache.base] Cache is empty. Rebuilding from storage backend...
INFO 2020-01-29 17:34:44,809 [pypicloud.cache.base] Cache repopulated
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x55f037928860 pid: 1 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 1)
spawned uWSGI worker 1 (pid: 10, cores: 1)
spawned uWSGI worker 2 (pid: 11, cores: 1)
spawned uWSGI worker 3 (pid: 12, cores: 1)
spawned uWSGI worker 4 (pid: 13, cores: 1)
spawned uWSGI worker 5 (pid: 14, cores: 1)
spawned uWSGI worker 6 (pid: 15, cores: 1)
spawned uWSGI worker 7 (pid: 16, cores: 1)
spawned uWSGI worker 8 (pid: 17, cores: 1)
spawned uWSGI worker 9 (pid: 18, cores: 1)
spawned uWSGI worker 10 (pid: 19, cores: 1)
spawned uWSGI worker 11 (pid: 20, cores: 1)
spawned uWSGI worker 12 (pid: 21, cores: 1)
spawned uWSGI worker 13 (pid: 22, cores: 1)
spawned uWSGI worker 14 (pid: 23, cores: 1)
spawned uWSGI worker 15 (pid: 24, cores: 1)
spawned uWSGI worker 16 (pid: 25, cores: 1)
spawned uWSGI worker 17 (pid: 26, cores: 1)
spawned uWSGI worker 18 (pid: 27, cores: 1)
spawned uWSGI worker 19 (pid: 28, cores: 1)
spawned uWSGI worker 20 (pid: 29, cores: 1)
spawned uWSGI http 1 (pid: 30)
[pid: 19|app: 0|req: 1/1] 10.204.16.12 () {30 vars in 356 bytes} [Wed Jan 29 17:34:50 2020] GET / => generated 2121 bytes in 38 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 11|app: 0|req: 1/2] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:35:00 2020] GET / => generated 2121 bytes in 61 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 29|app: 0|req: 1/3] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:35:10 2020] GET / => generated 2121 bytes in 36 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 17|app: 0|req: 1/4] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:35:14 2020] GET / => generated 2121 bytes in 35 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)

Uploading packages:
I have intentionally entered the wrong password for my pypicloud account and I get an auth failure error, so I believe authentication is working when uploading to pypicloud. That I'm receiving a GCS error leads me to believe authentication here isn't the issue.

Using python

$ python3 setup.py sdist upload -r pypicloud
running sdist
running egg_info
writing atlassianAutomation.egg-info/PKG-INFO
writing dependency_links to atlassianAutomation.egg-info/dependency_links.txt
writing top-level names to atlassianAutomation.egg-info/top_level.txt
reading manifest file 'atlassianAutomation.egg-info/SOURCES.txt'
writing manifest file 'atlassianAutomation.egg-info/SOURCES.txt'
running check
creating atlassianAutomation-0.0.11
creating atlassianAutomation-0.0.11/atlassianAutomation
creating atlassianAutomation-0.0.11/atlassianAutomation.egg-info
creating atlassianAutomation-0.0.11/atlassianAutomation/sdk
copying files to atlassianAutomation-0.0.11...
copying README.md -> atlassianAutomation-0.0.11
copying setup.py -> atlassianAutomation-0.0.11
copying atlassianAutomation/__init__.py -> atlassianAutomation-0.0.11/atlassianAutomation
copying atlassianAutomation/config.py -> atlassianAutomation-0.0.11/atlassianAutomation
copying atlassianAutomation/engine.py -> atlassianAutomation-0.0.11/atlassianAutomation
copying atlassianAutomation/exceptions.py -> atlassianAutomation-0.0.11/atlassianAutomation
copying atlassianAutomation.egg-info/PKG-INFO -> atlassianAutomation-0.0.11/atlassianAutomation.egg-info
copying atlassianAutomation.egg-info/SOURCES.txt -> atlassianAutomation-0.0.11/atlassianAutomation.egg-info
copying atlassianAutomation.egg-info/dependency_links.txt -> atlassianAutomation-0.0.11/atlassianAutomation.egg-info
copying atlassianAutomation.egg-info/top_level.txt -> atlassianAutomation-0.0.11/atlassianAutomation.egg-info
copying atlassianAutomation/sdk/__init__.py -> atlassianAutomation-0.0.11/atlassianAutomation/sdk
copying atlassianAutomation/sdk/atlassian.py -> atlassianAutomation-0.0.11/atlassianAutomation/sdk
copying atlassianAutomation/sdk/jira.py -> atlassianAutomation-0.0.11/atlassianAutomation/sdk
copying atlassianAutomation/sdk/utilities.py -> atlassianAutomation-0.0.11/atlassianAutomation/sdk
Writing atlassianAutomation-0.0.11/setup.cfg
Creating tar archive
removing 'atlassianAutomation-0.0.11' (and everything under it)
running upload
Submitting dist/atlassianAutomation-0.0.11.tar.gz to https://pypi.taosengineering.io/simple
Upload failed (500): Internal Server Error
error: Upload failed (500): Internal Server Error

Using Twine:

twine upload dist/* --repository-url https://pypicloud.mydomain.io/simple/ --verbose
Uploading distributions to https://pypicloud.mydomain.io/simple/
Enter your username: admin
Enter your password:
Uploading atlassianAutomation-0.0.11.tar.gz
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.5k/14.5k [00:01<00:00, 13.8kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 1 of 5
Uploading atlassianAutomation-0.0.11.tar.gz
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.5k/14.5k [00:00<00:00, 16.4kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 2 of 5
Uploading atlassianAutomation-0.0.11.tar.gz
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.5k/14.5k [00:00<00:00, 15.3kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 3 of 5
Uploading atlassianAutomation-0.0.11.tar.gz
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.5k/14.5k [00:00<00:00, 18.0kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 4 of 5
Uploading atlassianAutomation-0.0.11.tar.gz
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14.5k/14.5k [00:00<00:00, 17.4kB/s]
Received "500: Internal Server Error" Package upload appears to have failed.  Retry 5 of 5
Content received from server:
<html>
 <head>
  <title>500 Internal Server Error</title>
 </head>
 <body>
  <h1>500 Internal Server Error</h1>
  <br/><br/>
PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&amp;predefinedAcl=projectPrivate&amp;upload_id=AEnB2UoaxtswaTtbm-WkM6HOAk74BR8o50v0Kn49ZSUuuy4uf1ExI5awrMBvTH35r25Gd31S221Ycs3totMPeITdGrqMgQSpCXKQBSCjAnEE2-dgRdkkr6o: (&#x27;Request failed with status code&#x27;, 400, &#x27;Expected one of&#x27;, &lt;HTTPStatus.OK: 200&gt;, 308)


 </body>
</html>
HTTPError: 500 Server Error: Internal Server Error for url: https://pypicloud.mydomain.io/simple/

[pid: 29|app: 0|req: 13/135] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:46:10 2020] GET / => generated 2121 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
ERROR 2020-01-29 17:46:11,144 [pypicloud.views] PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&predefinedAcl=projectPrivate&upload_id=AEnB2UrzhhXQOUGUDfZiMKzPhQjaxUofkMWVFNA2-b8rure2LkxMlQBmq0vr50RnwVZ3eUgigfHmR37OxAWAoukracoUlRxrqt_SDD0Wjvu9A_7GevJ_-zg: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1263, in upload_from_file
    client, file_obj, content_type, size, num_retries, predefined_acl
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1173, in _do_upload
    client, stream, content_type, size, num_retries, predefined_acl
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1120, in _do_resumable_upload
    response = upload.transmit_next_chunk(transport)
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/requests/upload.py", line 427, in transmit_next_chunk
    self._process_response(response, len(payload))
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/_upload.py", line 597, in _process_response
    callback=self._make_invalid,
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/_helpers.py", line 96, in require_status_code
    *status_codes
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pyramid/tweens.py", line 41, in excview_tween
    response = handler(request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/router.py", line 148, in handle_request
    registry, request, context, context_iface, view_name
  File "/usr/local/lib/python3.6/site-packages/pyramid/view.py", line 667, in _call_view
    response = view_callable(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 169, in __call__
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 188, in attr_view
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 214, in predicate_wrapper
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/viewderivers.py", line 436, in rendered_view
    result = view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid_duh/params.py", line 306, in param_twiddler
    return fxn(**scope)
  File "/usr/local/lib/python3.6/site-packages/pypicloud/views/simple.py", line 39, in upload
    summary=summary,
  File "/usr/local/lib/python3.6/site-packages/pypicloud/cache/base.py", line 115, in upload
    self.storage.upload(new_pkg, data)
  File "/usr/local/lib/python3.6/site-packages/pypicloud/storage/gcs.py", line 176, in upload
    blob.upload_from_file(datastream, predefined_acl=self.object_acl)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1267, in upload_from_file
    _raise_from_invalid_response(exc)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 2156, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.BadRequest: 400 PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&predefinedAcl=projectPrivate&upload_id=AEnB2UrzhhXQOUGUDfZiMKzPhQjaxUofkMWVFNA2-b8rure2LkxMlQBmq0vr50RnwVZ3eUgigfHmR37OxAWAoukracoUlRxrqt_SDD0Wjvu9A_7GevJ_-zg: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)
[pid: 29|app: 0|req: 14/136] 10.200.2.49 () {50 vars in 903 bytes} [Wed Jan 29 17:46:10 2020] POST /simple/ => generated 517 bytes in 856 msecs (HTTP/1.1 500) 2 headers in 99 bytes (1 switches on core 0)
ERROR 2020-01-29 17:46:12,056 [pypicloud.views] PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&predefinedAcl=projectPrivate&upload_id=AEnB2UoHvtH-hs1w67dKbWk-oHHO3pG-Ff_nizCFnPy8cSJdrvDI9IIt6MgZ_YM--aek90WWhiIaCxGy-eQbtLUOreAiMQub2GHR-gwrtMVXnEcCHVwXvSg: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1263, in upload_from_file
    client, file_obj, content_type, size, num_retries, predefined_acl
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1173, in _do_upload
    client, stream, content_type, size, num_retries, predefined_acl
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1120, in _do_resumable_upload
    response = upload.transmit_next_chunk(transport)
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/requests/upload.py", line 427, in transmit_next_chunk
    self._process_response(response, len(payload))
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/_upload.py", line 597, in _process_response
    callback=self._make_invalid,
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/_helpers.py", line 96, in require_status_code
    *status_codes
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pyramid/tweens.py", line 41, in excview_tween
    response = handler(request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/router.py", line 148, in handle_request
    registry, request, context, context_iface, view_name
  File "/usr/local/lib/python3.6/site-packages/pyramid/view.py", line 667, in _call_view
    response = view_callable(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 169, in __call__
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 188, in attr_view
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 214, in predicate_wrapper
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/viewderivers.py", line 436, in rendered_view
    result = view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid_duh/params.py", line 306, in param_twiddler
    return fxn(**scope)
  File "/usr/local/lib/python3.6/site-packages/pypicloud/views/simple.py", line 39, in upload
    summary=summary,
  File "/usr/local/lib/python3.6/site-packages/pypicloud/cache/base.py", line 115, in upload
    self.storage.upload(new_pkg, data)
  File "/usr/local/lib/python3.6/site-packages/pypicloud/storage/gcs.py", line 176, in upload
    blob.upload_from_file(datastream, predefined_acl=self.object_acl)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1267, in upload_from_file
    _raise_from_invalid_response(exc)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 2156, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.BadRequest: 400 PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&predefinedAcl=projectPrivate&upload_id=AEnB2UoHvtH-hs1w67dKbWk-oHHO3pG-Ff_nizCFnPy8cSJdrvDI9IIt6MgZ_YM--aek90WWhiIaCxGy-eQbtLUOreAiMQub2GHR-gwrtMVXnEcCHVwXvSg: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)
[pid: 29|app: 0|req: 15/137] 10.200.2.49 () {50 vars in 903 bytes} [Wed Jan 29 17:46:11 2020] POST /simple/ => generated 517 bytes in 832 msecs (HTTP/1.1 500) 2 headers in 99 bytes (1 switches on core 0)
ERROR 2020-01-29 17:46:13,023 [pypicloud.views] PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&predefinedAcl=projectPrivate&upload_id=AEnB2UodbyrdG9kF7vpiQx45Lq81bq3DwCv3Y6i-E9WCm-eeo_Qssl4MAXuRJ4Kg8RslLYQwPg9J60Ktw_jdwqgX6pKp4YmVvw: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1263, in upload_from_file
    client, file_obj, content_type, size, num_retries, predefined_acl
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1173, in _do_upload
    client, stream, content_type, size, num_retries, predefined_acl
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1120, in _do_resumable_upload
    response = upload.transmit_next_chunk(transport)
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/requests/upload.py", line 427, in transmit_next_chunk
    self._process_response(response, len(payload))
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/_upload.py", line 597, in _process_response
    callback=self._make_invalid,
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/_helpers.py", line 96, in require_status_code
    *status_codes
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pyramid/tweens.py", line 41, in excview_tween
    response = handler(request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/router.py", line 148, in handle_request
    registry, request, context, context_iface, view_name
  File "/usr/local/lib/python3.6/site-packages/pyramid/view.py", line 667, in _call_view
    response = view_callable(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 169, in __call__
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 188, in attr_view
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 214, in predicate_wrapper
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/viewderivers.py", line 436, in rendered_view
    result = view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid_duh/params.py", line 306, in param_twiddler
    return fxn(**scope)
  File "/usr/local/lib/python3.6/site-packages/pypicloud/views/simple.py", line 39, in upload
    summary=summary,
  File "/usr/local/lib/python3.6/site-packages/pypicloud/cache/base.py", line 115, in upload
    self.storage.upload(new_pkg, data)
  File "/usr/local/lib/python3.6/site-packages/pypicloud/storage/gcs.py", line 176, in upload
    blob.upload_from_file(datastream, predefined_acl=self.object_acl)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1267, in upload_from_file
    _raise_from_invalid_response(exc)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 2156, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.BadRequest: 400 PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&predefinedAcl=projectPrivate&upload_id=AEnB2UodbyrdG9kF7vpiQx45Lq81bq3DwCv3Y6i-E9WCm-eeo_Qssl4MAXuRJ4Kg8RslLYQwPg9J60Ktw_jdwqgX6pKp4YmVvw: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)
[pid: 27|app: 0|req: 31/138] 10.200.2.49 () {50 vars in 901 bytes} [Wed Jan 29 17:46:12 2020] POST /simple/ => generated 496 bytes in 897 msecs (HTTP/1.1 500) 2 headers in 99 bytes (1 switches on core 0)
ERROR 2020-01-29 17:46:13,850 [pypicloud.views] PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&predefinedAcl=projectPrivate&upload_id=AEnB2UpukB3LHvrX--bbl1iNQNK1Xt1b751lDEeL1lvZ7vdk5aeX2TSTROs466s9MyW_AY09ONA6CWaUtxyHzC8cKqrhiQOBDA: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1263, in upload_from_file
    client, file_obj, content_type, size, num_retries, predefined_acl
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1173, in _do_upload
    client, stream, content_type, size, num_retries, predefined_acl
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1120, in _do_resumable_upload
    response = upload.transmit_next_chunk(transport)
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/requests/upload.py", line 427, in transmit_next_chunk
    self._process_response(response, len(payload))
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/_upload.py", line 597, in _process_response
    callback=self._make_invalid,
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/_helpers.py", line 96, in require_status_code
    *status_codes
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pyramid/tweens.py", line 41, in excview_tween
    response = handler(request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/router.py", line 148, in handle_request
    registry, request, context, context_iface, view_name
  File "/usr/local/lib/python3.6/site-packages/pyramid/view.py", line 667, in _call_view
    response = view_callable(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 169, in __call__
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 188, in attr_view
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 214, in predicate_wrapper
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/viewderivers.py", line 436, in rendered_view
    result = view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid_duh/params.py", line 306, in param_twiddler
    return fxn(**scope)
  File "/usr/local/lib/python3.6/site-packages/pypicloud/views/simple.py", line 39, in upload
    summary=summary,
  File "/usr/local/lib/python3.6/site-packages/pypicloud/cache/base.py", line 115, in upload
    self.storage.upload(new_pkg, data)
  File "/usr/local/lib/python3.6/site-packages/pypicloud/storage/gcs.py", line 176, in upload
    blob.upload_from_file(datastream, predefined_acl=self.object_acl)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1267, in upload_from_file
    _raise_from_invalid_response(exc)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 2156, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.BadRequest: 400 PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&predefinedAcl=projectPrivate&upload_id=AEnB2UpukB3LHvrX--bbl1iNQNK1Xt1b751lDEeL1lvZ7vdk5aeX2TSTROs466s9MyW_AY09ONA6CWaUtxyHzC8cKqrhiQOBDA: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)
[pid: 29|app: 0|req: 16/139] 10.200.2.49 () {50 vars in 902 bytes} [Wed Jan 29 17:46:13 2020] POST /simple/ => generated 496 bytes in 749 msecs (HTTP/1.1 500) 2 headers in 99 bytes (1 switches on core 0)
[pid: 27|app: 0|req: 32/140] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:46:14 2020] GET / => generated 2121 bytes in 3 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
ERROR 2020-01-29 17:46:14,707 [pypicloud.views] PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&predefinedAcl=projectPrivate&upload_id=AEnB2UoaxtswaTtbm-WkM6HOAk74BR8o50v0Kn49ZSUuuy4uf1ExI5awrMBvTH35r25Gd31S221Ycs3totMPeITdGrqMgQSpCXKQBSCjAnEE2-dgRdkkr6o: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1263, in upload_from_file
    client, file_obj, content_type, size, num_retries, predefined_acl
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1173, in _do_upload
    client, stream, content_type, size, num_retries, predefined_acl
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1120, in _do_resumable_upload
    response = upload.transmit_next_chunk(transport)
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/requests/upload.py", line 427, in transmit_next_chunk
    self._process_response(response, len(payload))
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/_upload.py", line 597, in _process_response
    callback=self._make_invalid,
  File "/usr/local/lib/python3.6/site-packages/google/resumable_media/_helpers.py", line 96, in require_status_code
    *status_codes
google.resumable_media.common.InvalidResponse: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pyramid/tweens.py", line 41, in excview_tween
    response = handler(request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/router.py", line 148, in handle_request
    registry, request, context, context_iface, view_name
  File "/usr/local/lib/python3.6/site-packages/pyramid/view.py", line 667, in _call_view
    response = view_callable(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 169, in __call__
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 188, in attr_view
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/config/views.py", line 214, in predicate_wrapper
    return view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid/viewderivers.py", line 436, in rendered_view
    result = view(context, request)
  File "/usr/local/lib/python3.6/site-packages/pyramid_duh/params.py", line 306, in param_twiddler
    return fxn(**scope)
  File "/usr/local/lib/python3.6/site-packages/pypicloud/views/simple.py", line 39, in upload
    summary=summary,
  File "/usr/local/lib/python3.6/site-packages/pypicloud/cache/base.py", line 115, in upload
    self.storage.upload(new_pkg, data)
  File "/usr/local/lib/python3.6/site-packages/pypicloud/storage/gcs.py", line 176, in upload
    blob.upload_from_file(datastream, predefined_acl=self.object_acl)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1267, in upload_from_file
    _raise_from_invalid_response(exc)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 2156, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
google.api_core.exceptions.BadRequest: 400 PUT https://storage.googleapis.com/upload/storage/v1/b/myorg-pypicloud/o?uploadType=resumable&predefinedAcl=projectPrivate&upload_id=AEnB2UoaxtswaTtbm-WkM6HOAk74BR8o50v0Kn49ZSUuuy4uf1ExI5awrMBvTH35r25Gd31S221Ycs3totMPeITdGrqMgQSpCXKQBSCjAnEE2-dgRdkkr6o: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, 308)
[pid: 29|app: 0|req: 17/141] 10.200.2.49 () {50 vars in 902 bytes} [Wed Jan 29 17:46:13 2020] POST /simple/ => generated 517 bytes in 783 msecs (HTTP/1.1 500) 2 headers in 99 bytes (1 switches on core 0)
[pid: 27|app: 0|req: 33/142] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:46:20 2020] GET / => generated 2121 bytes in 7 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 27|app: 0|req: 34/143] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:46:24 2020] GET / => generated 2121 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 22|app: 0|req: 39/144] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:46:30 2020] GET / => generated 2121 bytes in 3 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 22|app: 0|req: 40/145] 10.204.16.12 () {30 vars in 356 bytes} [Wed Jan 29 17:46:34 2020] GET / => generated 2121 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 22|app: 0|req: 41/146] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:46:40 2020] GET / => generated 2121 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 22|app: 0|req: 42/147] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:46:44 2020] GET / => generated 2121 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 22|app: 0|req: 43/148] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:46:50 2020] GET / => generated 2121 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 22|app: 0|req: 44/149] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:46:54 2020] GET / => generated 2121 bytes in 3 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 22|app: 0|req: 45/150] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:47:00 2020] GET / => generated 2121 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 16|app: 0|req: 5/151] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:47:04 2020] GET / => generated 2121 bytes in 5 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 17|app: 0|req: 18/152] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:47:10 2020] GET / => generated 2121 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)
[pid: 11|app: 0|req: 8/153] 10.204.16.12 () {30 vars in 357 bytes} [Wed Jan 29 17:47:14 2020] GET / => generated 2121 bytes in 2 msecs (HTTP/1.1 200) 2 headers in 81 bytes (1 switches on core 0)

Config file:

    [app:main]
    use = egg:pypicloud

    pyramid.reload_templates = False
    pyramid.debug_authorization = false
    pyramid.debug_notfound = false
    pyramid.debug_routematch = false
    pyramid.default_locale_name = en

    pypi.fallback = cache
    pypi.always_show_upstream =
    pypi.fallback_base_url = example.io                                                                                                                                                                                                          
    pypi.default_read = everyone                                                                                                                                                                                                                         
    pypi.default_write = authenticated                                                                                                                                                                                                                   
    pypi.cache_update = authenticated                                                                                                                                                                                                                    
    pypi.allow_overwrite =                                                                                                                                                                                                                               
    pypi.realm =                                                                                                                                                                                                                                         
    pypi.download_url =                                                                                                                                                                                                                                  
    pypi.stream_files =                                                                                                                                                                                                                                  
    pypi.package_max_age = 0


    pypi.storage = gcs
    storage.bucket = myorg-pypicloud

    storage.region_name =
    storage.gcp_service_account_json_filename = /etc/pypicloud/gcs-sa.json
    storage.gcp_project_id = myorg-engineering
    storage.prefix =
    storage.prepend_hash =
    storage.expire_after = 86400
    storage.redirect_urls = true
    storage.object_acl = project-private
    storage.storage_class = STANDARD


    pypi.auth = pypicloud.access.ConfigAccessBackend

    db.url = sqlite:////var/lib/pypicloud/db.sqlite
    db.graceful_reload =

    # List of admin names (space delimited)
    auth.admins = admin

    user.admin = $6$rounds=535000$FgJbxBA0Styh/l/X$QHNJCKlxswyvSzAkUzYIid1Hs.E3OfjGYDqAeqxP9hSdk8A1psg.BFXECR8aZ3eowpFUuN/v.8AD1zSkL6zs7.

    # For beaker
    ssession.encrypt_key = z9XbIqpxK62qZsZY70N8dItVRtyOAtY6ri73bz5ySTY=
    session.validate_key = xZwWw9L6m960EjN7AXdGaLCY3eCQvpCAJYqqW5eg46I=
    session.secure = false
    session.invalidate_corrupt = true

    ###
    # wsgi server configuration
    ###

    [uwsgi]
    paste = config:%p
    paste-logger = %p
    master = true
    processes = 20
    reload-mercy = 15
    worker-reload-mercy = 15
    max-requests = 1000
    enable-threads = true
    http = 0.0.0.0:8080
    uid = pypicloud
    gid = pypicloud

    ###
    # logging configuration
    # http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/logging.html
    ###

    [loggers]
    keys = root, botocore, pypicloud

    [handlers]
    keys = console

    [formatters]
    keys = generic

    [logger_root]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
    level = INFO                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
    handlers = console

    [logger_pypicloud]
    level = DEBUG
    qualname = pypicloud
    handlers =

    [logger_botocore]
    level = WARN
    qualname = botocore
    handlers =

    [handler_console]
    class = StreamHandler
    args = (sys.stderr,)
    level = NOTSET
    formatter = generic                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         

    [formatter_generic]
    format = %(levelname)s %(asctime)s [%(name)s] %(message)s
@stevearc
Copy link
Owner

This sounds to me like there's something going wrong with the GCS client, or how that library is being called. I haven't heard other reports of this, nor have I used GCS myself. One thing you could do to further debug this is try using the python library directly and see if you can find a way to successfully create the bucket, then see what is different about how pypicloud is doing that call. You can find it here:

pypicloud/pypicloud/storage/gcs.py

Lines 115 to 124 in 290c09e

client = cls._get_storage_client(settings)
bucket = client.bucket(bucket_name)
if not bucket.exists():
bucket.location = settings.get("storage.region_name")
LOG.info(
"Creating GCS bucket %s in location %s", bucket_name, bucket.location
)
bucket.create()

As a shot in the dark, you could also try the recently-added IAM signer functionality to see if that changes anything: #226

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stevearc @actual-nsnow