Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add extra data option to add_source #83

Merged
merged 6 commits into from
May 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -47,3 +47,5 @@ nosetests.xml
.cache/

.pytest_cache/

tmp
2 changes: 2 additions & 0 deletions src/pycrunch/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,8 @@

Session = elements.ElementSession

UnsafeSession = elements.UnsafeElementSession

__all__ = [
'cubes',
'elements',
Expand Down
30 changes: 30 additions & 0 deletions src/pycrunch/elements.py
Original file line number Diff line number Diff line change
Expand Up @@ -360,3 +360,33 @@ def password(self):
"`session.password` is being deprecated.", PendingDeprecationWarning
)
return self.__password


class UnsafeElementSession(ElementSession):
"""
A subclass of ElementSession that disables SSL/TLS verification.
This is mean to be used for development purposes only.
"""

def __init__(
self,
email=None,
password=None,
token=None,
site_url=None,
progress_tracking=None,
):
super(UnsafeElementSession, self).__init__(
email,
password,
token,
site_url,
progress_tracking,
)

# Skip SSL/TLS verification
self.verify = False

if self.token:
# Use Bearer token for authentication
self.headers.update({"Authorization": "Bearer {}".format(self.token)})
6 changes: 3 additions & 3 deletions src/pycrunch/importing.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

import six

from pycrunch import shoji, csvlib
from pycrunch import csvlib, shoji


class Importer(object):
Expand Down Expand Up @@ -53,13 +53,13 @@ def wait_for_batch_status(self, batch, status):
raise ValueError("The batch did not reach the '%s' state in the "
"given time. Please check again later." % status)

def add_source(self, ds, filename, fp, mimetype):
def add_source(self, ds, filename, fp, mimetype, data=None):
"""Create a new Source on the given dataset and return its URL."""
sources_url = ds.user_url.catalogs['sources']
# Don't call Catalog.post here (which would force application/json);
# we want requests.Session to set multipart/form-data with a boundary.
new_source_url = ds.session.post(
sources_url, files={"uploaded_file": (filename, fp, mimetype)}
sources_url, files={"uploaded_file": (filename, fp, mimetype)}, data=data
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this data argument? The POST method on the sources controller does not specify such argument.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sources controller now expects data in the POST body which is where we are sending the required attributes
for the crunchlake source.

data is not a sources controller specific attribute. It's request body data for the POST request.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, got it, this is not a data query param, but the request data

).headers["Location"]

if self.strict is not None:
Expand Down
2 changes: 1 addition & 1 deletion src/pycrunch/version.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
__version__ = '0.5.7'
__version__ = '0.5.8'
Loading