-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
.msg file upload #4961
base: master
Are you sure you want to change the base?
.msg file upload #4961
Changes from all commits
26986fd
f9c378a
c11860c
0ef0b1b
061dce1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
@@ -56,7 +56,8 @@ def __init__(self, allowed_mime_types: Iterable[str] | None = None): | |||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
def __call__(self, value: UploadedFile) -> None: | ||||||||||||||||||||||||||||||
head = value.read(2048) | ||||||||||||||||||||||||||||||
ext = value.name.split(".")[-1] | ||||||||||||||||||||||||||||||
file_name_parts = value.name.split(".") | ||||||||||||||||||||||||||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this way of grabbing the extension is actually quite flawed (it already was): try for example a file name Instead, what is more correct to use is from pathlib import Path
ext = Path(value.name).suffix[1:] which gives you the actual extension (without dot) |
||||||||||||||||||||||||||||||
ext = file_name_parts[-1] | ||||||||||||||||||||||||||||||
mime_type = magic.from_buffer(head, mime=True) | ||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
# gh #2520 | ||||||||||||||||||||||||||||||
|
@@ -76,6 +77,14 @@ def __call__(self, value: UploadedFile) -> None: | |||||||||||||||||||||||||||||
_("The provided file is not a valid file type.") | ||||||||||||||||||||||||||||||
) | ||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
if len(file_name_parts) == 1: | ||||||||||||||||||||||||||||||
raise serializers.ValidationError( | ||||||||||||||||||||||||||||||
_( | ||||||||||||||||||||||||||||||
"Could not determine the file type. Please make sure the file name " | ||||||||||||||||||||||||||||||
"has an extension." | ||||||||||||||||||||||||||||||
) | ||||||||||||||||||||||||||||||
) | ||||||||||||||||||||||||||||||
Comment on lines
+80
to
+86
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
this |
||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
# Contents is allowed. Do extension or submitted content_type agree? | ||||||||||||||||||||||||||||||
if value.content_type == "application/octet-stream": | ||||||||||||||||||||||||||||||
m = magic.Magic(extension=True) | ||||||||||||||||||||||||||||||
|
@@ -111,6 +120,11 @@ def __call__(self, value: UploadedFile) -> None: | |||||||||||||||||||||||||||||
"image/heif", | ||||||||||||||||||||||||||||||
): | ||||||||||||||||||||||||||||||
return | ||||||||||||||||||||||||||||||
# 4795 | ||||||||||||||||||||||||||||||
# The sdk cannot determine the file type of .msg files, which result into | ||||||||||||||||||||||||||||||
# content_type "". So we have to validate these for ourselves | ||||||||||||||||||||||||||||||
elif mime_type == "application/vnd.ms-outlook" and ext == "msg": | ||||||||||||||||||||||||||||||
return | ||||||||||||||||||||||||||||||
Comment on lines
+123
to
+127
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. rather than adding another hardcoded check here, we can do better, but we need to restructure the validator code a bit! First, start with figuring out the provided content type: provided_content_type = value.content_type or "application/octet-stream" and replace all The if value.content_type == "application/octet-stream": ... the extensions for the binary data are looked up and checked against the provided file extension. So, we invert the process here - given the binary data, what should the extension be, and validate this. |
||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
# gh #4658 | ||||||||||||||||||||||||||||||
# Windows use application/x-zip-compressed as a mimetype for .zip files, which | ||||||||||||||||||||||||||||||
|
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -338,14 +338,7 @@ class FileSerializer(serializers.Serializer): | |||||
originalName = serializers.CharField(trim_whitespace=False) | ||||||
size = serializers.IntegerField(min_value=0) | ||||||
storage = serializers.ChoiceField(choices=["url"]) | ||||||
type = serializers.CharField( | ||||||
error_messages={ | ||||||
"blank": _( | ||||||
"Could not determine the file type. Please make sure the file name " | ||||||
"has an extension." | ||||||
), | ||||||
} | ||||||
) | ||||||
type = serializers.CharField(required=False, allow_blank=True) | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
I would definitely still require the key to be present, but we can accept empty strings and process them during validation indeed. |
||||||
url = serializers.URLField() | ||||||
data = FileDataSerializer() # type: ignore | ||||||
|
||||||
|
Original file line number | Diff line number | Diff line change | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
@@ -202,6 +202,22 @@ def test_allowed_mime_types_for_csv_files(self): | |||||||||||||||||||
|
||||||||||||||||||||
validator(sample) | ||||||||||||||||||||
|
||||||||||||||||||||
def test_allowed_mime_types_for_msg_files(self): | ||||||||||||||||||||
valid_type = "application/vnd.ms-outlook" | ||||||||||||||||||||
msg_file = TEST_FILES / "test.msg" | ||||||||||||||||||||
validator = validators.MimeTypeValidator(allowed_mime_types=[valid_type]) | ||||||||||||||||||||
|
||||||||||||||||||||
# 4795 | ||||||||||||||||||||
# The sdk cannot determine the content_type for .msg files correctly. | ||||||||||||||||||||
# Because .msg is a windows specific file, and linux and MacOS don't know it. | ||||||||||||||||||||
# So we simulate the scenario where content_type is unknown | ||||||||||||||||||||
sample = SimpleUploadedFile( | ||||||||||||||||||||
"test.msg", | ||||||||||||||||||||
msg_file.read_bytes(), | ||||||||||||||||||||
) | ||||||||||||||||||||
Comment on lines
+214
to
+217
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This results in the file having
Suggested change
|
||||||||||||||||||||
|
||||||||||||||||||||
validator(sample) | ||||||||||||||||||||
|
||||||||||||||||||||
def test_validate_files_multiple_mime_types(self): | ||||||||||||||||||||
"""Assert that validation of files associated with multiple mime types works | ||||||||||||||||||||
|
||||||||||||||||||||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -940,7 +940,7 @@ def test_unknown_file_type(self): | |
# The frontend validation will *not* create a TemporaryFileUpload, | ||
# as the frontend will block the upload because of the invalid file type. | ||
# However the user could do an handcrafted API call. | ||
# For this reason, we manually create an invalid TemporaryFileUpload | ||
# For this reason, we manually try to create an invalid TemporaryFileUpload | ||
# and use it for the `api_value`: | ||
|
||
with open(TEST_FILES / "unknown-type", "rb") as infile: | ||
|
@@ -966,8 +966,8 @@ def test_unknown_file_type(self): | |
], | ||
) | ||
|
||
# Make sure the frontend did not create one: | ||
self.assertEqual(TemporaryFileUpload.objects.count(), 1) | ||
# Make sure that no temporary files were created | ||
self.assertEqual(TemporaryFileUpload.objects.count(), 0) | ||
Comment on lines
+969
to
+970
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't really understand what changed here? Why was there initially a temporary file upload and now not anymore? |
||
|
||
|
||
class SingleAddressNLTests(ValidationsTestCase): | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please exclude backend translations from PRs? They get sorted out during the release preparation.
They're a frequent source of merge conflicts otherwise.