Upgrade Azure Blob Storage SDK to v12

[pamelafox]

Fixes #2566

This PR upgrades the azure-storage-blob SDK from v2 to v12, which involved a lot of interface changes. I followed the migration guide @ https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/storage/azure-storage-blob/migration_guide.md and was able to get all the previous functionality working, at least according to the tests.

For ease of testing, I added a simple example app and a devcontainer.azure.json which brings in the Azurite local emulator. That means you can open this repo inside a Codespace or Dev Container with that configuration, and Azure Blob Storage will be running for you.

[pamelafox]

There's an admin test that is failing in CI that isn't failing locally, so I'm going to put some debugging in in the next commits to try to figure out what's happening.

[samuelhwilliams]

FYI, it is failing for me locally too 👀 Not that it helps much ... but I can possibly investigate a bit myself and see if I find anything.

[samuelhwilliams]

If it helps:

/Users/sam/work/personal/flask-admin/flask_admin/contrib/fileadmin/azure.py(220)read_file()
-> blob = self._container_client.get_blob_client(path).download_blob()
(Pdb) ll
215         def read_file(self, path):
216             path = self._ensure_blob_path(path)
217             if path is None:
218                 raise ValueError("No path provided")
219             breakpoint()
220  ->         blob = self._container_client.get_blob_client(path).download_blob()
221             return blob.readall()
(Pdb) pp path
'dummy.txt'
(Pdb) n
azure.core.exceptions.DeserializationError: Unable to deserialize response data. Data: undefined, bytearray
> /Users/sam/work/personal/flask-admin/flask_admin/contrib/fileadmin/azure.py(220)read_file()
-> blob = self._container_client.get_blob_client(path).download_blob()

[pamelafox]

Thanks! Might be a statefulness thing, I'll start a fresh env.

[LeXofLeviafan]

...Wouldn't it be better to include actual information in that error variable? I.e. make it an optional string (containing the error message if an error occurred).

The check would work the same, and the error could be logged properly before redirect (...as it probably should be, anyway.)

[pamelafox]

@samuelhwilliams It was an error due to the tests using an older Azure storage emulator, I've updated them to the official Microsoft hosted emulator now, and they're passing fine.
@LeXofLeviafan I agree that it would be nice if the errors were easier to debug. They should probably be logged as either warning or error, depending on whether they're likely user errors or server errors. That would be better in a different PR though, I think. I'm trying to only affect the Azure module for this one.

[pamelafox]

In the future, I could add Postgres and Mongo to this dev container too, to have a single container that can run all the tests. It should be fairly easy given tests.yaml has the services setup, just copying that to docker-compose.yaml.

[pamelafox]

I changed the handling of the importerror to make mypy happier. I looked at other optional modules and they all seemed to handle importerrors a bit differently. I found one that did it this way. This seems fine since you should import the module unless you're using it.

[pamelafox]

Hm, it appears I still don't have an approach for handling unimportable Azure modules that makes mypy happy. Let me know if you have any thoughts about the best practice for importing extra modules in tests (and skipping tests if they don't exist). I'll take another look Monday otherwise.

[samuelhwilliams]

I don't think we need to support tests running without azure installed - I'd probably be fine with you removing the try/except around that test import.

[pamelafox]

Okay, I've made it so that the tests assume you've got azure-blob-storage installed. I've also made the tests devcontainer bring in Postgres and Mongo too, so I was able to get all the tests passing in my dev container environment, without additional setup.

[samuelhwilliams]

One of the changes I made on the s3 admin side when bringing it up to date was to have __init__ take a client instance rather than parameters that get passed the client.

Do you think we should do something similar here and accept an instance of BlobServiceClient, or is it still fine to just use the connection string?

[pamelafox]

Oh I think thats nice, as I personally don't typically use connection strings (this was my first time using from_connection_string), so that gives developers more flexibility as to how they connect. I can make that change. That'd be breaking, right?

[samuelhwilliams]

It would be, but this is scheduled to go out for the v2 release where we're making a bunch of breaking changes, so I'm ok with it.

If you'd be happy to, feel free :) 🙏

[samuelhwilliams]

I'm not sure the hstore extension is being set up correctly for the devcontainer. At least, I just opened vscode for this PR and ran the tests, and it complained that hstore didn't exist. After manually creating it and re-running the tests seem good. Is that something you could check?

[samuelhwilliams]

I also think I'm getting one error in the devcontainer that feels odd: FAILED flask_admin/tests/fileadmin/test_fileadmin.py::TestLocalFileAdmin::test_file_admin - assert 200 == 302 🤔

But clearly it passes in CI, so...

[pamelafox]

@samuelhwilliams I put the hstore setup in the postAttachCommand, but that doesnt' run until really late in the process, so it's possible you ran the tests before that had been run. I can make it more robust by instead bringing in a Dockerfile for the postgis and putting the CREATE EXTENSIOn in there.

@samuelhwilliams Do you get that error when there are no changes locally? Sometimes errors happen with partially aborted tests due to dangling changes to dummy files.
And is that in Codespaces or VS Code Dev Containers locally?

[samuelhwilliams]

Apologies - both of these problems were on my end, my checkout of your latest changes failed so I still had an old version of the PR cached.

After resetting that, unfortunately I get a new error that looks unrelated - not able to install requirements/dev.txt, presumably due to some pinned package incompatibilities. I think that probably deserves to be out-of-scope for fixing in this PR though... After manually changing devcontainer to install requirements/tests.txt instead, which does work, there's still some test failures due to an apparent incompatibility somewhere related to Flask 3.1.0's partitioned cookies.

But I think, yeah, neither of these things are strictly this PR's problem...

[samuelhwilliams]

Running the example app locally, after uploading a file, when I click the file to trigger a download I'm getting an error:

Can you tell if this a fundamental issue or just related to the example app? Feels like it would be good to have the example app working fully in this regard.

Edit: Also one small other thing about the example app - most of the others have a landing page on / that has a link pointing to Flask-Admin. If you could add one of those it would be great 🙏

[pamelafox]

@samuelhwilliams I ran into both install issues. I've been commenting out numpy and manually installing Flask==3.0.0. I wasn't sure whether to tackle those in the PR or not.

[pamelafox]

@samuelhwilliams I'll check on the download error, that sounds like a flow I didn't do. (And perhaps isnt covered by tests, given they're passing). And add a link.

[samuelhwilliams]

@samuelhwilliams I ran into both install issues. I've been commenting out numpy and manually installing Flask==3.0.0. I wasn't sure whether to tackle those in the PR or not.

Ah ok that's reassuring for me at least. I think it's fine to leave it for a separate PR; I'll raise an issue on it...

@samuelhwilliams I'll check on the download error, that sounds like a flow I didn't do. (And perhaps isnt covered by tests, given they're passing). And add a link.

Brill, thanks!

[pamelafox]

The error with download_file looks like this Azurite issue:
Azure/Azurite#656
So I'm actually going to first see if the issue happens with a prod Azure account, and assuming it doesn't, I can shift focus to configuring Azurite correctly (as described in the issue).
That bit of the code was the trickiest to upgrade, so it's possible there's a lingering issue there.

[pamelafox]

Update:

• The download file code was originally generating a SAS URL and redirecting to that URL. To generate a SAS URL securely, you need to use keyless authentication, which is tricky to set up with Azurite. However, I realized we could just use flask's send_file after downloading the blob, since users dont need the URL, they just need the data, so I changed the code to use that. That's what I use for downloading blobs in my work projects.
• I changed the constructor to take in a BlobServiceClient, and put examples of two ways of making the client in the Azure example (one for prod with keyless auth, one for local with connection string).

[pamelafox]

FYI: there are no type annotations for AzureFileAdmin and S3FileAdmin currently. When we add them in, we get a mypy error about "multiple values for keyword storage". I think it's confused by the args/kwargs around it. So I have left off type annotations on AzureFileAdmin for now, though I added them to the storage class in a few places.