Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deployment of Python app fails because of usage of --no-dev parameter #2557

Open
MaartenUreel opened this issue Jan 5, 2025 · 23 comments · Fixed by #2561
Open

Deployment of Python app fails because of usage of --no-dev parameter #2557

MaartenUreel opened this issue Jan 5, 2025 · 23 comments · Fixed by #2561

Comments

@MaartenUreel
Copy link

MaartenUreel commented Jan 5, 2025 •
edited
Loading

Bug Report

Since a few hours, a deployment of a Python 3.12 app on Azure Web App fails.

On a working deployment, we saw this warning:

Running poetry install...
The `--no-dev` option is deprecated, use the `--only main` notation instead.
Installing dependencies from lock file

Now since the last deploy 24 hours ago, we see this message:

Running poetry install...
The option "--no-dev" does not exist

And the deployment fails.

It seems on working deployment, this version was pulled:

Running pip install poetry...
Collecting poetry
  Using cached poetry-1.8.5-py3-none-any.whl.metadata (6.9 kB)

While now it is this version:

Collecting poetry
  Using cached poetry-2.0.0-py3-none-any.whl.metadata (7.0 kB)

But the version of Poetry to use is not something I chose, it is part of the build process if I am not mistaken.
--no-dev is remove in version 2, so either the parameter needs to be replaced with the correct replacement or the version needs to be pinned to <2.
@MaartenUreel
Copy link
Author

I have worked around this on my own project by adding this to my pre-build script so that we do not rely on the built in Poetry handling:

# Temporary fix for Poetry, export requirements.txt
curl -sSL https://install.python-poetry.org | python3 -
/home/.local/bin/poetry self add poetry-plugin-export
/home/.local/bin/poetry export -f requirements.txt --output requirements.txt

@lloyd-lighthouse
Copy link

lloyd-lighthouse commented Jan 5, 2025 •
edited
Loading

im having the same issue.
where do you add this 'pre-build' script to?

Poetry was updated recently - https://github.com/python-poetry/poetry/blob/main/CHANGELOG.md#200---2025-01-05
My oryx build fails same as yours above.

Things I've tried

Adding a pre build command as an environment variable, that will install the poetry version that should work with Oryx.

PRE_BUILD_COMMAND curl -sSL https://install.python-poetry.org | POETRY_VERSION=1.8.5 python3 - && export PATH="/home/.local/bin:$PATH"

the log, you can clearly see the pre build command running and executing, however, oryx still seems to disregard it and installs its own version of poetry with Running pip install poetry...

Command: oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.12 -p virtualenv_name=antenv --log-file /tmp/build-debug.log  -i /tmp/8dd2ded15d6f615 --compress-destination-dir | tee /tmp/oryx-build.log
Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
You can report issues at https://github.com/Microsoft/Oryx/issues

Oryx Version: 0.2.20240424.1, Commit: d37b2225a252ab2c04b4726024d047cf01ea1318, ReleaseTagName: 20240424.1

Build Operation ID: 8b0ecd53c1488d14
Repository Commit : 1ecb1a44-c4e3-47fe-9876-47f0abb332eb
OS Type           : bullseye
Image Type        : githubactions

Detecting platforms...
Detected following platforms:
  python: 3.12.6
Version '3.12.6' of platform 'python' is not installed. Generating script to install it...

Using intermediate directory '/tmp/8dd2ded15d6f615'.

Copying files to the intermediate directory...
Done in 3 sec(s).

Source directory     : /tmp/8dd2ded15d6f615
Destination directory: /home/site/wwwroot


Downloading and extracting 'python' version '3.12.6' to '/tmp/oryx/platforms/python/3.12.6'...
Detected image debian flavor: bullseye.
Downloaded in 5 sec(s).
Verifying checksum...
Extracting contents...
performing sha512 checksum for: python...
Done in 29 sec(s).

image detector file exists, platform is python..
OS detector file exists, OS is bullseye..
Executing pre-build command...
Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/home/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.8.5)
Installing Poetry (1.8.5): Saving existing environment
Installing Poetry (1.8.5): Creating environment
Installing Poetry (1.8.5): Installing Poetry
Installing Poetry (1.8.5): Creating script
Installing Poetry (1.8.5): Done

Poetry (1.8.5) is installed now. Great!

To get started you need Poetry's bin directory (/home/.local/bin) in your `PATH`
environment variable.

Add `export PATH="/home/.local/bin:$PATH"` to your shell configuration file.

Alternatively, you can call Poetry explicitly with `/home/.local/bin/poetry`.

You can test that everything is set up by executing:

`poetry --version`

Finished executing pre-build command.
Python Version: /tmp/oryx/platforms/python/3.12.6/bin/python3.12
Creating directory for command manifest file if it does not exist
Removing existing manifest file
Python Virtual Environment: antenv
Creating virtual environment...
Activating virtual environment...
Running pip install poetry...
Collecting poetry
  Using cached poetry-2.0.0-py3-none-any.whl.metadata (7.0 kB)
Collecting build<2.0.0,>=1.2.1 (from poetry)
  Using cached build-1.2.2.post1-py3-none-any.whl.metadata (6.5 kB)
Collecting cachecontrol<0.15.0,>=0.14.0 (from cachecontrol[filecache]<0.15.0,>=0.14.0->poetry)
  Using cached cachecontrol-0.14.1-py3-none-any.whl.metadata (3.1 kB)
Collecting cleo<3.0.0,>=2.1.0 (from poetry)
  Using cached cleo-2.1.0-py3-none-any.whl.metadata (12 kB)
Collecting dulwich<0.23.0,>=0.22.6 (from poetry)
  Using cached dulwich-0.22.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.4 kB)
Collecting fastjsonschema<3.0.0,>=2.18.0 (from poetry)
  Using cached fastjsonschema-2.21.1-py3-none-any.whl.metadata (2.2 kB)
Collecting installer<0.8.0,>=0.7.0 (from poetry)
  Using cached installer-0.7.0-py3-none-any.whl.metadata (936 bytes)
Collecting keyring<26.0.0,>=25.1.0 (from poetry)
  Using cached keyring-25.6.0-py3-none-any.whl.metadata (20 kB)
Collecting packaging>=24.0 (from poetry)
  Using cached packaging-24.2-py3-none-any.whl.metadata (3.2 kB)
Collecting pkginfo<2.0,>=1.12 (from poetry)
  Using cached pkginfo-1.12.0-py3-none-any.whl.metadata (12 kB)
Collecting platformdirs<5,>=3.0.0 (from poetry)
  Using cached platformdirs-4.3.6-py3-none-any.whl.metadata (11 kB)
Collecting poetry-core==2.0.0 (from poetry)
  Using cached poetry_core-2.0.0-py3-none-any.whl.metadata (3.4 kB)
Collecting pyproject-hooks<2.0.0,>=1.0.0 (from poetry)
  Using cached pyproject_hooks-1.2.0-py3-none-any.whl.metadata (1.3 kB)
Collecting requests<3.0,>=2.26 (from poetry)
  Using cached requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting requests-toolbelt<2.0.0,>=1.0.0 (from poetry)
  Using cached requests_toolbelt-1.0.0-py2.py3-none-any.whl.metadata (14 kB)
Collecting shellingham<2.0,>=1.5 (from poetry)
  Using cached shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting tomlkit<1.0.0,>=0.11.4 (from poetry)
  Using cached tomlkit-0.13.2-py3-none-any.whl.metadata (2.7 kB)
Collecting trove-classifiers>=2022.5.19 (from poetry)
  Using cached trove_classifiers-2024.10.21.16-py3-none-any.whl.metadata (2.2 kB)
Collecting virtualenv<21.0.0,>=20.26.6 (from poetry)
  Using cached virtualenv-20.28.1-py3-none-any.whl.metadata (4.5 kB)
Collecting msgpack<2.0.0,>=0.5.2 (from cachecontrol<0.15.0,>=0.14.0->cachecontrol[filecache]<0.15.0,>=0.14.0->poetry)
  Using cached msgpack-1.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.4 kB)
Collecting filelock>=3.8.0 (from cachecontrol[filecache]<0.15.0,>=0.14.0->poetry)
  Using cached filelock-3.16.1-py3-none-any.whl.metadata (2.9 kB)
Collecting crashtest<0.5.0,>=0.4.1 (from cleo<3.0.0,>=2.1.0->poetry)
  Using cached crashtest-0.4.1-py3-none-any.whl.metadata (1.1 kB)
Collecting rapidfuzz<4.0.0,>=3.0.0 (from cleo<3.0.0,>=2.1.0->poetry)
  Using cached rapidfuzz-3.11.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
Collecting urllib3>=1.25 (from dulwich<0.23.0,>=0.22.6->poetry)
  Using cached urllib3-2.3.0-py3-none-any.whl.metadata (6.5 kB)
Collecting SecretStorage>=3.2 (from keyring<26.0.0,>=25.1.0->poetry)
  Using cached SecretStorage-3.3.3-py3-none-any.whl.metadata (4.0 kB)
Collecting jeepney>=0.4.2 (from keyring<26.0.0,>=25.1.0->poetry)
  Using cached jeepney-0.8.0-py3-none-any.whl.metadata (1.3 kB)
Collecting jaraco.classes (from keyring<26.0.0,>=25.1.0->poetry)
  Using cached jaraco.classes-3.4.0-py3-none-any.whl.metadata (2.6 kB)
Collecting jaraco.functools (from keyring<26.0.0,>=25.1.0->poetry)
  Using cached jaraco.functools-4.1.0-py3-none-any.whl.metadata (2.9 kB)
Collecting jaraco.context (from keyring<26.0.0,>=25.1.0->poetry)
  Using cached jaraco.context-6.0.1-py3-none-any.whl.metadata (4.1 kB)
Collecting charset-normalizer<4,>=2 (from requests<3.0,>=2.26->poetry)
  Using cached charset_normalizer-3.4.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (35 kB)
Collecting idna<4,>=2.5 (from requests<3.0,>=2.26->poetry)
  Using cached idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting certifi>=2017.4.17 (from requests<3.0,>=2.26->poetry)
  Using cached certifi-2024.12.14-py3-none-any.whl.metadata (2.3 kB)
Collecting distlib<1,>=0.3.7 (from virtualenv<21.0.0,>=20.26.6->poetry)
  Using cached distlib-0.3.9-py2.py3-none-any.whl.metadata (5.2 kB)
Collecting cryptography>=2.0 (from SecretStorage>=3.2->keyring<26.0.0,>=25.1.0->poetry)
  Using cached cryptography-44.0.0-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (5.7 kB)
Collecting more-itertools (from jaraco.classes->keyring<26.0.0,>=25.1.0->poetry)
  Using cached more_itertools-10.5.0-py3-none-any.whl.metadata (36 kB)
Collecting cffi>=1.12 (from cryptography>=2.0->SecretStorage>=3.2->keyring<26.0.0,>=25.1.0->poetry)
  Using cached cffi-1.17.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting pycparser (from cffi>=1.12->cryptography>=2.0->SecretStorage>=3.2->keyring<26.0.0,>=25.1.0->poetry)
  Using cached pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Using cached poetry-2.0.0-py3-none-any.whl (253 kB)
Using cached poetry_core-2.0.0-py3-none-any.whl (544 kB)
Using cached build-1.2.2.post1-py3-none-any.whl (22 kB)
Using cached cachecontrol-0.14.1-py3-none-any.whl (22 kB)
Using cached cleo-2.1.0-py3-none-any.whl (78 kB)
Using cached dulwich-0.22.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (976 kB)
Using cached fastjsonschema-2.21.1-py3-none-any.whl (23 kB)
Using cached installer-0.7.0-py3-none-any.whl (453 kB)
Using cached keyring-25.6.0-py3-none-any.whl (39 kB)
Using cached packaging-24.2-py3-none-any.whl (65 kB)
Using cached pkginfo-1.12.0-py3-none-any.whl (32 kB)
Using cached platformdirs-4.3.6-py3-none-any.whl (18 kB)
Using cached pyproject_hooks-1.2.0-py3-none-any.whl (10 kB)
Using cached requests-2.32.3-py3-none-any.whl (64 kB)
Using cached requests_toolbelt-1.0.0-py2.py3-none-any.whl (54 kB)
Using cached shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Using cached tomlkit-0.13.2-py3-none-any.whl (37 kB)
Using cached trove_classifiers-2024.10.21.16-py3-none-any.whl (13 kB)
Using cached virtualenv-20.28.1-py3-none-any.whl (4.3 MB)
Using cached certifi-2024.12.14-py3-none-any.whl (164 kB)
Using cached charset_normalizer-3.4.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (145 kB)
Using cached crashtest-0.4.1-py3-none-any.whl (7.6 kB)
Using cached distlib-0.3.9-py2.py3-none-any.whl (468 kB)
Using cached filelock-3.16.1-py3-none-any.whl (16 kB)
Using cached idna-3.10-py3-none-any.whl (70 kB)
Using cached jeepney-0.8.0-py3-none-any.whl (48 kB)
Using cached msgpack-1.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (401 kB)
Using cached rapidfuzz-3.11.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Using cached SecretStorage-3.3.3-py3-none-any.whl (15 kB)
Using cached urllib3-2.3.0-py3-none-any.whl (128 kB)
Using cached jaraco.classes-3.4.0-py3-none-any.whl (6.8 kB)
Using cached jaraco.context-6.0.1-py3-none-any.whl (6.8 kB)
Using cached jaraco.functools-4.1.0-py3-none-any.whl (10 kB)
Using cached cryptography-44.0.0-cp39-abi3-manylinux_2_28_x86_64.whl (4.2 MB)
Using cached more_itertools-10.5.0-py3-none-any.whl (60 kB)
Using cached cffi-1.17.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (479 kB)
Using cached pycparser-2.22-py3-none-any.whl (117 kB)
Installing collected packages: trove-classifiers, fastjsonschema, distlib, urllib3, tomlkit, shellingham, rapidfuzz, pyproject-hooks, pycparser, poetry-core, platformdirs, pkginfo, packaging, msgpack, more-itertools, jeepney, jaraco.context, installer, idna, filelock, crashtest, charset-normalizer, certifi, virtualenv, requests, jaraco.functools, jaraco.classes, dulwich, cleo, cffi, build, requests-toolbelt, cryptography, cachecontrol, SecretStorage, keyring, poetry
Successfully installed SecretStorage-3.3.3 build-1.2.2.post1 cachecontrol-0.14.1 certifi-2024.12.14 cffi-1.17.1 charset-normalizer-3.4.1 cleo-2.1.0 crashtest-0.4.1 cryptography-44.0.0 distlib-0.3.9 dulwich-0.22.7 fastjsonschema-2.21.1 filelock-3.16.1 idna-3.10 installer-0.7.0 jaraco.classes-3.4.0 jaraco.context-6.0.1 jaraco.functools-4.1.0 jeepney-0.8.0 keyring-25.6.0 more-itertools-10.5.0 msgpack-1.1.0 packaging-24.2 pkginfo-1.12.0 platformdirs-4.3.6 poetry-2.0.0 poetry-core-2.0.0 pycparser-2.22 pyproject-hooks-1.2.0 rapidfuzz-3.11.0 requests-2.32.3 requests-toolbelt-1.0.0 shellingham-1.5.4 tomlkit-0.13.2 trove-classifiers-2024.10.21.16 urllib3-2.3.0 virtualenv-20.28.1

[notice] A new release of pip is available: 24.2 -> 24.3.1
[notice] To update, run: pip install --upgrade pip
Running poetry install...

The option "--no-dev" does not exist
"2025-01-06 01:02:29"|WARNING|
The option "--no-dev" does not exist | Exit code: {pythonBuildExitCode} | Please review message | More information: https://aka.ms/troubleshoot-python
[notice] A new release of pip is available: 24.2 -> 24.3.1\n[notice] To update, run: pip install --upgrade pip\n/bin/bash -c "oryx build /tmp/zipdeploy/extracted -o /home/site/wwwroot --platform python --platform-version 3.12 -p virtualenv_name=antenv --log-file /tmp/build-debug.log  -i /tmp/8dd2ded15d6f615 --compress-destination-dir | tee /tmp/oryx-build.log ; exit $PIPESTATUS "

@lloyd-lighthouse
Copy link

Update:

I spent hours doing all sorts ....
I believe that this is impossible to fix from our side, until Microsoft resolve the issue themselves and force Oryx to either

a) Use Poetry 1.8.5
b) Remove anything in their script that refers to --no-dev

I spent a few hours waiting for someone from Azure Support to contact me, and when they finally did, I spent a bunch of time troubleshooting and doing pretty much the same things I'd tried already.

Only when we had exhausted all options, did the Support Admin take a breath and listen to what I thought was wrong.
As of now, there is a promise to 'coordinate with the internal team to ensure everything is aligned'.

My Interim Solution:

To remove anything to do with poetry from my project and revert to requirements.txt instead.
This is not ideal, as now for some reason my deployments from github actions are taking nearly 20mins (from 5mins), at this point a battle for another day.

Questions to Microsoft/Azure:

When will this ever get fixed?
How will I ever know if it does get fixed?
I want to use poetry again someday..... when?

I cant help but find this pretty humorous after nearly a full day dealing with it, and I'd gladly take any advice if there is a better alternative to what I've done above, I just literally had no time to deal with this today.

@Dan-White0
Copy link

I've also just run into this issue. What makes this even more frustrating is that it seems the --no-dev flag was listed as deprecated over 2.5 years ago!

@MaartenUreel
Copy link
Author

im having the same issue. where do you add this 'pre-build' script to?

I have an enviromnment variable PRE_BUILD_COMMAND set to cicd/pre_build.sh.

That script can then contain whatever you want to do before building.

There is also a POST_BUILD_COMMAND.

In my case, pre-build is minifying HTML, building TailwindCSS and now thus creating a requirements.txt file.

post-build is where you would typically cleanup stuff.

@Dan-White0
Copy link

Update:

I spent hours doing all sorts ....

Thanks for putting the time in to look at this, I've got it working now without poetry as well.

@tsukhu
Copy link

tsukhu commented Jan 7, 2025

We are too deep into using poetry and cannot revert to requirements.txt . By when do we expect a fix as we are blocked

@MaartenUreel
Copy link
Author

MaartenUreel commented Jan 7, 2025 •
edited
Loading

We are too deep into using poetry and cannot revert to requirements.txt . By when do we expect a fix as we are blocked

With my workaround, you can still use Poetry.
You just generate a requirements.txt on the build step with the export command, so that Oryx picks that up.

I also remain to use Poetry, I just added this export command to the pre-build step.

#2557 (comment)

You can also do this in a pre-commit or even manually on your development environment if you want / cannot use these pre-build commands.

Oryx will just check if there is a requirements.txt file, and use that. If not, it will check for pyproject.toml and use Poetry.

Nevertheless this needs to be fixed ofcourse.

tsukhu added a commit to tsukhu/Oryx that referenced this issue Jan 7, 2025
@tsukhu
Fix for issue microsoft#2557
3fcce74 
python 2.0 has removed --no-dev and we need to use `--without dev` instead
@tsukhu
Copy link

tsukhu commented Jan 7, 2025

I have worked around this on my own project by adding this to my pre-build script so that we do not rely on the built in Poetry handling:

# Temporary fix for Poetry, export requirements.txt
curl -sSL https://install.python-poetry.org | python3 -
/home/.local/bin/poetry self add poetry-plugin-export
/home/.local/bin/poetry export -f requirements.txt --output requirements.txt

Thanks @MaartenUreel for the workaround - In my case we have a mono repo hence more complexity . I have submitted a PR for the fix

@sarsharma sarsharma mentioned this issue Jan 7, 2025
Merged
@sarsharma sarsharma reopened this Jan 7, 2025
@sarsharma
Copy link
Member

Thank you for reporting this issue.
This is being actively looked into
Thanks

@tsukhu
Copy link

tsukhu commented Jan 8, 2025

Thank you for reporting this issue. This is being actively looked into Thanks

@sarsharma - thank you for the update. By when will this be released ?

@MaartenUreel
Copy link
Author

A pity that the fix is to pin to an old version, instead of just changing the flag as it literally said in the warning for months already...

The `--no-dev` option is deprecated, use the `--only main` notation instead.

@surenderssm
Copy link
Member

@tsukhu The rollout is currently underway, and with safe deployment practices in place, it should be available across regions in the coming days.

@surenderssm
Copy link
Member

@MaartenUreel Currently the focus is to revert to last known state, permanent fix will mostly align with your proposal. Thank you so much for your inputs.

A pity that the fix is to pin to an old version, instead of just changing the flag as it literally said in the warning for months already...

The `--no-dev` option is deprecated, use the `--only main` notation instead.

@lloyd-lighthouse
Copy link

Will you let us know when we can revert to using poetry?

@sarsharma
Copy link
Member

@lloyd-lighthouse yes, we will add an update here once the rollout is complete.

@yeshwanthjagannath
Copy link

We continue to get the issue on our end as of this morning at 9:00 AM MST. Look forward to an update.

@woter1832
Copy link

woter1832 commented Jan 12, 2025 •
edited
Loading

@sarsharma 19:05 UTC 0, 2025-01-12. The issue persists...

In my case, I am using Azure/static-web-apps-deploy@v1 (Region: West Europe) in a GitHub action. I assume your change to Oryx is not dynamically picked up by the action?

I've opened issue 1601 in the Az Static Web App repo.

@woter1832 woter1832 mentioned this issue Jan 12, 2025
Open
@lloyd-lighthouse
Copy link

Cmon team @microsoft, its been a whole week already.

@sarsharma
Copy link
Member

The fix should already be available in the following Azure App Service Regions:
East Asia
West Central US
West US 2
North Europe

We are actively working on rolling out the fix to the remaining regions.

@luismelo1999
Copy link

The issue still persists in West Europe too. Haven't been able to do any deploys for more than a week

@palvarezcordoba
Copy link

@sarsharma @surenderssm could you give us some info about the reason this is taking so long?

@tsukhu
Copy link

tsukhu commented Jan 14, 2025

Still waiting for the roll out , all our apps are blocked for a while now. This is the India region ....

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Revert poetry microsoft/Oryx
10 participants
@tsukhu @MaartenUreel @yeshwanthjagannath @surenderssm @palvarezcordoba @sarsharma @woter1832 @Dan-White0 @luismelo1999 @lloyd-lighthouse