Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python 3.13 transition #37197

Open
smoser opened this issue Dec 16, 2024 · 1 comment
Open

python 3.13 transition #37197

smoser opened this issue Dec 16, 2024 · 1 comment

Comments

@smoser
Copy link
Member

smoser commented Dec 16, 2024

Last week we bumped the 'provider priority' value for the python 3.13 subpackage of packages named 'py3-'.
That is to say that before last week, if you did apk add py3-lxml you'd end up with the python-3.12 version, a package named py3.12-lxml. This week, you'd end up with py3.13-lxml and its dependencies installed.

There are several packages that have an un-versioned python package listed as a build-depends, runtime-dependency or test dependency. Many of these packages will now be broken either in runtime or in the next build for reasons such as:

  1. they installed python modules (.py files) into /usr/lib/python3.12/site-packages or a virtual env's lib/python3.12/site-packages using 'python' or 'python3' to do the install.
  2. their build uses a python3 or python3-dev package , but does not support python-3.13

I searched through existing melange files for use of un-versioned python 3 packages.
There are very few cases where an un-versioned python dependency (python3-dev rather than python-3.XX-dev) should be used. Those are mostly limited to:

  1. build systems that use only python3 and no other python packages.
  2. runtime dependencies that use only the python executable and standard library. byobu is one such example.

To update packages, follow these bullet points:

  • Remove python-3 and python-3-dev references. Either replace them with python3 or python3-dev. In almost all cases, python3 should not be listed in the runtime or test dependencies.

  • Packages that simply use python or python3 in their build environment but do not produce packages with python files can continue to use python3.

  • If python3 is listed only in the build environment (not elsewhere), you may well be able to just delete it. If another package in the build environment needs python, it should be pulled in.

  • If a package has more than one python package (py3-) in its build environment, it should pick a specific version (3.13 if supported) and list the dependencies with py${{vars.py-version}}-pkg. This will guarantee that the dependencies are installed at a consistent version.

  • If there are more than one python build package (setuptools, wheel...) listed in the build environment, then replace those with py3.XX-build-base, then use the build-base package.

  • If a package has anything other than python3 listed in its runtime depends, or it uses a virtual env, then it needs to pick a specific python version.

  • If the build uses python3 or python by name, the easiest thing to do is
    to use a python-as-wrapper package to provide that executable.
    Add python-as-wrapper package and use it in asciidoc build. #36891 . It ensures that shbang will get
    the correct executable value, and melange will then generate the correct
    dependendency.

Where 'pick a specific version' is mentioned above, the preferred mechanism for
doing so is to use py-version variable.

vars:
  py-version: 3.13
...

environment:
  contents:
  packages:
   ..
   - py${{py-version}}-build-base
   - py${{py-version}}-
@smoser
Copy link
Member Author

smoser commented Dec 16, 2024

I put together a little python script find-pythons that searches build/run/test depends for un-versiondd python packages. The packages below were found to have some. The full output of that tool is python-rdeps.txt . The output lists where the dependency is (run/test/build as well as how many).

  • aom
  • apache-nifi
  • argo-cd-2.13
  • argo-workflows
  • audit
  • aws-c-common
  • aws-efs-csi-driver
  • az
  • bazel-5
  • bazel-6
  • bazel-7
  • bluez
  • boost
  • botan
  • btrfs-progs
  • bun
  • byobu
  • cassandra-reaper
  • chromium
  • cifs-utils
  • cilium-1.16
  • clamav-1.4
  • clamav
  • clang-15
  • clang-16
  • clang-17
  • clang-18
  • clang-19
  • clickhouse-24.10
  • conda-build
  • configurable-http-proxy
  • crun
  • cyrus-sasl
  • cython-0 - cython-0 - fix ftbfs after python3.13 transition #37222
  • dart
  • dask-gateway
  • datawire-envoy-1.31
  • deno
  • dnssec-root
  • dotnet-6
  • dotnet-7
  • dotnet-8
  • dotnet-9
  • doxygen
  • druid
  • dstat
  • enchant2
  • envoy-1.32
  • fatrace
  • fish
  • flink-1.20
  • font-lohit-beng-assamese
  • font-lohit-beng-bengali
  • font-lohit-beng-extra
  • font-tlwg
  • fontconfig
  • fontforge
  • frr-10.2
  • gdb
  • gdk-pixbuf
  • geckodriver
  • ggshield
  • gi-docgen
  • glib-gir
  • glib
  • glibc
  • glslang
  • gobject-introspection
  • gpsd
  • grafana-11.4
  • graphviz
  • grpc-1.66
  • gtest
  • gtk-3
  • gtk-4
  • gtk-doc
  • harbor-2.12
  • harfbuzz
  • heimdal
  • htop
  • httpie
  • i2c-tools
  • imath
  • ingress-nginx-controller-1.11
  • iso-codes
  • istio-envoy-1.24
  • itstool
  • jitsucom-jitsu
  • jo
  • jupyterhub-k8s-hub
  • jwt-tool
  • k8s-sidecar
  • kdash
  • kube-downscaler
  • kubeflow-jupyter-web-app
  • kubeflow-pipelines
  • kubeflow-volumes-web-app
  • ldb
  • libapr
  • libdwarf
  • libepoxy
  • libevdev
  • libevent
  • libglvnd
  • libgsf
  • libinput
  • liblogging
  • libnvme
  • libpsl
  • libpulsar
  • librelp
  • librsvg
  • libseccomp
  • libsecret
  • libsndfile
  • libxcb
  • libxml2
  • llvm-18
  • llvm-19
  • llvm-libcxx-15
  • llvm-libcxx-16
  • llvm-libcxx-17
  • llvm15
  • llvm16
  • llvm17
  • lttng-ust
  • mdbook
  • memtier-benchmark
  • mesa
  • mitmproxy
  • mlflow
  • modelmesh-runtime-adapter
  • mycli
  • neuvector-manager
  • neuvector-prometheus-exporter
  • nfs-utils
  • nftables
  • node-gyp
  • nodejs-16 nodejs - be explicit about which python is used in build environment. #37223
  • nodejs-18 nodejs - be explicit about which python is used in build environment. #37223
  • nodejs-19 nodejs - be explicit about which python is used in build environment. #37223
  • nodejs-20 nodejs - be explicit about which python is used in build environment. #37223
  • nodejs-21 nodejs - be explicit about which python is used in build environment. #37223
  • nodejs-22 nodejs - be explicit about which python is used in build environment. #37223
  • nodejs-23 nodejs - be explicit about which python is used in build environment. #37223
  • nri-kubernetes
  • nss
  • onetbb
  • openmp-16
  • openmp-17
  • openmpi
  • openpmix
  • openscap
  • opensearch-dashboards-2
  • openvpn
  • opus
  • patroni
  • pax-utils
  • percona-xtrabackup-8.4
  • pnpm
  • podman
  • proxysql
  • prrte
  • py3-cassandra-medusa
  • py3-google-cloud-bigquery
  • py3-google-cloud-bigtable
  • py3-google-cloud-pubsub
  • py3-grpcio-opentracing
  • py3-jupyter-lsp
  • py3-jupyter-server-terminals
  • py3-jupyter-server
  • py3-jupyterhub-firstuseauthenticator
  • py3-jupyterhub-hmacauthenticator
  • py3-jupyterhub-idle-culler
  • py3-jupyterhub-ldapauthenticator
  • py3-jupyterhub-ltiauthenticator
  • py3-jupyterhub-nativeauthenticator
  • py3-jupyterhub-tmpauthenticator
  • py3-jupyterlab-server
  • py3-kubernetes
  • py3-notebook-shim
  • py3-pgcli
  • py3-pgspecial
  • py3-psycopg
  • py3-pulsar-client
  • py3-setuptools-wheel
  • py3-tensorflow-data-validation
  • py3-tensorflow-io-gcs-filesystem
  • py3-tensorflow-model-analysis
  • py3-tensorflow-serving-api
  • pylint
  • qemu
  • qt6-qtbase
  • rabbitmq-server-4.0
  • re2c
  • reflex
  • rethinkdb
  • rpm
  • rsyslog
  • rust-1.69
  • rust-1.70
  • rust-1.71
  • rust-1.72
  • rust-1.73
  • rust-1.74
  • rust-1.75
  • rust-1.76
  • rust-1.77
  • rust-1.78
  • rust-1.79
  • rust-1.80
  • rust-1.81
  • rust-1.82
  • rust-1.83
  • rustup
  • samba
  • scap-security-guide
  • scons
  • selenium
  • semgrep
  • shaderc
  • smartypants
  • snyk-cli
  • souffle
  • spark-3.5
  • spirv-tools
  • sqlmap
  • sqlpad
  • ssh-import-id
  • statsd
  • stunnel
  • subunit
  • superset
  • supervisor
  • suricata-update
  • suricata
  • systemd
  • talloc
  • tclap
  • tdb
  • teleport
  • tevent
  • tiff
  • tomcat-native
  • tree-sitter
  • trino
  • unixodbc
  • valgrind
  • varnish-modules
  • varnish
  • vim
  • wabt
  • wasi-sdk
  • wasmedge
  • wayland-protocols
  • websockify
  • xcb-proto
  • xfsprogs
  • xkeyboard-config
  • yamllint
  • yunikorn-web
  • zetasql

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant