feat(python): split out temp wheel builds

[hsheth2]

This should enable wheels to be available within ~2 minutes of the commit, which is significantly faster than the ~18 minutes we currently get with Vercel builds.

The new site also comes with a nicer index page, which gives you a brief tutorial on how to use it.

The new dedicated Python wheels site is deployed with Cloudflare pages, which means it will not be generated for PRs from forks. We can look into using pull_request_target for those in the future.

I intend to remove Python wheels from the docs site in the future, once this change has baked for a while.

This PR also updates our CI pipelines across our Python builds - upgrading versions, fixing various bugs with flags, etc

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[sgomezvillamor]

which means it will not be generated for PRs from forks

isn't that the usual way of working?
should we consider adapting to work directly on the original GitHub project instead?

[hsheth2]

@sgomezvillamor I primarily create branches on the datahub-project/datahub repo, and other folks with committer access can do the same. This has other benefits e.g. cypress cloud will show logs of exactly what tests failed and will show screenshots.

[sgomezvillamor]

LGTM
Just minor suggestions

[sgomezvillamor]

The copy_wheels.py script could be replaced with an inline rsync command. This change would bring the list of directories to be synced closer to the gradle dependencies of the task, reducing the risk of misalignment.

[hsheth2]

yup - will note that down for a follow up

[sgomezvillamor]

Commit hash, commit message, branch name... all this info is available in the github actions context.
We can pass these details as arguments to the gradle build, which would streamline this script

[hsheth2]

we pull some of these from the github env variable context. however, I wanted it to work similarly across local dev and CI, so we have both variants