Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MLCOMPUTE-2001 | Skip UI port selection, pod tpl generation, and logs for driver on k8s tron #151

Merged
merged 3 commits into from
Oct 10, 2024

Conversation

chi-yelp
Copy link
Contributor

@chi-yelp chi-yelp commented Oct 9, 2024

Did the following changes to reduce unnecessary noise from Spark jobs in the Tron deployment logs:

  • Allow passing in a constructer flag is_driver_on_k8s_tron to SparkConfBuilder
  • When is_driver_on_k8s_tron is true:
    • Skip Spark argument deprecation warnings
    • Skip trying to pick an available ephemeral port (always use the first port for the driver pod)
    • Skip generating pod template

(There will be a follow up PR in the paasta repo)

Test: build and install the package in the paasta repo

service_configuration_lib/spark_config.py Outdated Show resolved Hide resolved
@chi-yelp chi-yelp force-pushed the u/chi/cleanup_setup_tron_namespace branch from b653216 to 7ad9da0 Compare October 10, 2024 12:28
@chi-yelp chi-yelp merged commit f79b9e0 into master Oct 10, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants