Skip to content

Commit

Permalink
[SPARK-45670][CORE] SparkSubmit does not support `--total-executor-co…
Browse files Browse the repository at this point in the history
…res` when deploying on K8s

### What changes were proposed in this pull request?

Remove Kubernetes from the support list of `--total-executor-cores` in SparkSubmit

### Why are the changes needed?

`--total-executor-cores` does not take effect in Spark on K8s, [the
comments from original PR](apache#19717 (comment)) also proves that

### Does this PR introduce _any_ user-facing change?

The output of `spark-submit --help` changed

```patch
...
-  Spark standalone and Kubernetes only:
+  Spark standalone only:
    --total-executor-cores NUM  Total cores for all executors.
...
```
### How was this patch tested?

Pass GA and review.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes apache#43536 from pan3793/tec.

Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
pan3793 authored and dongjoon-hyun committed Oct 26, 2023
1 parent 8c11804 commit 5364392
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -674,7 +674,7 @@ private[spark] class SparkSubmit extends Logging {
confKey = EXECUTOR_CORES.key),
OptionAssigner(args.executorMemory, STANDALONE | YARN | KUBERNETES, ALL_DEPLOY_MODES,
confKey = EXECUTOR_MEMORY.key),
OptionAssigner(args.totalExecutorCores, STANDALONE | KUBERNETES, ALL_DEPLOY_MODES,
OptionAssigner(args.totalExecutorCores, STANDALONE, ALL_DEPLOY_MODES,
confKey = CORES_MAX.key),
OptionAssigner(args.files, LOCAL | STANDALONE | KUBERNETES, ALL_DEPLOY_MODES,
confKey = FILES.key),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -566,7 +566,7 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
| --kill SUBMISSION_ID If given, kills the driver specified.
| --status SUBMISSION_ID If given, requests the status of the driver specified.
|
| Spark standalone and Kubernetes only:
| Spark standalone only:
| --total-executor-cores NUM Total cores for all executors.
|
| Spark standalone, YARN and Kubernetes only:
Expand Down

0 comments on commit 5364392

Please sign in to comment.