-
Notifications
You must be signed in to change notification settings - Fork 8.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
YARN-11571. [GPG] Add Information About YARN GPG in Federation.md (#6158
) Contributed by Shilun Fan. Reviewed-by: Inigo Goiri <[email protected]> Signed-off-by: Shilun Fan <[email protected]>
- Loading branch information
Showing
2 changed files
with
89 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -384,7 +384,7 @@ Optional: | |
| |`yarn.federation.state-store.heartbeat-interval-secs` | `60` | The rate at which RMs report their membership to the federation to the central state-store. | | ||
|
|
||
|
|
||
| ###ON ROUTER: | ||
| ### ON ROUTER: | ||
|
|
||
| These are extra configurations that should appear in the **conf/yarn-site.xml** at each Router. | ||
|
|
||
|
|
@@ -465,7 +465,88 @@ If we want to use JCache, we can configure `yarn.federation.cache.class` to `org | |
| This is a Cache implemented based on the Guava framework. | ||
| If we want to use it, we can configure `yarn.federation.cache.class` to `org.apache.hadoop.yarn.server.federation.cache.FederationGuavaCache`. | ||
|
|
||
| ###ON NMs: | ||
| ### ON GPG: | ||
|
|
||
| GlobalPolicyGenerator, abbreviated as "GPG," is used for the automatic generation of global policies for subClusters. | ||
|
|
||
| These are extra configurations that should appear in the **conf/yarn-site.xml** for GPG. We allow only one GPG. | ||
|
|
||
| Optional: | ||
|
|
||
| | Property | Example | Description | | ||
| |:------------------------------------------------------------------|:---------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `yarn.federation.gpg.scheduled.executor.threads` | `10` | The number of threads to use for the GPG scheduled executor service. default is 10. | | ||
| | `yarn.federation.gpg.subcluster.cleaner.interval-ms` | `-1` | The interval at which the subcluster cleaner runs, -1 means disabled | | ||
| | `yarn.federation.gpg.subcluster.heartbeat.expiration-ms` | `30m` | The expiration time for a subcluster heartbeat, default is 30 minutes. | | ||
| | `yarn.federation.gpg.application.cleaner.class` | `org.apache.hadoop.yarn.server.globalpolicygenerator.DefaultApplicationCleaner` | The application cleaner class to use. | | ||
| | `yarn.federation.gpg.application.cleaner.interval-ms` | `-1` | The interval at which the application cleaner runs, -1 means disabled | | ||
| | `yarn.federation.gpg.application.cleaner.contact.router.spec` | `3,10,600000` | Should have three values separated by comma: minimal success retries, maximum total retry, retry interval (ms). | | ||
| | `yarn.federation.gpg.policy.generator.interval` | `1h` | The interval at which the policy generator runs, default is one hour. | | ||
| | `yarn.federation.gpg.policy.generator.class` | `org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator.NoOpGlobalPolicy` | The configured policy generator class, runs NoOpGlobalPolicy by default. | | ||
| | `yarn.federation.gpg.policy.generator.readonly` | `false` | Whether or not the policy generator is running in read only (won't modify policies), default is false.` | | ||
| | `yarn.federation.gpg.policy.generator.blacklist` | | Which sub-clusters the policy generator should blacklist. | | ||
| | `yarn.federation.gpg.policy.generator.load-based.pending.minimum` | `100` | The minimum number of pending applications in the subCluster. | | ||
| | `yarn.federation.gpg.policy.generator.load-based.pending.maximum` | `1000` | The maximum number of pending applications in the subCluster. | | ||
| | `yarn.federation.gpg.policy.generator.load-based.weight.minimum` | `0` | If a subCluster has a very high load, we will assign this value to the subCluster. The default value is 0, which means that we no longer assign appliaction to this subCluster. | | ||
| | `yarn.federation.gpg.policy.generator.load-based.edit.maximum` | `3` | This value represents the number of subClusters we want to calculate. default is 3. | | ||
| | `yarn.federation.gpg.policy.generator.load-based.scaling` | `LINEAR` | We provide 4 calculation methods: NONE, LINEAR, QUADRATIC, LOG. | | ||
| | `yarn.federation.gpg.webapp.address` | `0.0.0.0:8069` | The address of the GPG web application. | | ||
| | `yarn.federation.gpg.webapp.https.address` | `0.0.0.0:8070` | The https address of the GPG web application. | | ||
|
|
||
| - yarn.federation.gpg.application.cleaner.contact.router.spec | ||
|
|
||
| Specifications on how (many times) to contact Router for apps. We need to | ||
| do this because Router might return partial application list because some | ||
| sub-cluster RM is not responsive (e.g. failing over). Should have three values separated by comma: minimal success retries, | ||
| maximum total retry, retry interval (ms). | ||
|
|
||
| - yarn.federation.gpg.policy.generator.load-based.scaling | ||
|
|
||
| Note, this calculation method is when the number of Pending Applications in | ||
| the subCluster is less than yarn.federation.gpg.policy.generator.load-based.pending.maximum. | ||
|
|
||
| maxPendingVal = `yarn.federation.gpg.policy.generator.load-based.pending.maximum` - | ||
| `yarn.federation.gpg.policy.generator.load-based.pending.minimum` | ||
|
|
||
| curPendingVal = `Pending Applications in the subCluster` - | ||
| `yarn.federation.gpg.policy.generator.load-based.pending.minimum` | ||
|
|
||
| No calculation is required, and the weight is 1 at this time. | ||
|
|
||
| - LINEAR: | ||
| For linear computation, | ||
| we will use (maxPendingVal - curPendingVal) / (maxPendingVal). | ||
|
|
||
| - QUADRATIC: | ||
| Calculated using quadratic, We will calculate quadratic for maxPendingVal, curPendingVal, | ||
| then use this formula = (maxPendingVal - curPendingVal) / (maxPendingVal). | ||
|
|
||
| - LOG(LOGARITHM): | ||
| Calculated using logarithm, We will calculate logarithm for maxPendingVal, curPendingVal, | ||
| then use this formula = (maxPendingVal - curPendingVal) / (maxPendingVal). | ||
|
|
||
| LINEAR is used by default. | ||
|
|
||
| Security: | ||
|
|
||
| Kerberos supported in GPG. | ||
|
|
||
| | Property | Example | Description | | ||
| |:--------------------------------------------------|:--------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `yarn.federation.gpg.keytab.file` | | The keytab file used by GPG to login as its service principal. The principal name is configured with 'yarn.federation.gpg.kerberos.principal.hostname'. | | ||
| | `yarn.federation.gpg.kerberos.principal` | | The GPG service principal. This is typically set to GPG/_[email protected]. GPG will substitute _HOST with its own fully qualified hostname at startup. The _HOST placeholder allows using the same configuration setting on GPG in setup. | | ||
| | `yarn.federation.gpg.kerberos.principal.hostname` | | Optional. The hostname for the GPG containing this configuration file. Will be different for each machine. Defaults to current hostname. | | ||
|
|
||
| Enabling CORS support: | ||
|
|
||
| To enable cross-origin support (CORS) for the Yarn Router, please set the following configuration parameters: | ||
|
|
||
| | Property | Example | Description | | ||
| |---------------------------------------------------|---------------------------------------------------------------|----------------------------------------------------------------------------------------------------------| | ||
| | `hadoop.http.filter.initializers` | `org.apache.hadoop.security.HttpCrossOriginFilterInitializer` | Optional. Set the filter to HttpCrossOriginFilterInitializer, Configure this parameter in core-site.xml. | | ||
| | `yarn.federation.gpg.webapp.cross-origin.enabled` | `true` | Optional. Enable/disable CORS filter.Configure this parameter in yarn-site.xml. | | ||
|
|
||
| ### ON NMs: | ||
|
|
||
| These are extra configurations that should appear in the **conf/yarn-site.xml** at each NodeManager. | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters