diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md index a61325238447f..66c79c94cc9f9 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md @@ -384,7 +384,7 @@ Optional: |`yarn.federation.state-store.heartbeat-interval-secs` | `60` | The rate at which RMs report their membership to the federation to the central state-store. | -###ON ROUTER: +### ON ROUTER: These are extra configurations that should appear in the **conf/yarn-site.xml** at each Router. @@ -465,7 +465,88 @@ If we want to use JCache, we can configure `yarn.federation.cache.class` to `org This is a Cache implemented based on the Guava framework. If we want to use it, we can configure `yarn.federation.cache.class` to `org.apache.hadoop.yarn.server.federation.cache.FederationGuavaCache`. -###ON NMs: +### ON GPG: + +GlobalPolicyGenerator, abbreviated as "GPG," is used for the automatic generation of global policies for subClusters. + +These are extra configurations that should appear in the **conf/yarn-site.xml** for GPG. We allow only one GPG. + +Optional: + +| Property | Example | Description | +|:------------------------------------------------------------------|:---------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `yarn.federation.gpg.scheduled.executor.threads` | `10` | The number of threads to use for the GPG scheduled executor service. default is 10. | +| `yarn.federation.gpg.subcluster.cleaner.interval-ms` | `-1` | The interval at which the subcluster cleaner runs, -1 means disabled | +| `yarn.federation.gpg.subcluster.heartbeat.expiration-ms` | `30m` | The expiration time for a subcluster heartbeat, default is 30 minutes. | +| `yarn.federation.gpg.application.cleaner.class` | `org.apache.hadoop.yarn.server.globalpolicygenerator.DefaultApplicationCleaner` | The application cleaner class to use. | +| `yarn.federation.gpg.application.cleaner.interval-ms` | `-1` | The interval at which the application cleaner runs, -1 means disabled | +| `yarn.federation.gpg.application.cleaner.contact.router.spec` | `3,10,600000` | Should have three values separated by comma: minimal success retries, maximum total retry, retry interval (ms). | +| `yarn.federation.gpg.policy.generator.interval` | `1h` | The interval at which the policy generator runs, default is one hour. | +| `yarn.federation.gpg.policy.generator.class` | `org.apache.hadoop.yarn.server.globalpolicygenerator.policygenerator.NoOpGlobalPolicy` | The configured policy generator class, runs NoOpGlobalPolicy by default. | +| `yarn.federation.gpg.policy.generator.readonly` | `false` | Whether or not the policy generator is running in read only (won't modify policies), default is false.` | +| `yarn.federation.gpg.policy.generator.blacklist` | | Which sub-clusters the policy generator should blacklist. | +| `yarn.federation.gpg.policy.generator.load-based.pending.minimum` | `100` | The minimum number of pending applications in the subCluster. | +| `yarn.federation.gpg.policy.generator.load-based.pending.maximum` | `1000` | The maximum number of pending applications in the subCluster. | +| `yarn.federation.gpg.policy.generator.load-based.weight.minimum` | `0` | If a subCluster has a very high load, we will assign this value to the subCluster. The default value is 0, which means that we no longer assign appliaction to this subCluster. | +| `yarn.federation.gpg.policy.generator.load-based.edit.maximum` | `3` | This value represents the number of subClusters we want to calculate. default is 3. | +| `yarn.federation.gpg.policy.generator.load-based.scaling` | `LINEAR` | We provide 4 calculation methods: NONE, LINEAR, QUADRATIC, LOG. | +| `yarn.federation.gpg.webapp.address` | `0.0.0.0:8069` | The address of the GPG web application. | +| `yarn.federation.gpg.webapp.https.address` | `0.0.0.0:8070` | The https address of the GPG web application. | + +- yarn.federation.gpg.application.cleaner.contact.router.spec + +Specifications on how (many times) to contact Router for apps. We need to +do this because Router might return partial application list because some +sub-cluster RM is not responsive (e.g. failing over). Should have three values separated by comma: minimal success retries, +maximum total retry, retry interval (ms). + +- yarn.federation.gpg.policy.generator.load-based.scaling + +Note, this calculation method is when the number of Pending Applications in +the subCluster is less than yarn.federation.gpg.policy.generator.load-based.pending.maximum. + +maxPendingVal = `yarn.federation.gpg.policy.generator.load-based.pending.maximum` - +`yarn.federation.gpg.policy.generator.load-based.pending.minimum` + +curPendingVal = `Pending Applications in the subCluster` - +`yarn.federation.gpg.policy.generator.load-based.pending.minimum` + +No calculation is required, and the weight is 1 at this time. + +- LINEAR: + For linear computation, + we will use (maxPendingVal - curPendingVal) / (maxPendingVal). + +- QUADRATIC: + Calculated using quadratic, We will calculate quadratic for maxPendingVal, curPendingVal, + then use this formula = (maxPendingVal - curPendingVal) / (maxPendingVal). + +- LOG(LOGARITHM): + Calculated using logarithm, We will calculate logarithm for maxPendingVal, curPendingVal, + then use this formula = (maxPendingVal - curPendingVal) / (maxPendingVal). + +LINEAR is used by default. + +Security: + +Kerberos supported in GPG. + +| Property | Example | Description | +|:--------------------------------------------------|:--------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `yarn.federation.gpg.keytab.file` | | The keytab file used by GPG to login as its service principal. The principal name is configured with 'yarn.federation.gpg.kerberos.principal.hostname'. | +| `yarn.federation.gpg.kerberos.principal` | | The GPG service principal. This is typically set to GPG/_HOST@REALM.TLD. GPG will substitute _HOST with its own fully qualified hostname at startup. The _HOST placeholder allows using the same configuration setting on GPG in setup. | +| `yarn.federation.gpg.kerberos.principal.hostname` | | Optional. The hostname for the GPG containing this configuration file. Will be different for each machine. Defaults to current hostname. | + +Enabling CORS support: + +To enable cross-origin support (CORS) for the Yarn Router, please set the following configuration parameters: + +| Property | Example | Description | +|---------------------------------------------------|---------------------------------------------------------------|----------------------------------------------------------------------------------------------------------| +| `hadoop.http.filter.initializers` | `org.apache.hadoop.security.HttpCrossOriginFilterInitializer` | Optional. Set the filter to HttpCrossOriginFilterInitializer, Configure this parameter in core-site.xml. | +| `yarn.federation.gpg.webapp.cross-origin.enabled` | `true` | Optional. Enable/disable CORS filter.Configure this parameter in yarn-site.xml. | + +### ON NMs: These are extra configurations that should appear in the **conf/yarn-site.xml** at each NodeManager. diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md index b4d5386a7987f..0b309b6c376f6 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnUI2.md @@ -36,12 +36,13 @@ Please note that, If you run YARN daemons locally in your machine for test purpo you need the following configurations added to `yarn-site.xml` to enable cross origin (CORS) support. -| Configuration Property | Value | Description | -|:---- |:---- |:---- | +| Configuration Property | Value | Description | +|:---- |:---- |:-----------------------------------------| | `yarn.timeline-service.http-cross-origin.enabled` | true | Enable CORS support for Timeline Server | -| `yarn.resourcemanager.webapp.cross-origin.enabled` | true | Enable CORS support for Resource Manager | -| `yarn.nodemanager.webapp.cross-origin.enabled` | true | Enable CORS support for Node Manager | -| `yarn.router.webapp.cross-origin.enabled` | true | Enable CORS support for Yarn Router | +| `yarn.resourcemanager.webapp.cross-origin.enabled` | true | Enable CORS support for Resource Manager | +| `yarn.nodemanager.webapp.cross-origin.enabled` | true | Enable CORS support for Node Manager | +| `yarn.router.webapp.cross-origin.enabled` | true | Enable CORS support for Yarn Router | +| `yarn.federation.gpg.webapp.cross-origin.enabled` | true | Enable CORS support for Yarn GPG | Also please ensure that CORS related configurations are enabled in `core-site.xml`. Kindly refer [here](../../hadoop-project-dist/hadoop-common/HttpAuthentication.html)