tcWebHooksPlugin 2.0.0-rc.1

Bugfix Threading Issues on large TeamCity instances

On large systems with thousands of build agents, and many webhooks, the TeamCity thread pool can be filled up processing webhook executions.
This prevents certain aspects of TeamCity from functioning. In the case reported, this was preventing agents from registering.
This release has the following changes from 1.2.2.

Minimum support TeamCity version is 2019.1

This is the minimum version of TeamCity that supports creating a dedicated threadpool with a configurable size. To be able to request TeamCity to create a threadpool, we needed to update to this minimum version of TeamCity.

Removed the tcWebHooks agent plugin.

The agent plugin was forcing all agents to connect, download the agent plugins, restart and then re-connect. On large systems this process puts additional load on the default TeamCity thread pool.

Don't queue tasks for webhooks that won't execute.

Previously, every configured webhook would be submitted to the queue for processing for every build event. Most of these would then be skipped. This is because the majority of webhooks are only configured to run on the build finished event. With this change, only webhooks that are configured to actually run for that build event are queued for processing.
This has the consequence that the skipped count of webhooks will be significantly reduced, because webhooks are no longer assessed for every build event. Previously, most events were "701 :: Webhook not enabled for build state 'event'". These are now not logged and don't appear in the statistics.

tcWebHooks runs webhooks in its own ThreadPool (ExecutorService).

tcWebHooks now has a dedicated threadpool from which webhooks are executed. This leaves the default threadpool available for other TeamCity activities.

The threadpool defaults to mininum of 1 worker thread, maximum of 50 worker threads and a queue of 3000 items.
The pool size determines the number of webhooks that can be executed simultaneously. The pool will automatically scale up from min up to the max size when items are added. The queue size determines the number of webhooks that can be queued for execution once the pool is already processing the maximum number of webhooks.

The new ThreadPool is configurable by adding a dedicatedThreadPool element in the webhooks section of main-config.xml. For example:

<webhooks>
  <dedicatedThreadPool enabled="true" minPoolSize="10" maxPoolSize="100" queueSize="5000" />
</webhooks>

Note: Changes to the size of pool and queue will not take effect until TeamCity is restarted.

These settings have the following effects:

1. enabled="true" - This enables the use of a separate thread pool. If set to false, tcWebHooks will revert the behaviour and use the default thread pool like tcWebHooks version 1.2.2 did.
2. minPoolSize="10" - Sets the minimum pool size to 10. Without this value, the default is 1.
3. maxPoolSize="100" - Sets the maximum webhooks workers to 100 at a time. Without this value, the default is 50.
4. queueSize="5000" - Set the maximum number of webhooks that can be queued waiting to be executed. Without this value, the default is 3000.

If the pool and queue both fill up, webhooks will error with code 802 :: WebHook Executor Queue full. Exception was: Unable to put new task into the queue because maximum capacity is exceeded. Current queue size: 1000. Executor info: Thread factory "WebHookThread-" {corePoolSize=10; maxPoolSize=50; activeCount=50}

This can been seen in the error list on the WebHooks Administation tab.

These errors are also logged to the teamcity-server.log

[2023-03-27 08:33:04,891] ERROR - jetbrains.buildServer.SERVER - WebHookThreadingExecutorImpl :: trackingId: 68c207ae-829b-4bee-9a33-d6cb91b6e78b :: projectId: TestTopProject :: webhookId: id_968610231 :: templateId: json-redux, errorCode: 802, errorMessage: Unable to put new task into the queue because maximum capacity is exceeded. Current queue size: 1000. Executor info: Thread factory "WebHookThread-" {corePoolSize=10; maxPoolSize=50; activeCount=50}