Proxy - WebApp Health Check Performed on Server Doesn't Respect Custom Proxy

[FinnianDempsey]

Team

• I've assigned a team label to this issue

Severity

Workaround Exists

Version

Found in 2022.1.2133 but confirmed in latest

Latest Version

I could reproduce the problem in the latest build

What happened?

When configuring a Custom Proxy for Octopus Web Requests, the proxy isn't used when performing an Azure WebApp health check directly on the Octopus Server.

Proxy is used when using the proxy server configured in Internet Explorer.

Reproduction

• Configure a custom proxy for Octopus WebRequests without configuring it in Windows
• Perform a health check of an Azure WebApp directly on the Server.
• Fail to see entries in access logs.

Error and Stacktrace

Error    |     An attempt was made to access a socket in a way forbidden by its access permissions. (login.microsoftonline.com:443)
16:45:01   Error    |     System.Net.Http.HttpRequestException
16:45:01   Error    |     at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
16:45:01   Error    |     at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
16:45:01   Error    |     at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
16:45:01   Error    |     at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
16:45:01   Error    |     at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
16:45:01   Error    |     at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
16:45:01   Error    |     at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
16:45:01   Error    |     at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
16:45:01   Error    |     at Microsoft.Identity.Core.Http.HttpManager.ExecuteAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, HttpMethod method)
16:45:01   Error    |     at Microsoft.Identity.Core.Http.HttpManager.ExecuteWithRetryAsync(Uri endpoint, IDictionary`2 headers, HttpContent body, HttpMethod method, RequestContext requestContext, Boolean doNotThrow, Boolean retry)
16:45:01   Error    |     at Microsoft.Identity.Core.Http.HttpManager.SendGetAsync(Uri endpoint, IDictionary`2 headers, RequestContext requestContext)
16:45:01   Error    |     at Microsoft.Identity.Core.OAuth2.OAuthClient.ExecuteRequestAsync()
16:45:01   Error    |     at Microsoft.Identity.Core.OAuth2.OAuthClient.GetResponseAsync[T](Boolean respondToDeviceAuthChallenge)
16:45:01   Error    |     at Microsoft.Identity.Core.OAuth2.OAuthClient.GetResponseAsync[T]()
16:45:01   Error    |     at Microsoft.IdentityModel.Clients.ActiveDirectory.InstanceDiscovery.DiscoverAsync(Uri authority, Boolean validateAuthority, RequestContext requestContext)
16:45:01   Error    |     at Microsoft.IdentityModel.Clients.ActiveDirectory.InstanceDiscovery.GetMetadataEntryAsync(Uri authority, Boolean validateAuthority, RequestContext requestContext)
16:45:01   Error    |     at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Instance.Authenticator.UpdateFromTemplateAsync(RequestContext requestContext)
16:45:01   Error    |     at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.PreRunAsync()
16:45:01   Error    |     at Microsoft.IdentityModel.Clients.ActiveDirectory.Internal.Flows.AcquireTokenHandlerBase.RunAsync()
16:45:01   Error    |     at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenForClientCommonAsync(String resource, ClientKey clientKey)
16:45:01   Error    |     at Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext.AcquireTokenAsync(String resource, ClientCredential clientCredential)
16:45:01   Error    |     at Microsoft.Rest.Azure.Authentication.Internal.MemoryApplicationAuthenticationProvider.AuthenticateAsync(String clientId, String audience, AuthenticationContext context)
16:45:01   Error    |     at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, String clientId, IApplicationAuthenticationProvider authenticationProvider, ActiveDirectoryServiceSettings settings, TokenCache cache)
16:45:01   Error    |     at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, ClientCredential credential, ActiveDirectoryServiceSettings settings, TokenCache cache)
16:45:01   Error    |     at Microsoft.Rest.Azure.Authentication.ApplicationTokenProvider.LoginSilentAsync(String domain, String clientId, String secret, ActiveDirectoryServiceSettings settings, TokenCache cache)
16:45:01   Error    |     at Microsoft.Azure.Management.ResourceManager.Fluent.Authentication.AzureCredentials.ProcessHttpRequestAsync(HttpRequestMessage request, CancellationToken cancellationToken)
16:45:01   Error    |     at Microsoft.Azure.Management.AppService.Fluent.WebAppsOperations.GetWithHttpMessagesAsync(String resourceGroupName, String name, Dictionary`2 customHeaders, CancellationToken cancellationToken)
16:45:01   Error    |     at Microsoft.Azure.Management.AppService.Fluent.WebAppsOperationsExtensions.GetAsync(IWebAppsOperations operations, String resourceGroupName, String name, CancellationToken cancellationToken)
16:45:01   Error    |     at Microsoft.Azure.Management.AppService.Fluent.WebAppsImpl.GetInnerByGroupAsync(String groupName, String name, CancellationToken cancellationToken)
16:45:01   Error    |     at Microsoft.Azure.Management.AppService.Fluent.WebAppsImpl.GetByResourceGroupAsync(String groupName, String name, CancellationToken cancellationToken)
16:45:01   Error    |     at Calamari.AzureAppService.HealthCheckBehaviour.ConfirmWebAppExists(ServicePrincipalAccount servicePrincipal, String resourceGroupName, String siteAndSlotName) in C:\buildAgent\work\cdb95c8a359b9bc9\source\Calamari\HealthCheckCommand.cs:line 40
16:45:01   Error    |     at Calamari.Common.Plumbing.Pipeline.PipelineCommand.ExecuteBehaviour(RunningDeployment context, IBehaviour behaviour)
16:45:01   Error    |     at Calamari.Common.Plumbing.Pipeline.PipelineCommand.Execute(ILifetimeScope lifetimeScope, IVariables variables)
16:45:01   Error    |     at Calamari.Common.Plumbing.Pipeline.PipelineCommand.Execute(ILifetimeScope lifetimeScope, IVariables variables)
16:45:01   Error    |     at Calamari.Common.CalamariFlavourProgramAsync.Run(String[] args)
16:45:01   Error    |     --Inner Exception--
16:45:01   Error    |     An attempt was made to access a socket in a way forbidden by its access permissions.
16:45:01   Error    |     System.Net.Sockets.SocketException
16:45:01   Error    |     at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
16:45:01   Error    |     at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
16:45:01   Error    |     at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|283_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
16:45:01   Error    |     at System.Net.Http.HttpConnectionPool.DefaultConnectAsync(SocketsHttpConnectionContext context, CancellationToken cancellationToken)
16:45:01   Error    |     at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)

More Information

Previous issue:
#6958

Workaround

Either run the Azure WebApp health check from Tentacle or configure the Proxy within Windows:

[mjhilton]

Starting work on this now. I'm using Squid proxy in a Docker container as my proxy for testing. You can run up an instance with:

docker run --rm --name squidproxy -p 3128:3128 datadog/squid

then set your proxy to http://localhost:3128 and watch the access logs to validate requests are proxied via the server, using:

docker exec squidproxy tail -f /var/log/squid/access.log

[mjhilton]

I've identified the root cause of this.

There's a bug in the underlying Azure SDK library we use to run the Health Check, which doesn't respect the Proxy settings that have we configured when establishing a connection to Azure.

The health-check uses a specific method in the SDK which the deployment behaviours don't use, which is why this shows up only for the health-check.

The replacement library is still in beta, but it sounds like the old version which we're using is in maintenance mode and not receiving further fixes.

I've checked in with MS on that linked issue, but in the meantime I'm re-writing the health-check to use the newer SDK. The health check is low enough complexity that I'm happy to use the beta version, but I don't want to touch the deployment code itself until the new SDK stabilises.

[mjhilton]

The fix has been merged into the 2022.1 stream and is currently building a release package. It will be merged forward to 2022.2 and vNext shortly thereafter.

[octoreleasebot]

Release Note: Fixed an issue where the configured Web Request Proxy was not respected in Health Checks for Azure App Service deployment targets

[Octobob]

🎉 The fix for this issue has been released in:

Release stream	Release
2022.1	2022.1.2875
2022.2	2022.2.6764
2022.3	2022.3.2651
2022.4+	all releases