How to Customize Cluster Destination Address Based on Route Path at Runtime

[tawman]

Looking for an existing configuration setting or extension point to control cluster destination addresses at run-time. I have a working proof on concept for my use case.

Business Use Case

I have a use case that involves spinning up temporary customer specific Azure Docker container instances as part of a workflow. The Docker container instances will be deleted once the workflow task has completed. There will be thousands of these container instances created and deleted every day.

Currently, you can only generate self-signed SSL certs for web assets running on an Azure Docker container because it does not support custom domain names. This led me to discover reverse-proxy, and build a working proof of concept to put the Docker instances behind a vanity url with wildcard SSL certificate.

• Sample Azure Docker container instance web asset: http://docker-container-instance-**SLUG**.southcentralus.azurecontainer.io:8080/**SLUG**/index.html
• Desired reverse-proxy representation: https://some.vanity.url.com/**SLUG**/index.html

What I found missing in reverse-proxy was the ability to influence how the cluster destination address is customized at runtime based on the route requested. It is not practical for me to pre-configure the routes and clusters due to the ephemeral nature of the Docker container instances, and I could not find an extension point to write my own provider.

I was able to make a local patch to the MakeDestinationAddress method to achieve my desired results. The custom reverse-proxy build only works for my use case, but that is fine since it is the only traffic I will support with it hosted as an Azure App Service.

Example Docker Contain Instances

http://docker-container-instance-4d89c7a3-950c-4f91-87df-810dbf5e6651.southcentralus.azurecontainer.io:8080/4d89c7a3-950c-4f91-87df-810dbf5e6651/index.html
http://docker-container-instance-0a4af0c7-3844-4164-9b5f-0b767a54f749.southcentralus.azurecontainer.io:8080/0a4af0c7-3844-4164-9b5f-0b767a54f749/index.html
http://docker-container-instance-1b08148d-d1c0-43d4-85d5-10e58cce1c1d.southcentralus.azurecontainer.io:8080/1b08148d-d1c0-43d4-85d5-10e58cce1c1d/index.html
http://docker-container-instance-71a1333a-2ea0-42d4-8ba8-3ecac28633b7.southcentralus.azurecontainer.io:8080/71a1333a-2ea0-42d4-8ba8-3ecac28633b7/index.html

Example Reverse Proxy URLs

https://some.vanity.url.com/4d89c7a3-950c-4f91-87df-810dbf5e6651/index.html
https://some.vanity.url.com/0a4af0c7-3844-4164-9b5f-0b767a54f749/index.html
https://some.vanity.url.com/1b08148d-d1c0-43d4-85d5-10e58cce1c1d/index.html
https://some.vanity.url.com/71a1333a-2ea0-42d4-8ba8-3ecac28633b7/index.html

appsettings.json

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "AllowedHosts": "*",
  "ReverseProxy": {
    "Routes": {
      "route1": {
        "ClusterId": "cluster1",
        "Match": {
          "Path": "{**catch-all}"
        }
      }
    },
    "Clusters": {
      "cluster1": {
        "Destinations": {
          "cluster1/destination1": {
            "Address": "http://docker-container-instance-{0}.southcentralus.azurecontainer.io:8080"
          }
        }
      }
    }
  }
}

Local Code Patch

---------------- src/ReverseProxy/Utilities/RequestUtilities.cs ----------------
index db38d8c..d5ba647 100644
@@ -1,10 +1,11 @@
 // Copyright (c) Microsoft Corporation.
 // Licensed under the MIT License.
 
 using System;
 using System.Collections.Generic;
+using System.Linq;
 using System.Net.Http;
 using Microsoft.AspNetCore.Http;
 using Microsoft.Extensions.Primitives;
 using Microsoft.Net.Http.Headers;
 
@@ -43,19 +44,21 @@ namespace Yarp.ReverseProxy.Utilities
         /// e.g. "http://example.com:80/path/prefix"</param>
         /// <param name="path">The path to append.</param>
         /// <param name="query">The query to append</param>
         public static Uri MakeDestinationAddress(string destinationPrefix, PathString path, QueryString query)
         {
-            ReadOnlySpan<char> prefixSpan = destinationPrefix;
+            var prefixSpan = destinationPrefix;
 
             if (path.HasValue && destinationPrefix.EndsWith('/'))
             {
                 // When PathString has a value it always starts with a '/'. Avoid double slashes when concatenating.
                 prefixSpan = prefixSpan[0..^1];
             }
 
-            var targetAddress = string.Concat(prefixSpan, path.ToUriComponent(), query.ToUriComponent());
+            var siteNameSuffix = path.Value?.Split('/').Skip(1).FirstOrDefault();
+            var siteName = string.Format(prefixSpan, siteNameSuffix);
+            var targetAddress = string.Concat(siteName, path.ToUriComponent(), query.ToUriComponent());
 
             return new Uri(targetAddress, UriKind.Absolute);
         }

[Tratcher]

I agree your scenario doesn't make sense for pre-configured routes. Nor would it work with any of the built in features like session affinity, load balancing, health checks, etc. Instead, take a look at https://microsoft.github.io/reverse-proxy/articles/direct-proxying.html.

[tawman]

Thank you @Tratcher as that is exactly what I was looking for... so many good examples that I missed that one. Works out of the box for my use case

--------------- samples/ReverseProxy.Direct.Sample/Startup.cs ----------------
index af7c3b6..00a53cd 100644
@@ -1,9 +1,10 @@
 // Copyright (c) Microsoft Corporation.
 // Licensed under the MIT License.
 
 using System;
+using System.Linq;
 using System.Net;
 using System.Net.Http;
 using System.Threading.Tasks;
 using Microsoft.AspNetCore.Builder;
 using Microsoft.AspNetCore.Http;
@@ -50,11 +51,11 @@ namespace Yarp.Sample
             {
                 // When using IHttpProxy for direct proxying you are responsible for routing, destination discovery, load balancing, affinity, etc..
                 // For an alternate example that includes those features see BasicYarpSample.
                 endpoints.Map("/{**catch-all}", async httpContext =>
                 {
-                    await httpProxy.ProxyAsync(httpContext, "https://example.com", httpClient, requestOptions, transformer);
+                    await httpProxy.ProxyAsync(httpContext, "http://docker-container-instance-{0}.southcentralus.azurecontainer.io:8080", httpClient, requestOptions, transformer);
                     var errorFeature = httpContext.Features.Get<IProxyErrorFeature>();
                     
                     // Check if the proxy operation was successful
                     if (errorFeature != null)
                     {
@@ -82,17 +83,18 @@ namespace Yarp.Sample
             /// <param name="proxyRequest">The outgoing proxy request.</param>
             /// <param name="destinationPrefix">The uri prefix for the selected destination server which can be used to create
             /// the RequestUri.</param>
             public override async ValueTask TransformRequestAsync(HttpContext httpContext, HttpRequestMessage proxyRequest, string destinationPrefix)
             {
+                var siteNameSuffix = httpContext.Request.Path.Value?.Split('/').Skip(1).FirstOrDefault();
+                destinationPrefix = string.Format(destinationPrefix, siteNameSuffix);
+
                 // Copy all request headers
                 await base.TransformRequestAsync(httpContext, proxyRequest, destinationPrefix);
 
                 // Customize the query string:
                 var queryContext = new QueryTransformContext(httpContext.Request);
-                queryContext.Collection.Remove("param1");
-                queryContext.Collection["area"] = "xx2";
 
                 // Assign the custom uri. Be careful about extra slashes when concatenating here.
                 proxyRequest.RequestUri = new Uri(destinationPrefix + httpContext.Request.Path + queryContext.QueryString);
 
                 // Suppress the original request header, use the one from the destination Uri.

[shookietea]

Just want to double check as that URL is broken. Was this moved to https://microsoft.github.io/reverse-proxy/articles/direct-forwarding.html ?

[Tratcher]

Yes, that's the new correct link.

[hamidrezashahnazari]

is Direct-Forwarding can just use for HTTP/1.1? if so what should we do with HTTP/2 and gRPC

[Tratcher]

Direct forwarding works for all protocols.