Dapr.Extensions.Configuration.AddDaprSecretStore() fails if the sidecar and app are started separately

[juris-greitans]

Expected Behavior

When starting the application after Dapr sidecar, AddDaprSecretStore() should not time out and throw exceptions:

using Dapr.Client;
using Dapr.Extensions.Configuration;

var builder = WebApplication.CreateBuilder(args);

using var client = new DaprClientBuilder().Build();
builder.Configuration.AddDaprSecretStore("secrets", client, TimeSpan.FromSeconds(5));

var app = builder.Build();
app.Run();

Actual Behavior

When running the sidecar and the app separately, exception is thrown:

# Run in separate terminal
$ dapr run --app-id "app" --app-port "5200"

and then the app:

$ dotnet run
Building...
Unhandled exception. System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Dapr.Client.DaprClientGrpc.WaitForSidecarAsync(CancellationToken cancellationToken)
   at Dapr.Extensions.Configuration.DaprSecretStore.DaprSecretStoreConfigurationProvider.LoadAsync()
   at Dapr.Extensions.Configuration.DaprSecretStore.DaprSecretStoreConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
   at Dapr.Extensions.Configuration.DaprSecretStoreConfigurationExtensions.AddDaprSecretStore(IConfigurationBuilder configurationBuilder, String store, DaprClient client, TimeSpan sidecarWaitTimeout, IReadOnlyDictionary`2 metadata)
   at Program.<Main>$(String[] args) in C:\src\dapr-bug-001\src\app\Program.cs:line 7

Steps to Reproduce the Problem

I created a Github repo containing minimal .NET 7 project to repeat the issue. The README.md contains more details.
In short:

1. Run the sidecar, but do not specify the application, e.g. dapr run --app-id "app" --app-port "5200" --components-path "components"
2. Run the application after sidecar has started, e.g. dotnet run

It seems to me that this behavior was first described in #779 - the sidecar is waiting when the app will start listening on its port, but the app is waiting for sidecar's healthz/outbound endpoint to return 2xx statuscode during startup.
Also #983 could be related.

Release Note

RELEASE NOTE:

[geertdoornbos]

When you're more specific about the dapr ports being used, it works:

dapr run --app-id "app" --app-port "5200" --components-path "components" --dapr-http-port 3500 --dapr-grpc-port 50001

Apparently, the CLI does not use the default ports.

Or set the desired port numbers for the dapr client:

using var client = new DaprClientBuilder()
    .UseHttpEndpoint("http://127.0.0.1:3500")
    .UseGrpcEndpoint("http://127.0.0.1:50001")
    .Build();

[JasonRodman]

Or set the desired port numbers for the dapr client:

using var client = new DaprClientBuilder()
    .UseHttpEndpoint("http://127.0.0.1:3500")
    .UseGrpcEndpoint("http://127.0.0.1:50001")
    .Build();

This doesn't work. I looked into the SDK code and it is setting these by default already, so setting them yourself to the same ports it already uses does nothing. It still causes the sidecar to block waiting for the app to run. Has anyone else looked into this to see if there are any other options to get this to work? It makes it impossible to use the secret store component.

[geertdoornbos]

It is a (working) example of how you can set the ports you want (even if it's the default). It must match sidecar http/grpc.

[JasonRodman]

It is a (working) example of how you can set the ports you want (even if it's the default). It must match sidecar http/grpc.

I double checked the environment variables in the pods and it matches them. It still doesn't fix the issue. I think the problem is more that the AddDaprSecretStore method is not waiting long enough for the dapr sidecar to load. I upped the timeout and it seems to have solved it for the moment.

[evanvenell]

Is there any update on this issue?

[WhitWaldo]

@evanvenell Is this an issue you're presently experiencing? The original issue was opened several Dapr versions ago and both the runtime and client have seen significant changes since then.

[phuot]

Is there any update on this issue? I also experiencing the same from time to time.

[WhitWaldo]

@phuot There are changes being made to the runtime as part of 1.15 (due to be released later this month) that change the behavior between the SDKs and the runtime. Specifically, rather than the SDK polling the runtime for readiness and then passing data, the runtime should trap all inbound requests and then process them when ready.

Until that work is completed, I hesitate to make any changes on the SDK since they'd be obsolete when that change is rolled out to all the building blocks (instead of its initial targeting of the Workflow block).