Resource Discovery error: [queriedSubscriptions] cannot be Null.

[akasprzok]

Heya!
I've got both resource discovery and scraper running, but seem to be misconfiguring something.

Seeing this error in resource discovery when queried by scraper:

[20:12:00 WRN] {"DependencyId": null, "DependencyType": "Azure Resource Graph", "DependencyName": "Query", "DependencyData": "Resources\n| where type =~ 'microsoft.storage/storageaccounts'\n| project subscriptionId, resourceGroup, type, name\n| order by subscriptionId asc, resourceGroup asc, type asc, name asc", "TargetName": "Query", "ResultCode": null, "IsSuccessful": false, "StartTime": "2022-12-22T20:12:00.1882357 +00:00", "Duration": "00:00:00.3495306", "Context": {"Query": "Resources\n| where type =~ 'microsoft.storage/storageaccounts'\n| project subscriptionId, resourceGroup, type, name\n| order by subscriptionId asc, resourceGroup asc, type asc, name asc", "QueryName": "storage-account-landscape", "TelemetryType": "Dependency"}, "$type": "DependencyLogEntry"}
[20:12:00 INF] Executed action Promitor.Agents.ResourceDiscovery.Controllers.v2.DiscoveryV2Controller.Get (Promitor.Agents.ResourceDiscovery) in 475.4835ms
[20:12:00 FTL] [queriedSubscriptions] cannot be Null. (Parameter 'queriedSubscriptions')
System.ArgumentNullException: [queriedSubscriptions] cannot be Null. (Parameter 'queriedSubscriptions')
   at GuardNet.Guard.For[TException](Func`1 predicate, TException exception)
   at Promitor.Agents.ResourceDiscovery.Graph.Exceptions.QueryContainsInvalidSubscriptionException..ctor(List`1 queriedSubscriptions) in /src/Promitor.Agents.ResourceDiscovery/Exceptions/QueryContainsInvalidSubscriptionException.cs:line 15
   at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.<>c__DisplayClass16_0`1.<<InteractWithAzureResourceGraphAsync>b__1>d.MoveNext() in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 154
--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.InteractWithAzureResourceGraphAsync[TResponse](String queryName, String query, Func`2 interactionFunc, List`1 targetSubscriptions) in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 181
   at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.QueryAsync(String queryName, String query, Int32 pageSize, Int32 currentPage, List`1 targetSubscriptions) in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 106
   at Promitor.Agents.ResourceDiscovery.Graph.AzureResourceGraph.QueryTargetSubscriptionsAsync(String queryName, String query, Int32 pageSize, Int32 currentPage) in /src/Promitor.Agents.ResourceDiscovery/AzureResourceGraph.cs:line 75
   at Promitor.Agents.ResourceDiscovery.Graph.Repositories.AzureResourceRepository.GetResourcesAsync(String resourceDiscoveryGroupName, Int32 pageSize, Int32 currentPage) in /src/Promitor.Agents.ResourceDiscovery/Repositories/AzureResourceRepository.cs:line 78
   at Promitor.Agents.ResourceDiscovery.Graph.Repositories.CachedAzureResourceRepository.GetResourcesAsync(String resourceDiscoveryGroupName, Int32 pageSize, Int32 currentPage) in /src/Promitor.Agents.ResourceDiscovery/Repositories/CachedAzureResourceRepository.cs:line 52
   at Promitor.Agents.ResourceDiscovery.Controllers.v2.DiscoveryV2Controller.Get(String resourceDiscoveryGroup, Int32 pageSize, Int32 currentPage) in /src/Promitor.Agents.ResourceDiscovery/v2/DiscoveryV2Controller.cs:line 83
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Prometheus.Client.HttpRequestDurations.HttpRequestDurationsMiddleware.Invoke(HttpContext context)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Promitor.Agents.Core.Middleware.AgentVersionMiddleware.Invoke(HttpContext httpContext) in /src/Promitor.Agents.Core/AgentVersionMiddleware.cs:line 39
   at Arcus.WebApi.Logging.Correlation.CorrelationMiddleware.Invoke(HttpContext httpContext, HttpCorrelation service)
   at Arcus.WebApi.Logging.RequestTrackingMiddleware.TrackRequest(HttpContext httpContext, Exclude attributeExcludeFilter, StatusCodeRange[] attributeTrackedStatusCodes)
   at Arcus.WebApi.Logging.RequestTrackingMiddleware.TrackRequest(HttpContext httpContext, Exclude attributeExcludeFilter, StatusCodeRange[] attributeTrackedStatusCodes)
   at Arcus.WebApi.Logging.RequestTrackingMiddleware.Invoke(HttpContext httpContext)
   at Arcus.WebApi.Logging.ExceptionHandlingMiddleware.Invoke(HttpContext context, ILoggerFactory loggerFactory)
[20:12:00 INF] Request finished HTTP/1.1 GET http://stable-azure-exporter-resource-discovery:8889/api/v2/resources/groups/storage-account-landscape/discover?currentPage=1 - - - 500 0 - 481.7548ms

Same for the cosmosdb query (see configuration yamls below).

resource-discovery-declaration.yaml:

    version: v1
    azureLandscape:
      tenantId: <tenant_id>
      subscriptions:
      - <subscription_id>
    resourceDiscoveryGroups:
    - name: api-gateways
      type: ApiManagement
    - name: app-plan-landscape
      type: AppPlan
    - name: container-instances
      type: ContainerInstance
    - name: container-registry-landscape
      type: ContainerRegistry
    - name: cosmos-accounts
      type: CosmosDb
    - name: device-provisioning-service
      type: DeviceProvisioningService
    - name: event-hubs-landscape
      type: EventHubs
    - name: service-bus-landscape
      type: ServiceBusQueue
    - name: storage-account-landscape
      type: StorageAccount

metrics-declaration.yaml:

    version: v1
  azureMetadata:
    tenantId: <tenant_id>
    subscriptionId: <subscription_id>
    resourceGroupName: <resource_group_name>
  metricDefaults:
    aggregation:
      interval: 00:05:00
    scraping:
      schedule: "0 * * ? * *"
  metrics:
  - name: azure_cosmos_db_total_requests
    description: "Demo cosmos query"
    resourceType: CosmosDb
    azureMetricConfiguration:
      metricName: TotalRequests
      aggregation:
        type: Count
    resourceDiscoveryGroups:
    - name: cosmos-accounts
  - name: azure_storage_account_capacity
    description: "The average capacity used in the storage account"
    resourceType: StorageAccount
    azureMetricConfiguration:
      metricName: UsedCapacity
      aggregation:
        type: Average
    resourceDiscoveryGroups:
    - name: storage-account-landscape

I'm not a NET native, but this reads to me like the subscription might not be getting passed to the query correctly?

[tomkerkhove]

Sorry for the slow response but I was on vacation.

It looks like it indeed, are you sure that the configured subscription ID is valid and that the intention of your configuration is correct? It looks like the subscription is empty.

Would you mind sharing the startup logs? It should show what it detects to be configured.

[akasprzok]

Argh, I apologize, I since figured this out.

Was indeed an invalid subscription ID.

[tomkerkhove]

Can you elaborate on what the subscription ID looked like? Because I might want to improve the exception handling as it seems it did not detect a subscription ID.

[akasprzok]

It was a valid UUID however it was not the subscription ID. Quite odd.

I'm sorry, I can't give you more info at this point as we decided to use a different exporter.

[tomkerkhove]

No worries - May I ask what exporter and why you moved away from Promitor?

[akasprzok]

We ended up using https://github.com/webdevops/azure-metrics-exporter.
The config was easier to wrangle, configuration via serviceMonitor means we could iterate quickly. Also packaged as a single container.