.NET 8 AOT Lambda w/ X-Ray

[tmschlot]

When we add X-Ray to a working AOT-compiled Lambda function, we get the following exception:

Unhandled Exception: System.MissingMethodException: No parameterless constructor defined for type 'System.Collections.Generic.Dictionary`2[System.String,Amazon.XRay.Recorder.Handlers.AwsSdk.Entities.AWSServiceHandler]'.
    at System.ActivatorImplementation.CreateInstance(Type, Boolean) + 0x108
    at ThirdParty.LitJson.JsonMapper.ReadValue(Type inst_type, JsonReader reader) + 0x437
    at ThirdParty.LitJson.JsonMapper.ReadValue(Type inst_type, JsonReader reader) + 0x537
    at ThirdParty.LitJson.JsonMapper.ToObject[T](TextReader reader) + 0x66
    at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineHandler.GetAWSServiceHandlerManifest(Stream stream) + 0x66
    at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineHandler.GetDefaultAWSWhitelist() + 0x44
    at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineHandler..ctor() + 0x21
    at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineCustomizer.Customize(Type serviceClientType, RuntimePipeline pipeline) + 0x79
    at Amazon.Runtime.Internal.RuntimePipelineCustomizerRegistry.ApplyCustomizations(Type type, RuntimePipeline pipeline) + 0xb0
    at LambdaFunction.<Main>d__15.MoveNext() + 0x134

This .NET 7 sample has X-Ray included, but this commit removes it from the .NET 8 sample. There is no explanation for why it was removed, though. Was it this same issue? I'm just trying to figure out if I'm doing something wrong or if a gap remains with the X-Ray support.

I have attempted to "root" various assemblies and modify trim settings to make this work. The only way I have found to get around the problem is to not publish with AOT and to a self-contained executable. Any other info would be helpful.

Thanks.

[Beau-Gosse-dev]

Thanks for opening this issue. Could you please provide the output of the dotnet publish command so that we can see what trimmer warnings (if any) are being shown?

[tmschlot]

After updating some references -- and rooting another assembly (System.Linq.Expressions) -- I got past the error described above and now see...

Unhandled Exception: System.ArgumentException: The value "JsonData object" is not of type "Amazon.XRay.Recorder.Handlers.AwsSdk.Entities.AWSServiceHandler" and cannot be used in this generic collection. (Parameter 'value')
   at System.ThrowHelper.ThrowWrongValueTypeArgumentException[T](T, Type) + 0x17
   at System.Collections.Generic.Dictionary`2.System.Collections.IDictionary.Add(Object, Object) + 0xbb
   at ThirdParty.LitJson.JsonMapper.ReadValue(Type inst_type, JsonReader reader) + 0x5f0
   at ThirdParty.LitJson.JsonMapper.ReadValue(Type inst_type, JsonReader reader) + 0x55e
   at ThirdParty.LitJson.JsonMapper.ToObject[T](TextReader reader) + 0x5f
   at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineHandler.GetAWSServiceHandlerManifest(Stream stream) + 0x6d
   at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineHandler.GetDefaultAWSWhitelist() + 0x47
   at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineHandler..ctor() + 0x1e
   at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineCustomizer.Customize(Type serviceClientType, RuntimePipeline pipeline) + 0x77
   at Amazon.Runtime.Internal.RuntimePipelineCustomizerRegistry.ApplyCustomizations(Type type, RuntimePipeline pipeline) + 0xad
   at LambdaFunction.<Main>d__14.MoveNext() + 0x73

Based on the stack trace, it appears to be a couple of steps further along.

The trim warnings I see are...

... docker run: /tmp/dotnet/.nuget/packages/amazon.lambda.runtimesupport/1.10.0/lib/net8.0/Amazon.Lambda.RuntimeSupport.dll : warning IL3053: Assembly 'Amazon.Lambda.RuntimeSupport' produced AOT analysis warnings. [/tmp/source/src/EventPublisher/EventPublisher.csproj]
... docker run: /tmp/dotnet/.nuget/packages/awsxrayrecorder.handlers.awssdk/2.12.0/lib/net6.0/AWSXRayRecorder.Handlers.AwsSdk.dll : warning IL2104: Assembly 'AWSXRayRecorder.Handlers.AwsSdk' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries [/tmp/source/src/EventPublisher/EventPublisher.csproj]
... docker run: /tmp/dotnet/.nuget/packages/awsxrayrecorder.core/2.14.0/lib/net6.0/AWSXRayRecorder.Core.dll : warning IL2104: Assembly 'AWSXRayRecorder.Core' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries [/tmp/source/src/EventPublisher/EventPublisher.csproj]   
... docker run: /tmp/dotnet/.nuget/packages/awsxrayrecorder.core/2.14.0/lib/net6.0/AWSXRayRecorder.Core.dll : warning IL3053: Assembly 'AWSXRayRecorder.Core' produced AOT analysis warnings. [/tmp/source/src/EventPublisher/EventPublisher.csproj]
... docker run: /_/sdk/src/Core/AWSXRayRecorderImpl.cs(746): warning IL3000: Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl.PopulateContexts(): 'System.Reflection.Assembly.Location.get' always returns an empty string for assemblies embedded in a single-file app. If the path to the app directory is needed, consider calling 'System.AppContext.BaseDirectory'. [/tmp/source/src/EventPublisher/EventPublisher.csproj]
... docker run: /tmp/dotnet/.nuget/packages/awssdk.core/3.7.302.10/lib/net8.0/AWSSDK.Core.dll : warning IL3053: Assembly 'AWSSDK.Core' produced AOT analysis warnings. [/tmp/source/src/EventPublisher/EventPublisher.csproj]
... docker run: /tmp/dotnet/.nuget/packages/awssdk.core/3.7.302.10/lib/net8.0/AWSSDK.Core.dll : warning IL2104: Assembly 'AWSSDK.Core' produced trim warnings. For more information see https://aka.ms/dotnet-illink/libraries [/tmp/source/src/EventPublisher/EventPublisher.csproj]
... docker run: /tmp/dotnet/.nuget/packages/runtime.linux-x64.microsoft.dotnet.ilcompiler/8.0.3/framework/System.Linq.Expressions.dll : warning IL3053: Assembly 'System.Linq.Expressions' produced AOT analysis warnings. [/tmp/source/src/EventPublisher/EventPublisher.csproj]

And the assemblies I've rooted are...

    <ItemGroup>
        <TrimmerRootAssembly Include="Amazon.Lambda.RuntimeSupport" />
        <TrimmerRootAssembly Include="AWSSDK.Core" />
        <TrimmerRootAssembly Include="AWSXRayRecorder.Core" />
        <TrimmerRootAssembly Include="AWSXRayRecorder.Handlers.AwsSdk" />
        <TrimmerRootAssembly Include="System.Linq.Expressions" />
    </ItemGroup>

I don't see any other things to root based on the trim warnings. Any other ideas?

[Beau-Gosse-dev]

Your code is now throwing at this line of code it looks like: https://github.com/aws/aws-xray-sdk-dotnet/blob/0867b0601a62cebfd744ba857f03284fd5d2cc81/sdk/src/Handlers/AwsSdk/Internal/XRayPipelineHandler.cs#L293

Which calls this: https://github.com/aws/aws-xray-sdk-dotnet/blob/0867b0601a62cebfd744ba857f03284fd5d2cc81/sdk/src/Core/ThirdParty/LitJson/JsonMapper.cs#L862

It seems like a similar issue was opened here: aws/aws-xray-sdk-dotnet#295

The xray libraries haven't been made trim-safe as you can see by the requirement to use TrimmerRootAssembly (which isn't a fool-proof fix, since runtime types could change). One common misconception is that the assembly showing the trimming warning is what needs to be added to the TrimmerRootAssembly, however it's really just saying this assembly is using unsafe code and we don't know what type that assembly will need at runtime. For example, you could imagine an assembly having this code: Type type = Type.GetType(Console.ReadLine()); We don't know what will come into the ReadLine. This shows that even if you can run though your code once without exception, it's no guarantee that you won't run into future runtime trimming errors. The only guarantee is if there are no trim warnings at build time.

It seems like the code is trying to convert this file (or a similar file) into a dictionary of known AWS Service types, which is why this error only seems to come up once you try to trace calls to another service. (Basic xray for an isolated Lambda seem to work fine for me.)

I would have thought those types were in AWSXRayRecorder.Handlers.AwsSdk but it seems like something is getting trimmed out and the ThirdParty JSON parser is getting confused. It might be best to ask the xray people about this by either commenting on the above issue or creating a new one.

[tmschlot]

Thanks for the help, @Beau-Gosse-dev. I'll follow up with the X-Ray people.

[burciugi]

@tmschlot did you find any workarounds?