NullReferenceException is thrown when processing TaskTimedOutException from IAmazonStepFunctions client

[agotakozak]

In our project, we use the AWS X-Ray SDK for .NET to instrument our lambda function that uses IAmazonStepFunctions client to report task success or failure with SendTaskSuccessAsync and SendTaskFailureAsync to Amazon StepFunctions.

In some cases, the SendTaskSuccessAsync or SendTaskFailureAsync call throws a TaskTimedOutException, and that causes the SDK to throw a NullReferenceException when trying to serialize an ExceptionDescriptor upon Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineHandler.ProcessEndRequest is called.

Based on the exception, I could narrow it down to the ExceptionDescriptorExporter function in the JsonSegmentMarshaller class, but I could not pinpoint the exact line of code that throws the exception due to the fact that we use AOT for our lambdas and I can only debug with logging.

The stack trace:

"System.NullReferenceException: Object reference not set to an instance of an object.
   at Amazon.XRay.Recorder.Core.Internal.Emitters.JsonSegmentMarshaller.ExceptionDescriptorExporter(ExceptionDescriptor descriptor, JsonWriter writer) + 0x276
   at bootstrap!<BaseAddress>+0x12444ba
   at ThirdParty.LitJson.JsonMapper.<>c__DisplayClass40_0`1.<RegisterExporter>b__0(Object obj, JsonWriter writer) + 0x68
   at ThirdParty.LitJson.JsonMapper.WriteValue(Object obj, JsonWriter writer, Boolean writer_is_private, Int32 depth) + 0x71f
   at ThirdParty.LitJson.JsonMapper.WriteValue(Object obj, JsonWriter writer, Boolean writer_is_private, Int32 depth) + 0x4ed
   at ThirdParty.LitJson.JsonMapper.ToJson(Object obj, JsonWriter writer) + 0x3e
   at Amazon.XRay.Recorder.Core.Internal.Emitters.JsonSegmentMarshaller.CauseExporter(Cause cause, JsonWriter writer) + 0x1cc
   at bootstrap!<BaseAddress>+0x12444ba
   at ThirdParty.LitJson.JsonMapper.<>c__DisplayClass40_0`1.<RegisterExporter>b__0(Object obj, JsonWriter writer) + 0x68
   at ThirdParty.LitJson.JsonMapper.WriteValue(Object obj, JsonWriter writer, Boolean writer_is_private, Int32 depth) + 0x71f
   at ThirdParty.LitJson.JsonMapper.ToJson(Object obj, JsonWriter writer) + 0x3e
   at Amazon.XRay.Recorder.Core.Internal.Emitters.JsonSegmentMarshaller.WriteEntityFields(Entity entity, JsonWriter writer) + 0x3d0
   at Amazon.XRay.Recorder.Core.Internal.Emitters.JsonSegmentMarshaller.EntityExporter(Entity entity, JsonWriter writer) + 0x34
   at bootstrap!<BaseAddress>+0x12444ba
   at ThirdParty.LitJson.JsonMapper.<>c__DisplayClass40_0`1.<RegisterExporter>b__0(Object obj, JsonWriter writer) + 0x68
   at ThirdParty.LitJson.JsonMapper.WriteValue(Object obj, JsonWriter writer, Boolean writer_is_private, Int32 depth) + 0x876
   at ThirdParty.LitJson.JsonMapper.ToJson(Object obj) + 0xd7
   at Amazon.XRay.Recorder.Core.Internal.Emitters.JsonSegmentMarshaller.Marshall(Entity segment) + 0x35
   at Amazon.XRay.Recorder.Core.Internal.Emitters.UdpSegmentEmitter.Send(Entity segment) + 0x6c
   at Amazon.XRay.Recorder.Core.Strategies.DefaultStreamingStrategy.Stream(Entity entity, ISegmentEmitter emitter) + 0x351
   at Amazon.XRay.Recorder.Core.Strategies.DefaultStreamingStrategy.Stream(Entity entity, ISegmentEmitter emitter) + 0xd2
   at Amazon.XRay.Recorder.Core.AWSXRayRecorder.EndFacadeSegment() + 0x11c
  at Amazon.XRay.Recorder.Core.AWSXRayRecorder.ProcessEndSubsegmentInLambdaContext(Nullable`1 timestamp) + 0x217
   at Amazon.XRay.Recorder.Core.AWSXRayRecorder.EndSubsegment(Nullable`1 timestamp) + 0xbd
   at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineHandler.ProcessEndRequest(IExecutionContext executionContext) + 0x7ab
   at Amazon.XRay.Recorder.Handlers.AwsSdk.Internal.XRayPipelineHandler.<InvokeAsync>d__31`1.MoveNext() + 0x476
--- End of stack trace from previous location ---
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext() + 0x1e3
--- End of stack trace from previous location ---
   at Amazon.Runtime.Internal.CallbackHandler.<InvokeAsync>d__9`1.MoveNext() + 0x1e3
--- End of stack trace from previous location ---
   at Amazon.Runtime.Internal.ErrorCallbackHandler.<InvokeAsync>d__5`1.MoveNext() + 0x26a
--- End of stack trace from previous location ---
   at Amazon.Runtime.Internal.MetricsHandler.<InvokeAsync>d__1`1.MoveNext() + 0x5ef
--- End of stack trace from previous location ---
   at Diligent.BoardsCloud.UploadService.ConversionCompletion.Lambda.Services.ConversionCompletionService.<SendTaskSuccessAsync>d__8.MoveNext() + 0x35b
--- End of stack trace from previous location ---
   at Diligent.BoardsCloud.UploadService.ConversionCompletion.Lambda.Services.ConversionCompletionService.<SendCallbackIfConversionCompletedAsync>d__5.MoveNext() + 0xd01"

Used nuget packages:

• AWSXRayRecorder.Core v2.15.0
• AWSXRayRecorder.Handlers.AwsSdk v2.13.0
• AWSXRayRecorder.Handlers.System.Net v2.12.0
• AWSSDK.StepFunctions v3.7.402.27

The instrumentation code:

var recorder = new AWSXRayRecorderBuilder().Build();
AWSXRayRecorder.InitializeInstance(recorder: recorder);
AWSSDKHandler.RegisterXRayForAllServices();

Our target framework is net9.0.