ERROR: transform: send_aws_s3: data could not be added to batch

[viraj-lunani]

We are getting the above error on the following transform

local sub = import "../../substation.libsonnet";
{
  concurrency: 2,
  transforms: [
      sub.tf.send.aws.s3(settings={
        bucket_name: "<bucket1>",
        file_path: {
          prefix: "...",
          uuid: true,
          time_format: "2006/01/02/15/04"
        },
        auxiliary_transforms: sub.pattern.tf.fmt.jsonl
      }),
    ]
    +[
      sub.tf.send.aws.s3(settings={
        bucket_name: "<bucket2>",
        file_path: {
          prefix: "...",
          uuid: true,
          time_format: "2006/01/02/15/04"
        },
        auxiliary_transforms: internal.tf.fmt.jsonl(
            batch={
                size: 10000000,
                count: 5000
            }
        ),
        batch: { 
          size: 10000000,
          count: 5000
        }
      }),
    ]
}

The jsonl function looks like

      jsonl(batch): [
        sub.pattern.tf.conditional(
          condition=sub.cnd.meta.negate({ inspector: sub.cnd.fmt.json() }),
          transform=sub.tf.util.drop(),
        ),
        sub.tf.agg.to.string({
          separator: '\n',
          batch: batch,
        }),
        sub.tf.str.append({ suffix: '\n' }),
      ],

After digging into the source code it seems to be some sort of batch misconfiguration. I thought originally that it was a misconfiguration with a mismatch in batch settings between the sub.pattern.tf.fmt.jsonl since this uses an agg transform that has batch settings but after trying to reproduce the error locally with two file send

{
    transforms: [
        sub.tf.send.file(settings={
        file_path: {
          prefix: "out2/",
          uuid: true,
          time_format: "2006/01/02/15/04"
        },
        auxiliary_transforms: sub.pattern.tf.fmt.jsonl,
      })
    ]
    + [
        sub.tf.send.file(settings={
        file_path: {
          prefix: "out/",
          uuid: true,
          time_format: "2006/01/02/15/04"
        },
        auxiliary_transforms: sub.pattern.tf.fmt.jsonl,
        batch: { 
          size: 10000000,
          count: 5000
        }
      })
    ]
}

I am not getting the same error. Any ideas on what the issue could be here? We are on v1.1.0

[viraj-lunani]

Answered in #218

[viraj-lunani]

Hey just wanted to follow up on this. We are running into this issue with another log source of ours. It only happens occasionally. I have followed the guidance on #218 and adjusted the batch size of the jsonl agg and the send function to match (this can be seen in the config above) but we still seem to occasionally be seeing "data could not be added to batch error".

[jshlbrd]

@viraj-lunani If you haven't already, then review the compiled JSON that you have deployed in production (in AWS AppConfig) to confirm if the batch sizes are correct. If the configuration deployed in production doesn't match the Jsonnet configuration, then your CI / CD pipeline may have a bug; otherwise, the data source may be producing individual events that are larger than the batch settings in your configuration.

If you think you've found a bug and want us to review, then we'll need these items:

• Full Jsonnet configuration
• Compiled JSON configuration
• Samples from the data source (ideally the log entry that causes the issue)

We also accept pull requests for bug fixes. 👍

[viraj-lunani]

Still curious why this result wasn't reproduced when running locally

[jshlbrd]

@viraj-lunani Your configurations are different and may produce different results. You said this is causing errors in production ...

local sub = import "../../substation.libsonnet";
{
  concurrency: 2,
  transforms: [
      sub.tf.send.aws.s3(settings={
        bucket_name: "<bucket1>",
        file_path: {
          prefix: "...",
          uuid: true,
          time_format: "2006/01/02/15/04"
        },
        auxiliary_transforms: sub.pattern.tf.fmt.jsonl
      }),
    ]
    +[
      sub.tf.send.aws.s3(settings={
        bucket_name: "<bucket2>",
        file_path: {
          prefix: "...",
          uuid: true,
          time_format: "2006/01/02/15/04"
        },
        auxiliary_transforms: internal.tf.fmt.jsonl(
            batch={
                size: 10000000,
                count: 5000
            }
        ),
        batch: { 
          size: 10000000,
          count: 5000
        }
      }),
    ]
}

... and this is not causing errors locally ...

{
    transforms: [
        sub.tf.send.file(settings={
        file_path: {
          prefix: "out2/",
          uuid: true,
          time_format: "2006/01/02/15/04"
        },
        auxiliary_transforms: sub.pattern.tf.fmt.jsonl,
      })
    ]
    + [
        sub.tf.send.file(settings={
        file_path: {
          prefix: "out/",
          uuid: true,
          time_format: "2006/01/02/15/04"
        },
        auxiliary_transforms: sub.pattern.tf.fmt.jsonl,
        batch: { 
          size: 10000000,
          count: 5000
        }
      })
    ]
}

These are different (and are mixing use of the JSONL pattern and your custom function) so the results can be different. The results can also be different if the input isn't the same. The best way to test for this type of error is by using an identical local configuration to what is in production and testing with the same input data.