Commits on Oct 4, 2024

1. Fix NPEs in TypesAccumulator …

   We've seen exceptions in spark executors like:
   
   ```
   java.lang.NullPointerException: Cannot invoke "scala.collection.mutable.Set.isEmpty()" because the return value of "com.snowplowanalytics.snowplow.rdbloader.transformer.batch.spark.TypesAccumulator.accum()" is null
   ```
   
   The error is coming from our Spark Accumulator for accumulating Iglu
   types. This is similar to [an issue previously seen][1] in Spark's own
   `CollectionAccumulator`. That issue [was fixed in Spark][2] by making
   the accumulator's internal state non-final, and synchronizing access to
   the internal state. So here we make the exact same change to our own
   Accumulator.
   
   It is a rare race condition which is hard to reproduce.
   
   [1]: https://issues.apache.org/jira/browse/SPARK-20977
   [2]: apache/spark#31540

   

   istreeter authored and spenes committed Oct 4, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for d540e99
   • Browse repository at this point

   Copy the full SHA

   d540e99 View commit details

   Browse the repository at this point in the history