Azure Synapse Spark Notebook: UDF fails with strange conversion error

[SacredSkull]

I have a very simple UDF, which I have gradually commented out in an attempt to try and fix.
Every version of it has failed however - here is a really simple version:

Func<Column, Column> submitToCosmos = Functions.Udf<Column, Column>(row => {

    string? exceptionMessage = null;

    // try
    // {
    //     CosmosContainer.UpsertItemAsync(item,
    //         new PartitionKey($"{AggregationName}/{string.Join('/', aggregationPartitionSourceKeysList.Select(pKey => item[pKey]))}")).Wait();
    // }
    // catch (CosmosException cosmosException)
    // {
    //     exceptionMessage = cosmosException.Message;
    // }

    return Functions.Lit(exceptionMessage);
});

And I am calling it with this:

var data= spark
    .Read()
    .Format("jdbc")
    .Options(new Dictionary<string, string>
    {
        {"driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver"},
        {"user", "..."},
        {"password", "..."},
        {"url", $"jdbc:sqlserver://{sqlPoolName}.sql.azuresynapse.net:1433;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.sql.azuresynapse.net;loginTimeout=300;;databaseName=mydbname;"},
        {"dbtable", "dbo.mytable"},
    })
    .Load();

data
    .WithColumn("cosmosOpResult", submitToCosmos(data["column_I_want"]))
    .Count();

Any version of this code returns the following error:

cannot convert from Microsoft.Spark.Sql.Column [/usr/local/lib/sparkdotnet/.nuget/packages/microsoft.spark/2.0.0/lib/netstandard2.1/Microsoft.Spark.dll]' to Microsoft.Spark.Sql.Column

I should note that this isn't really the code I want, but I'm trying to get something working. I'm currently converting PySpark code to C# as the Azure Cosmos python client is much slower than the C# implementation.

The end goal is to take a full row and push it to Cosmos - and return the status of any failure or null if successful.

I had initially set up this UDF to work on an entire row, but it fails with the same strange cannot convert from Microsoft.Spark.Sql.Column error:

Func<Column, Column> submitToCosmos = Functions.Udf<Row, string>(row => {
    // ... convert row into dict ...
    // ... call cosmos client and push data up ...
    // finally return cosmos exception message (or null)
})

var data= spark
    .Read()
    .Format("jdbc")
    .Options(new Dictionary<string, string>
    {
        {"driver", "com.microsoft.sqlserver.jdbc.SQLServerDriver"},
        {"user", "..."},
        {"password", "..."},
        {"url", $"jdbc:sqlserver://{sqlPoolName}.sql.azuresynapse.net:1433;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.sql.azuresynapse.net;loginTimeout=300;;databaseName=mydbname;"},
        {"dbtable", "dbo.mytable"},
    })
    .Load();

data
    .WithColumn("cosmosOpResult", submitToCosmos(data.Schema()))
    .Count();

So to summarise, my UDFs are not working and are throwing strange conversion errors between the same type (Microsoft.Spark.Sql.Column). I think if I get the basic version of the UDF working I should be able to get going with the full fat UDF I need.

I've just checked there and the Spark pool is Spark 3.1 and .NET Spark 2.0 which seems to line up with the error message.

Oh, and additionally, I have tried to eliminate serialisation by doing as the docs suggest and having the UDF in a different cell to the calling code - same error.