Checking of column has empty array #1070
mosharafMS
started this conversation in
General
Replies: 1 comment
-
solved it! found a When function in the Microsoft.Spark.Sql.Functions class. This way, I'll use identical way to pyspark |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
In pyspark when having an array column, I can check if the array Size is 0 and replace the column with null value like this
.withColumn('joinedColumns',when(size(df.joinedColumns)==0, None).otherwise(df.joinedColumns))
However in C# doing the same with this line
df=df.WithColumn("JoinedTables",((Size(df["JoinedTables"]) == 0) ? Lit(null) : df["JoinedTables"]));
or
df=df.WithColumn("JoinedTables",((Size(df["JoinedTables"]) == 0) ? null : df["JoinedTables"]));
It returns error
(1,34): error CS0029: Cannot implicitly convert type 'Microsoft.Spark.Sql.Column' to 'bool'
Schema is like this
|-- IsCopyStatement: boolean (nullable = true)
|-- IsDeleteStatement: boolean (nullable = true)
|-- IsInsertStatement: boolean (nullable = true)
|-- IsSelectStatement: boolean (nullable = true)
|-- IsUpdateStatement: boolean (nullable = true)
|-- JoinedColumns: array (nullable = true)
| |-- element: string (containsNull = true)
|-- JoinedTables: array (nullable = true)
| |-- element: string (containsNull = true)
|-- SQLCommand: string (nullable = true)
|-- SelectStatementFrom: array (nullable = true)
| |-- element: string (containsNull = true)
Beta Was this translation helpful? Give feedback.
All reactions