Dictionary filter skips null values when evaluating not-equals.

[asfimport]

This was discovered in Spark, see SPARK-26677. From the Spark PR:

// Repeat the values to get dictionary encoding.
Seq(Some("A"), Some("A"), None).toDF.repartition(1).write.mode("overwrite").parquet("/tmp/foo")
spark.read.parquet("/tmp/foo").where("NOT (value <=> 'A')").show()
+-----+
|value|
+-----+
+-----+

// Use plain encoding.
Seq(Some("A"), None).toDF.repartition(1).write.mode("overwrite").parquet("/tmp/bar")
spark.read.parquet("/tmp/bar").where("NOT (value <=> 'A')").show()
+-----+
|value|
+-----+
| null|
+-----+

This is a correctness issue.

Reporter: Ryan Blue / @rdblue
Assignee: Ryan Blue / @rdblue

Related issues:

• Release Parquet Java 1.10.1 (blocks)
• Incorrect results of not(eqNullSafe) when data read from Parquet file (causes)

PRs and other links:

• GitHub Pull Request #603

_{Note: This issue was originally created as PARQUET-1510. Please see the migration documentation for further details.}

[asfimport]

Dongjoon Hyun / @dongjoon-hyun:
Ur, @rdblue, is this Improvement instead of Bug?

[asfimport]

Ryan Blue / @rdblue:
Fixed metadata.