Bump Apache Avro to 1.9.1

[asfimport]

Reporter: Fokko Driesprong / @Fokko
Assignee: Fokko Driesprong / @Fokko

Related issues:

• Upgrade to Avro 1.9.1 (is duplicated by)

_{Note: This issue was originally created as PARQUET-1645. Please see the migration documentation for further details.}

[asfimport]

Gabor Szadovszky / @gszadovszky:
@Fokko, is it really targeted to 1.11.0?

[asfimport]

Michael Heuer:
I am very curious about this – Parquet vs Avro version incompatibilities have been a source of major headache for us downstream of Apache Spark.  Will Spark be able to accept Avro 1.9.1 and Parquet 1.11.0 upgrades simultaneously?

[asfimport]

Fokko Driesprong / @Fokko:
Thanks for tagging me in here. Allow me to elaborate.

If you both update Parquet and Avro it will be fine. The problem is that Parquet used Avro's Jackson public API, and this public API has been removed in the 1.9 branch. You can see it in the PR here: 9d6fb45#diff-600376dffeb79835ede4a0b285078036

In Spark, I'd love to update Avro to the latest version since it contains a lot of updates and patches a lot of CVE's. This weekend I'll compile Spark with Parquet 1.11-SNAPSHOT and Avro 1.9 to see what we're running into.

[asfimport]

Ismaël Mejía / @iemejia:
We should definitely align Parquet (which may imply also changes) in Spark.

There might be another issue because of hive leaking old avro versions on Spark too. So this should be probably fixed on Hive's master and 'backported' to the 2.x branch of Hive too. Hopefully we can get this done before the final Spark 3 release.

For ref.
https://issues.apache.org/jira/browse/SPARK-27733
https://issues.apache.org/jira/browse/HIVE-21737