spark-avro 3.2.0 doesn't work with spark 2.2.0 (abstract OutputWriter.write)

[gnmerritt]

I'm trying to upgrade to the recently released spark 2.2.0. I'm using spark-avro version 3.2.0 and I get the following error when trying to write to an avro file.

org.apache.spark.SparkException: Job aborted.
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:215)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:173)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:173)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65)
...
Caused by: java.lang.AbstractMethodError: org.apache.spark.sql.execution.datasources.OutputWriter.write(Lorg/apache/spark/sql/catalyst/InternalRow;)V
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:327)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:258)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:256)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1375)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:261)
	... 8 more

At first glance it appeared to be related to #208 but on closer inspection the problem here is the OutputWriter.write method and not OutputWriterFactory.getFileExtension which was causing trouble with the spark 2.1.0 upgrade.

Happy to provide more info or help debug, just let me know!

[squito]

I'm seeing this too. I think its from SPARK-19085 apache/spark@b3d3962

You get a compile error if you try to compile after changing the spark version to 2.2.0:

info] Compiling 5 Scala sources to /Users/irashid/github/pub/spark-avro/target/scala-2.11/classes...
[error] /Users/irashid/github/pub/spark-avro/src/main/scala/com/databricks/spark/avro/AvroOutputWriter.scala:41: class AvroOutputWriter needs to be abstract, since method write in class OutputWriter of type (row: org.apache.spark.sql.catalyst.InternalRow)Unit is not defined
[error] (Note that org.apache.spark.sql.catalyst.InternalRow does not match org.apache.spark.sql.Row)
[error] private[avro] class AvroOutputWriter(
[error]                     ^
[error] /Users/irashid/github/pub/spark-avro/src/main/scala/com/databricks/spark/avro/AvroOutputWriter.scala:69: method write overrides nothing.
[error] Note: the super classes of class AvroOutputWriter contain the following, non final members named write:
[error] def write(row: org.apache.spark.sql.catalyst.InternalRow): Unit
[error]   override def write(row: Row): Unit = {
[error]                ^
[error] two errors found
[error] (compile:compileIncremental) Compilation failed

this was discussed in the PR for spark: apache/spark#16479 (comment)

[marcintustin]

I'm also hitting this, would love to see the PR merged

[sathish-io]

I got the same problem, any fix coming soon ?

[gnmerritt]

We're using the fix proposed in #242 and it seems to be working fine. You'll just have to build spark-avro with the patch applied and point your maven/gradle/whatever at the custom build rather than the released version.

[ritesh-dineout]

Is there any confirmation when this PR will be merged. We are facing this in production and would appreciate if this is released sooner.

[airawat]

We have similar urgency as well. Do share when this PR will be merged.

[ljank]

Since the PR has been already merged, is there any roadmap/timeline for the release? Custom build is always an option, but going the clean way feels way better :) Thank you!

[nightscape]

If you need it now, you can use the Jitpack build like this:

spark-shell --repositories https://jitpack.io --packages com.github.databricks:spark-avro:204864b6cf 

[jung-kim]

I understand there are work arounds, but what is blocking the release of this fix?

[omervk]

+1 waiting for the release. meanwhile using @nightscape's suggestion.

[reflog]

@nightscape's solution works!

[mateo41]

I know other people have asked, but is there a timeline for when the spark-avro release?

[rxin]

We want to make a release, although the eng team is pretty swamped at the moment.

For now please use the workaround provided above.