Skip to content

Backward incompatible change introduced in 1.8 #2176

@asfimport

Description

@asfimport

ThriftSchemaConverter#convert(StructType struct) introduced a backward incompatible logic in Parquet 1.8, If StructType's deprecated constructor is used to create StructTypes, and this convert method is used, then the client receives the following exception:

org.apache.parquet.ShouldNeverHappenException: Encountered UNKNOWN StructOrUnionType

	at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.isUnion(ThriftSchemaConvertVisitor.java:342)
	at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.visit(ThriftSchemaConvertVisitor.java:218)
	at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.visit(ThriftSchemaConvertVisitor.java:74)
	at org.apache.parquet.thrift.struct.ThriftType$StructType.accept(ThriftType.java:269)
	at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.convert(ThriftSchemaConvertVisitor.java:93)
	at org.apache.parquet.thrift.ThriftSchemaConverter.convert(ThriftSchemaConverter.java:75)

To illustrate the issue, here's a test case, which passes on older versions, but fails on master:

  @Test
  public void testIncompatibleThriftConverterChange() {
    ThriftSchemaConverter converter = new ThriftSchemaConverter();

    ThriftType.StructType structType = new ThriftType.StructType(
      asList(
        new ThriftField("a", (short)1, REQUIRED, new ThriftType.StringType()),
        new ThriftField("b", (short) 2, REQUIRED, new ThriftType.StringType())
      )
    );
    converter.convert(structType);
  }

Reporter: Nándor Kollár / @nandorKollar
Assignee: Nándor Kollár / @nandorKollar

Related issues:

PRs and other links:

Note: This issue was originally created as PARQUET-1305. Please see the migration documentation for further details.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions