-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Closed
Milestone
Description
ThriftSchemaConverter#convert(StructType struct) introduced a backward incompatible logic in Parquet 1.8, If StructType's deprecated constructor is used to create StructTypes, and this convert method is used, then the client receives the following exception:
org.apache.parquet.ShouldNeverHappenException: Encountered UNKNOWN StructOrUnionType
at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.isUnion(ThriftSchemaConvertVisitor.java:342)
at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.visit(ThriftSchemaConvertVisitor.java:218)
at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.visit(ThriftSchemaConvertVisitor.java:74)
at org.apache.parquet.thrift.struct.ThriftType$StructType.accept(ThriftType.java:269)
at org.apache.parquet.thrift.ThriftSchemaConvertVisitor.convert(ThriftSchemaConvertVisitor.java:93)
at org.apache.parquet.thrift.ThriftSchemaConverter.convert(ThriftSchemaConverter.java:75)To illustrate the issue, here's a test case, which passes on older versions, but fails on master:
@Test
public void testIncompatibleThriftConverterChange() {
ThriftSchemaConverter converter = new ThriftSchemaConverter();
ThriftType.StructType structType = new ThriftType.StructType(
asList(
new ThriftField("a", (short)1, REQUIRED, new ThriftType.StringType()),
new ThriftField("b", (short) 2, REQUIRED, new ThriftType.StringType())
)
);
converter.convert(structType);
}Reporter: Nándor Kollár / @nandorKollar
Assignee: Nándor Kollár / @nandorKollar
Related issues:
PRs and other links:
Note: This issue was originally created as PARQUET-1305. Please see the migration documentation for further details.