Skip to content

[SPARK-21985][PySpark] PairDeserializer is broken for double-zipped RDDs #19226

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

[SPARK-21985][PySpark] PairDeserializer is broken for double-zipped RDDs #19226

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4a9eb93
remove check and add tests
aray Sep 14, 2017
0d64a6d
woops
aray Sep 14, 2017
e99ed23
convert batches to list in PairDeserializer so we can check the len a…
aray Sep 14, 2017
66477f8
update doc and test
aray Sep 14, 2017
54b7fd0
only convert to list if __len__ not available, set number of partitio…
aray Sep 15, 2017
f6d42f4
style
aray Sep 15, 2017
ff1417e
add comment per review
aray Sep 15, 2017
5282ee5
nit comment
aray Sep 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions python/pyspark/serializers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,8 @@ def _load_stream_without_unbatching(self, stream):
key_batch_stream = self.key_ser._load_stream_without_unbatching(stream)
val_batch_stream = self.val_ser._load_stream_without_unbatching(stream)
for (key_batch, val_batch) in zip(key_batch_stream, val_batch_stream):
key_batch = list(key_batch)
val_batch = list(val_batch)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we fix the doc in Serializer._load_stream_without_unbatching to say, it returns iterator of iterables?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 66477f8

Copy link
Member

@HyukjinKwon HyukjinKwon Sep 14, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I had to be clear. Actually, I meant if Serializer._load_stream_without_unbatching works as documented an iterator of deserialized batches (lists), everything should have worked fine. So, I think the reverse is actually more correct because PairDeserializer and CartesianDeserializer do not follow this.

I am okay with the current change too but I believe the reverse is better because I think we could prevent such issues in the future and make the things simpler. WDYT @aray and @holdenk ?

if len(key_batch) != len(val_batch):
raise ValueError("Can not deserialize PairRDD with different number of items"
" in batches: (%d, %d)" % (len(key_batch), len(val_batch)))
Expand Down
12 changes: 12 additions & 0 deletions python/pyspark/tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -644,6 +644,18 @@ def test_cartesian_chaining(self):
set([(x, (y, y)) for x in range(10) for y in range(10)])
)

def test_zip_chaining(self):
# Tests for SPARK-21985
rdd = self.sc.parallelize(range(10), 2)
Copy link
Member

@HyukjinKwon HyukjinKwon Sep 14, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test case already passes without this change, doesn't it?

self.assertSetEqual(
set(rdd.zip(rdd).zip(rdd).collect()),
set([((x, x), x) for x in range(10)])
)
self.assertSetEqual(
set(rdd.zip(rdd.zip(rdd)).collect()),
set([(x, (x, x)) for x in range(10)])
)

def test_deleting_input_files(self):
# Regression test for SPARK-1025
tempFile = tempfile.NamedTemporaryFile(delete=False)
Expand Down