PARQUET-1320: Fix potential direct memory leak for snappy decompressor

[caneGuy]

When use NonBlockedDecompressorStream for decompress.
If we do not get any full gc for old gen.we may failed by off-heap memory leak

This patch fix this potential leak.

[caneGuy]

Can one of the owner help review this?Thanks

[rdblue]

@caneGuy, can you fix the test failures? Also, can you more clearly describe the situation this is intended to fix and how the fix works?

[rdblue]

Why was this introduced?

[caneGuy]

Used for reset below.We can remove this.But i think for the purpose of this patch,we should also refactor reset api.
Any suggestion?Thanks @rdblue

[rdblue]

Can you elaborate more than "used for reset below"? What was the intent? Also, why do you think the reset API needs to be refactored?

[caneGuy]

Sorry, @rdblue since i can't find the actual failure test case.
This patcth fix this situation, when full gc do not happen, the buffer allocated by NonBlockedDecompressorStream will not release the off-heap memory related with this direct buffer.
Here, this patch call clean whenever the buffer can be garbage collected.

[caneGuy]

I think the failure was caused by
BUILD FAILED /home/travis/build/apache/parquet-mr/thrift-0.9.3/lib/java/build.xml:298: java.net.ConnectException: Connection timed out (Connection timed out) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589)

[wangyum]

@caneGuy Can you trigger a retest?

[caneGuy]

Sorry @wangyum i don't know how to retest.Could you help trigger?
I will check the contribute document later.Thanks

[wangyum]

If you are trying to build a pull-request, you can mark it closed and then mark it opened. This will trigger a new build.
travis-ci/travis-ci#576 (comment)

[caneGuy]

Thanks @wangyum done!

[nputnam]

I was having an issue with a new service we're building that processes a bunch of snappy encoded parquet files OOM'ing. This patch seemed to resolve the issue for us. I also played around with having the cleaner only run on reset, didn't seem to resolve the issue.

[caneGuy]

May be i can refactor later @nputnam
And compressor may also have this issue