Snappy Decompressor/Compressor may cause direct memory leak

[asfimport]

In out production environment, we encountered a direct memory oom issues caused by the direct buffer not released in time.

After carefully checked the code, it seems that the some methods of SnappyDecompressor/SnappyCompressor would not release the direct buffer manually. If too much direct memory allocated and no GC happens, this bug may result in direct memory oom.

Moreover, if the -XX:+DisableImplicitGC jvm option is specified, the direct memory oom would happen easily for large datasets.

Seems that the problem still exist in the latest code.

Environment: parquet-1.8.1

spark2.1
Reporter: liupengcheng / @liupc
Assignee: liupengcheng / @liupc

Related issues:

• Fast clean unused direct memory when decompress (is duplicated by)
• TestSnappy() throws OOM exception with Parquet-1485 change (is related to)

PRs and other links:

• GitHub Pull Request #581

_{Note: This issue was originally created as PARQUET-1485. Please see the migration documentation for further details.}

[asfimport]

Nate Putnam:
I've also ran into this issue. Applying the patch outlined in https://issues.apache.org/jira/browse/PARQUET-1320 fixed the issue for me. 

[asfimport]

liupengcheng / @liupc:
[~nputnam] OK, seems that it's my colleague's PR. we are pointing to the same issue.

[asfimport]

liupengcheng / @liupc:
User 'liupengcheng' has created a pull request for this issue: #581

[asfimport]

Yuming Wang / @wangyum:
Could we backport this patch for parquet-1.10.x?

[asfimport]

liupengcheng / @liupc:
@wangyum I think we currently dont backport the patch too all prior versions is for consideration of maintenance, and hope users can keep up new versions. Actually, you can backport this by yourself.