[SPARK-4945] [SQL] Add overwrite option support for SchemaRDD.saveAsParquetFile

[chenghao-intel]

No description provided.

[SparkQA]

Test build #24755 has started for PR 3780 at commit b776284.

• This patch merges cleanly.

[SparkQA]

Test build #24755 has finished for PR 3780 at commit b776284.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24755/
Test FAILed.

[SparkQA]

Test build #24758 has started for PR 3780 at commit 52adf46.

• This patch merges cleanly.

[SparkQA]

Test build #24758 has finished for PR 3780 at commit 52adf46.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24758/
Test FAILed.

[SparkQA]

Test build #24802 has started for PR 3780 at commit a7a380f.

• This patch merges cleanly.

[SparkQA]

Test build #24802 has finished for PR 3780 at commit a7a380f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24802/
Test PASSed.

[cloud-fan]

Is it safe to replace create with createEmpty? I see create will do some check and then call createEmpty

[chenghao-intel]

It should be safe, as we believe the logical plan has been resolved already during the analysis phase.

[marmbrus]

/cc @rxin

More API questions.

[SparkQA]

Test build #25409 has started for PR 3780 at commit 72c4a4b.

• This patch merges cleanly.

[SparkQA]

Test build #25409 has finished for PR 3780 at commit 72c4a4b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25409/
Test PASSed.

[rxin]

put this on the previous line?

[rxin]

The changes lgtm. I left couple small comments. Do you mind fixing those?

[marmbrus]

@rxin what about the question of append, overwrite, error? Do we want to expose all three for these types of interfaces?

[chenghao-intel]

For my understanding, insertInto works in the ways of create new, append or overwrite already, but it doesn't give user an option, which allow throwing exception when the file already existed. Probably that's why we need the saveAsParquetFile, which in the semantic of create new if the file doesn't exist, or overwrite provided by this PR.

[SparkQA]

Test build #25860 has started for PR 3780 at commit 8782fef.

• This patch merges cleanly.

[chenghao-intel]

Or probably we can cleanup those APIs, kind of like:

trait Mode
object Append extends Mode
object Create extends Mode
object CreateOrOverwrite extends Mode
object CreateOrAppend extends Mode

def outputFile(path: String, format: FileFormat=defaultFormat, mode: Mode = CreateOrOverwrite)
def insertIntoFile(path: String, format: FileFormat) = outputFile(path, format, CreateOrOverwrite)
def toCsv(path: String, mode: Mode) = outputFile(path, CSV, mode)
def toParquet(path: String, mode: Mode) = outputFile(path, Parquet, mode)

Any ideas?

[SparkQA]

Test build #25860 has finished for PR 3780 at commit 8782fef.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25860/
Test PASSed.

[marmbrus]

I kind of like a proposal like that, but we need to make it Java friendly as well.

[chenghao-intel]

OK, I will close this.