[SPARK-9270] [PySpark] allow --name option in pyspark

[piaozhexiu]

This is continuation of #7512 which added --name option to spark-shell. This PR adds the same option to pyspark.

Note that --conf spark.app.name in command-line has no effect in spark-shell and pyspark. Instead, --name must be used. This is in fact inconsistency with spark-sql which doesn't accept --name option while it accepts --conf spark.app.name. I am not fixing this inconsistency in this PR. IMO, one of --name and --conf spark.app.name is needed not both. But since I cannot decide which to choose, I am not making any change here.

[srowen]

That seems fine if it still sets the app name as desired and improves consistency.

[SparkQA]

Test build #38189 has finished for PR 7610 at commit 190638b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[sarutak]

retest this please.

[sarutak]

retest this please.

[vanzin]

Note that --conf spark.app.name in command-line has no effect in spark-shell and pyspark. Instead, --name must be used.

This is not critical, but not great either; it points at some inconsistency in the code somewhere. From looking at SparkSubmit.scala, it should work, but I haven't explicitly tested it.

spark-sql which doesn't accept --name

Is that true? SparkSQLEnv has code to explicitly handle app names set by the user. It that doesn't work, it sounds like a bug.

[SparkQA]

Test build #38224 has finished for PR 7610 at commit 190638b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[piaozhexiu]

@vanzin thank you for your comments!

1. As for spark-sql, sorry I was wrong. spark-sql works with both --name and --conf spark.app.name. So we're good here.

2. As to why --conf spark.app.name doesn't work in pyspark, if you look at context.py, line #135 overwrites spark.app.name in conf if appName is passed via a constructor parameter.

So if you run the following cmd-

pyspark --conf spark.app.name=MyName

It invokes the following cmd b/c --name PySparkShell is added by default-

spark-submit --name PySparkShell --conf spark.app.name=MyName

Since PySparkShell ends up being passed as a parameter to the SparkContext constructor, it overwrites spark.app.name. So when we get to line #153 later, spark.app.name is no longer MyName but PySparkShell.

In summary, --name always wins and is always added to the cmd. So --conf spark.app.name never takes effect.

[srowen]

So the Python part is OK but now does spark-sql need a similar treatment?

[piaozhexiu]

@srowen spark-sql actually already works with both --name and --conf spark.app.name. I was mistaken earlier.

In short, --name works with every one (spark-shell, spark-sql, pyspark) while --conf spark.app.name only works with spark-sql.

[SparkQA]

Test build #38232 has finished for PR 7610 at commit ccef949.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[piaozhexiu]

Hmm, jenkins unstable?

[vanzin]

As to why --conf spark.app.name doesn't work in pyspark, if you look at context.py, line #135

Hmm, it may not work, but I don't think that's the cause. With your changes, that line should never be reached when starting the shell. What I think is happening is:

• SparkSubmit sets the app name in SparkSubmit.scala, L470 (processing the option defined in L405).
• In SparkSubmit.scala L555, the conf is read but at that point spark.app.name is already set, so it's not overwritten.

So it seems like an ordering issue in SparkSubmit.scala. In any case, it doesn't seem important enough to change just for this particular edge case. The change LGTM.

[SparkQA]

Test build #38256 has finished for PR 7610 at commit 763e86d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[davies]

Should we support bin/pyspark --name MyName ? spark-shell support that.

[piaozhexiu]

--name MyName works. So the final command will be like this:

spark-submit pyspark-shell-main --name "PySparkShell" --name "MyName" <other args>

Then, MyName takes effect since it comes later than PySparkShell.

[piaozhexiu]

Just to be clear, this is how --name is supported in spark-shell too (#7512). I am following the pattern introduced by that patch.

[vanzin]

Merged to master. Thanks!