Spark Submit changes and test

[foxish]

cc @mccheah @liyinan926 @apache-spark-on-k8s/contributors

[liyinan926]

LGTM with one minor comment.

[liyinan926]

Add one for spark.driver.memory.

[foxish]

Done. Thanks!

[ifilonenko]

Are we missing testing logic for app resources or is that going to be a follow up PR?

[foxish]

@ifilonenko Added the check for the jar that's submitted. Did you mean something else? The implementation details wrt resources should be covered under org/apache/spark/deploy/k8s/submit/ClientSuite.scala and submission step tests I think.

[ifilonenko]

@foxish that was part of it, yes. But coverage in regards to this:

spark/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

Lines 636 to 653 in 0add46a

	childMainClass = "org.apache.spark.deploy.k8s.submit.Client"
	if (args.isPython) {
	childArgs ++= Array("--primary-py-file", args.primaryResource)
	childArgs ++= Array("--main-class", "org.apache.spark.deploy.PythonRunner")
	if (args.pyFiles != null) {
	childArgs ++= Array("--other-py-files", args.pyFiles)
	}
	} else if (args.isR) {
	childArgs ++= Array("--primary-r-file", args.primaryResource)
	childArgs ++= Array("--main-class", "org.apache.spark.deploy.RRunner")
	} else {
	childArgs ++= Array("--primary-java-resource", args.primaryResource)
	childArgs ++= Array("--main-class", args.mainClass)
	}
	args.childArgs.foreach { arg =>
	childArgs += "--arg"
	childArgs += arg
	}

[foxish]

Ah! I see.. I think those should be separate unit tests in a subsequent PR.
I was looking at this from the perspective of the upstreaming, and initially, we're not going to have python/R support.

[foxish]

rerun integration test please

[foxish]

rerun unit tests please

[foxish]

rerun integration test please

[foxish]

rerun integration test please