You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-1566] consolidate programming guide, and general doc updates
This is a fairly large PR to clean up and update the docs for 1.0. The major changes are:
* A unified programming guide for all languages replaces language-specific ones and shows language-specific info in tabs
* New programming guide sections on key-value pairs, unit testing, input formats beyond text, migrating from 0.9, and passing functions to Spark
* Spark-submit guide moved to a separate page and expanded slightly
* Various cleanups of the menu system, security docs, and others
* Updated look of title bar to differentiate the docs from previous Spark versions
You can find the updated docs at http://people.apache.org/~matei/1.0-docs/_site/ and in particular http://people.apache.org/~matei/1.0-docs/_site/programming-guide.html.
Author: Matei Zaharia <[email protected]>
Closes#896 from mateiz/1.0-docs and squashes the following commits:
03e6853 [Matei Zaharia] Some tweaks to configuration and YARN docs
0779508 [Matei Zaharia] tweak
ef671d4 [Matei Zaharia] Keep frames in JavaDoc links, and other small tweaks
1bf4112 [Matei Zaharia] Review comments
4414f88 [Matei Zaharia] tweaks
d04e979 [Matei Zaharia] Fix some old links to Java guide
a34ed33 [Matei Zaharia] tweak
541bb3b [Matei Zaharia] miscellaneous changes
fcefdec [Matei Zaharia] Moved submitting apps to separate doc
61d72b4 [Matei Zaharia] stuff
181f217 [Matei Zaharia] migration guide, remove old language guides
e11a0da [Matei Zaharia] Add more API functions
6a030a9 [Matei Zaharia] tweaks
8db0ae3 [Matei Zaharia] Added key-value pairs section
318d2c9 [Matei Zaharia] tweaks
1c81477 [Matei Zaharia] New section on basics and function syntax
e38f559 [Matei Zaharia] Actually added programming guide to Git
a33d6fe [Matei Zaharia] First pass at updating programming guide to support all languages, plus other tweaks throughout
3b6a876 [Matei Zaharia] More CSS tweaks
01ec8bf [Matei Zaharia] More CSS tweaks
e6d252e [Matei Zaharia] Change color of doc title bar to differentiate from 0.9.0
Copy file name to clipboardExpand all lines: docs/bagel-programming-guide.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -21,7 +21,7 @@ To use Bagel in your program, add the following SBT or Maven dependency:
21
21
22
22
# Programming Model
23
23
24
-
Bagel operates on a graph represented as a [distributed dataset](scala-programming-guide.html) of (K, V) pairs, where keys are vertex IDs and values are vertices plus their associated state. In each superstep, Bagel runs a user-specified compute function on each vertex that takes as input the current vertex state and a list of messages sent to that vertex during the previous superstep, and returns the new vertex state and a list of outgoing messages.
24
+
Bagel operates on a graph represented as a [distributed dataset](programming-guide.html) of (K, V) pairs, where keys are vertex IDs and values are vertices plus their associated state. In each superstep, Bagel runs a user-specified compute function on each vertex that takes as input the current vertex state and a list of messages sent to that vertex during the previous superstep, and returns the new vertex state and a list of outgoing messages.
25
25
26
26
For example, we can use Bagel to implement PageRank. Here, vertices represent pages, edges represent links between pages, and messages represent shares of PageRank sent to the pages that a particular page links to.
If you don't run this, you may see errors like the following:
19
21
@@ -25,9 +27,9 @@ If you don't run this, you may see errors like the following:
25
27
26
28
You can fix this by setting the `MAVEN_OPTS` variable as discussed before.
27
29
28
-
*Note:For Java 1.8 and above this step is not required.*
30
+
**Note:***For Java 8 and above this step is not required.*
29
31
30
-
##Specifying the Hadoop version ##
32
+
# Specifying the Hadoop Version
31
33
32
34
Because HDFS is not protocol-compatible across versions, if you want to read from HDFS, you'll need to build Spark against the specific HDFS version in your environment. You can do this through the "hadoop.version" property. If unset, Spark will build against Hadoop 1.0.4 by default. Note that certain build profiles are required for particular Hadoop versions:
33
35
@@ -46,14 +48,16 @@ Because HDFS is not protocol-compatible across versions, if you want to read fro
46
48
47
49
For Apache Hadoop versions 1.x, Cloudera CDH "mr1" distributions, and other Hadoop versions without YARN, use:
For Apache Hadoop 2.x, 0.23.x, Cloudera CDH, and other Hadoop versions with YARN, you can enable the "yarn-alpha" or "yarn" profile and optionally set the "yarn.version" property if it is different from "hadoop.version". The additional build profile required depends on the YARN version:
59
63
@@ -69,75 +73,77 @@ For Apache Hadoop 2.x, 0.23.x, Cloudera CDH, and other Hadoop versions with YARN
Tests are run by default via the [ScalaTest Maven plugin](http://www.scalatest.org/user_guide/using_the_scalatest_maven_plugin). Some of the require Spark to be packaged first, so always run `mvn package` with `-DskipTests` the first time. You can then run the tests with `mvn -Dhadoop.version=... test`.
96
102
97
103
The ScalaTest plugin also supports running only a specific test suite as follows:
98
104
99
-
$ mvn -Dhadoop.version=... -DwildcardSuites=org.apache.spark.repl.ReplSuite test
105
+
mvn -Dhadoop.version=... -DwildcardSuites=org.apache.spark.repl.ReplSuite test
100
106
101
107
102
-
##Continuous Compilation ##
108
+
# Continuous Compilation
103
109
104
110
We use the scala-maven-plugin which supports incremental and continuous compilation. E.g.
105
111
106
-
$ mvn scala:cc
112
+
mvn scala:cc
107
113
108
114
should run continuous compilation (i.e. wait for changes). However, this has not been tested extensively.
109
115
110
-
##Using With IntelliJ IDEA ##
116
+
# Using With IntelliJ IDEA
111
117
112
118
This setup works fine in IntelliJ IDEA 11.1.4. After opening the project via the pom.xml file in the project root folder, you only need to activate either the hadoop1 or hadoop2 profile in the "Maven Properties" popout. We have not tried Eclipse/Scala IDE with this.
113
119
114
-
##Building Spark Debian Packages ##
120
+
# Building Spark Debian Packages
115
121
116
-
The maven build includes support for building a Debian package containing the assembly 'fat-jar', PySpark, and the necessary scripts and configuration files. This can be created by specifying the following:
122
+
The Maven build includes support for building a Debian package containing the assembly 'fat-jar', PySpark, and the necessary scripts and configuration files. This can be created by specifying the following:
117
123
118
-
$ mvn -Pdeb -DskipTests clean package
124
+
mvn -Pdeb -DskipTests clean package
119
125
120
126
The debian package can then be found under assembly/target. We added the short commit hash to the file name so that we can distinguish individual packages built for SNAPSHOT versions.
121
127
122
-
##Running java 8 test suites.
128
+
# Running Java 8 Test Suites
123
129
124
-
Running only java 8 tests and nothing else.
130
+
Running only Java 8 tests and nothing else.
125
131
126
-
$ mvn install -DskipTests -Pjava8-tests
132
+
mvn install -DskipTests -Pjava8-tests
127
133
128
-
Java 8 tests are run when -Pjava8-tests profile is enabled, they will run in spite of -DskipTests.
134
+
Java 8 tests are run when `-Pjava8-tests` profile is enabled, they will run in spite of `-DskipTests`.
129
135
For these tests to run your system must have a JDK 8 installation.
130
136
If you have JDK 8 installed but it is not the system default, you can set JAVA_HOME to point to JDK 8 before running the tests.
131
137
132
-
##Building for PySpark on YARN ##
138
+
# Building for PySpark on YARN
133
139
134
-
PySpark on YARN is only supported if the jar is built with maven. Further, there is a known problem
135
-
with building this assembly jar on Red Hat based operating systems (see SPARK-1753). If you wish to
140
+
PySpark on YARN is only supported if the jar is built with Maven. Further, there is a known problem
141
+
with building this assembly jar on Red Hat based operating systems (see [SPARK-1753](https://issues.apache.org/jira/browse/SPARK-1753)). If you wish to
136
142
run PySpark on a YARN cluster with Red Hat installed, we recommend that you build the jar elsewhere,
137
143
then ship it over to the cluster. We are investigating the exact cause for this.
138
144
139
-
##Packaging without Hadoop dependencies for deployment on YARN ##
145
+
# Packaging without Hadoop Dependencies for YARN
140
146
141
-
The assembly jar produced by "mvn package" will, by default, include all of Spark's dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor classpaths: the version packaged in the Spark assembly and the version on each node, included with yarn.application.classpath. The "hadoop-provided" profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself.
147
+
The assembly jar produced by `mvn package` will, by default, include all of Spark's dependencies, including Hadoop and some of its ecosystem projects. On YARN deployments, this causes multiple versions of these to appear on executor classpaths: the version packaged in the Spark assembly and the version on each node, included with yarn.application.classpath. The `hadoop-provided` profile builds the assembly without including Hadoop-ecosystem projects, like ZooKeeper and Hadoop itself.
have assembly plugins. When creating assembly jars, list Spark and Hadoop
62
-
as `provided` dependencies; these need not be bundled since they are provided by
63
-
the cluster manager at runtime. Once you have an assembled jar you can call the `bin/spark-submit`
64
-
script as shown here while passing your jar.
65
-
66
-
For Python, you can use the `pyFiles` argument of SparkContext
67
-
or its `addPyFile` method to add `.py`, `.zip` or `.egg` files to be distributed.
68
-
69
-
### Launching Applications with Spark submit
70
-
71
-
Once a user application is bundled, it can be launched using the `spark-submit` script located in
72
-
the bin directory. This script takes care of setting up the classpath with Spark and its
73
-
dependencies, and can support different cluster managers and deploy modes that Spark supports:
74
-
75
-
./bin/spark-submit \
76
-
--class <main-class>
77
-
--master <master-url> \
78
-
--deploy-mode <deploy-mode> \
79
-
... // other options
80
-
<application-jar>
81
-
[application-arguments]
82
-
83
-
main-class: The entry point for your application (e.g. org.apache.spark.examples.SparkPi)
84
-
master-url: The URL of the master node (e.g. spark://23.195.26.187:7077)
85
-
deploy-mode: Whether to deploy this application within the cluster or from an external client (e.g. client)
86
-
application-jar: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an `hdfs://` path or a `file://` path that is present on all nodes.
87
-
application-arguments: Space delimited arguments passed to the main method of <main-class>, if any
88
-
89
-
To enumerate all options available to `spark-submit` run it with the `--help` flag. Here are a few
90
-
examples of common options:
91
-
92
-
{% highlight bash %}
93
-
# Run application locally
94
-
./bin/spark-submit \
95
-
--class org.apache.spark.examples.SparkPi
96
-
--master local[8]\
97
-
/path/to/examples.jar \
98
-
100
99
-
100
-
# Run on a Spark standalone cluster
101
-
./bin/spark-submit \
102
-
--class org.apache.spark.examples.SparkPi
103
-
--master spark://207.184.161.138:7077 \
104
-
--executor-memory 20G \
105
-
--total-executor-cores 100 \
106
-
/path/to/examples.jar \
107
-
1000
108
-
109
-
# Run on a YARN cluster
110
-
HADOOP_CONF_DIR=XX ./bin/spark-submit \
111
-
--class org.apache.spark.examples.SparkPi
112
-
--master yarn-cluster \ # can also be `yarn-client` for client mode
113
-
--executor-memory 20G \
114
-
--num-executors 50 \
115
-
/path/to/examples.jar \
116
-
1000
117
-
{% endhighlight %}
118
-
119
-
### Loading Configurations from a File
120
-
121
-
The `spark-submit` script can load default [Spark configuration values](configuration.html) from a
122
-
properties file and pass them on to your application. By default it will read configuration options
123
-
from `conf/spark-defaults.conf`. For more detail, see the section on
0 commit comments