(DOCSP-21715) Remove RDD support (#97)

* (DOCSP-21715) Remove RDD support

Author: zach-carr

source/faq.txt

@@ -8,7 +8,7 @@ How can I achieve data locality?
 --------------------------------
 
 For any MongoDB deployment, the Mongo Spark Connector sets the
-preferred location for an RDD to be where the data is:
+preferred location for a DataFrame or Dataset to be where the data is:
 
 - For a non sharded system, it sets the preferred location to be the
   hostname(s) of the standalone or the replica set.
@@ -30,89 +30,10 @@ To promote data locality,
   To partition the data by shard use the 
   :ref:`conf-shardedpartitioner`.
 
-How do I interact with Spark Streams?
--------------------------------------
-
-Spark streams can be considered as a potentially infinite source of
-RDDs. Therefore, anything you can do with an RDD, you can do with the
-results of a Spark Stream.
-
-For an example, see :mongo-spark:`SparkStreams.scala
-</blob/master/examples/src/test/scala/tour/SparkStreams.scala>`
-
 How do I resolve ``Unrecognized pipeline stage name`` Error?
 ------------------------------------------------------------
 
 In MongoDB deployments with mixed versions of :binary:`~bin.mongod`, it is
 possible to get an ``Unrecognized pipeline stage name: '$sample'``
 error. To mitigate this situation, explicitly configure the partitioner
 to use and define the Schema when using DataFrames.
-
-How do I use MongoDB BSON types that are unsupported in Spark?
---------------------------------------------------------------
-
-Some custom MongoDB BSON types, such as ``ObjectId``, are unsupported 
-in Spark.
-
-The MongoDB Spark Connector converts custom MongoDB data types to and 
-from extended JSON-like representations of those data types that are 
-compatible with Spark. See :ref:`<bson-spark-datatypes>` for a list of 
-custom MongoDB types and their Spark counterparts.
-
-Spark Datasets
-~~~~~~~~~~~~~~
-
-To create a standard Dataset with custom MongoDB data types, use 
-``fieldTypes`` helpers:
-
-.. code-block:: scala
-   
-   import com.mongodb.spark.sql.fieldTypes
- 
-   case class MyData(id: fieldTypes.ObjectId, a: Int)
-   val ds = spark.createDataset(Seq(MyData(fieldTypes.ObjectId(new ObjectId()), 99)))
-   ds.show()
-
-The preceding example creates a Dataset containing the following fields 
-and data types:
-
-- The ``id`` field is a custom MongoDB BSON type, ``ObjectId``, defined 
-  by ``fieldTypes.ObjectId``.
-
-- The ``a`` field is an ``Int``, a data type available in Spark.
-
-Spark DataFrames
-~~~~~~~~~~~~~~~~
-
-To create a DataFrame with custom MongoDB data types, you must supply 
-those types when you create the RDD and schema:
-
-- Create RDDs using custom MongoDB BSON types 
-  (e.g. ``ObjectId``). The Spark Connector handles converting 
-  those custom types into Spark-compatible data types.
-
-- Declare schemas using the ``StructFields`` helpers for data types 
-  that are not natively supported by Spark 
-  (e.g. ``StructFields.objectId``). Refer to 
-  :ref:`<bson-spark-datatypes>` for the mapping between BSON and custom 
-  MongoDB Spark types.
-
-.. code-block:: scala
-   
-   import org.apache.spark.sql.Row
-   import org.apache.spark.sql.types.{StructType, StructField, IntegerType}
-   import com.mongodb.spark.sql.helpers.StructFields
- 
-   val data = Seq(Row(Row(new ObjectId().toHexString()), 99))
-   val rdd = spark.sparkContext.parallelize(data)
-   val schema = StructType(List(StructFields.objectId("id", true), StructField("a", IntegerType, true)))
-   val df = spark.createDataFrame(rdd, schema)
-   df.show()
-
-The preceding example creates a DataFrame containing the following 
-fields and data types:
-
-- The ``id`` field is a custom MongoDB BSON type, ``ObjectId``, defined 
-  by ``StructFields.objectId``.
-
-- The ``a`` field is an ``Int``, a data type available in Spark.

source/getting-started.txt

@@ -15,8 +15,6 @@ Prerequisites
 
 .. include:: /includes/list-prerequisites.rst
 
-- Java 8 or later.
-
 .. _pyspark-shell:
 .. _scala-getting-started:
 .. _python-basics:

source/includes/bson-type-consideration.rst

@@ -0,0 +1,12 @@
+.. code-block:: javascript
+
+   { "_id" : ObjectId("585024d558bef808ed84fc3e"), "name" : "Bilbo Baggins", "age" : 50 }
+   { "_id" : ObjectId("585024d558bef808ed84fc3f"), "name" : "Gandalf", "age" : 1000 }
+   { "_id" : ObjectId("585024d558bef808ed84fc40"), "name" : "Thorin", "age" : 195 }
+   { "_id" : ObjectId("585024d558bef808ed84fc41"), "name" : "Balin", "age" : 178 }
+   { "_id" : ObjectId("585024d558bef808ed84fc42"), "name" : "Kíli", "age" : 77 }
+   { "_id" : ObjectId("585024d558bef808ed84fc43"), "name" : "Dwalin", "age" : 169 }
+   { "_id" : ObjectId("585024d558bef808ed84fc44"), "name" : "Óin", "age" : 167 }
+   { "_id" : ObjectId("585024d558bef808ed84fc45"), "name" : "Glóin", "age" : 158 }
+   { "_id" : ObjectId("585024d558bef808ed84fc46"), "name" : "Fíli", "age" : 82 }
+   { "_id" : ObjectId("585024d558bef808ed84fc47"), "name" : "Bombur" }

source/includes/characters-example-collection.rst

@@ -3,7 +3,7 @@ content: |
    - the ``--packages`` option to download the MongoDB Spark Connector
      package.  The following package is available:
 
-     - ``mongo-spark-connector_{+scala-version+}`` for use with Scala 2.12.x
+     - ``mongo-spark-connector``
 
    - the ``--conf`` option to configure the MongoDB Spark Connnector.
      These settings configure the ``SparkConf`` object.
@@ -39,7 +39,7 @@ content: |
 
       ./bin/spark-shell --conf "spark.mongodb.read.uri=mongodb://127.0.0.1/test.myCollection?readPreference=primaryPreferred" \
                         --conf "spark.mongodb.write.uri=mongodb://127.0.0.1/test.myCollection" \
-                        --packages org.mongodb.spark:mongo-spark-connector_{+scala-version+}:{+current-version+}
+                        --packages org.mongodb.spark:mongo-spark-connector:{+current-version+}
  
    .. include:: /includes/extracts/list-configuration-explanation.rst
 
@@ -56,7 +56,7 @@ content: |
 
       ./bin/pyspark --conf "spark.mongodb.read.uri=mongodb://127.0.0.1/test.myCollection?readPreference=primaryPreferred" \
                     --conf "spark.mongodb.write.uri=mongodb://127.0.0.1/test.myCollection" \
-                    --packages org.mongodb.spark:mongo-spark-connector_{+scala-version+}:{+current-version+}
+                    --packages org.mongodb.spark:mongo-spark-connector:{+current-version+}
  
    .. include:: /includes/extracts/list-configuration-explanation.rst
 
@@ -73,7 +73,7 @@ content: |
 
       ./bin/sparkR  --conf "spark.mongodb.read.uri=mongodb://127.0.0.1/test.myCollection?readPreference=primaryPreferred" \
                     --conf "spark.mongodb.write.uri=mongodb://127.0.0.1/test.myCollection" \
-                    --packages org.mongodb.spark:mongo-spark-connector_{+scala-version+}:{+current-version+}
+                    --packages org.mongodb.spark:mongo-spark-connector:{+current-version+}
 
    .. include:: /includes/extracts/list-configuration-explanation.rst
 ...

source/includes/extracts-command-line.yaml

@@ -4,4 +4,6 @@
 
 - Running MongoDB instance (version 4.0 or later).
 
-- Spark version 3.1 or later
+- Spark version 3.1 or later.
+
+- Java 8 or later.

source/includes/list-prerequisites.rst

@@ -0,0 +1,6 @@
+.. important::
+
+   In version 10.0.0 and later of the Connector, use the format 
+   ``mongodb`` to read from and write to MongoDB:
+
+   ``df = spark.read.format("mongodb").load()``

source/includes/new-format-name.rst

@@ -1,2 +1,2 @@
 Provide the Spark Core, Spark SQL, and MongoDB Spark Connector
-dependencies to your dependency management tool.
+dependencies to your dependency management tool.