DOCSP-18193 Add a Wildcard Configuration section (#191)

* DOCSP-18193 Add a Wildcard Configuration section

* Apply suggestions from code review

Co-authored-by: Melissa Mahoney <[email protected]>

* DOCSP-18193 updates for review feedback

* DOCSP-18193 minor correction

Co-authored-by: Melissa Mahoney <[email protected]>

Author: kanchana-mongodb

source/config/config-data-lake.txt

@@ -23,17 +23,8 @@ format, see :ref:`datalake-configuration-file`.
 
 You can retrieve and update the {+data-lake-short+} configuration by 
 :ref:`connecting <gst-connect-adl>` a :binary:`~bin.mongo` shell to the 
-{+dl+}. You can also update your {+dl+} from the |service| UI:
-
-1. From the |service| UI, select :guilabel:`Data Lake` from the 
-   left-hand navigation. 
-
-#. Click :guilabel:`Configuration` for the {+data-lake-short+} that you 
-   want to update.
-
-#. Make necessary changes to the :ref:`storage configuration 
-   <datalake-configuration-file>` and click :guilabel:`Save` for the 
-   changes to take effect.
+{+dl+}. You can also update your {+dl+} from the |service| UI. See 
+:ref:`datalake-setstorageconfig` for more information.
 
 .. note::
 
@@ -64,30 +55,37 @@ Set or Update {+data-lake-short+} Configuration
 -----------------------------------------------
 
 Once connected to the {+data-lake-short+}, you can use the following
-database commands to set or update the {+data-lake-short+} configuration:
+database commands to set or update the {+data-lake-short+} 
+configuration:
 
 .. code-block:: javascript
 
    use admin
    db.runCommand( { "storageSetConfig" : <config> } )
 
-Replace ``<config>`` with the {+data-lake-short+} configuration. For complete 
-documentation on the configuration fields and format, see 
-:ref:`datalake-configuration-format`. You can validate your :ref:`configuration 
-<datalake-configuration-file>` before setting or updating the 
-{+data-lake-short+} configuration by running the :ref:`storageValidateConfig 
-<datalake-validatestorageconfig>` command.
+Replace ``<config>`` with the {+data-lake-short+} configuration. For 
+complete documentation on the configuration fields and format, see 
+:ref:`datalake-configuration-format`. You can validate your 
+:ref:`configuration <datalake-configuration-file>` before setting or 
+updating the {+dl+} configuration by running the 
+:ref:`storageValidateConfig <datalake-validatestorageconfig>` command.
 
 To set or update the storage configuration through the |service| UI: 
 
-1. Click :guilabel:`Configuration` for your {+dl+} to view the 
-   {+dl+} storage configuration. 
+1. From the |service| UI, select :guilabel:`Data Lake` from the 
+   left-hand navigation. 
+
+#. Click :guilabel:`Configuration` for the {+data-lake-short+} that you 
+   want to update.
 
    .. figure:: /images/set-update-config-ui.png
       :figwidth: 600px
       :alt: Image highlighting the Configuration button.
 
-2. Make changes to your storage configuration and click :guilabel:`Save`.
+#. Make any necessary changes to the :ref:`storage configuration 
+   <datalake-configuration-file>`.
+   
+#. Click :guilabel:`Save` for the changes to take effect.
 
 .. _datalake-validatestorageconfig:
 
@@ -102,8 +100,8 @@ You can run the following command to validate your {+data-lake-short+}
    use admin
    db.runCommand( { "storageValidateConfig" : <config> } )
 
-Replace ``<config>`` with the {+data-lake-short+} configuration. For complete 
-documentation on the configuration fields and format, see 
+Replace ``<config>`` with the {+data-lake-short+} configuration. For 
+complete documentation on the configuration fields and format, see 
 :ref:`datalake-configuration-format`.
 
 The command returns the following if your {+dl+} configuration is valid:
@@ -159,15 +157,212 @@ configuration.
    </security-add-mongodb-users/#atlasAdmin>` role has the 
    ``storageSetConfig`` privilege by default.
 
-To generate a {+data-lake-short+} configuration, connect to the {+data-lake-short+} 
-and run the following database commands: 
+To generate a {+data-lake-short+} configuration, connect to the 
+{+dl+} and run the following database commands: 
 
 .. code-block:: javascript 
 
    use admin 
    db.runCommand( { "storageGenerateConfig" : 1 } )
 
-For complete documentation on the configuration fields and format, see :ref:`datalake-configuration-format`. 
+For complete documentation on the configuration fields and format, see 
+:ref:`datalake-configuration-format`. 
+
+.. _generate-wildcard-collections:
+
+Generate Wildcard Collections 
+-----------------------------
+
+You can dynamically generate collection names that map to data in your 
+|s3| bucket or |service| cluster. To dynamically generate collection 
+names, specify the wildcard, ``*``, as the value for the collection 
+name setting in your {+dl+} storage configuration. You can't 
+dynamically generate collection names in your {+dl+} storage 
+configuration that map to data in your |http| or |https| data store. 
+
+You can use the :ref:`storageSetConfig <datalake-setstorageconfig>` 
+command to configure the settings for generating wildcard (``*``) 
+collections. 
+
+To learn more about the configuration settings for generating wildcard 
+collections, click on the tab for your data store:
+
+.. tabs:: 
+
+   .. tab:: S3 
+      :tabid: s3 
+
+      To generate wildcard collections in your {+dl+} storage 
+      configuration that map to data in your |s3| bucket, configure the 
+      following settings in your {+dl+} storage configuration:
+
+      - Specify ``*`` as the value for the 
+        :datalakeconf:`databases.[n].collections.name` setting.
+
+      - Specify the ``collectionName()`` function as the value for 
+        the :datalakeconf:`databases.[n].collections.[n].dataSources.[n].path` 
+        setting. 
+
+      - *Optional*. Specify the maximum number of collections to 
+        include in the database in the 
+        :datalakeconf:`databases.[n].maxWildcardCollections` setting. 
+        By default, {+adl+} generates up to ``100`` wildcard 
+        collections in the database.
+
+      .. example:: 
+
+         .. code-block:: json 
+            :copyable: false 
+            :emphasize-lines: 6
+
+            "databases" : [
+              {
+                "name" : "<db-name>",
+                "collections" : [
+                  {
+                    "name" : "*",
+                    "dataSources" : [
+                      {
+                        "storeName" : "<s3-store-name>",
+                        "path" : "{collectionName()}"
+                      }
+                    ]
+                  }
+                ], 
+                "maxWildcardCollections" : <integer>,
+              }
+            ]  
+
+      You can also use the :ref:`dl-create-collection-views-cmd` 
+      administration command and the {+adl+} User Interface |json| 
+      Editor to configure the settings for generating wildcard 
+      collections. You can't use the {+adl+} User Interface Visual 
+      Editor to configure the settings for generating wildcard 
+      collections.
+
+   .. tab:: Atlas  
+      :tabid: atlas
+
+      For the |service| data store, you can generate the following 
+      wildcard collections and databases in your {+dl+} storage configuration:
+      
+      - Wildcard collections for a specific database 
+      
+      - Wildcard databases with one wildcard collection
+
+      You can also dynamically generate collection names that match 
+      a regex pattern. 
+
+      .. tabs:: 
+
+         .. tab:: Wildcard Collections 
+            :tabid: wildcardColls 
+
+            To generate wildcard collections in your {+dl+} storage 
+            configuration that map to data in your |service| cluster, 
+            configure the following settings in your {+dl+} storage 
+            configuration:
+
+            - Specify ``*`` as the value for the 
+              :datalakeconf:`databases.[n].collections.name` setting.
+
+            - Omit the :datalakeconf:`databases.[n].collections.[n].dataSources.[n].collection` 
+              setting. 
+
+            - *Optional*. Use the :datalakeconf:`databases.[n].collections.[n].dataSources.[n].collectionRegex` 
+              setting to generate wildcard collection names that match 
+              a regex pattern.
+
+            .. example:: 
+
+               .. code-block:: json 
+                  :copyable: false 
+                  :emphasize-lines: 6
+
+                  "databases" : [
+                    {
+                      "name" : "<db-name>",
+                      "collections" : [
+                        {
+                          "name" : "*",
+                          "dataSources" : [
+                            {
+                              "storeName" : "<atlas-store-name>",
+                              "database" : "<atlas-db-name>",
+                              "collectionRegex" : "<regex-pattern>"
+                            }
+                          ]
+                        }
+                      ]
+                    }
+                  ]             
+
+            You can also use the :ref:`dl-create-collection-views-cmd` 
+            administration command and the {+adl+} User Interface to 
+            configure the settings for generating wildcard collections. 
+
+         .. tab:: Wildcard Databases 
+            :tabid: wildcardDbs
+
+            To dynamically generate databases with one wildcard 
+            collection in your {+dl+} storage configuration, configure 
+            the following settings in your {+dl+} storage configuration:
+
+            - Specify ``*`` as the value for the 
+              :datalakeconf:`databases.[n].name` setting.
+
+            - Specify ``*`` as the value for the 
+              :datalakeconf:`databases.[n].collections.name` setting.
+
+            - Omit the :datalakeconf:`databases.[n].collections.[n].dataSources.[n].database` 
+              and :datalakeconf:`databases.[n].collections.[n].dataSources.[n].collection` settings. 
+
+            - *Optional*. Use the :datalakeconf:`databases.[n].collections.[n].dataSources.[n].collectionRegex` 
+              setting to generate wildcard collection names that match 
+              a regex pattern.
+
+            .. example:: 
+
+               .. code-block:: json 
+                  :copyable: false 
+                  :emphasize-lines: 3,6
+
+                  "databases" : [
+                    {
+                      "name" : "*",
+                      "collections" : [
+                        {
+                          "name" : "*",
+                          "dataSources" : [
+                            {
+                              "storeName" : "<atlas-store-name>",
+                              "collectionRegex" : "<regex-pattern>"
+                            }
+                          ]
+                        }
+                      ]
+                    }
+                  ]
+
+            You can use the :ref:`dl-create-collection-views-cmd` 
+            administration command also to configure the settings for 
+            generating wildcard collection for wildcard databases. You 
+            can't use the {+adl+} User Interface to configure the 
+            settings for generating wildcard collection for wildcard 
+            databases.
+
+            Dynamically generated databases:
+
+            - Can exist alongside explicitly defined databases. 
+              However, {+adl+} won't include dynamically generated 
+              databases with names that conflict with databases that 
+              are explicitly defined in the storage configuration.
+            - Can only be from a single |service| cluster. {+adl+} 
+              won't dynamically generate databases from multiple 
+              |service| clusters or other data stores.
+
+To learn more about the configuration settings, see 
+:ref:`datalake-configuration-file`.
 
 .. toctree::
    :titlesonly: