Reverse ETL 2.0 docs

src/_data/sidenav/main.yml

@@ -184,8 +184,14 @@ sections:
     section:
     - path: /connections/reverse-etl
       title: Reverse ETL Overview
+    - path: /connections/reverse-etl/setup
+      title: Set Up Reverse ETL
+    - path: /connections/reverse-etl/manage-retl
+      title: Manage Reverse ETL Syncs
+    - path: /connections/reverse-etl/system
+      title: Reverse ETL System
     - path: /connections/reverse-etl/reverse-etl-catalog
-      title: Reverse ETL Catalog
+      title: Reverse ETL Destination Catalog
     - section_title: Reverse ETL Source Setup Guides
       slug: connections/reverse-etl/reverse-etl-source-setup-guides
       section:

src/connections/reverse-etl/faq.md

@@ -0,0 +1,25 @@
+---
+title: Reverse ETL FAQ
+beta: false
+---
+
+Get answers to some frequently asked Reverse ETL questions. 
+
+## Why do my sync results show *No records extracted* when I select *Updated records* after I enable the mapping? 
+It's expected that when you select **Updated records**, the records do not change after the first sync. During the first sync, the reverse ETL system calculates a snapshot of all the results and creates records in the `_segment_reverse_etl` schema. All the records are considered as *Added records* instead of *Updated records* at this time. The records can only meet the *Updated records* condition when the underlying values change after the first sync completes.
+
+## Can I be notified when Reverse ETL syncs fail?
+Yes, you can sign up for Reverse ETL sync notifications.
+
+To receive Reverse ETL sync notifications: 
+1. Navigate to **Settings > User Preferences**.
+2. Select **Reverse ETL** in the **Activity Notifications** section.
+3. Enable the toggle for **Reverse ETL Sync Failed**.
+
+In case of consecutive failures, Segment sends notifications for every sync failure. Segment doesn't send notifications for partial failures.
+
+## Does Segment use Transport Layer Security (TLS) for the connection between Snowflake and Segment?
+Segment uses the [gosnowflake library](https://pkg.go.dev/github.com/snowflakedb/gosnowflake#pkg-variables){:target="_blank"} to connect with Snowflake, which internally uses TLS for the HTTP transport.
+
+## Can I have multiple queries in the Query Builder?
+No. In Reverse ETL, Segment executes queries in a [common table expression](https://cloud.google.com/bigquery/docs/reference/standard-sql/query-syntax#with_clause){:target="_blank”}, which can only bind the results from **one single** subquery. If there are multiple semicolons `;` in the query, they'll be treated as several subqueries (even if the second part is only an inline comment) and cause syntax errors.

src/connections/reverse-etl/manage-retl.md

@@ -0,0 +1,141 @@
+---
+title: Manage Reverse ETL Syncs
+beta: false
+---
+
+View your sync history, gain insights into sync statuses, and restart or replay failed or partially successful syncs.
+
+## Sync overview
+On the Reverse ETL sync overview tab for your destination, you can see information about your recent Reverse ETL syncs at a glance, search for recent syncs, and quickly access the mappings and models that power Reverse ETL.
+
+![A screenshot of the sync overview page, which includes one failed sync and three successful syncs.](images/sync-overview.png)
+
+You can view the following information about each sync: 
+- **Latest sync**: The progress of your latest sync: syncs can either be **In progress**, **Successful**, or **Failed**. Also included is the timestamp of the sync start time.
+- **Mapping**: The named mapping that powered the sync and a hyperlink to the mapping's overview page. 
+- **Model**: The name that you gave the SQL query used to withdraw information from your warehouse, with a hyperlink to the model overview page. Below the model name, you can see the warehouse source that Segment extracts information from.
+- **Action**: The Action that your destination uses to map information from your warehouse to your downstream destination. 
+- **Mapping status**: The status of your mapping: either **Enabled** or **Disabled**. 
+
+You can also filter the sync overview table to return only the syncs that match your criteria. 
+
+You can filter for the following sync attributes: 
+- **Sync status**: The status of your sync: In progress, Successful, Partially successful, or Failed.
+- **Start time**: Select a predefined time period, or create a custom date range. 
+- **Model**: The model connected to your sync. 
+- **Destination**: Select one or more of your connected destinations. 
+- **Mapping status**: The status of your mapping: either **Enabled** or **Disabled**. 
+
+## Sync history
+Check the status of your data extractions and see details of your syncs. Click into failed records to view additional details on the error, sample payloads to help you debug the issue, and recommended actions.
+
+To check the status of your extractions:
+1. Navigate to **Connections > Destinations** and select the **Reverse ETL** tab.
+2. Select the destination you want to view.
+3. Select the mapping you want to view.  
+4. Click the sync you want to view to get details of the sync. You can view:
+    * The status of the sync.
+    * Details of how long it took for the sync to complete.
+    * How many total records were extracted, as well as a breakdown of the number of records added, updated, and deleted.
+    * The load results - how many successful records were synced as well as how many records were updated, deleted, or are new.
+5. If your sync failed, click the failed reason to get more details on the error and view sample payloads to help troubleshoot the issue.
+
+> info "Segment automatically retries events that were extracted but failed to load"
+> Segment retries events for 14 days following a total or partial sync failure. Before loading the failed records on a subsequent sync, Segment checks for the latest changes in your data to ensure the data loaded into your warehouse isn't stale. If the error causing the load failure is coming from an upstream tool, you can fix the error in the upstream tool to ensure the record loads on the next sync. 
+
+## Reset syncs
+You can reset your syncs so that your data is synced from the beginning. This means that Segment resyncs your entire dataset for the model. During the next sync, all records extracted by the model are sent to your destination, not just the records that changed since the last sync.
+
+To reset a sync:
+1. Select the three dots next to **Sync now**.
+2. Select **Reset sync**. 
+3. Click **I understand what happens when I reset a sync state**. 
+4. Click **Reset sync**.
+
+## Replays
+You can choose to replay syncs. To replay a specific sync, contact [[email protected]](mailto:[email protected]). Keep in mind that triggering a replay resyncs all records for a given sync.
+
+## Alerting
+You can opt in to receive email, Slack, and in-app alerts about Reverse ETL sync failures and partial successes. 
+
+To subscribe to alerts: 
+1. Navigate to **Settings > User Preferences**. 
+2. Select **Reverse ETL** in the **Activity Notifications** section.
+3. Click the Reverse ETL sync status that you'd like to receive notifications for. You can select one or more of the following sync statuses:
+    - **Reverse ETL sync failed**: Receive a notification when your Reverse ETL sync fails.
+    - **Reverse ETL sync partial success**: Receive a notification when your Reverse ETL sync is partially successful.
+4. Select one or more of the following alert options: 
+    - **Enable email notifications**: Enter an email address or alias that should receive alerts.
+    - **Enable Slack notifications**: Enter a webhook URL and Slack channel name.
+    - **Enable in-app notifications**: Select this option to see an in-app notification.
+5. Click **Create alert**.
+
+> success ""
+> If you opted to receive notifications by email, you can click **View active email addresses** to see the email addresses that are currently signed up to receive notifications. 
+
+## Supported object and arrays 
+
+When you set up destination actions in Reverse ETL, depending on the destination, some [mapping fields](/docs/connections/reverse-etl/setup/#step-4-create-mappings) may require data as an [object](/docs/connections/reverse-etl/manage-retl/#object-mapping) or [array](/docs/connections/reverse-etl/manage-retl/#array-mapping). 
+
+### Object mapping
+You can send data to a mapping field that requires object data. An example of object mapping is an `Order completed` model with a `Products` column that’s in object format. 
+
+Example: 
+
+```json    
+    {
+        "product": {
+            "id": 0001,
+            "color": "pink",
+            "name": "tshirt",
+            "revenue": 20,
+            "inventory": 500
+        }
+    }
+```
+
+To send data to a mapping field that requires object data, you can choose between these two options: 
+
+Option | Details
+------ | --------
+Customize object | This enables you to manually set up the mapping fields with any data from the model. If the model contains some object data, you can select properties within the object to set up the mappings as well.
+Select object | This enables you to send all nested properties within an object. The model needs to provide data in the format of the object. 
+
+> success ""
+> Certain object mapping fields have a fixed list of properties they can accept. If the names of the nested properties in your object don't match with the destination properties, the data won't send. Segment recommends you to use **Customize Object** to ensure your mapping is successful.
+
+
+### Array mapping
+To send data to a mapping field that requires array data, the model must provide data in the format of an array of objects. An example is an `Order completed` model with a `Product purchased` column that’s in an array format.
+
+Example: 
+
+```json    
+    [
+    {
+        "currency": "USD",
+        "price": 40,
+        "productName": "jacket",
+        "purchaseTime": "2021-12-17 23:43:47.102",
+        "quantity": 1
+    },
+    {
+        "currency": "USD",
+        "price": 5,
+        "productName": "socks",
+        "quantity": 2
+    }
+    ]
+```    
+
+To send data to a mapping field that requires array data, you can choose between these two options: 
+
+Option | Details
+------ | --------
+Customize array | This enables you to select the specific nested properties to send to the destination. 
+Select array | This enables you to send all nested properties within the array.
+
+> success ""
+> Certain array mapping fields have a fixed list of properties they can accept. If the names of the nested properties in your array don't match the destination properties, the data won't send. Segment recommends you to use the **Customize array** option to ensure your mapping is successful.
+
+Objects in an array don't need to have the same properties. If a user selects a missing property in the input object for a mapping field, the output object will miss the property.

src/connections/reverse-etl/reverse-etl-catalog.md

@@ -1,8 +1,14 @@
 ---
 title: Reverse ETL Catalog
+beta: false
 ---
 
-These destinations support [Reverse ETL](/docs/connections/reverse-etl/). If you don’t see your destination listed in the Reverse ETL catalog, use the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/) to send data from your Reverse ETL warehouse to other destinations listed in the [catalog](/docs/connections/destinations/catalog/).  
+Reverse ETL supports the entire Segment destination catalog - 30+ Actions destinations are natively supported and all other destinations are supported through the [Segment Connections](#segment-connections-destination) destination. 
+
+> success ""
+> Twilio Engage Premier Subscriptions users can use the [Segment Profiles](/docs/connections/destinations/catalog/actions-segment-profiles/) destination to enrich their warehouse data.  
+
+The following destinations natively support [Reverse ETL](/docs/connections/reverse-etl/). If you don’t see your destination listed in the Reverse ETL catalog, use the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/) to send data from your Reverse ETL warehouse to other destinations listed in the [catalog](/docs/connections/destinations/catalog/).  
 
 <div class="destinations-catalog">
       <div class="flex flex--wrap waffle waffle--xlarge">
@@ -37,3 +43,26 @@ These destinations support [Reverse ETL](/docs/connections/reverse-etl/). If you
       </div>
     </div>
 
+## Segment Connections destination
+If you don’t see your destination listed in the Reverse ETL catalog, use the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/) to send data from your Reverse ETL warehouse to other destinations listed in the [catalog](/docs/connections/destinations/catalog/).  
+
+The Segment Connections destination enables you to mold data extracted from your warehouse in [Segment Spec](/docs/connections/spec/) API calls that are then processed by [Segment’s HTTP Tracking API](/docs/connections/sources/catalog/libraries/server/http-api/). The requests hit Segment’s servers, and then Segment routes your data to any destination you want. Get started with the [Segment Connections destination](/docs/connections/destinations/catalog/actions-segment/). 	
+
+> warning ""
+> The Segment Connections destination sends data to Segment’s Tracking API, which has cost implications. New users count as new MTUs and each call counts as an API call. For information on how Segment calculates MTUs and API calls, please see [MTUs, Throughput and Billing](/docs/guides/usage-and-billing/mtus-and-throughput/).
+
+## Send data to Engage with Segment Profiles
+Engage Premier Subscriptions users can use Reverse ETL to sync subscription data from warehouses to destinations.
+
+To get started with using Reverse ETL for subscriptions:
+1. Navigate to **Engage > Audiences** and select the **Profile explorer** tab.
+2. Click **Manage subscription statuses** and select **Update subscription statuses**.
+3. Select **Sync with RETL** as the method to update your subscription statuses.
+4. Click **Configure**.
+5. In the Reverse ETL catalog, select the Reverse ETL source you want to use.
+6. Set up the source. Refer to the [add a source](/docs/connections/reverse-etl/setup/#step-1-add-a-source) section for more details on how to set up the source.
+7. Add the Segment Profiles destination as your Reverse ETL destination. Refer to [add a destination](/docs/connections/reverse-etl/setup/#step-3-add-a-destination) for more details on how to set up the destination.
+8. Once your destination is set, go to the **Mappings** tab of your destination and click **Add Mapping**.
+9. Select the model you want to use and then select **Send Subscriptions**.
+10. Click **Create Mapping**.
+11. Follow the steps in the [Create Mappings](/docs/connections/reverse-etl/setup/#step-4-create-mappings) section to set your mappings.

src/connections/reverse-etl/reverse-etl-source-setup-guides/azure-setup.md

@@ -61,7 +61,7 @@ To set up Azure as your Reverse ETL source:
 5. Navigate to **Connections > Sources** and select the **Reverse ETL** tab. 
 6. Click **+ Add Reverse ETL source**. 
 7. Select **Azure** and click **Add Source**. 
-8. Enter the configuration settings for you Azure source based on the information from Step 3. 
+8. Enter the configuration settings for your Azure source based on the information from Step 3. 
     * Hostname: 
         * Use `xxxxxxx.sql.azuresynapse.net` if you’re connecting to a dedicated SQL pool in Synapse workspace. 
         * Use `xxxxxxx.database.windows.net` if you’re connecting to a dedicated SQL pool (formerly SQL DW)
@@ -72,5 +72,5 @@ To set up Azure as your Reverse ETL source:
 9. Click **Test Connection** to see if the connection works. If the connection fails, make sure you have the right permissions and credentials, then try again.
 10. Click **Add source** if the test connection is successful. 
 
-After you've successfully added your Azure source, [add a model](/docs/connections/reverse-etl/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. 
+After you've successfully added your Azure source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide. 
 

src/connections/reverse-etl/reverse-etl-source-setup-guides/bigquery-setup.md

@@ -28,7 +28,10 @@ To set up the Segment BigQuery connector:
 20. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
 6. Click **Add source** if the test connection is successful.
 
-After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl#step-2-add-a-model).
+After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model).
+
+> info "BigQuery Reverse ETL sources support Segment's dbt extension"
+> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.
 
 ## Constructing your own role or policy
 When you construct your own role or policy, Segment needs the following permissions:
@@ -48,3 +51,5 @@ Permission | Details
 `bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
 
 The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset. 
+
+After you've successfully added your BigQuery source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.