Skip to content

DB migration with fallback steps and more error log example. Closes #330 #332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,37 @@ pg_restore: error: could not execute query: ERROR: deadlock detected - 1
pg_restore: error: could not execute query: ERROR: operator does not exist: public.ltree = public.ltree - 264
```

### Fallback Procedure

If you encounter errors during `pg_restore`, or if the migration must be aborted at this stage, follow these steps to safely revert changes:

1. Clean Up Replication Artifacts

- **Delete Replication Slots** from the source database:

```sql
SELECT * FROM pg_replication_slots;
SELECT pg_drop_replication_slot('<slot_name>');
```

- **Delete the Publication** created on the source database:

```sql
SELECT * FROM pg_publication;
DROP PUBLICATION <pubname>;
```

2. Terminate New Resources

- **Delete the newly created TED stack** and associated **RDS DB instance** used for the target database.

3. Disable Logical Replication

- Revert the parameter `rds.logical_replication` to `0` on the source database, as done in [Step 1: Enable DB Logical Replication](#step-1-enable-db-logical-replication).
- Reboot the source DB instance to apply changes.

> [!WARNING]
> Ensure all cleanup steps are completed before attempting a fresh migration to avoid conflicts or residual configuration issues.

## Step 11: Create Subscription in the Target DB Instance

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
---
title: Discover ServiceNow Custom Tables
sidebar_label: Discover ServiceNow Custom Tables
---
This guide explains how to discover and manage records from custom ServiceNow tables in Turbot Guardrails.

## What you'll learn

- How to enable discovery of custom ServiceNow tables in Guardrails
- How to configure which tables to discover and filter records
- How to set up event-driven updates for custom table records

## Time to complete

10-15 minutes

## Prerequisites

Before starting this guide, make sure you have:

- [Imported your ServiceNow instance into Guardrails](/guardrails/docs/guides/servicenow/import-servicenow-instance)
- Installed the `@turbot/servicenow-custom` mod in your workspace

# Discovering ServiceNow Custom Tables

Turbot Guardrails can discover and manage records from any specified ServiceNow table, extending visibility beyond the pre-configured Application, Cost Center, and User tables. This allows you to bring data from your custom or other standard ServiceNow tables into Guardrails as `ServiceNow > Custom > Record` resources.

## Prerequisites

### Associate your ServiceNow instance to Turbot Guardrails

Before you get started, ensure you have completed the steps in the [Importing a ServiceNow instance into Guardrails](/guardrails/docs/guides/servicenow/import-servicenow-instance) guide to associate your ServiceNow instance with Turbot Guardrails.

### Turbot Guardrails Mod Installation

The `@turbot/servicenow-custom` mod must be installed in your Guardrails workspace. This mod provides the necessary resource types, policies, and controls for discovering custom table records. Ensure it is installed and the mod's `Installed` control is in the `OK` state.

## Enabling Custom Table Discovery

To enable the discovery of records from specific ServiceNow tables, you need to configure the relevant policies. The primary policies involved are:

- `ServiceNow > Custom > Table > CMDB`
- Enables the discovery mechanism for custom tables.
- Set this policy to `Enforce: Enabled` at the scope of your ServiceNow instance resource or higher.
- `ServiceNow > Custom > Table > CMDB > Tables`
- Specifies the list of ServiceNow table names you want Guardrails to discover records from.
- Provide a YAML list of table names (e.g., `["u_custom_table", "cmdb_ci_storage_volume"]`).
- Defaults to an empty list `[]`.
- > [!IMPORTANT]
> Removing a table name from this list will result in the deletion of the corresponding `ServiceNow > Custom > Table` resource and all its child `ServiceNow > Custom > Record` resources from the Guardrails CMDB.
- `ServiceNow > Custom > Record > CMDB > Query`
- Allows filtering of records discovered from the specified tables using a ServiceNow encoded query string.
- Paste the encoded query string as the policy value.
- Defaults to `""` (empty string), meaning no filter is applied.
- `ServiceNow > Custom > Record > CMDB > Title`
- Specifies the data key(s) used to retrieve the title for discovered `ServiceNow > Custom > Record` resources in Guardrails.
- Provide an array of strings representing field names in order of preference (e.g., `["name", "display_name", "sys_id"]`). Guardrails uses the first field in the list that contains a non-empty value.
- This allows for fallback options if preferred fields are missing or empty.
- Defaults to `["name", "display_value", "display_name", "title", "label", "short_description", "number", "sys_name", "sys_title", "sys_id"]`.
- You can customize this list by creating a new policy setting.

### Business Rule for Event-Driven Updates (Optional)

Similar to the pre-configured table sync, you can enable event-driven updates for discovered custom tables:

- `ServiceNow > Custom > Table > Business Rule`
- Configures ServiceNow Business Rules for event handling of record changes (new, updated, deleted) in the discovered tables.
- Set to `Enforce: Configured` to enable automatic setup and management.
- Requires `ServiceNow > Config > System Properties` to be set to `Enforce: Configured` as described in the [ServiceNow sync prerequisites](/guardrails/docs/guides/servicenow/servicenow-to-guardrails-sync#prerequisites).
- `ServiceNow > Custom > Table > Business Rule > Name`
- Allows customization of the Business Rule name.

## Example: Discovering the Server Table (`cmdb_ci_server`)

To discover records from the standard ServiceNow Server table `cmdb_ci_server`:

1. Set `ServiceNow > Custom > Table > CMDB` to `Enforce: Enabled`.
2. Set `ServiceNow > Custom > Table > CMDB > Tables` to:
```yaml
- cmdb_ci_server
```
3. (Optional) To only discover active servers, set `ServiceNow > Custom > Record > CMDB > Query` to `active=true` (or the relevant field/value for server status).
4. (Optional) To enable real-time updates, set `ServiceNow > Custom > Table > Business Rule` to `Enforce: Configured`.

Once configured, Guardrails will begin discovering records from the `cmdb_ci_server` table. These records will appear in your Guardrails inventory under the associated ServiceNow instance resource as `ServiceNow > Custom > Record` resources.

## Next Steps

- Explore the discovered `ServiceNow > Custom > Record` resources in your Guardrails inventory.
- Utilize the data from these custom records in Guardrails policies and calculated policies for context-aware automation.

For more details on the specific policies and controls introduced, refer to the `servicenow-custom` mod changelog [[1]](https://turbot.com/guardrails/changelog/servicenow-custom-v5-0-0).

We want to hear from you! Join our [Slack Community](https://turbot.com/community/join) `#guardrails` channel to ask questions and share feedback.
112 changes: 112 additions & 0 deletions docs/guides/servicenow/discovery-servicenow-custom-tables/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
---
title: Discover ServiceNow Custom Tables
sidebar_label: Discover ServiceNow Custom Tables
---

# Discovering ServiceNow Custom Tables

In this guide, you will:

- Learn how to discover and manage records from custom ServiceNow tables in Turbot Guardrails
- Configure table discovery, filtering, and record title display settings
- Set up optional business rules for custom table management

Turbot Guardrails can discover and manage records from any specified ServiceNow table, extending visibility beyond the pre-configured Application, Cost Center, and User tables. This allows you to bring data from your custom or other standard ServiceNow tables into Guardrails as `ServiceNow > Custom > Record` resources.

## Prerequisites

- An active ServiceNow instance integrated with Turbot Guardrails following the [Importing a ServiceNow instance into Guardrails](/guardrails/docs/guides/servicenow/import-servicenow-instance) guide.
- The `@turbot/servicenow-custom` mod installed in your Guardrails workspace. See [Install a Mod](/guardrails/docs/guides/configuring-guardrails/install-mod).
- Administrator access to your ServiceNow instance.
- Administrator access to your Turbot Guardrails workspace.

## Enabling Custom Table Discovery

To enable discovery of records from ServiceNow custom tables, configure the following policies:

### ServiceNow > Custom > Table > CMDB

- Enables the discovery for custom tables.
- Set this policy to `Enforce: Enabled` at the scope of your ServiceNow instance resource or higher.

### ServiceNow > Custom > Table > CMDB > Tables

- Specifies the list of ServiceNow table names you want Guardrails to discover records from.
- Provide a YAML list of table names e.g. `["u_custom_table", "cmdb_ci_storage_volume"]`.
- Defaults to an empty list `[]`.

> [!IMPORTANT]
> Removing a table name from this list will result in the deletion of the corresponding `ServiceNow > Custom > Table` resource and all its child `ServiceNow > Custom > Record` resources from the Guardrails CMDB.

## Set Optional Policies

The following two policies allow you to control which records are discovered and how they are displayed in Guardrails:

### ServiceNow > Custom > Record > CMDB > Query

- Allows you to refine which records are discovered by applying [ServiceNow encoded query string](https://www.servicenow.com/docs/bundle/yokohama-platform-user-interface/page/use/using-lists/concept/c_EncodedQueryStrings.html).
- Useful for limiting discovery to specific records that match your criteria.
- For example, filter by status, category, or any other field available in the ServiceNow table
- Defaults to `""` (empty string) which means all records will be discovered.

### ServiceNow > Custom > Record > CMDB > Title

- Specifies the data key(s) used to retrieve the title for discovered `ServiceNow > Custom > Record` resources in Guardrails.
- Provide an array of strings representing field names in order of preference e.g. `["name", "display_name", "sys_id"]`. Guardrails uses the first field in the list that contains a non-empty value.
- This allows for fallback options if preferred fields are missing or empty.
- Defaults to `["name", "display_value", "display_name", "title", "label", "short_description", "number", "sys_name", "sys_title", "sys_id"]`.
- Customize by creating a new policy setting.

<!-- ### Business Rule for Event-Driven Updates (Optional) -->

## (Optional) Configure Real-time Business Rules

To enable real-time updates when records change in ServiceNow, you can configure Business Rules. This allows Guardrails to automatically sync changes as they happen in ServiceNow.

Before configuring Business Rules, ensure you have:

1. Set `ServiceNow > Config > System Properties` to `Enforce: Configured` as described in the [ServiceNow sync prerequisites](/guardrails/docs/guides/servicenow/servicenow-to-guardrails-sync#prerequisites).
2. Administrator access to create Business Rules in ServiceNow

### ServiceNow > Custom > Table > Business Rule

Configures ServiceNow Business Rules for real-time event handling of record changes (new, updated, deleted) in the discovered tables.

- Set to `Enforce: Configured` to enable automatic setup and management.


### ServiceNow > Custom > Table > Business Rule > Name

- Allows customization of the Business Rule name.

## Example: Discovering the Server Table

To discover records from the standard ServiceNow Server table `cmdb_ci_server`, follow these steps:

1. Set `ServiceNow > Custom > Table > CMDB` to `Enforce: Enabled`.
2. Set `ServiceNow > Custom > Table > CMDB > Tables` with yaml value as `- cmdb_ci_server`

![ServiceNow Discovery Policy Settings](/images/docs/guardrails/guides/servicenow/discovery-servicenow-custom-tables/policy-setting.png)

3. (Optional) To only discover active servers, set `ServiceNow > Custom > Record > CMDB > Query` to `active=true` (or the relevant field/value for server status).
4. (Optional) To enable real-time updates, set `ServiceNow > Custom > Table > Business Rule` to `Enforce: Configured`.

![ServiceNow Business Rule Setting](/images/docs/guardrails/guides/servicenow/discovery-servicenow-custom-tables/business-rule-policy.png)

Guardrails will now discover records from `cmdb_ci_server`. These records will appear in your Guardrails inventory under the ServiceNow instance resource as `ServiceNow > Custom > Record` resources.


## Next Steps

Please see the following resources to learn more about ServiceNow integrations:

- [Enable ServiceNow to Guardrails Sync](/guardrails/docs/guides/servicenow/servicenow-to-guardrails-sync)
- [Enable Guardrails to ServiceNow Sync](/guardrails/docs/guides/servicenow/guardrails-to-servicenow-sync)

## Troubleshooting

| Issue | Description | Guide |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
| Common errors | Common issues that may prevent controls from running include network connectivity problems, permission issues, and API rate limits. These can cause controls to enter an error state. | Refer to [Common Troubleshooting](/guardrails/docs/guides/troubleshooting) for detailed resolution steps. |
| Further Assistance | If you encounter further issues with Calculated Policies, please open a ticket with us and attach the relevant information to assist you more efficiently. | [Open Support Ticket](https://support.turbot.com) |
| Community Support | We want to hear from you! Join our [Slack Community](https://turbot.com/community/join) `#guardrails` channel to ask questions and share feedback. | [Join Slack Community](https://turbot.com/community/join) |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
9 changes: 3 additions & 6 deletions docs/guides/servicenow/guardrails-to-servicenow-sync/index.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
---
title: "Enable Guardrails sync to ServiceNow"
template: Documentation
nav:
title: "Enable Guardrails sync to ServiceNow"
order: 20
title: Enable Guardrails to ServiceNow Sync
sidebar_label: Enable Guardrails to ServiceNow Sync
---

# Enable Guardrails sync to ServiceNow
# Enable Guardrails to ServiceNow Sync

## Prerequisites

Expand Down
9 changes: 3 additions & 6 deletions docs/guides/servicenow/import-servicenow-instance/index.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
---
title: "Importing a ServiceNow instance into Guardrails"
template: Documentation
nav:
title: "Importing Accounts"
order: 20
title: Import ServiceNow Instance
sidebar_label: Import ServiceNow Instance
---

# Importing a ServiceNow instance into Guardrails
# Importing a ServiceNow Instance in Guardrails

## Prerequisites to import a ServiceNow instance

Expand Down
30 changes: 27 additions & 3 deletions docs/guides/servicenow/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ Turbot Guardrails is deeply integrated with [ServiceNow](https://servicenow.com)

1. Continuously sync ServiceNow resources to Turbot Guardrails
2. Continuously sync cloud resources with Turbot Guardrails to ServiceNow
3. Discover and manage records from any ServiceNow table


## 1. Continuously sync ServiceNow resources to Turbot Guardrails
ServiceNow resources such Applications, Cost Centers, Users, etc tables can sync to Turbot Guardrails. This integration bridges the gap between your business context with your cloud & security controls.
Expand All @@ -29,7 +31,7 @@ Syncing ServiceNow CMDB data brings critical context to your cloud resources in
* And many other cloud and security automation use cases

**To get started:**
* [Import a ServiceNow Instance](https://turbot.com/guardrails/docs/guides/servicenow/import-servicenow-instance) into a Guardrails Folder.
* [Import a ServiceNow Instance](/guardrails/docs/guides/servicenow/import-servicenow-instance) into a Guardrails Folder.
* [Enable ServiceNow sync to Guardrails](https://turbot.com/guardrails/docs/guides/servicenow/servicenow-to-guardrails-sync) to configure real-time syncing.

Additional context about the feature is in the announcement post [ServiceNow + Guardrails: Context-aware cloud & security automation](https://turbot.com/guardrails/blog/2023/12/context-aware-guardrails-servicenow-integration).
Expand All @@ -50,8 +52,30 @@ Continuously syncing your cloud resources into ServiceNow enhances the CMDB into
* Enrich context: surface tags as key resource data.

**To get started:**
* [Import a ServiceNow Instance](https://turbot.com/guardrails/docs/guides/servicenow/import-servicenow-instance) into a Guardrails Folder.
* [Enable Guardrails sync to ServiceNow](https://turbot.com/guardrails/docs/guides/servicenow/guardrails-to-servicenow-sync) to configure real-time syncing.
* [Import a ServiceNow Instance](/guardrails/docs/guides/servicenow/import-servicenow-instance) into a Guardrails Folder.
* [Enable Guardrails sync to ServiceNow](/guardrails/docs/guides/servicenow/guardrails-to-servicenow-sync) to configure real-time syncing.

## 3. Discover and manage records from any ServiceNow table

Guardrails can discover and manage records from any ServiceNow table, extending visibility beyond the pre-configured Application, Cost Center, and User tables. This allows you to bring data from your custom or other standard ServiceNow tables into Guardrails as ServiceNow Custom Record resources.

When enabled, Guardrails configures and manages:
1. Guardrails provides the ServiceNow Custom mod, with policies and controls to support discovering and managing any ServiceNow table
2. Event handlers (Business Rules) for each discovered table to handle real-time updates as records are created, modified, and deleted
3. Audit trail of all activity on your discovered tables to see what happened, who made the change, when it occurred, and what configuration differences were made

Discovering custom tables enhances your cloud and security automation capabilities:

* Extend visibility to any ServiceNow table data
* Filter which records to discover using ServiceNow queries
* Configure how records are displayed in Guardrails
* Enable real-time updates through Business Rules

**To get started:**

* [Import a ServiceNow Instance](/guardrails/docs/guides/servicenow/import-servicenow-instance) into a Guardrails Folder
* [Discover ServiceNow Custom Tables](/guardrails/docs/guides/servicenow/discovery-servicenow-custom-tables) to configure table discovery and record management


Additional context about the feature is in the announcement post [Continuous Cloud CMDB Sync with ServiceNow & Turbot Guardrails](https://turbot.com/guardrails/blog/2023/12/cmdb-sync-guardrails-servicenow-integration).

Expand Down
Loading