diff --git a/docs/integrations/data-ingestion/clickpipes/postgres/source/alloydb.md b/docs/integrations/data-ingestion/clickpipes/postgres/source/alloydb.md new file mode 100644 index 00000000000..69a1b3f75b4 --- /dev/null +++ b/docs/integrations/data-ingestion/clickpipes/postgres/source/alloydb.md @@ -0,0 +1,154 @@ +--- +sidebar_label: 'AlloyDB Postgres' +description: 'Set up an AlloyDB Postgres instance as a source for ClickPipes' +slug: /integrations/clickpipes/postgres/source/alloydb +title: 'AlloyDB Postgres Source Setup Guide' +doc_type: 'guide' +--- + +import edit_instance from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/1_edit_instance.png'; +import set_flags from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/2_set_flags.png'; +import verify_logical_replication from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/3_verify_logical_replication.png'; +import configure_network_security from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/4_configure_network_security.png'; +import configure_network_security2 from '@site/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/5_configure_network_security.png'; +import Image from '@theme/IdealImage'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# AlloyDB Postgres source setup guide + +## Supported versions {#supported-versions} + +To propagate data from your AlloyDB instance to ClickHouse Cloud using ClickPipes, your instance must be configured for **logical replication**. This is supported **from AlloyDB Version 14**. + +## Enable logical replication {#enable-logical-replication} + +To check if logical replication is enabled in your AlloyDB instance, run the following query against your primary instance: + +```sql +SHOW wal_level; +``` + +If the result is `logical`, logical replication is already enabled and you can skip to the [next step](#creating-clickpipes-user-and-granting-permissions). If the result is `replica`, you must set the [`alloydb.enable_pglogical`](https://cloud.google.com/alloydb/docs/reference/alloydb-flags#alloydb.enable_pglogical) and [`alloydb.logical_decoding`](https://cloud.google.com/alloydb/docs/reference/alloydb-flags#alloydb.logical_decoding) flags to `on` in the primary instance. + +:::warning +As noted in the [AlloyDB flags documentation](https://cloud.google.com/alloydb/docs/reference/alloydb-flags), modifying the flags that enable logical replication requires a restart of the primary instance. +::: + +To enable these flags: + +1. In the Google Cloud Console, navigate to the AlloyDB [Clusters](https://console.cloud.google.com/alloydb/clusters) page. From the **Actions** menu for your primary instance, click **Edit**. + + Edit primary instance configuration + +2. Scroll down to **Advanced configuration options** and expand the section. Under **Flags**, click **Add a database flag**. + - Add the [`allowdb.enable_pglogical`](https://cloud.google.com/alloydb/docs/reference/alloydb-flags#alloydb.enable_pglogical) flag and set its value to `on` + - Add the [`alloydb.logical_decoding`](https://cloud.google.com/alloydb/docs/reference/alloydb-flags#alloydb.logical_decoding) flag and set its value to `on` + + Set allowdb.enable_pglogical and alloydb.logical_decoding flags to on + +3. Click **Update instance** to save the configuration changes. It's important to note that this action **triggers a restart of the primary instance.** + +4. Once the status of the instance changes from `Updating` to `Ready`, run the following query against your primary instance to verify that logical replication is enabled: + + ```sql + SHOW wal_level; + ``` + + The result should be `logical`. + + Verify that logical replication is enabled + +## Create a ClickPipes user and manage replication permissions {#create-a-clickpipes-user-and-manage-replication-permissions} + +Connect to your AlloyDB instance as an admin user and execute the following commands: + +1. Create a dedicated user for ClickPipes: + + ```sql + CREATE USER clickpipes_user PASSWORD 'some-password'; + ``` + +2. Grant the dedicated user permissions on the schema(s) you want to replicate. + + ```sql + GRANT USAGE ON SCHEMA "public" TO clickpipes_user; + GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO clickpipes_user; + ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO clickpipes_user; + ``` + + The example above shows permissions for the `public` schema. Repeat the sequence of commands for each schema you want to replicate using ClickPipes. + +3. Grant the dedicated user permissions to manage replication: + + ```sql + ALTER ROLE clickpipes_user REPLICATION; + ``` + +4. Create a [publication](https://www.postgresql.org/docs/current/logical-replication-publication.html) with the tables you want to replicate. We strongly recommend only including the tables you need in the publication to avoid performance overhead. + + :::warning + All tables included in the publication must either have a **primary key** defined _or_ have its **replica identity** configured to `FULL`. See the [Postgres FAQs](../faq.md#how-should-i-scope-my-publications-when-setting-up-replication-how-should-i-scope-my-publications-when-setting-up-replication) for guidance on scoping. + ::: + + - To create a publication for specific tables: + + ```sql + CREATE PUBLICATION clickpipes FOR TABLE table_to_replicate, table_to_replicate2; + ``` + + - To create a publication for all tables in a specific schema: + + ```sql + CREATE PUBLICATION clickpipes FOR TABLES IN SCHEMA "public"; + ``` + + The `clickpipes` publication will contain the set of change events generated from the specified tables, and will later be used to ingest the replication stream. + +## Configure network security {#configure-network-security} + +:::note +ClickPipes does not support Private Service Connect (PSC) connections. If you do not allow public access to your AlloyDB instance, you can [use an SSH tunnel](#configure-network-security) to connect securely. PSC will be supported in the future. +::: + +Next, you must allow connections to your AlloyDB instance from ClickPipes. + + + + +1. In the Google Cloud Console, navigate to the AlloyDB [Clusters](https://console.cloud.google.com/alloydb/clusters) page. Select your primary instance to open the **Overview** page. + +2. Scroll down to **Instances in your cluster** and click **Edit primary**. + +3. Check the **Enable Public IP** checkbox to allow connections to the instance over the public internet. Under **Authorized external networks**, enter the [list of ClickPipes static IP addresses](../../../clickpipes/index.md#list-of-static-ips-list-of-static-ips) for the region your service is deployed in. + + Configure networking for public access with an IP allowlist + + :::note + AlloyDB expects addresses to be specified in [CIDR notation](https://cloud.google.com/alloydb/docs/connection-overview#public-ip). You can adapt the provided list of ClickPipes static IP addresses to follow this notation by appending `/32` to each address. + ::: + +4. Under **Network Security**, select **Require SSL Encryption (default)** (if not already selected). + +5. Click **Update instance** to save the network security configuration changes. + + + + +If you do not allow public access to your AlloyDB instance, you must first set up an SSH bastion host to securely tunnel your connection. To set up an SSH bastion host on Google Cloud Platform: + +1. Create and start a Google Compute Engine (GCE) instance following the [official documentation](https://cloud.google.com/compute/docs/instances/create-start-instance). + - Ensure the GCE instance is in the same Virtual Private Network (VPC) as your AlloyDB instance. + - Ensure the GCE instance has a [static public IP address](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address). You’ll use this IP address when connecting ClickPipes to your SSH bastion host. + +2. Update the firewall rules of the SSH bastion host to allow traffic from the [list of ClickPipes static IP addresses](../../../clickpipes/index.md#list-of-static-ips-list-of-static-ips) for the region your service is deployed in. + +3. Update the firewall rules of AlloyDB to allow traffic from the SSH bastion host. + + + + +## What's next? {#whats-next} + +You can now [create your ClickPipe](../index.md) and start ingesting data from your Postgres instance into ClickHouse Cloud. +Make sure to note down the connection details you used while setting up your Postgres instance as you will need them during the ClickPipe creation process. diff --git a/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/1_edit_instance.png b/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/1_edit_instance.png new file mode 100644 index 00000000000..284e59c53a3 Binary files /dev/null and b/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/1_edit_instance.png differ diff --git a/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/2_set_flags.png b/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/2_set_flags.png new file mode 100644 index 00000000000..4656457b11c Binary files /dev/null and b/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/2_set_flags.png differ diff --git a/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/3_verify_logical_replication.png b/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/3_verify_logical_replication.png new file mode 100644 index 00000000000..189c67a6d22 Binary files /dev/null and b/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/3_verify_logical_replication.png differ diff --git a/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/4_configure_network_security.png b/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/4_configure_network_security.png new file mode 100644 index 00000000000..12cf4780b0c Binary files /dev/null and b/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/4_configure_network_security.png differ diff --git a/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/5_configure_network_security.png b/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/5_configure_network_security.png new file mode 100644 index 00000000000..d7d4b8a7931 Binary files /dev/null and b/static/images/integrations/data-ingestion/clickpipes/postgres/source/alloydb/5_configure_network_security.png differ