Skip to content

Add PGFS documentation for Google Cloud Storage #6820

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 19, 2025
Merged

Add PGFS documentation for Google Cloud Storage #6820

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions advocacy_docs/edb-postgres-ai/ai-accelerator/pgfs/functions/gcs.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
title: "Pipelines PGFS with Google Cloud Storage"
navTitle: "Google Cloud storage"
description: "PGFS options and credentials with Google Cloud Storage."
---


## Overview: Google Cloud Storage
PGFS uses the `gs:` prefix to indicate an Google Cloud Storage bucket.

The general syntax for using GCS is this:
```sql
select pgfs.create_storage_location(
'storage_location_name',
'gs://bucket_name',
credentials => '{}'::JSONB
);
```

### The `credentials` argument in JSON format offers the following settings:
| Option | Description |
|------------------------------------|------------------------------------------|
| `google_application_credentials` | Path to the application credentials file |
| `google_service_account_key_file` | Path to the service account key file |

See the [Google Cloud documentation](https://cloud.google.com/iam/docs/keys-create-delete#creating) for more information on how to manage service account keys.

These options can also be set up via the equivalent environment variables to facilitate authentication in managed environments such as Google Kubernetes Engine.

## Example: private GCS bucket

```sql
SELECT pgfs.create_storage_location('edb_ai_example_images', 'gs://my-company-ai-images',
credentials => '{"google_service_account_key_file": "/var/run/gcs.json"}'
);
```

## Example: authentication in GKE

Ensure that the `GOOGLE_APPLICATION_CREDENTIALS` or the `GOOGLE_SERVICE_ACCOUNT_KEY_FILE` environment variable
is set on your PostgreSQL pod. Then, PGFS will automatically pick them up:

```sql
SELECT pgfs.create_storage_location('edb_ai_example_images', 'gs://my-company-ai-images');
```
30 changes: 15 additions & 15 deletions advocacy_docs/edb-postgres-ai/ai-accelerator/reference/pgfs.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: "Reference documentation for EDB Postgres AI - AI Accelerator Pipel
deepToC: true
---

This reference documentation for EDB Postgres AI - AI Accelerator Pipelines PGFS includes information on the functions and views available in the [pgfs](../pgfs) extension. These functions give aidb access to S3-compatible file systems and local file systems.
This reference documentation for EDB Postgres AI - AI Accelerator Pipelines PGFS includes information on the functions and views available in the [pgfs](../pgfs) extension. These functions give aidb access to S3-compatible file systems, Google Cloud Storage buckets and local file systems.

## pgfs

Expand Down Expand Up @@ -44,13 +44,13 @@ Creates a storage location in the database.

#### Parameters

| Parameter | Type | Default | Description |
|---------------|-------|---------|---------------------------------------------------------|
| `name` | text | | Name for storage location |
| `url` | text | | URL for this storage location (prefix `s3:` or `file:`) |
| `msl_id` | uuid | | Unused |
| `options` | jsonb | | Options for the storage location |
| `credentials` | jsonb | | Credentials for the storage location |
| Parameter | Type | Default | Description |
|---------------|-------|---------|-----------------------------------------------------------------|
| `name` | text | | Name for storage location |
| `url` | text | | URL for this storage location (prefix `s3:`, `gs:`, or `file:`) |
| `msl_id` | uuid | | Unused |
| `options` | jsonb | | Options for the storage location |
| `credentials` | jsonb | | Credentials for the storage location |

#### Example

Expand All @@ -64,13 +64,13 @@ Creates a storage location in the database and associates it with a foreign tabl

#### Parameters

| Parameter | Type | Default | Description |
|-------------------------|------|---------|---------------------------------------------------------|
| `storage_location_name` | text | | Name for storage location |
| `url` | text | | URL for this storage location (prefix `s3:` or `file:`) |
| `msl_id` | uuid | | Unused |
| `options` | json | | Options for the storage location |
| `credentials` | json | | Credentials for the storage location |
| Parameter | Type | Default | Description |
|-------------------------|------|---------|----------------------------------------------------------------|
| `storage_location_name` | text | | Name for storage location |
| `url` | text | | URL for this storage location (prefix `s3:`, `gs:` or `file:`) |
| `msl_id` | uuid | | Unused |
| `options` | json | | Options for the storage location |
| `credentials` | json | | Credentials for the storage location |

#### Example

Expand Down