Skip to content

Conversation

@brandur
Copy link
Contributor

@brandur brandur commented Mar 6, 2025

This one aims to give us a workable resolution to one of our most common
problems with sqlc. Namely, that although it allows substitution for
parameters that work with a prepared query, it can't replace arbitrary
parts of a SQL query, leading to operations that aren't possible so that
we either don't do them or end up degrading to raw SQL that's only
checked at runtime.

Here, we add a sqlctemplate package that's designed to be run from
inside custom implementations of sqlc's DBTX interface so that it it
runs after sqlc's generated code but before the query goes to Postgres.

In sqlc code, templates look like this:

-- name: JobCountByState :one
SELECT count(*)
FROM /* TEMPLATE: schema */river_job
WHERE state = @state;

The template replacement is modeled as a comment so that it doesn't
interfere with with sqlc's parsing of SQL syntax. The above is valid SQL
with or without the template, but with it, sqlctemplate can add an
arbitrary schema name to the queried table.

It also supports a form of syntax where a value is required for SQL to
be valid. For example, WHERE and ORDER BY clauses both require a
value for them to be valid. Here, a stand in value is provide between
template tags. It's processed by sqlc's parser, but then replace by the
template engine before the SQL is executed:

-- name: JobList :many
SELECT *
FROM river_job
WHERE /* TEMPLATE_BEGIN: where_clause */ 1 /* TEMPLATE_END */
ORDER BY /* TEMPLATE_BEGIN: order_by_clause */ id /* TEMPLATE_END */
LIMIT @max::int;

Template values are injected via context (don't love this, but there's
no other way in getting information down to a layer below DBTX):

ctx = sqlctemplate.WithTemplates(ctx, map[string]sqlctemplate.Replacement{
    "order_by_clause": {Value: params.OrderByClause},
    "where_clause":    {Value: params.WhereClause},
}, params.NamedArgs)

jobs, err := dbsqlc.New().JobList(ctx, e.dbtx, params.Max)

The template engine is written to be root out as many error as possible
by noticing if a template replacement is passed that doesn't have an
equivalent template in SQL, or if a template in SQL is present for which
there's no replacement.

Named args are support in templates similar to how sqlc supports them.
This allows pgx's prepared statement cache to continue to operate as it
did before, thereby keeping everything fast.

Lastly, I should note that templates are meant as a utility of last
resort. All effort should be made to resolve problems via mainstream
sqlc, and only bring in templates when there's no other option.

@brandur brandur force-pushed the brandur-sqlc-templates branch from d78d533 to b054626 Compare March 6, 2025 02:19
Copy link
Contributor

@bgentry bgentry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid, this should help with a few different areas including unlocking full support for schemas and some other stuff in the works. Let's get it over the line :shipit:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume you're intending to fill this out a bit?

Copy link
Contributor Author

@brandur brandur Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, just wanted to get your reaction before I spent too much time on it in case we decided to go a different direction.

This one aims to give us a workable resolution to one of our most common
problems with sqlc. Namely, that although it allows substitution for
parameters that work with a prepared query, it can't replace arbitrary
parts of a SQL query, leading to operations that aren't possible so that
we either don't do them or end up degrading to raw SQL that's only
checked at runtime.

Here, we add a `sqlctemplate` package that's designed to be run from
inside custom implementations of sqlc's `DBTX` interface so that it it
runs after sqlc's generated code but before the query goes to Postgres.

In sqlc code, templates look like this:

    -- name: JobCountByState :one
    SELECT count(*)
    FROM /* TEMPLATE: schema */river_job
    WHERE state = @State;

The template replacement is modeled as a comment so that it doesn't
interfere with with sqlc's parsing of SQL syntax. The above is valid SQL
with or without the template, but with it, `sqlctemplate` can add an
arbitrary schema name to the queried table.

It also supports a form of syntax where a value is required for SQL to
be valid. For example, `WHERE` and `ORDER BY` clauses both require a
value for them to be valid. Here, a stand in value is provide between
template tags. It's processed by sqlc's parser, but then replace by the
template engine before the SQL is executed:

    -- name: JobList :many
    SELECT *
    FROM river_job
    WHERE /* TEMPLATE_BEGIN: where_clause */ 1 /* TEMPLATE_END */
    ORDER BY /* TEMPLATE_BEGIN: order_by_clause */ id /* TEMPLATE_END */
    LIMIT @max::int;

Template values are injected via context (don't love this, but there's
no other way in getting information down to a layer below `DBTX`):

    ctx = sqlctemplate.WithTemplates(ctx, map[string]sqlctemplate.Replacement{
        "order_by_clause": {Value: params.OrderByClause},
        "where_clause":    {Value: params.WhereClause},
    }, params.NamedArgs)

    jobs, err := dbsqlc.New().JobList(ctx, e.dbtx, params.Max)

The template engine is written to be root out as many error as possible
by noticing if a template replacement is passed that doesn't have an
equivalent template in SQL, or if a template in SQL is present for which
there's no replacement.

Named args are support in templates similar to how sqlc supports them.
This allows pgx's prepared statement cache to continue to operate as it
did before, thereby keeping everything fast.

Lastly, I should note that templates are meant as a utility of last
resort. All effort should be made to resolve problems via mainstream
sqlc, and only bring in templates when there's no other option.
@brandur
Copy link
Contributor Author

brandur commented Mar 7, 2025

Going to keep this around for now as reference for a while longer, but superseded by #794.

@brandur brandur marked this pull request as draft March 7, 2025 04:40
@brandur
Copy link
Contributor Author

brandur commented Mar 8, 2025

Got the rest of this into #795 and #798, so closing it out.

@brandur brandur closed this Mar 8, 2025
@brandur brandur deleted the brandur-sqlc-templates branch March 8, 2025 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants