Skip to content

Support credentials and SSL with SQLAlchemy via HTTP/DB URIs #400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 3, 2022

Conversation

amotl
Copy link
Member

@amotl amotl commented Mar 17, 2021

Hi there,

in order to resolve crate/sqlalchemy-cratedb#116, this patch improves some bits when connecting to CrateDB, specifically in scenarios when preferring to connect using a SQLAlchemy DB URI instead of separate connect_args. This is convenient for programs which can be configured using a single DB URI as database connection option, like Apache Superset.

  • 064e341: Accept credentials within the HTTP URI. Example: http://foo:[email protected]:4200/
  • 52e715d: Enable SSL using SQLAlchemy DB URI. Example: crate://cratedb.example.org/?ssl=true

With kind regards,
Andreas.

P.S.: 30fc28e, 93f33b9 and 160aa1e are only maintenance commits without feature significance.

@amotl amotl force-pushed the amo/connect-uri-credentials-https branch from 1491673 to 1f8965e Compare March 17, 2021 18:21
@amotl amotl force-pushed the amo/connect-uri-credentials-https branch from 1f8965e to 03ff23c Compare March 29, 2021 13:02
@amotl amotl force-pushed the amo/connect-uri-credentials-https branch from 03ff23c to ccdfffd Compare April 26, 2022 16:46
@amotl amotl changed the title Improve connect arguments to support credentials and SSL mode with SQLAlchemy Support credentials and SSL with SQLAlchemy via HTTP/DB URIs Apr 26, 2022
@amotl amotl marked this pull request as ready for review April 26, 2022 17:59
@amotl amotl force-pushed the amo/connect-uri-credentials-https branch 2 times, most recently from 160aa1e to 1b1136a Compare April 26, 2022 21:24
@amotl amotl requested review from mfussenegger, seut and SStorm April 27, 2022 07:15
Copy link
Member

@mfussenegger mfussenegger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P.S.: 30fc28e, 93f33b9 and 160aa1e are only maintenance commits without feature significance

Would be cool to split maintenance stuff and features into separate PRs in the future. It is much easier to review that way. Github doesn't support adding suggestions when looking at an individual commit.

Copy link
Member

@mfussenegger mfussenegger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes should probably also get documented somewhere in

An SQLAlchemy database is represented by special type of *Uniform Resource
Locator* (URL) called a `database URL`_.
The simplest database URL for CrateDB looks like this::
crate://<HOST>
Here, ``<HOST>`` is the node *host string*.
A host string looks like this::
<HOST_ADDR>:<PORT>
Here, ``<HOST_ADDR>`` is the hostname or IP address of the CrateDB node and
``<PORT>`` is a valid `psql.port`_ number.
Example host strings:
- ``localhost:4200``
- ``crate-1.vm.example.com:4200``
- ``198.51.100.1:4200``
.. TIP::
If ``<HOST>`` is blank (i.e. just ``crate://``) then ``localhost:4200`` will
be assumed.
Getting a connection
--------------------
Create an engine
................
You can connect to CrateDB using the ``create_engine`` method. This method
takes a `database URL`_.
Import the ``sa`` module, like so:
>>> import sqlalchemy as sa
To connect to ``localhost:4200``, you can do this::
>>> engine = sa.create_engine('crate://')
To connect to ``crate-1.vm.example.com:4200``, you would do this:
>>> engine = sa.create_engine('crate://crate-1.vm.example.com:4200')
If your CrateDB cluster has multiple nodes, however, we recommend that you
configure all of them. You can do that by specifying the ``crate://`` database
URL and passing in a list of :ref:`host strings <database-urls>` passed using
the ``connect_args`` argument, like so::
>>> engine = sa.create_engine('crate://', connect_args={
... 'servers': ['198.51.100.1:4200', '198.51.100.2:4200']
... })
When you do this, the Database API layer will use its :ref:`round-robin
<multiple-nodes>` implementation.
The client validates `SSL server certificates`_ by default. For further
adjusting this behaviour, SSL verification options can be passed in by using
the ``connect_args`` dictionary. For example, use ``ca_cert`` for providing
a path to the CA certificate used for signing the server certificate::
>>> engine = sa.create_engine(
... 'crate://',
... connect_args={
... 'servers': ['198.51.100.1:4200', '198.51.100.2:4200'],
... 'ca_cert': '<PATH_TO_CA_CERT>',
... }
... )
In order to disable SSL verification, use ``verify_ssl_cert = False``, like::
>>> engine = sa.create_engine(
... 'crate://',
... connect_args={
... 'servers': ['198.51.100.1:4200', '198.51.100.2:4200'],
... 'verify_ssl_cert': False,
... }
... )

Otherwise I think this looks good

@amotl amotl force-pushed the amo/connect-uri-credentials-https branch from dae941f to c08ecd1 Compare April 28, 2022 11:38
@amotl
Copy link
Member Author

amotl commented Apr 28, 2022

The changes should probably also get documented somewhere.

Thank you for your attention. I've just added 8fa85a5 and c08ecd1, intending to improve the corresponding documentation about the new capabilities to connect by URI.

@amotl amotl requested a review from mfussenegger April 28, 2022 12:00
amotl added 6 commits May 3, 2022 13:37
ResourceWarning: unclosed <socket.socket fd=12, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 52797), raddr=('127.0.0.1', 44209)>
Occasionally, the test suite would croak like:

  crate.client.exceptions.ConnectionError: No more Servers available, exception from last server: HTTPConnectionPool(host='127.0.0.1', port=44209): Read timed out. (read timeout=2)

By removing the "timeout=2" parameter, that error might go away on CI.
@amotl amotl force-pushed the amo/connect-uri-credentials-https branch from c08ecd1 to 69eea5d Compare May 3, 2022 11:38
@amotl amotl merged commit 80cefb3 into master May 3, 2022
@amotl amotl deleted the amo/connect-uri-credentials-https branch May 3, 2022 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support HTTPS URLs when specifying a SqlAlchemy URI
2 participants