Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support using a pre-existing sqlx pool for postgres (and others as appropriate) #199

Open
derekleverenz opened this issue Dec 17, 2024 · 2 comments

Comments

@derekleverenz
Copy link

derekleverenz commented Dec 17, 2024

SQLx provides database pool implementations of postgres and other dbs and is commonly used in the rust ecosystem. In the case of using datafusion-table-providers to expose postgres in datafusion, but also doing other postgres operations with sqlx, it would be ideal to have datafusion-table-providers use the same pool as the rest of the application.

It looks like it would be possible to either replace the underlying bb8 pool with a sqlx one or abstract it so that either could be used.

I'm happy to take a stab at implementing this myself but want to get feedback before I go and do the work

@phillipleblanc
Copy link
Collaborator

Yeah, the PostgresConnectionPool struct in src/sql/db_connection_pool/postgrespool.rs encapsulates the BB8 pool and returns instances of PostgresConnection in src/sql/db_connection_pool/dbconnection/postgresconn.rs that wraps the actual bb8::PooledConnection.

I don't see much point having runtime code to select between the two, so it might be better to add a new feature sqlx that when enabled switches the PostgresConnectionPool/PostgresConnection to use the sqlx-based pool.

Happy to review any draft/in-progress code once you have an approach you'd like feedback on.

@derekleverenz
Copy link
Author

Looks like this will be a bigger task than I thought initially, using sqlx will also involve replacing the tokio_postgres client and redoing the rows_to_arrow implementation for postgres. Still doable I think just more involved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants