Skip to content

Latest commit

 

History

History
17 lines (12 loc) · 1.51 KB

Federation.md

File metadata and controls

17 lines (12 loc) · 1.51 KB

Enabling Query Federation

We use accio to rewrite a federated query into multiple single-source ones, and combine the result locally with datafusion.

Currently, this feature is only enabled by building connectorx from source code as follows:

  1. Clone connectorx: git clone [email protected]:sfu-db/connector-x.git.
  2. Build connectorx from source follows the instruction. Note, for the final step, build wheel with build-python-wheel-fed command instead of build-python-wheel.
  3. Install connectorx: pip install ${YOUR_LOCAL_CONNECTORX_PATH}/connectorx-python/target/wheels/${YOUR_WHEEL_FILE}.
  4. Clone accio: git clone [email protected]:sfu-db/accio.git.
  5. Build accio: cd accio/rewriter && mvn package -Dmaven.test.skip=true.
  6. Move the jar file to location ${YOUR_LOCAL_PYTHON_PATH}/site-packages/connectorx/dependencies/federated-rewriter.jar
  7. Configure accio and set the configuration path as FED_CONFIG_PATH. Example configurations can be found here.
  8. Run federated query using connectorx!

Alternatively, accio provides wrappers that can directly run federated queries on various query engines. In particular, it uses connectorx as the data fetching method when datafusion or polars are the federation engine. For more details, check out here.