Skip to content

Commit

Permalink
[FEAT] Add initial Spark Connect support (#3261)
Browse files Browse the repository at this point in the history
Implements basic Spark Connect functionality in Daft with the following:

- Add `daft-connect` crate to handle Spark Connect protocol
- Implement configuration management via Spark Connect API
- Add Python bindings with `connect_start()` function
- Add integration tests for config operations

Currently supports:
- Basic session management
- Config operations (Set, Get, GetWithDefault, GetOption, GetAll, Unset)
- Error handling and status reporting

Notable changes:
- New dependency on `spark-connect` protocol
- Added `tracing` for debugging and monitoring
- Integration with existing Daft infrastructure

Some operations like execute_plan, analyze_plan, and artifact handling
are
currently unimplemented and will return appropriate error messages.
  • Loading branch information
andrewgazelka authored Nov 11, 2024
1 parent f290f40 commit 16f5a8c
Show file tree
Hide file tree
Showing 17 changed files with 1,033 additions and 35 deletions.
203 changes: 180 additions & 23 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 16f5a8c

Please sign in to comment.