v0.1.4

New features

Implement interactive REPL for viewing sources and testing data
Implement continue ingestion for postgres
Support for aliases in SQL
Support LIKE operator
Improve Test framework to use prebuilt docker container and validate REST schema
Add version field to Record
Add count API

Bug fixes

Propagate nullables along SQL schema
Update samples with public ecr image
Fix postgres replication messages handling on update
Id in LmdbCache can be duplicated for different records
Remove duplicate dependencies and bump a few libs.
Display user friendly postgres connection failure
Handle error return from thread when intialize grpc + rest
Typos and links in proto files
Add error msg for primary key missing/mismatching

Full Changelog: v0.1.3...v0.1.4

v0.1.3

New features:

• Automatic key generation
• New e2e test framework
• Health endpoint for REST and GRPC
• Optional Push events

Bug Fixes

• Errors propagation from connector threads.
• Synchronize sources startup
• Improve errors reporting
• Increase max readers on lmdb cache
• Hanging dag tests

Full Changelog: v0.1.2...v0.1.3

v0.1.2

New features:

• Improved CLI experience
• GRPC authentication
• Apply filters in GRPC event stream
• SQL aggregation
• Parse Ethereum ABI and events and publish as tables
• CLI init command
• Pretty print schema
• Table and column names validation
• Schema registry support for Debezium
• Schema mapping for Snowflake
• Columns selection from source

Bug fixes:

• Commits synchronization across multiple sources
• Dozer App and Pipeline errors reporting
• Cache benchmark panicking after duplicate insert
• Validate RDS user for postgres replication
• Handle event name conflicts
• Handle Ethereum data conversion for long strings
• Correctly reuse connection
• NULL support for SQL expressions
• NULL support in cache
• Allow conversion from NULL when field is nullable
• Numbers serialization in REST API
• Reduced schema update calls for snowflake ingestion
• Secondary index calculated properly during querying

Development Build

Features

• Integrate odbc connector for snowflake #191 (Karolis Gudiškis)
• introduce generic GRPC queries (#194) (VG)

Chores

• switch to stable (#188) (VG)
• Remove duplicate get_key method (#195) (Bei Chu)
• Remove an unnecessary unwrap (#196) (Bei Chu)
• fix channel & safe generation (#198) (VG)
• fix artifact path (VG)
• fix path (VG)

Commits

• 0cfa354: Implement sync with initial snapshot (Karolis Gudiškis) #191
• c2f2f62: Implement read from stream (Karolis Gudiškis) #191
• baf0e91: Refactor some logic in stream consumer (Karolis Gudiškis) #191
• 36bacc6: fix connector name (Karolis Gudiškis)
• 0e19345: Fix snowflake library name (Karolis Gudiškis)
• ef41ed6: Make driver name configurable (Karolis Gudiškis)
• 764b184: Make snowflake feature configurable (Karolis Gudiškis)
• 99a410f: Fix some unwraps (Karolis Gudiškis)
• 2a482f6: Fix compilation (Karolis Gudiškis)
• 6a6f7eb: Add readme file (Karolis Gudiškis)

Development Release - v0.1.1

Bug Fixes

• Redirect events to the right endpoint

Support

Contact us at https://getdozer.io

Development Release - v0.1.0

Bug Fixes

• Handled groupby with no dimensions fields (#68)
• first flag in CacheIterator was not set correctly (#101)
• CacheIterator descending from past db end (#105)
• Fix cache changes on update operation (#116)
• Benches report missing the comparison (#183)
• Fix schema validation when only some tables are used (#174)
• Wait until all schema initialized before start grpc server (#187)
• Secondary index key should never be empty (#200)
• Make dynamic server optional

Documentation

• Add build dependences in README (#96)

Features

• Reorganise into a mono repo (#6)
• Merge pipeline code (#7)
• Merge api (#8)
• Implement first version of storage (#17)
• Refactor connector * iterator behavior (#24)
• Update issue templates
• Implement type conversion for postgresql (#25)
• Implement basic cache operations (#48)
• Implement secondary indexes (#49)
• Implement query bench (#50)
• Implement proper schema id usage in snapshotter and refactor mapping (#51)
• Implement simple orchestrator (#61)
• Init dozer ci (#77)
• Include concurrency and multiple restore keys (#82)
• Implement multi ingestion sources (#78)
• Implement continue of replication messages stream handling (#94)
• Dozer-api REST (#86)
• Allows to specify state store options (#112)
• Add FullText IndexType (#121)
• Multi-key indexer for state store (#127)
• Implement e2e tests (#130)
• Supports for transactions across multiple threads (#132)
• Implement auth module and middleware (#109)
• Implement the Aggregation Builder (#129)
• Remove schema registry (#138)
• Implement plan generation (#149)
• Write github action for Long running tests, Performance benchmarks, Code vulnerability, Code coverage etc (#146)
• Add import google libs when generate proto with special type (#165)
• Implement schema validator for postgresql (#128)
• Implement indexed record store (#181)
• Implement eth connector (#178)
• Implement cursor modify operations (#182)
• Introduce generic GRPC queries (#194)
• Support descending fields in sorted inverted index (#202)
• Implement release process (#204)

Miscellaneous Tasks

• Include lock file
• Refactor name and minor changes (#16)
• Convert REST to GRPC (#20)
• Fix warnings (#63)
• Fix schema insert and warnings (#72)
• Remove sql scripts (#76)
• Run on every branch
• Use slices instead (#79)
• Fix dozer.yaml (#84)
• Restore branch after refactor (#81)
• Switch error management to anyhow (#85)
• Fix clippy warnings (#88)
• Add log4rs file + logging for processors (#90)
• Improve ci build time (#89)
• Generate schema_id in Sink (#98)
• Make clippy mandatory (#99)
• Refactor cache params (#104)
• Use unwrap instead of ? in tests.
• Record.get_value() returning Result (#115)
• Moved files (#122)
• Inject pipeline_details (#123)
• Make minor improvements to CLI experience. (#135)
• Remove debug messages (#141)
• Make run the default command (#142)
• Refactor indexes into multiple dbs (#167)
• Use 1.64.0 (#177)
• Introduce array types, uint and remove record type (#175)
• Refactor ingestion to connectors & ingestor (#179)
• Introduce a few orchestrator improvements (#184)
• Switch to stable (#188)
• Remove duplicate get_key method (#195)
• Remove an unnecessary unwrap (#196)
• Fix channel & safe generation (#198)
• Bulid secondary index key only from values. (#197)
• Use ipv4 address as default for grpc (#203)
• Update nightly benches script (#201)

Refactor

• Refactor e2e (#64)
• Improve cache interface (#83)
• Build Processing logic on Schema update (#92)
• Dynamic schema support for groupby (#95)
• Remove references to lmdb::Database (#100)
• Use dynamic query value instead of field (#117)
• Migrated to thiserror for dozer-sql and dozer-core (#124)
• Introduce typed errors (#125)
• Refactor errors and reuse exported libs (#126)
• Encapsulate arc in lmdb-sys and tests cleanup (#134)
• Change IndexDefinition from struct to enum (#147)
• Remove state store (#180)
• Unify the Expression builders (#160)

Ref

• Groupby refactor + processor impl (#56)

Support

Contact us at https://getdozer.io