[Replicated] crosscluster/logical: enable automatic bidi replication #225
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This patch introduces two options to CREATE LOGICALLY REPLICATED TABLES:
- UNIDIRECTIONAL: to specify a unidectional ldr stream with a fast offline scan
- BIDIRECTIONAL ON {uri}: which customers will use to set up a bidirectional
stream with one command. The uri provided by the customer connects to the
source of the reverse stream, aka the destination of the main stream.
The PARENT option will be used internally to set up the reverse stream.
Specifically, when the destination sets up the reverse stream, it will pass its
jobID as the parent for the reverse stream, which the reverse stream will use
to prevent creating duplicate reverse streams and to configure a stream that
replicates from an offline source. Customers should not directly use it.
Epic: none
Release note: none
This destination will use this api to set up a reverse stream from a cursor. Epic: none Release note: none
For mysterious resons, the resolver cannot fetch offline descriptors unless the mutable flag is passed. Epic: none Release note: none
This patch allows a user to create an LDR stream that replicates from an offline table. Epic: none Release note: none
This patch enables a user to create a bidirectional replication stream with one
command. When the user runs `CREATE LOGICALLY REPLICATED TABLE tabB FROM tabA
ON uriA WITH BIDIRECTIONAL ON uriB`, the following will happen:
- on cluster B, during job planning, construct the reverse stream cmd, while we
still have access to original LDR cmd. This command will have the following
form: `CREATE LOGICAL REPLICATION STREAM FROM TABLE tabB ON 'uriB' INTO TABLE
tabA WITH CURSOR = $1, PARENT = '{og job id}';`
- on cluster B, create table tabB and begin an offline scan, as normal
- on cluster B, once the offline scan completes, but before tabB is published,
set up the reverse stream from B to A at a cursor time after the initial scan
completes but before the tabB has come online. This cursor time prevents data
looping and ensures the replication of all future foreground writes.
- on cluster B, steady state replication begins.
Epic: none
Release note (sql change): when a user runs CREATE LOGICALLY REPLICATED TABLE,
they must specify one of the following options:
- UNIDIRECTIONAL: setups a unidirectional stream with fast initial scan
- BIDIRECTIONAL ON {dest uri}: sets up a bidirectional stream from the original
dest to the original source.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replicated from original PR cockroachdb#138297
Original author: msbutler
Original creation date: 2025-01-06T15:51:16Z
Original reviewers: msbutler, jeffswenson
Original description:
This patch enables a user to create a bidirectional replication stream with one
command. When the user runs
CREATE LOGICALLY REPLICATED TABLE tabB FROM tabA ON uriA WITH BIDIRECTIONAL ON uriB, the following will happen:still have access to original LDR cmd. This command will have the following
form:
CREATE LOGICAL REPLICATION STREAM FROM TABLE tabB ON 'uriB' INTO TABLE tabA WITH CURSOR = $1, PARENT = '{og job id}';set up the reverse stream from B to A at a cursor time after the initial scan
completes but before the tabB has come online. This cursor time prevents data
looping and ensures the replication of all future foreground writes.
Epic: none
Release note (sql change): when a user runs CREATE LOGICALLY REPLICATED TABLE,
they must specify one of the following options:
dest to the original source.