Skip to content

Commit

Permalink
Fix various problems with rollout dashboard's cache and task processing.
Browse files Browse the repository at this point in the history
As per https://dfinity.atlassian.net/browse/DRE-303 and
https://dfinity.atlassian.net/browse/DRE-304 , in certain cases where the
administrator has intervened in the rollout (via clearing tasks for retry,
or marking them as failed, or marking them as successful), the cache gets
out of synchronization with the actual Airflow state, because these task
changes do not update the task dates (which is what the cache relies on,
to prevent transferring tens of megabytes every 5 seconds).

This implements the use of an incremental log parser that fishes out
tasks updated in the last update window.  With that, we can deduce when
a rollout needs full task update or just certain tasks.

Alas, some tasks do not log updates even when they execute (chiefly the
tasks implemented by Airflow logic and null operators), so we still must
hit Airflow with an incremental "after this date" query (actually, 3)
to obtain that information.  We may be able to optimize this to three
queries for N rollouts rather than three * N, but that is not in the
cards for this PR.

This has been verified manually through replication of the error conditions
in a local Airflow instance.
  • Loading branch information
DFINITYManu committed Oct 23, 2024
1 parent 9e1607f commit e4eb1d2
Show file tree
Hide file tree
Showing 5 changed files with 1,137 additions and 546 deletions.
7 changes: 7 additions & 0 deletions rollout-dashboard/server/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions rollout-dashboard/server/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ futures = "0.3.30"
indexmap = { version = "2.3.0", features = ["serde"] }
lazy_static = "1.5.0"
log = "0.4.22"
querystring = "1.1.0"
regex = "1.10.5"
reqwest = { version = "0.12.5", features = ["json", "cookies"] }
serde = { version = "1.0.203", features = ["derive", "std"] }
Expand Down
Loading

0 comments on commit e4eb1d2

Please sign in to comment.