Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address Tracker for subscriptions to changed UTXOs notifications #427

Merged
merged 135 commits into from
Apr 9, 2024
Merged
Show file tree
Hide file tree
Changes from 122 commits
Commits
Show all changes
135 commits
Select commit Hold shift + click to select a range
1778f59
Add an integration test covering UTXOs propagation
tiram88 Nov 13, 2023
6151986
Refactor daemon_utxos_propagation_test()
tiram88 Nov 14, 2023
e4a47df
Add heap profiling feature
tiram88 Nov 20, 2023
0f676ef
Cover VirtualDaaScoreChanged notifications in test
tiram88 Nov 20, 2023
ac252c0
Merge branch 'master' into rpc-memory-benchmark
tiram88 Nov 22, 2023
5373479
Assert all changed UTXOs
tiram88 Nov 22, 2023
d35520d
Use UtxosChangedScope ctor
tiram88 Nov 27, 2023
8f54f84
Make active non-blanket UtxosChanged subscription unique
tiram88 Nov 28, 2023
6782a8d
Refactor broadcaster subscription unregistering
tiram88 Nov 28, 2023
d952b06
Add Display to Connection trait
tiram88 Nov 28, 2023
904559f
Save creation of UtxosChanged address vec on listener unregistration
tiram88 Nov 30, 2023
3be0af0
Reduce UtxosChangedSubscription memory footprint on mutation
tiram88 Dec 2, 2023
deca955
Add gRPC client timeout to every request & remove tpc keep alive
tiram88 Dec 2, 2023
72d4074
Reconnect gRPC client on broken pipe
tiram88 Dec 2, 2023
df0351d
Disable gRPC server http2 keepalive
tiram88 Dec 2, 2023
34a1a6d
Add a test benchmarking UtxosChanged notifications memory footprint
tiram88 Dec 3, 2023
d9ab82e
Merge branch 'master' into rpc-memory-benchmark
tiram88 Dec 3, 2023
1669e47
Fix log
tiram88 Dec 3, 2023
17539a2
Merge branch 'master' into rpc-memory-benchmark
tiram88 Dec 10, 2023
a993a99
Merge branch 'master' into rpc-memory-benchmark
tiram88 Dec 10, 2023
bd635e2
Merge branch 'master' into rpc-memory-benchmark
tiram88 Dec 10, 2023
83b74c0
Merge branch 'master' into rpc-memory-benchmark
tiram88 Dec 11, 2023
33515c2
Remove tokio dependency in crate notify
tiram88 Dec 11, 2023
fd0c45e
Add a UtxosChanged mutation policy to the notification system
tiram88 Dec 12, 2023
8ede4f1
Share subscriptions between listeners and broadcasters
tiram88 Dec 16, 2023
34c5ec4
Replace UtxoAddress with Address
tiram88 Dec 16, 2023
903a9a8
Refactor Single::mutate
tiram88 Dec 17, 2023
9977c27
Refactor `UtxosChangedSubscription` internals
tiram88 Dec 17, 2023
5518180
Make `UtxosChangedSubscription::hash()` cheaper to compute
tiram88 Dec 17, 2023
ffc5f26
mimalloc
michaelsutton Dec 19, 2023
e9a2508
mimalloc disable purge_decommits
michaelsutton Dec 19, 2023
56a80d3
Merge branch 'master' into rpc-memory-benchmark
tiram88 Dec 21, 2023
4a8ae1c
Add to simpa and integration tests
someone235 Dec 21, 2023
a9efe14
remove redundant unix and replace with win/linux/mac
someone235 Dec 21, 2023
7478786
Add comment
someone235 Dec 21, 2023
738771b
Add comment
someone235 Dec 21, 2023
b07a92e
Sort out features
someone235 Dec 21, 2023
26136ad
Merge remote-tracking branch 'someone235/mimalloc' into rpc-memory-be…
tiram88 Dec 21, 2023
30a1827
Merge branch 'master' into rpc-memory-benchmark
tiram88 Dec 26, 2023
99be373
Merge branch 'master' into rpc-memory-benchmark
tiram88 Dec 26, 2023
1f1449e
Merge branch 'master' into rpc-memory-benchmark
tiram88 Dec 31, 2023
a628a42
Merge branch 'master' into rpc-memory-benchmark
tiram88 Jan 5, 2024
fde4286
Remove request timeout from `heap` feature
tiram88 Jan 5, 2024
00712ca
Enhance `Broadcaster` memory release and logs
tiram88 Jan 5, 2024
314f218
Track global `UtxosChangedSubscription` count
tiram88 Jan 5, 2024
6f61749
Fix heap profile
tiram88 Jan 5, 2024
1b9db37
Merge branch 'master' into rpc-memory-benchmark
tiram88 Jan 8, 2024
d0455b4
Let the number of broadcasters in gRPC server be configurable
tiram88 Jan 8, 2024
4c84f0d
Identify an active and filtering UtxosChangedSubscription with a List…
tiram88 Jan 9, 2024
7b34e70
Give address.rs its folder
tiram88 Jan 14, 2024
a24f395
Merge branch 'master' into rpc-memory-benchmark
tiram88 Jan 15, 2024
ea58a68
Address tracker, indexing and counting registrations
tiram88 Jan 16, 2024
b8e5dad
Merge branch 'rpc-memory-benchmark' into address-tracker-subscriptions
tiram88 Jan 16, 2024
3362171
Add a sharable subscription context to the notification system
tiram88 Jan 16, 2024
edbfd83
Merge branch 'master' into rpc-memory-benchmark
tiram88 Jan 16, 2024
59307d1
Merge branch 'rpc-memory-benchmark' into address-tracker-subscriptions
tiram88 Jan 16, 2024
57f22fe
Use a subscription context in `Notification` trait and in `single::Ut…
tiram88 Jan 19, 2024
d092158
Add an index counter and use short names
tiram88 Jan 19, 2024
d33808d
Use a subscription context in `compounded::UtxosChangedSubscription` …
tiram88 Jan 21, 2024
95711bc
Merge branch 'master' into rpc-memory-benchmark
tiram88 Jan 21, 2024
32a8104
Merge branch 'rpc-memory-benchmark' into address-tracker-subscriptions
tiram88 Jan 21, 2024
0a9dfc1
Merge branch 'master' into rpc-memory-benchmark
tiram88 Jan 23, 2024
5cd6d9c
Merge branch 'rpc-memory-benchmark' into address-tracker-subscriptions
tiram88 Jan 23, 2024
e817591
Rely on hash sets & maps instead of sorted vectors in single and comp…
tiram88 Jan 26, 2024
7e02425
fix lint
tiram88 Jan 26, 2024
4188e3b
Merge branch 'master' into rpc-memory-benchmark
tiram88 Jan 26, 2024
035241e
Merge branch 'rpc-memory-benchmark' into address-tracker-subscriptions
tiram88 Jan 26, 2024
47a766c
Add an optional maximum capacity to the address tracker
tiram88 Jan 27, 2024
7bbb705
Introduce a mutation outcome
tiram88 Jan 28, 2024
22acaa0
Remove unneeded CompoundedClone::clone_arc
tiram88 Jan 28, 2024
0c220aa
Provide inner mutability to `Indexes` and `Counters`
tiram88 Jan 30, 2024
e400e02
Restore the filtering of UtxosChanged notifications based on compound…
tiram88 Jan 30, 2024
b5a3d7e
Measure memory footprint of CounterMap
tiram88 Jan 31, 2024
c3ec099
Extend `UtxosChangedSubscription` inner mutability to its state
tiram88 Feb 2, 2024
94d2c1b
Group all wildcard `UtxosChangedSubscription` in broadcaster plan
tiram88 Feb 2, 2024
52b3047
Have kaspad use a single `SubscriptionContext`
tiram88 Feb 4, 2024
48d7e7a
Add event_type() to Scope
tiram88 Feb 4, 2024
f93fb1d
Reduce the number of mutation clones
tiram88 Feb 4, 2024
bb0e0d7
Log some memory stats in a file
tiram88 Feb 4, 2024
e898e62
Consume the address vector of UtxosChangedScope
tiram88 Feb 4, 2024
1390eb7
Retain the original address vector of a UtxosChanged mutation along t…
tiram88 Feb 5, 2024
f11c1dd
Enhance the termination of all gRPC server connections
tiram88 Feb 6, 2024
a053db4
Put `UtxosChangedSubscription` state and indexes under the same lock
tiram88 Feb 6, 2024
fced58e
Some Tracker lock fairness enhancements
tiram88 Feb 7, 2024
b9ca754
Preallocate static UtxosChanged subscriptions
tiram88 Feb 9, 2024
20c9369
Address new lint rules
tiram88 Feb 12, 2024
8f9a9b8
Move memory_monitor
tiram88 Feb 13, 2024
0e33998
Silent on simnet
tiram88 Feb 13, 2024
ca7d82e
Add a shutdown listener to `Core`
tiram88 Feb 13, 2024
e565047
Add a `Task` trait and implement some tasks
tiram88 Feb 13, 2024
0e4ef51
New daemon memory benchmark running in its own child process
tiram88 Feb 13, 2024
e3a2b0d
Refactor `ClientPool` with a start fn returning a vector of `JoinHandle`
tiram88 Feb 13, 2024
95d47c0
Add start and shutdown signaling to `ClientPool`
tiram88 Feb 15, 2024
f121ab8
Add full miner, tx sender, subscriber tasks and all their sub-tasks
tiram88 Feb 15, 2024
cd05339
Use the tasks in fn utxos_changed_subscriptions_client
tiram88 Feb 15, 2024
03521e4
Cleaning
tiram88 Feb 15, 2024
84d9491
Fix a rare case potentially preventing subscriber tasks to exit
tiram88 Feb 15, 2024
b9dd478
Fill the mempool up to the target
tiram88 Feb 15, 2024
0da4794
Run actual memory benchmarks
tiram88 Feb 15, 2024
dce1bce
Merge branch 'kaspanet:master' into address-tracker-subscriptions
tiram88 Feb 15, 2024
1ef567b
Add a main task to `TasksRunner`
tiram88 Feb 15, 2024
a97fd53
Move tasks
tiram88 Feb 15, 2024
502666f
Move tasks (2)
tiram88 Feb 15, 2024
630bff0
Rename full to group
tiram88 Feb 15, 2024
201a7ef
Rename full to group (2)
tiram88 Feb 15, 2024
385efab
Fix cargo syntax error
tiram88 Feb 15, 2024
b88855e
Merge branch 'master' into address-tracker-subscriptions
tiram88 Feb 15, 2024
d4e997f
Add a stopper status to some tasks
tiram88 Feb 15, 2024
2efa5ae
Let the main task run before adding sub tasks that need it alive
tiram88 Feb 15, 2024
ec242c8
Mempool benchmark based on tasks
tiram88 Feb 15, 2024
9eb468f
Small adjustments on the utxos changed subscribe benchmark
tiram88 Feb 15, 2024
e257fc3
Prevent a race condition
tiram88 Feb 26, 2024
d77f3e2
Refactor
tiram88 Feb 26, 2024
5a7cc69
Move the core shutdown request signaling into `RpcCoreService`
tiram88 Feb 26, 2024
3af4cc8
Add a signal indicating the gRPC server has started
tiram88 Feb 26, 2024
a041348
Recycle emptied tracker entries
tiram88 Feb 27, 2024
1cb277f
Add `max-tracked-addresses` argument
tiram88 Feb 27, 2024
5b1f140
Rename `UtxosChangedMutationPolicy` `AllOrNothing` to `Wildcard`
tiram88 Feb 28, 2024
7eb32ba
Merge branch 'master' into address-tracker-subscriptions
tiram88 Feb 28, 2024
f1d0136
Cleaning: remove R&D code
tiram88 Mar 2, 2024
cc80d22
Merge branch 'master' into address-tracker-subscriptions
tiram88 Mar 21, 2024
9b6694d
Merge branch 'master' into address-tracker-subscriptions
tiram88 Mar 26, 2024
d7eda15
Merge branch 'master' into rpc-memory-benchmark
tiram88 Apr 2, 2024
aa4f0fb
Merge branch 'rpc-memory-benchmark' into address-tracker-subscriptions
tiram88 Apr 2, 2024
8b286c2
Some comments and documentation
tiram88 Apr 4, 2024
2a3fd52
Use a preset listener id in direct mode
tiram88 Apr 5, 2024
1bdd04a
Add lower and upper bounds to the tracker max address count & change …
tiram88 Apr 6, 2024
37c8de6
For each event type the notifier can have at most one subscriber
tiram88 Apr 6, 2024
400ef0c
Add and document `GrpcClient::connect_with_args`
tiram88 Apr 6, 2024
3152d39
Some doc
tiram88 Apr 7, 2024
1a3d140
Complete `UtxosChangedMutationPolicy` description
tiram88 Apr 8, 2024
e1dd475
Validate --max-tracked-addresses argument
tiram88 Apr 8, 2024
aae65d8
Merge branch 'master' into address-tracker-subscriptions
michaelsutton Apr 9, 2024
91ebd83
remove unused AddressesHash
michaelsutton Apr 9, 2024
82a9fe9
fix minor warnings under `devnet-prealloc` feature code
michaelsutton Apr 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,13 @@ members = [
"utils",
"utils/tower",
"rothschild",
"metrics/perf_monitor",
"metrics/core",
"metrics/perf_monitor",
"utils/alloc",
]

[workspace.package]
rust-version = "1.77.0"
version = "0.13.5"
authors = ["Kaspa developers"]
license = "MIT/Apache-2.0"
Expand Down
23 changes: 14 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -251,25 +251,30 @@ cargo run --release --bin kaspad -- --testnet
<details>

<summary>
Use a configuration file
Using a configuration file
</summary>

```bash
cargo run --release --bin kaspad -- --configfile /path/to/configfile.toml
# or
cargo run --release --bin kaspad -- -C /path/to/configfile.toml
```
The config file should be a list of \<CLI argument\> = \<value\> separated by newlines.
- The config file should be a list of \<CLI argument\> = \<value\> separated by newlines.
- Whitespace around the `=` is fine, `arg=value` and `arg = value` are both parsed correctly.
- Values with special characters like `.` or `=` will require quoting the value i.e \<CLI argument\> = "\<value\>".
- Arguments with multiple values should be surrounded with brackets like `addpeer = ["10.0.0.1", "1.2.3.4"]`.

For example:
```
testnet=true
utxoindex=false
disable-upnp=true
perf-metrics=true
appdir="some-dir"
testnet-suffix=11
testnet = true
utxoindex = false
disable-upnp = true
perf-metrics = true
appdir = "some-dir"
netsuffix = 11
addpeer = ["10.0.0.1", "1.2.3.4"]
```
note that some arguments have no space between words and some use kebab-case, pass the --help flag to view all possible arguments
Pass the `--help` flag to view all possible arguments

```bash
cargo run --release --bin kaspad -- --help
Expand Down
1 change: 1 addition & 0 deletions cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
name = "kaspa-cli"
description = "Kaspa CLI"
keywords = ["kaspa", "wallet", "cli", "rpc"]
rust-version.workspace = true
version.workspace = true
edition.workspace = true
authors.workspace = true
Expand Down
1 change: 1 addition & 0 deletions components/addressmanager/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[package]
name = "kaspa-addressmanager"
description = "Kaspa address manager"
rust-version.workspace = true
version.workspace = true
edition.workspace = true
authors.workspace = true
Expand Down
1 change: 1 addition & 0 deletions components/connectionmanager/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[package]
name = "kaspa-connectionmanager"
description = "Kaspa connection manager"
rust-version.workspace = true
version.workspace = true
edition.workspace = true
authors.workspace = true
Expand Down
1 change: 1 addition & 0 deletions components/consensusmanager/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[package]
name = "kaspa-consensusmanager"
description = "Kaspa consensus library"
rust-version.workspace = true
version.workspace = true
edition.workspace = true
authors.workspace = true
Expand Down
2 changes: 2 additions & 0 deletions components/consensusmanager/src/session.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,11 @@ pub use tokio::task::spawn_blocking;

use crate::BlockProcessingBatch;

#[allow(dead_code)]
#[derive(Clone)]
pub struct SessionOwnedReadGuard(Arc<RfRwLockOwnedReadGuard>);

#[allow(dead_code)]
pub struct SessionReadGuard<'a>(RfRwLockReadGuard<'a>);

pub struct SessionWriteGuard<'a>(RfRwLockWriteGuard<'a>);
Expand Down
1 change: 1 addition & 0 deletions consensus/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[package]
name = "kaspa-consensus"
description = "Kaspa consensus library"
rust-version.workspace = true
version.workspace = true
edition.workspace = true
authors.workspace = true
Expand Down
1 change: 1 addition & 0 deletions consensus/core/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[package]
name = "kaspa-consensus-core"
description = "Kaspa consensus core"
rust-version.workspace = true
version.workspace = true
edition.workspace = true
authors.workspace = true
Expand Down
4 changes: 2 additions & 2 deletions consensus/core/src/api/stats.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
use borsh::{BorshDeserialize, BorshSchema, BorshSerialize};
use borsh::{BorshDeserialize, BorshSerialize};
use serde::{Deserialize, Serialize};

#[derive(Clone, Debug, Serialize, Deserialize, BorshSerialize, BorshDeserialize, BorshSchema, Default)]
#[derive(Clone, Debug, Serialize, Deserialize, BorshSerialize, BorshDeserialize, Default)]
#[serde(rename_all = "camelCase")]
pub struct BlockCount {
pub header_count: u64,
Expand Down
4 changes: 2 additions & 2 deletions consensus/core/src/header.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use crate::{hashing, BlueWorkType};
use borsh::{BorshDeserialize, BorshSchema, BorshSerialize};
use borsh::{BorshDeserialize, BorshSerialize};
use js_sys::{Array, Object};
use kaspa_hashes::Hash;
use kaspa_utils::hex::ToHex;
Expand All @@ -8,7 +8,7 @@ use serde_wasm_bindgen::*;
use wasm_bindgen::prelude::*;
use workflow_wasm::prelude::*;

#[derive(Clone, Debug, Serialize, Deserialize, BorshSerialize, BorshDeserialize, BorshSchema)]
#[derive(Clone, Debug, Serialize, Deserialize, BorshSerialize, BorshDeserialize)]
#[serde(rename_all = "camelCase")]
#[wasm_bindgen(inspectable)]
pub struct Header {
Expand Down
Loading