Bounding node RAM -- implementing a cache policy which tracks dynamic…

… data sizes (kaspanet#351) * cache inner struct: prep for tracking size * initial cache policy and mem size estimator * use new cache policies * fix storage calcs * update TN11 genesis * temp: limit max block parents in virtual processor * use more conservative constants (wip) * ensure sufficient cache allocs for high level stores * increase caches related to daa window * Squashed commit of the following: commit a4ba784 Author: Ori Newman <[email protected]> Date: Tue Dec 26 16:22:36 2023 +0200 Add headers first validation to simpa (kaspanet#364) commit fb1aa5f Author: Ori Newman <[email protected]> Date: Tue Dec 26 16:07:19 2023 +0200 Move to mimalloc (kaspanet#361) * mimalloc * mimalloc disable purge_decommits * Add to simpa and integration tests * remove redundant unix and replace with win/linux/mac * Add comment * Add comment * Sort out features * Get rid of ctor --------- Co-authored-by: Michael Sutton <[email protected]> commit b6f7e09 Author: aspect <[email protected]> Date: Tue Dec 26 13:38:24 2023 +0200 Cumulative PR for Wallet & Metrics Refactoring (kaspanet#360) * external access to kaspad core rpc services API * rename wallet event members for serde serialization misc wallet functionality update * wip - event renaming * wip - event serialization tuneup * remove default wrpc listening ports on kaspad Args + fix port mapping bug for automatic JSON RPC port mapping. * Box wallet events * introduce RpcCtl + isolate wRPC from DynRpcApi * debug trait for secret * de-couple wRPC from the runtiem wallet making RPC bindings generic * convert wallet_list to return Vec<WalletDescriptor> * user-friendly error message on decryption error * UtxoProcessor - ignore task shutdown if not running * wallet storage data versioning * wallet - propagate wallet title * sort traits on WalletDescriptor * wallet - use precise decimal conversion when parsing strings to sompi * display kaspa on get-balance-by-address-rpc * wallet - rename settings fns to have try_ prefixes * rename library names "kaspad" and "kaspa_cli" to "kaspad_lib" and "kaspa_cli_lib" to prevent build collisions * no-unsafe-eval feature * allow for chrome-extension schema in the RPC url * wallet - expose bip32 wordlist for external access * wallet - delegate rpc ownership to utxo_processor; reorder rpc disconnect to occur before utxo_processor shutdown; * wallet - migrate wallet settings loading to cli * rename RpcCtlOp (Open, Close) to RpcState (Opened, Closed) * remove automatic RPC disconnect from wallet stop() to allow retention of the RPC channel after closing the wallet * allow UtxoProcessor to bind to an already existing/connected RPC channel, allow retention of a connected RPC channel past UtxoProcessor shutdown. (API centric defensive functionality) * code formatting * fix clippy warnings * migrate chrome-extension to use ws:// schema by default * wallet list formatting * staging for WRS 0.7.0 * add bip32 word list iterator * update WRS to 0.7.0 * fix rustdoc errors * wip - initial WalletApi trait implementation * wallet - macros and account descriptors * wallet - ping_call * cleanup * wip - wallet api cleanup & testing * kaspa-metrics as a separate crate * wip - metrics cleanup * cargo.lock update * async/sync api changes for storage * merge * master merge * wip * wip * wip * wallet - (wip) changes to AccountDescriptor and Wallet API * metrics - make sink future return as optional * wip - refactor wallet open; improve wallet event ordering; cleanup * wip (debugging UtxoChanged notifications) - fix issuance during failed tx * adding workflow-rs as github dependency (allows CI) * revert github deps for WRS * metrics * refactor metrics to include bdi counts and peer counts * wallet api updates refactoring transaction access interface misc updates to wallet APIs refactor wallet open sequence refactor legacy accounts * wallet framework - fix a but resulting in the early issuane of UTXO maturity events. Refactor handling of outgoing transaction UTXOs (sending outgoing tx ACKs on UtxoChange removal), minor fixes in the Transaction file-system storage. Add "outgoing" property to account balances (to track in-flight transactions being submitted until acceptance). Introduce new Events::Change that gets fired specifically when receiving change of the currently outgoing transaction. Initial steps toward Daa property wrapper. * npm 0.13.0 * metrics - cleanup after RPC refactoring * rename rust-side of wasm bindings as naming conventions interfere with rust use (make fns more explicitly named) * make metrics data formatting fns as pub * expose log levels in kaspa-core::log * rename metrics & improve metrics fetch task * add API calls to pre-configure wallet framework stroage folders * cumulative updates for Cargo.lock * Improve address error messages for human readability * PrvKeyData enumeration, initial account transfer support (wip), wallet descriptor handling * wallet - change error to retain PrvKeyDataId instead of a String * propagate bip39 flag to runtime account and it's descriptor * wip - keydata refactoring * metrics refactoring (bandwidth) * wip (cleanup) * misc improvements to metrics * code formatting * wRPC bandwidth monitoring + scaffolding for global bandwidth metrics * bump WRS to 0.9.0 * metrics - add mempool_size and convert some items to u32/f32 * metrics cleanup * wallet and account rename, remove "title" from accounts * add missing mempool size metric * UTXO "stasis" mode for new coinbase transactions * remove useless Option from the rpc url parser * fix daa score in maturity state check * add name_or_id to PrvKeyDataInfo * wallet RPC Connect/Disconnect and GetStatus API calls * cleanup * support for 12-word bip39 mnemonics * add a hidden --daemon flag to kaspad * refactor get_daa_score_timestamp helper to return Vec<u64> instead of Response struct * Add Scan event * implement support for unixtime in transactions * kaspa value formatting functions * fix a deadlock due to re-entrant mutex lock when loading multiple accounts from storage * introduce utxo stasis mode + make balance carry utxo counts as well as the totals * refactor wallet events, replace reset() with close() where appropriate * manual debug impl for tx components (display script as hex, address as address etc) * fix incorrect gas assignment in WASM bindings * updates for wallet event handling * refactor sign_with_multiple to return Err(tx) when partially signed * refactor Storage and Wallet descriptors * fix account derivation index handling in DerivationCapable * track acceptance DAA stamp in outgoing transaction record * temporary harness for embedding script engine directly into tx generator (used for debugging) * convert PendingTransaction Utxo entries into HashSet * rename pending tx fns * outgoing tx wrapper around pending tx * refactor notification events during wallet ops * as_ref() for PrvKeyData * wip - initial RpcCoreMocks for wallet framework tests * wallet api messages update for StorageDescriptor usage * update AddressDerivationMeta to track addresses from index 0 * add AHash and async_channel to wallet_core crate * allow Args to be used externally for daemon argument pre-validation * impl Hash for UtxoEntryReference and rename "fake()" to "simulated()" * update wallet events * expose tx validator check_scripts() as pub * update wallet wasm fns for wallet and storage descriptors * rename kaspa-metrics to kaspa-metrics-core * code formatting * code formatting * migrate transaction console formatting to cli/extensions * refactor wallet events * use ahash for faster processing in wallet hash maps (utxo context) * wallet - add SendError handling for error::Error * wallet event refactoring, scan/discovery process refactoring, wallet change secret api handling * refactoring: transfer and deep derivation scan * transaction record refactoring * wallet - support for cross-account transfers in generator * wallet - outgoing transaction impl * wallet - refactor maturity handling * wallet/utxo - initial tests scaffolding * minor optimization of the discovery/scan process * wallet - utxo processor outgoing tx and discovery timestamp unixtime handling * refactor utxo context to use output transactions, refactor transfers, compound and change notifications * code formatting + Cargo.lock update * Signed enum to represent Fully or Partially signed tx * bind p2p and grpc bandwidth metrics * misc clippy fixes * cleanup * remove --daemon as Args::parse() offers external mutations * remove previously implemented mnemonic variant in favor of client-side mnemonic creation * remove AccessContextT, convert to wallet_secret, wip WalletApi traits, refactor account args * code formatting * separate runtime wallet into multiple modules * wallet - revert create_accounts to process a single account at a time * cleanup * wallet - JSON import and export + preliminary docs * address - AddressList - fix TryFrom deadlock caused by recursive trait invocation * wallet - api rustdoc + cleanup, transaction calls * wallet docs * change address debug formatting * adding cli rpc addpeer * wip - wallet restructuring * wip - restructure wallet modules * wip - wallet restructuring - cleanup * wip - AccountKind refactoring * wallet - refactoring storage serialization & encryption * fold in PR kaspanet#349 * wallet framework - misc helper fns and cleanup * storage unit tests + prv key data module restructuring * wallet - account factory registrations * Remove panics caused by sudden loss of connection while submitting transactions. * cleanup * cleanup * cleanup + docs * docs * adjust metrics formatting * cleanup * bump workflow-rs to 0.10.1 * fix wallet storage unit tests * fix generator unit tests * convert missing listener_id panic to error * docs * code formatting * lints * misc updates following github reviews * refactor AccountKind to use fixedstr and implement Copy * refactor AccountKind to use fixedstr (str64) and re-implement Copy, removing clone() where necessary * change js-bound fn names from *as* to *to* to follow Rust conventions * replace AssocPrvKeyDataIds iter().any() to contains() * remove vec![] in MetricsGroup iterator fn; add TODO comment on Metric::group() * scope magic numbers within their associated structs * add try_fully_signed() and try_partially_signed() fns to Signed that allow recovery of the tx * refactor local storage cache to use RwLock instead of a Mutex * refactor serialization as per github feedback (partial trait impl) * Optimize AssocPrvKeyDataIds to use Either in the IntoIter impl, removing allocations. * Refactor to bypass aux buffer creation when storing wallet on native platforms. * impl custom Debug for storage structs to display data in hex * adding misc SAFETY comments on unsafe ops * TODO comments for later SignerT+multisig refactoring * Change panic!() to unreachable!() to make it clearer that a state should not be reached. * rename TransactionRecord::unixtime to unixtime_msec * fix wrong references in comments * Fix min_cosigner_index handling in multisig fns * add comments to UtxoContext::insert() error handling * bind AccountStorable:Storable for AccountStorage, migrate Storable into a separate module, add comments to MAGIC values in some locations * make framework init functions unsafe * update workflow-rs to 0.10.2 * fix comments * fix data cloning when getting collection counts * adding TODO notes to wasm macros and KAS value decimal parsing --------- Co-authored-by: Surinder Singh Matoo <[email protected]> commit 10d0ebf Author: D-Stacks <[email protected]> Date: Tue Dec 26 10:43:59 2023 +0100 make step 3. of linux installation not fail copy and paste (kaspanet#363) remove the space to escape the newline instead. * reduce children cache policy * reduce caches 25% * increase tx cache * fix gd alloc and slightly increase other numbers * fix a cache alloc bug by redesigning the cache and adding a min units option + fix bps constants * fix simpa short cmd op name colision * move impls to inner struct * comments, docs and technical refactoring * refactor MemSizeEstimator * refactor bool to MemMode * memsize docs * remove unneeded min limits * comments and name enhancements * rename `CachePolicy::Units` to `CachePolicy::Count` to avoid confusion with `MemMode` * use estimated num of unique headers in validate pruning point headers cache * cache policy builder * add todos * median window has different size + comments * read ghostdag data from compact access if possible * remove wrong upper bound for reachability sets
smartgoo · Dec 28, 2023 · 1064c54 · 1064c54
1 parent 74779d4
commit 1064c54
Show file tree

Hide file tree

Showing 51 changed files with 887 additions and 297 deletions.
diff --git a/Cargo.lock b/Cargo.lock
diff --git a/components/addressmanager/src/lib.rs b/components/addressmanager/src/lib.rs
@@ -15,7 +15,7 @@ use itertools::{
 };
 use kaspa_consensus_core::config::Config;
 use kaspa_core::{debug, info, task::tick::TickService, time::unix_now, warn};
-use kaspa_database::prelude::{StoreResultExtensions, DB};
+use kaspa_database::prelude::{CachePolicy, StoreResultExtensions, DB};
 use kaspa_utils::networking::IpAddress;
 use local_ip_address::list_afinet_netifas;
 use parking_lot::Mutex;
@@ -61,7 +61,7 @@ pub struct AddressManager {
 impl AddressManager {
     pub fn new(config: Arc<Config>, db: Arc<DB>, tick_service: Arc<TickService>) -> (Arc<Mutex<Self>>, Option<Extender>) {
         let mut instance = Self {
-            banned_address_store: DbBannedAddressesStore::new(db.clone(), MAX_ADDRESSES as u64),
+            banned_address_store: DbBannedAddressesStore::new(db.clone(), CachePolicy::Count(MAX_ADDRESSES)),
             address_store: address_store_with_cache::new(db),
             local_net_addresses: Vec::new(),
             config,
@@ -337,7 +337,7 @@ mod address_store_with_cache {
     };
 
     use itertools::Itertools;
-    use kaspa_database::prelude::DB;
+    use kaspa_database::prelude::{CachePolicy, DB};
     use kaspa_utils::networking::PrefixBucket;
     use rand::{
         distributions::{WeightedError, WeightedIndex},
@@ -359,7 +359,8 @@ mod address_store_with_cache {
 
     impl Store {
         fn new(db: Arc<DB>) -> Self {
-            let db_store = DbAddressesStore::new(db, 0);
+            // We manage the cache ourselves on this level, so we disable the inner builtin cache
+            let db_store = DbAddressesStore::new(db, CachePolicy::Empty);
             let mut addresses = HashMap::new();
             for (key, entry) in db_store.iterator().map(|res| res.unwrap()) {
                 addresses.insert(key, entry);

diff --git a/components/addressmanager/src/stores/address_store.rs b/components/addressmanager/src/stores/address_store.rs
@@ -1,9 +1,10 @@
 use kaspa_database::{
     prelude::DB,
+    prelude::{CachePolicy, StoreError, StoreResult},
     prelude::{CachedDbAccess, DirectDbWriter},
-    prelude::{StoreError, StoreResult},
     registry::DatabaseStorePrefixes,
 };
+use kaspa_utils::mem_size::MemSizeEstimator;
 use serde::{Deserialize, Serialize};
 use std::net::Ipv6Addr;
 use std::{error::Error, fmt::Display, sync::Arc};
@@ -17,6 +18,12 @@ pub struct Entry {
     pub address: NetAddress,
 }
 
+impl MemSizeEstimator for Entry {
+    fn estimate_mem_units(&self) -> usize {
+        1
+    }
+}
+
 pub trait AddressesStoreReader {
     fn get(&self, key: AddressKey) -> Result<Entry, StoreError>;
 }
@@ -74,8 +81,8 @@ pub struct DbAddressesStore {
 }
 
 impl DbAddressesStore {
-    pub fn new(db: Arc<DB>, cache_size: u64) -> Self {
-        Self { db: Arc::clone(&db), access: CachedDbAccess::new(db, cache_size, DatabaseStorePrefixes::Addresses.into()) }
+    pub fn new(db: Arc<DB>, cache_policy: CachePolicy) -> Self {
+        Self { db: Arc::clone(&db), access: CachedDbAccess::new(db, cache_policy, DatabaseStorePrefixes::Addresses.into()) }
     }
 
     pub fn iterator(&self) -> impl Iterator<Item = Result<(AddressKey, Entry), Box<dyn Error>>> + '_ {

diff --git a/components/addressmanager/src/stores/banned_address_store.rs b/components/addressmanager/src/stores/banned_address_store.rs
@@ -1,15 +1,22 @@
 use kaspa_database::{
+    prelude::{CachePolicy, StoreError, StoreResult},
     prelude::{CachedDbAccess, DirectDbWriter, DB},
-    prelude::{StoreError, StoreResult},
     registry::DatabaseStorePrefixes,
 };
+use kaspa_utils::mem_size::MemSizeEstimator;
 use serde::{Deserialize, Serialize};
 use std::net::{IpAddr, Ipv6Addr};
 use std::{error::Error, fmt::Display, sync::Arc};
 
 #[derive(Clone, Copy, Serialize, Deserialize)]
 pub struct ConnectionBanTimestamp(pub u64);
 
+impl MemSizeEstimator for ConnectionBanTimestamp {
+    fn estimate_mem_units(&self) -> usize {
+        1
+    }
+}
+
 pub trait BannedAddressesStoreReader {
     fn get(&self, address: IpAddr) -> Result<ConnectionBanTimestamp, StoreError>;
 }
@@ -70,8 +77,8 @@ pub struct DbBannedAddressesStore {
 }
 
 impl DbBannedAddressesStore {
-    pub fn new(db: Arc<DB>, cache_size: u64) -> Self {
-        Self { db: Arc::clone(&db), access: CachedDbAccess::new(db, cache_size, DatabaseStorePrefixes::BannedAddresses.into()) }
+    pub fn new(db: Arc<DB>, cache_policy: CachePolicy) -> Self {
+        Self { db: Arc::clone(&db), access: CachedDbAccess::new(db, cache_policy, DatabaseStorePrefixes::BannedAddresses.into()) }
     }
 
     pub fn iterator(&self) -> impl Iterator<Item = Result<(IpAddr, ConnectionBanTimestamp), Box<dyn Error>>> + '_ {

diff --git a/consensus/core/src/blockstatus.rs b/consensus/core/src/blockstatus.rs
@@ -1,3 +1,4 @@
+use kaspa_utils::mem_size::MemSizeEstimator;
 use serde::{Deserialize, Serialize};
 
 #[derive(Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Debug)]
@@ -20,6 +21,12 @@ pub enum BlockStatus {
     StatusHeaderOnly,
 }
 
+impl MemSizeEstimator for BlockStatus {
+    fn estimate_mem_units(&self) -> usize {
+        1
+    }
+}
+
 impl BlockStatus {
     pub fn has_block_header(self) -> bool {
         matches!(

diff --git a/consensus/core/src/config/bps.rs b/consensus/core/src/config/bps.rs
@@ -58,8 +58,6 @@ impl<const BPS: u64> Bps<BPS> {
         let val = (Self::ghostdag_k() / 2) as u8;
         if val < 10 {
             10
-        // TODO (TEMP): uncomment when restarting TN11 or when implementing a TN11 HF
-        /*
         } else if val > 16 {
             // We currently limit the number of parents by 16 in order to preserve processing performance
             // and to prevent number of parent references per network round from growing quadratically with
@@ -69,21 +67,19 @@ impl<const BPS: u64> Bps<BPS> {
             // be merged after a log number of rounds. For mainnet this requires an increase to the value of GHOSTDAG
             // K accompanied by a short security analysis, or moving to the parameterless DAGKNIGHT.
             16
-         */
         } else {
             val
         }
     }
 
     pub const fn mergeset_size_limit() -> u64 {
-        Self::ghostdag_k() as u64 * 10
-    }
-
-    // TODO (TEMP): rename to mergeset_size_limit when restarting TN11 or when implementing a TN11 HF
-    pub const fn _mergeset_size_limit() -> u64 {
         let val = Self::ghostdag_k() as u64 * 2;
         if val < 180 {
             180
+        } else if val > 512 {
+            // Bounded relatively low because we have storage complexity of O(#headers * mergeset_limit) coming
+            // from reachability and GHOSTDAG stores (GHOSTDAG is avoidable but reachability is must).
+            512
         } else {
             val
         }
@@ -97,15 +93,24 @@ impl<const BPS: u64> Bps<BPS> {
         BPS * NEW_FINALITY_DURATION
     }
 
+    /// Limit used to previously calculate the pruning depth.
+    const fn prev_mergeset_size_limit() -> u64 {
+        Self::ghostdag_k() as u64 * 10
+    }
+
     pub const fn pruning_depth() -> u64 {
         // Based on the analysis at https://github.com/kaspanet/docs/blob/main/Reference/prunality/Prunality.pdf
         // and on the decomposition of merge depth (rule R-I therein) from finality depth (φ)
         // We add an additional merge depth unit as a safety margin for anticone finalization
         Self::finality_depth()
             + Self::merge_depth_bound() * 2
-            + 4 * Self::mergeset_size_limit() * Self::ghostdag_k() as u64
+            + 4 * Self::prev_mergeset_size_limit() * Self::ghostdag_k() as u64
             + 2 * Self::ghostdag_k() as u64
             + 2
+
+        // TODO (HF or restart of TN11):
+        // Return `Self::finality_depth() * 3` and assert that this value is equal or larger than the above expression.
+        // This will give us a round easy number to track which is not sensitive to minor changes in other related params.
     }
 
     pub const fn pruning_proof_m() -> u64 {

diff --git a/consensus/core/src/config/constants.rs b/consensus/core/src/config/constants.rs
@@ -110,11 +110,7 @@ pub mod perf {
     //! The constants in this module should all be revisited if mainnet consensus parameters change.
     //!
 
-    use crate::{config::params::Params, header::Header, BlueWorkType};
-    use kaspa_hashes::Hash;
-    use std::mem::size_of;
-
-    use super::consensus::NETWORK_DELAY_BOUND;
+    use crate::config::params::Params;
 
     /// The default target depth for reachability reindexes.
     pub const DEFAULT_REINDEX_DEPTH: u64 = 100;
@@ -123,32 +119,29 @@ pub mod perf {
     /// algorithm to encounter for blocks out of the selected chain.
     pub const DEFAULT_REINDEX_SLACK: u64 = 1 << 12;
 
-    const BASELINE_HEADER_DATA_CACHE_SIZE: u64 = 10_000;
-    const BASELINE_BLOCK_DATA_CACHE_SIZE: u64 = 200;
-    const BASELINE_BLOCK_WINDOW_CACHE_SIZE: u64 = 2000;
-    const BASELINE_UTXOSET_CACHE_SIZE: u64 = 10_000;
+    const BASELINE_HEADER_DATA_CACHE_SIZE: usize = 10_000;
+    const BASELINE_BLOCK_DATA_CACHE_SIZE: usize = 200;
+    const BASELINE_BLOCK_WINDOW_CACHE_SIZE: usize = 2000;
+    const BASELINE_UTXOSET_CACHE_SIZE: usize = 10_000;
 
     #[derive(Clone, Debug)]
     pub struct PerfParams {
         //
         // Cache sizes
         //
-        /// Preferred cache size for headers store
-        pub headers_cache_size: u64,
-
         /// Preferred cache size for header-related data
-        pub header_data_cache_size: u64,
+        pub header_data_cache_size: usize,
 
         /// Preferred cache size for block-body-related data which
         /// is typically orders-of magnitude larger than header data
         /// (Note this cannot be set to high due to severe memory consumption)
-        pub block_data_cache_size: u64,
+        pub block_data_cache_size: usize,
 
         /// Preferred cache size for UTXO-related data
-        pub utxo_set_cache_size: u64,
+        pub utxo_set_cache_size: usize,
 
         /// Preferred cache size for block-window-related data
-        pub block_window_cache_size: u64,
+        pub block_window_cache_size: usize,
 
         //
         // Thread-pools
@@ -163,7 +156,6 @@ pub mod perf {
     }
 
     pub const PERF_PARAMS: PerfParams = PerfParams {
-        headers_cache_size: BASELINE_HEADER_DATA_CACHE_SIZE,
         header_data_cache_size: BASELINE_HEADER_DATA_CACHE_SIZE,
         block_data_cache_size: BASELINE_BLOCK_DATA_CACHE_SIZE,
         utxo_set_cache_size: BASELINE_UTXOSET_CACHE_SIZE,
@@ -174,41 +166,10 @@ pub mod perf {
 
     impl PerfParams {
         pub fn adjust_to_consensus_params(&mut self, consensus_params: &Params) {
-            self.block_data_cache_size *= consensus_params.bps().clamp(1, 10); // Allow caching up to 10x over the baseline
-            self.block_window_cache_size = calculate_difficulty_window_cache_size(consensus_params);
-            self.headers_cache_size = calculate_headers_cache_size(consensus_params);
+            // Allow caching up to 10x over the baseline
+            self.block_data_cache_size *= consensus_params.bps().clamp(1, 10) as usize;
         }
     }
-
-    /// Bounds the cache size according to the "memory budget" (represented in bytes) and the approximate size of each unit in bytes
-    pub fn bounded_cache_size(desired_size: u64, memory_budget_bytes: u64, approx_unit_bytes: usize) -> u64 {
-        let max_cache_size = memory_budget_bytes / approx_unit_bytes as u64;
-        u64::min(desired_size, max_cache_size)
-    }
-
-    pub fn calculate_difficulty_window_cache_size(consensus_params: &Params) -> u64 {
-        let window_memory_budget = 250_000_000u64; // 250MB
-        let single_window_byte_size = consensus_params.difficulty_window_size(0) * (size_of::<Hash>() + size_of::<BlueWorkType>());
-        bounded_cache_size(BASELINE_BLOCK_WINDOW_CACHE_SIZE, window_memory_budget, single_window_byte_size)
-    }
-
-    pub fn calculate_headers_cache_size(consensus_params: &Params) -> u64 {
-        let headers_memory_budget = 500_000_000u64; // 500MB
-        let approx_header_byte_size = approx_header_parents(consensus_params) * size_of::<Hash>() + size_of::<Header>();
-        bounded_cache_size(consensus_params.bps() * BASELINE_HEADER_DATA_CACHE_SIZE, headers_memory_budget, approx_header_byte_size)
-    }
-
-    pub fn approx_direct_header_parents(consensus_params: &Params) -> usize {
-        consensus_params.bps() as usize * NETWORK_DELAY_BOUND as usize
-    }
-
-    pub fn approx_header_parents(consensus_params: &Params) -> usize {
-        approx_direct_header_parents(consensus_params) * 4 // 4x for multi-levels
-    }
-
-    pub fn approx_mergeset_size(consensus_params: &Params) -> usize {
-        consensus_params.bps() as usize * NETWORK_DELAY_BOUND as usize
-    }
 }
 
 #[cfg(test)]

diff --git a/consensus/core/src/tx.rs b/consensus/core/src/tx.rs
@@ -2,6 +2,7 @@ mod script_public_key;
 
 use borsh::{BorshDeserialize, BorshSchema, BorshSerialize};
 use kaspa_utils::hex::ToHex;
+use kaspa_utils::mem_size::MemSizeEstimator;
 use kaspa_utils::{serde_bytes, serde_bytes_fixed_ref};
 pub use script_public_key::{scriptvec, ScriptPublicKey, ScriptPublicKeyVersion, ScriptPublicKeys, ScriptVec, SCRIPT_VECTOR_SIZE};
 use serde::{Deserialize, Serialize};
@@ -44,6 +45,12 @@ impl UtxoEntry {
     }
 }
 
+impl MemSizeEstimator for UtxoEntry {
+    fn estimate_mem_units(&self) -> usize {
+        1
+    }
+}
+
 pub type TransactionIndexType = u32;
 
 /// Represents a Kaspa transaction outpoint

diff --git a/consensus/core/src/utxo/utxo_diff.rs b/consensus/core/src/utxo/utxo_diff.rs
@@ -3,8 +3,9 @@ use super::{
     utxo_error::{UtxoAlgebraError, UtxoResult},
 };
 use crate::tx::{TransactionOutpoint, UtxoEntry, VerifiableTransaction};
+use kaspa_utils::mem_size::MemSizeEstimator;
 use serde::{Deserialize, Serialize};
-use std::collections::hash_map::Entry::Vacant;
+use std::{collections::hash_map::Entry::Vacant, mem::size_of};
 
 pub trait ImmutableUtxoDiff {
     fn added(&self) -> &UtxoCollection;
@@ -17,6 +18,12 @@ pub struct UtxoDiff {
     pub remove: UtxoCollection,
 }
 
+impl MemSizeEstimator for UtxoDiff {
+    fn estimate_mem_bytes(&self) -> usize {
+        size_of::<Self>() + (self.add.len() + self.remove.len()) * (size_of::<TransactionOutpoint>() + size_of::<UtxoEntry>())
+    }
+}
+
 impl<T: ImmutableUtxoDiff> ImmutableUtxoDiff for &T {
     fn added(&self) -> &UtxoCollection {
         (*self).added()