perf(pruner): delete history indices by changeset keys

[shekhirin]

Closes #9271

h/t @rkrasiuk for the idea

This PR improves the performance of history pruning.

Explanation

How history works

We have two tables for account history: AccountChangeSets and AccountsHistory. Same for storages.

• *ChangeSets stores a block number as a key, and the previous value of the account or storage slot before the specified block as a value.
• *History stores an inverse index to changesets – address or storage slot as a key, and block numbers where it was changed as a value. As an optimization, we don't store all block numbers in one row but instead shard them into multiple rows.

  reth/crates/storage/db/src/tables/mod.rs

  Lines 321 to 339 in c4589ec

  	/// Stores pointers to block changeset with changes for each account key.
  	///
  	/// Last shard key of the storage will contain `u64::MAX` `BlockNumber`,
  	/// this would allows us small optimization on db access when change is in plain state.
  	///
  	/// Imagine having shards as:
  	/// * `Address | 100`
  	/// * `Address | u64::MAX`
  	///
  	/// What we need to find is number that is one greater than N. Db `seek` function allows us to fetch
  	/// the shard that equal or more than asked. For example:
  	/// * For N=50 we would get first shard.
  	/// * for N=150 we would get second shard.
  	/// * If max block number is 200 and we ask for N=250 we would fetch last shard and
  	/// know that needed entry is in `AccountPlainState`.
  	/// * If there were no shard we would get `None` entry or entry of different storage key.
  	///
  	/// Code example can be found in `reth_provider::HistoricalStateProviderRef`
  	table AccountsHistory<Key = ShardedKey<Address>, Value = BlockNumberList>;

When a user wants to get a historical state of an account or storage slot at block N, we first query *History table to get the shard with the block number where the change at the block <= N (closest to N) has happened. If it's found, we do a lookup into *ChangeSets table.

reth/crates/storage/provider/src/providers/state/historical.rs

Lines 136 to 197 in c4589ec

	fn history_info<T, K>(
	&self,
	key: K,
	key_filter: impl Fn(&K) -> bool,
	lowest_available_block_number: Option<BlockNumber>,
	) -> ProviderResult<HistoryInfo>
	where
	T: Table<Key = K, Value = BlockNumberList>,
	{
	let mut cursor = self.tx.cursor_read::<T>()?;
	// Lookup the history chunk in the history index. If they key does not appear in the
	// index, the first chunk for the next key will be returned so we filter out chunks that
	// have a different key.
	if let Some(chunk) = cursor.seek(key)?.filter(|(key, _)| key_filter(key)).map(|x| x.1 .0) {
	// Get the rank of the first entry before or equal to our block.
	let mut rank = chunk.rank(self.block_number);
	// Adjust the rank, so that we have the rank of the first entry strictly before our
	// block (not equal to it).
	if rank.checked_sub(1).and_then(|rank| chunk.select(rank)) == Some(self.block_number) {
	rank -= 1
	};
	let block_number = chunk.select(rank);
	// If our block is before the first entry in the index chunk and this first entry
	// doesn't equal to our block, it might be before the first write ever. To check, we
	// look at the previous entry and check if the key is the same.
	// This check is worth it, the `cursor.prev()` check is rarely triggered (the if will
	// short-circuit) and when it passes we save a full seek into the changeset/plain state
	// table.
	if rank == 0 &&
	block_number != Some(self.block_number) &&
	!cursor.prev()?.is_some_and(|(key, _)| key_filter(&key))
	{
	if let (Some(_), Some(block_number)) = (lowest_available_block_number, block_number)
	{
	// The key may have been written, but due to pruning we may not have changesets
	// and history, so we need to make a changeset lookup.
	Ok(HistoryInfo::InChangeset(block_number))
	} else {
	// The key is written to, but only after our block.
	Ok(HistoryInfo::NotYetWritten)
	}
	} else if let Some(block_number) = block_number {
	// The chunk contains an entry for a write after our block, return it.
	Ok(HistoryInfo::InChangeset(block_number))
	} else {
	// The chunk does not contain an entry for a write after our block. This can only
	// happen if this is the last chunk and so we need to look in the plain state.
	Ok(HistoryInfo::InPlainState)
	}
	} else if lowest_available_block_number.is_some() {
	// The key may have been written, but due to pruning we may not have changesets and
	// history, so we need to make a plain state lookup.
	Ok(HistoryInfo::MaybeInPlainState)
	} else {
	// The key has not been written to at all.
	Ok(HistoryInfo::NotYetWritten)
	}
	}

How history pruning worked before

1. Given a range of blocks N..=M, delete all entries in *ChangeSets table within that range. Record the highest deleted block number.
2. Walk through the entries in *History table. Take those shards that contain block numbers less than or equal to the highest deleted block number from step 1. Decode them, filter block numbers so that only higher ones are kept, encode back and update the shard in the database. Jump to the next key if there are no more shards with relevant block numbers left for the current key.

How history pruning works now

1. Given a range of blocks N..=M, delete all entries in *ChangeSets table within that range. Record the highest deleted block number and keys (addresses or storage slots) that were deleted).
2. Sort keys from step 1.
3. Walk through the entries in the *History table, seeking to keys from step 1, and doing the same kind of "decode, filter, encode back, update" as before.

An enormous difference here comes from the fact that we're seeking to only those keys that need an update and don't walk through the whole *History table every time.

Performance

Before

After

[joshieDo]

Suggested change

[shekhirin]

thanks boss

[emhane]

if you don't leave the white space, then std and core libs are ordered alphabetically with rest of deps. it's more mature rust to put std and core deps up top, for developers who care ab std/no-std code - which includes us since we included the no-std option in several crates, like reth-evm

[shekhirin]

hmm I disagree, I also prefer to have all imports in one place ordered alphabetically.

for developers who care ab std/no-std code

#![no_std] exists for this reason, and developers shouldn't judge the crate no_std guarantees purely by deps it uses

[shekhirin]

we really seem to not have any consistency with it 😅

➜  reth git:(alexey/history-pruning-perf) ✗ rg --multiline --type rust 'use std.*;\nuse' | wc -l
     204
➜  reth git:(alexey/history-pruning-perf) ✗ rg --multiline --type rust 'use std.*;\n\nuse' | wc -l
     132

[emhane]

using miri as a reference https://github.com/rust-lang/miri/blob/521422adc7404ed5236cf2ad801ec8d1ea3b8818/cargo-miri/src/util.rs#L1-L13

[shekhirin]

it doesn't look like a rule that everyone follows even in core Rust team https://github.com/rust-lang/rust/blob/59a4f02f836f74c4cf08f47d76c9f6069a2f8276/compiler/rustc_ast/src/ast.rs#L21-L42

anyway, there are no std import in this file now, and I believe with #9141 (comment) we will have this as a fmt rule

[emhane]

partial review

[mattsse]

I don't enough context on this
defer to @rkrasiuk
changes and docs make sense though

[joshieDo]

how many usually? maybe par_sorted_unstable can help here as well

[shekhirin]

will leave a comment about it #9312 (comment)

[shekhirin]

think no need for parallel here, but can bench and compare later

[emhane]

why not max perf from start? is there much overhead in using the par sort?

[shekhirin]

it can actually make the perf worse, the unit of work should be big enough to get a benefit from parallelization. i believe here it's not that big, given the size of the array and the simple nature of sorting

[emhane]

why not max perf from start? is there much overhead in using the par sort?