Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/state db layout by path #40

Open
wants to merge 30 commits into
base: flatdb/base
Choose a base branch
from
Open

Feature/state db layout by path #40

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
9a7af7d
Adding state by path storage draft work in progress
damian-orzechowski Jan 11, 2023
67198d7
Further changes and tests
damian-orzechowski Jan 13, 2023
0bf9ff9
Further changes / fixes and sync work
damian-orzechowski Jan 17, 2023
78b0d47
Refactoring, state reader and state provider changes, snap sync & hea…
damian-orzechowski Feb 2, 2023
681fb48
Fixes to fast sync / healing, further changes and refactoring.
damian-orzechowski Feb 3, 2023
b7ac0f9
Don't do oddity encoding for full paths.
damian-orzechowski Feb 3, 2023
97b73bc
Updated tests for by path state.
damian-orzechowski Feb 7, 2023
ec8cedf
StateReader requires 2nd trie store - storage trie unchanged (hash ba…
damian-orzechowski Feb 9, 2023
1619d50
Added benchmarks for path based trie store.
damian-orzechowski Feb 9, 2023
226f4b1
Do not use leaf cache during snap sync.
damian-orzechowski Feb 9, 2023
360b99f
Set by-path trie store when initalizing blockchain.
damian-orzechowski Feb 9, 2023
0e3025a
Change by path storage history approach.
damian-orzechowski Feb 23, 2023
daf28e3
Added limit for history of changes.
damian-orzechowski Feb 24, 2023
bada91f
Merged master
damian-orzechowski Feb 28, 2023
03bfb30
Small fixes after merging master.
damian-orzechowski Mar 3, 2023
1359e4a
Fixed checking for existing node - fast sync / healing.
damian-orzechowski Mar 6, 2023
98d1d1b
State root save fix.
damian-orzechowski Mar 8, 2023
1e331de
Pass correct store to storage provider.
damian-orzechowski Mar 8, 2023
76db8b6
Fixes for finding best full state (initial sync).
damian-orzechowski Mar 13, 2023
5fc783b
Merged master
damian-orzechowski Mar 15, 2023
b82b726
Fixes post master merge. Fixes for ConnectNodes - extending node scen…
damian-orzechowski Mar 20, 2023
f102ea2
Further changes to API to support separate trie store for storage - t…
damian-orzechowski Mar 28, 2023
dd3fa74
Added new node cache for path based trie store. Fixes for sync and ro…
damian-orzechowski Mar 28, 2023
4cc62da
Small fixes for cache.
damian-orzechowski Mar 29, 2023
c26363b
Reworked Get Account and node cache.
damian-orzechowski Apr 5, 2023
9ab4ffe
refac
tanishqjasoria Apr 3, 2023
e759156
Further rework for cache/persist of path based state.
damian-orzechowski Apr 6, 2023
0751852
Resolved conflicts
damian-orzechowski Apr 6, 2023
c90f5d1
Refactoring of persistance.
damian-orzechowski Apr 8, 2023
6d4823e
update tests
tanishqjasoria Apr 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions src/Nethermind/Benchmarks.sln
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Nethermind.Network.Discover
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Nethermind.Network.Enr", "Nethermind.Network.Enr\Nethermind.Network.Enr.csproj", "{333B909F-DD34-42BE-A924-9C17D2D5CA31}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Nethermind.Db.Rocks", "Nethermind.Db.Rocks\Nethermind.Db.Rocks.csproj", "{DF2D18AF-CFD8-41D4-88F4-22296C7999EC}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Expand Down Expand Up @@ -275,6 +277,10 @@ Global
{333B909F-DD34-42BE-A924-9C17D2D5CA31}.Debug|Any CPU.Build.0 = Debug|Any CPU
{333B909F-DD34-42BE-A924-9C17D2D5CA31}.Release|Any CPU.ActiveCfg = Release|Any CPU
{333B909F-DD34-42BE-A924-9C17D2D5CA31}.Release|Any CPU.Build.0 = Release|Any CPU
{DF2D18AF-CFD8-41D4-88F4-22296C7999EC}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{DF2D18AF-CFD8-41D4-88F4-22296C7999EC}.Debug|Any CPU.Build.0 = Debug|Any CPU
{DF2D18AF-CFD8-41D4-88F4-22296C7999EC}.Release|Any CPU.ActiveCfg = Release|Any CPU
{DF2D18AF-CFD8-41D4-88F4-22296C7999EC}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down Expand Up @@ -317,6 +323,7 @@ Global
{C03B1A82-43BE-49F0-967B-4FB97A60C225} = {57CE427C-455B-47A9-85D6-CEF379DC00F6}
{808DF0CF-6C13-45B5-805F-476C4909A0D4} = {57CE427C-455B-47A9-85D6-CEF379DC00F6}
{333B909F-DD34-42BE-A924-9C17D2D5CA31} = {57CE427C-455B-47A9-85D6-CEF379DC00F6}
{DF2D18AF-CFD8-41D4-88F4-22296C7999EC} = {57CE427C-455B-47A9-85D6-CEF379DC00F6}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {14335C6A-B827-4069-8D11-9DC9398E51F3}
Expand Down
1 change: 1 addition & 0 deletions src/Nethermind/Directory.Packages.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@
<PackageVersion Include="System.Runtime.Caching" Version="7.0.0" />
<PackageVersion Include="System.Security.Cryptography.ProtectedData" Version="7.0.1" />
<PackageVersion Include="System.Text.Encodings.Web" Version="7.0.0" />
<PackageVersion Include="Umbraco.CSharpTest.Net.Collections" Version="15.0.0" />
<PackageVersion Include="Websocket.Client" Version="4.5.2" />
<PackageVersion Include="YamlDotNet" Version="13.0.1" />
</ItemGroup>
Expand Down
3 changes: 2 additions & 1 deletion src/Nethermind/Ethereum.Test.Base/BlockchainTestBase.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,12 @@ protected async Task<EthereumTestResult> RunTest(BlockchainTest test, Stopwatch?
IEthereumEcdsa ecdsa = new EthereumEcdsa(specProvider.ChainId, _logManager);

TrieStore trieStore = new(stateDb, _logManager);
TrieStore storageTrieStore = new(stateDb, _logManager);
IStateProvider stateProvider = new StateProvider(trieStore, codeDb, _logManager);
MemDb blockInfoDb = new MemDb();
IBlockTree blockTree = new BlockTree(new MemDb(), new MemDb(), blockInfoDb, new ChainLevelInfoRepository(blockInfoDb), specProvider, NullBloomStorage.Instance, _logManager);
ITransactionComparerProvider transactionComparerProvider = new TransactionComparerProvider(specProvider, blockTree);
IStateReader stateReader = new StateReader(trieStore, codeDb, _logManager);
IStateReader stateReader = new StateReader(trieStore, storageTrieStore, codeDb, _logManager);
IChainHeadInfoProvider chainHeadInfoProvider = new ChainHeadInfoProvider(specProvider, blockTree, stateReader);
ITxPool transactionPool = new TxPool(ecdsa, chainHeadInfoProvider, new TxPoolConfig(), new TxValidator(specProvider.ChainId), _logManager, transactionComparerProvider.GetDefaultComparer());

Expand Down
3 changes: 2 additions & 1 deletion ...tAbstraction.Test/AccountAbstractionRpcModuleTests.TestAccountAbstractionRpcBlockchain.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ protected override IBlockProducer CreateTestBlockProducer(TxPoolTxSource txPoolT
DbProvider,
BlockTree,
ReadOnlyTrieStore,
ReadOnlyStorageTrieStore,
SpecProvider,
BlockValidator,
NoBlockRewards.Instance,
Expand Down Expand Up @@ -214,7 +215,7 @@ protected override BlockProcessor CreateBlockProcessor()
UserOperationSimulator[entryPoint] = new(
UserOperationTxBuilder[entryPoint],
ReadOnlyState,
new ReadOnlyTxProcessingEnvFactory(DbProvider, ReadOnlyTrieStore, BlockTree, SpecProvider, LogManager),
new ReadOnlyTxProcessingEnvFactory(DbProvider, ReadOnlyTrieStore, ReadOnlyStorageTrieStore, BlockTree, SpecProvider, LogManager),
EntryPointContractAbi,
entryPoint!,
WhitelistedPayamsters,
Expand Down
1 change: 1 addition & 0 deletions src/Nethermind/Nethermind.AccountAbstraction/AccountAbstractionPlugin.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ private UserOperationSimulator UserOperationSimulator(Address entryPoint)
ReadOnlyTxProcessingEnvFactory readOnlyTxProcessingEnvFactory = new(
getFromApi.DbProvider,
getFromApi.ReadOnlyTrieStore,
getFromApi.ReadOnlyStorageTrieStore,
getFromApi.BlockTree,
getFromApi.SpecProvider,
getFromApi.LogManager);
Expand Down
1 change: 1 addition & 0 deletions src/Nethermind/Nethermind.Api/IApiWithBlockchain.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ public interface IApiWithBlockchain : IApiWithStores, IBlockchainBridgeFactory
IHeaderValidator? HeaderValidator { get; set; }
IManualBlockProductionTrigger ManualBlockProductionTrigger { get; }
IReadOnlyTrieStore? ReadOnlyTrieStore { get; set; }
IReadOnlyTrieStore? ReadOnlyStorageTrieStore { get; set; }
IRewardCalculatorSource? RewardCalculatorSource { get; set; }
/// <summary>
/// PoS switcher for The Merge
Expand Down
2 changes: 2 additions & 0 deletions src/Nethermind/Nethermind.Api/NethermindApi.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,7 @@ public IBlockchainBridge CreateBlockchainBridge()
ReadOnlyTxProcessingEnv readOnlyTxProcessingEnv = new(
_readOnlyDbProvider,
ReadOnlyTrieStore,
ReadOnlyStorageTrieStore,
readOnlyTree,
SpecProvider,
LogManager);
Expand Down Expand Up @@ -196,6 +197,7 @@ public ISealEngine SealEngine
public ITransactionProcessor? TransactionProcessor { get; set; }
public ITrieStore? TrieStore { get; set; }
public IReadOnlyTrieStore? ReadOnlyTrieStore { get; set; }
public IReadOnlyTrieStore? ReadOnlyStorageTrieStore { get; set; }
public ITxSender? TxSender { get; set; }
public INonceManager? NonceManager { get; set; }
public ITxPool? TxPool { get; set; }
Expand Down
1 change: 1 addition & 0 deletions src/Nethermind/Nethermind.AuRa.Test/Contract/AuRaContractGasLimitOverrideTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ protected override BlockProcessor CreateBlockProcessor()
BlockGasLimitContract gasLimitContract = new(AbiEncoder.Instance, blockGasLimitContractTransition.Value, blockGasLimitContractTransition.Key,
new ReadOnlyTxProcessingEnv(
DbProvider,
new TrieStoreByPath(DbProvider.StateDb, LimboLogs.Instance).AsReadOnly(),
new TrieStore(DbProvider.StateDb, LimboLogs.Instance).AsReadOnly(),
BlockTree, SpecProvider, LimboLogs.Instance));

Expand Down
2 changes: 1 addition & 1 deletion src/Nethermind/Nethermind.AuRa.Test/Contract/TxPriorityContractTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,7 @@ protected override TxPoolTxSource CreateTxPoolTxSource()
TxPoolTxSource txPoolTxSource = base.CreateTxPoolTxSource();

TxPriorityContract = new TxPriorityContract(AbiEncoder.Instance, TestItem.AddressA,
new ReadOnlyTxProcessingEnv(DbProvider, TrieStore.AsReadOnly(), BlockTree, SpecProvider, LimboLogs.Instance));
new ReadOnlyTxProcessingEnv(DbProvider, TrieStore.AsReadOnly(), ReadOnlyStorageTrieStore, BlockTree, SpecProvider, LimboLogs.Instance));

Priorities = new DictionaryContractDataStore<TxPriorityContract.Destination>(
new TxPriorityContract.DestinationSortedListContractDataStoreCollection(),
Expand Down
1 change: 1 addition & 0 deletions src/Nethermind/Nethermind.AuRa.Test/Transactions/TxCertifierFilterTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ protected override BlockProcessor CreateBlockProcessor()
AbiEncoder abiEncoder = AbiEncoder.Instance;
ReadOnlyTransactionProcessorSource = new ReadOnlyTxProcessingEnv(
DbProvider,
new TrieStoreByPath(DbProvider.StateDb, LimboLogs.Instance).AsReadOnly(),
new TrieStore(DbProvider.StateDb, LimboLogs.Instance).AsReadOnly(),
BlockTree, SpecProvider,
LimboLogs.Instance);
Expand Down
6 changes: 4 additions & 2 deletions src/Nethermind/Nethermind.AuRa.Test/Transactions/TxPermissionFilterTest.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -276,16 +276,18 @@ protected override BlockProcessor CreateBlockProcessor()
TransactionPermissionContractVersions =
new LruCache<KeccakKey, UInt256>(PermissionBasedTxFilter.Cache.MaxCacheSize, nameof(TransactionPermissionContract));

IReadOnlyTrieStore trieStore = new TrieStore(DbProvider.StateDb, LimboLogs.Instance).AsReadOnly();
IReadOnlyTrieStore trieStore = new TrieStoreByPath(DbProvider.StateDb, LimboLogs.Instance).AsReadOnly();
IReadOnlyTrieStore storageTrieStore = new TrieStore(DbProvider.StateDb, LimboLogs.Instance).AsReadOnly();
IReadOnlyTxProcessorSource txProcessorSource = new ReadOnlyTxProcessingEnv(
DbProvider,
trieStore,
storageTrieStore,
BlockTree,
SpecProvider,
LimboLogs.Instance);

VersionedTransactionPermissionContract transactionPermissionContract = new(AbiEncoder.Instance, _contractAddress, 1,
new ReadOnlyTxProcessingEnv(DbProvider, trieStore, BlockTree, SpecProvider, LimboLogs.Instance), TransactionPermissionContractVersions, LimboLogs.Instance, SpecProvider);
new ReadOnlyTxProcessingEnv(DbProvider, trieStore, storageTrieStore, BlockTree, SpecProvider, LimboLogs.Instance), TransactionPermissionContractVersions, LimboLogs.Instance, SpecProvider);

TxPermissionFilterCache = new PermissionBasedTxFilter.Cache();
PermissionBasedTxFilter = new PermissionBasedTxFilter(transactionPermissionContract, TxPermissionFilterCache, LimboLogs.Instance);
Expand Down
3 changes: 3 additions & 0 deletions src/Nethermind/Nethermind.Benchmark/Nethermind.Benchmark.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@
<ProjectReference Include="..\Nethermind.Consensus.Ethash\Nethermind.Consensus.Ethash.csproj" />
<ProjectReference Include="..\Nethermind.Core.Test\Nethermind.Core.Test.csproj" />
<ProjectReference Include="..\Nethermind.Core\Nethermind.Core.csproj" />
<ProjectReference Include="..\Nethermind.Db.Rocks\Nethermind.Db.Rocks.csproj" />
<ProjectReference Include="..\Nethermind.Db\Nethermind.Db.csproj" />
<ProjectReference Include="..\Nethermind.HashLib\Nethermind.HashLib.csproj" />
<ProjectReference Include="..\Nethermind.Network.Stats\Nethermind.Network.Stats.csproj" />
</ItemGroup>
</Project>
Loading