Dictionary v2 #2257

jiqiang90 · 2024-02-19T03:17:29Z

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist

I have tested locally
I have performed a self review of my changes
Updated any relevant documentation
Linked to any relevant issues
I have added tests relevant to my changes
Any dependent changes have been merged and published in downstream modules
My code is up to date with the base branch
I have updated relevant changelogs. We suggest using chan

packages/node-core/src/configure/NodeConfig.ts

packages/node-core/src/indexer/blockDispatcher/block-dispatcher.ts

packages/node-core/src/indexer/dictionary/types.ts

stwiname · 2024-02-20T01:52:38Z

packages/node-core/src/indexer/dictionary/utils.ts

These need tests

packages/node-core/src/indexer/dictionary/coreDictionary.ts

packages/node/src/indexer/dictionary/v2/substrateDictionaryV2.ts

packages/node-core/src/indexer/dictionary/v2/dictionaryV2.ts

packages/node-core/src/indexer/fetch.service.ts

packages/node-core/src/indexer/dictionary/dictionary.service.ts

stwiname

It would be good to use IBlock with UnfinalizedBlocks. It should be able to simplify implementation. https://github.com/subquery/subql/blob/main/packages/node/src/indexer/unfinalizedBlocks.service.ts#L17

packages/node-core/src/indexer/dictionary/v2/dictionaryV2.ts

packages/node-core/src/indexer/dictionary/v2/types.ts

packages/node-core/src/indexer/dictionary/v2/dictionaryV2.ts

packages/node-core/src/utils/blocks.spec.ts

Co-authored-by: Scott Twiname <[email protected]>

packages/types-core/src/project/types.ts

packages/node-core/src/indexer/blockDispatcher/worker-block-dispatcher.ts

packages/node-core/src/indexer/dictionary/coreDictionary.ts

packages/node-core/src/indexer/dictionary/dictionary.service.spec.ts

packages/node-core/src/indexer/dictionary/dictionary.service.ts

packages/node-core/src/indexer/dictionary/v2/dictionaryV2.ts

stwiname · 2024-03-03T21:52:55Z

packages/node/src/indexer/unfinalizedBlocks.service.ts

@@ -36,11 +36,6 @@ export class UnfinalizedBlocksService extends BaseUnfinalizedBlocksService<
    super(nodeConfig, storeCache);
  }

-  @mainThreadOnly()


We should also remove substrateHeaderToHeader function

Beside this place been used, I think a few other places are using is it

subql/packages/node/src/indexer/fetch.service.ts

Line 68 in 6a4d55f

const header = substrateHeaderToHeader(finalizedHeader);

subql/packages/node/src/indexer/unfinalizedBlocks.service.ts

Line 51 in 6a4d55f

return substrateHeaderToHeader(

packages/node/src/indexer/worker/worker.unfinalizedBlocks.service.ts

packages/node/src/subcommands/testing.service.ts

stwiname · 2024-03-03T21:55:32Z

packages/node/src/utils/substrate.ts

+    getHeader: () => {
+      return {
+        hash: blockHeader.block.header.hash.toString(),
+        height: blockHeader.block.header.number.toNumber(),
+        parentHash: blockHeader.block.header.parentHash.toString(),
+      };


This can be put into a function to dedupe with https://github.com/subquery/subql/pull/2257/files#diff-622aec34f349237d6d2b7b613e44aedbe25468c88ba6597f2051bb9c825d55f4R396-R398

Might be a good idea to unify getHeader: () return type with Header https://github.com/subquery/subql/pull/2257/files#diff-8138bbd5bd4b60a1dea77c9b7b6577d99ff57d83838c9a49fe66024a1793b153R23, so substrateHeaderToHeader can be re-used

Co-authored-by: Scott Twiname <[email protected]>

stwiname

A lot of the headers have 2023 in the files, should be 2024

packages/node-core/src/indexer/dictionary/v1/dictionaryV1.ts

packages/node-core/src/indexer/dictionary/v2/dictionaryV2.ts

packages/node/src/indexer/dictionary/utils.ts

Co-authored-by: Scott Twiname <[email protected]>

packages/types-core/src/project/types.ts

jiqiang90 and others added 5 commits February 16, 2024 12:25

node-core and types

cdf9be4

remove deprecated dictionary

34ac73c

need to fix runtime

9c1ab64

fix getDsProcessor

d5ff192

Tidy up

2099ae3

stwiname reviewed Feb 20, 2024

View reviewed changes

stwiname and others added 5 commits February 20, 2024 15:49

Remove dictionary version enum, tidy up generics and imports

c701c0d

runtime test and fix

424c0a1

move tests, and change getDictionary behaviour

3bb5beb

Merge branch 'main' into dictionary-v2

093bee9

Add changelogs

3f3ddca

stwiname reviewed Feb 26, 2024

View reviewed changes

packages/node-core/src/indexer/dictionary/coreDictionary.ts Outdated Show resolved Hide resolved

Tidy up unused code

df8107d

stwiname reviewed Feb 26, 2024

View reviewed changes

jiqiang90 and others added 5 commits February 27, 2024 17:11

Update queryEndBlock and add retry logic

312c160

return undefined for invalid dictionary index

6f0504b

add test for v2

aee65ea

Merge branch 'main' into dictionary-v2

a883e20

Simplify generics

f2ad093

stwiname reviewed Feb 28, 2024

View reviewed changes

jiqiang90 and others added 7 commits February 29, 2024 14:32

Update packages/node-core/src/indexer/dictionary/v2/dictionaryV2.ts

a15f82a

Co-authored-by: Scott Twiname <[email protected]>

Update packages/node-core/src/indexer/dictionary/v2/dictionaryV2.ts

3bf4a0a

Co-authored-by: Scott Twiname <[email protected]>

generalised getData and other fixes

850f9b4

Unfinalized service use IBlock

3135927

Merge branch 'main' into dictionary-v2

48a5459

remove TODO

bf45f44

Merge remote-tracking branch 'origin/dictionary-v2' into dictionary-v2

cd8bd30

stwiname reviewed Mar 3, 2024

View reviewed changes

jiqiang90 and others added 2 commits March 4, 2024 11:11

Update packages/node-core/src/indexer/dictionary/v2/dictionaryV2.ts

c769870

Co-authored-by: Scott Twiname <[email protected]>

Update packages/node-core/src/indexer/dictionary/v2/dictionaryV2.ts

0ea7865

Co-authored-by: Scott Twiname <[email protected]>

jiqiang90 and others added 8 commits March 7, 2024 15:52

Update packages/node-core/src/indexer/dictionary/dictionary.service.ts

a8e9a1e

Co-authored-by: Scott Twiname <[email protected]>

update for validateChainMeta

2200f16

Merge remote-tracking branch 'origin/main' into dictionary-v2

592698a

remove entryscript from test

541719f

fix self import

811f45c

remove

b70acaf

rollback

06b43a9

space

6c032bc

stwiname reviewed Mar 8, 2024

View reviewed changes

packages/node-core/src/indexer/dictionary/v1/dictionaryV1.ts Outdated Show resolved Hide resolved

packages/node-core/src/indexer/dictionary/v2/dictionaryV2.ts Outdated Show resolved Hide resolved

packages/node/src/indexer/dictionary/utils.ts Outdated Show resolved Hide resolved

Update packages/node-core/src/indexer/dictionary/v1/dictionaryV1.ts

f7f16d2

Co-authored-by: Scott Twiname <[email protected]>

stwiname reviewed Mar 8, 2024

View reviewed changes

packages/types-core/src/project/types.ts Show resolved Hide resolved

jiqiang90 added 15 commits March 8, 2024 14:54

header

42d48b5

header

b3f5fe3

remove SubstrateDsInterface

b252154

update dictionary type in common

93064d1

use SubstrateDataSource type

4af1570

fix dictionary test

488c881

add test for substrate dictionary v1

ec77209

rename test file

5fe37aa

add test and queryMapValidByHeight

2b678f8

dictionary getData test

cad42de

fix tests

65381fc

fix tests

f37a9b3

fix metadata log error msg

52eaa6c

tidy up test

891cff8

Fix unfinalizedBlocks and StoreOp tests

971562f

stwiname approved these changes Mar 12, 2024

View reviewed changes

stwiname merged commit 0d900f1 into main Mar 12, 2024
1 of 2 checks passed

stwiname mentioned this pull request Mar 21, 2024

Bring back changes from https://github.com/subquery/subql/pull/2225 #2320

Merged

9 tasks

stwiname deleted the dictionary-v2 branch March 21, 2024 23:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dictionary v2 #2257

Dictionary v2 #2257

jiqiang90 commented Feb 19, 2024

stwiname Feb 20, 2024

stwiname left a comment

stwiname Mar 3, 2024

jiqiang90 Mar 4, 2024 •

edited

Loading

stwiname Mar 3, 2024

jiqiang90 Mar 4, 2024

stwiname left a comment

Dictionary v2 #2257

Dictionary v2 #2257

Conversation

jiqiang90 commented Feb 19, 2024

Description

Type of change

Checklist

stwiname Feb 20, 2024

Choose a reason for hiding this comment

stwiname left a comment

Choose a reason for hiding this comment

stwiname Mar 3, 2024

Choose a reason for hiding this comment

jiqiang90 Mar 4, 2024 • edited Loading

Choose a reason for hiding this comment

stwiname Mar 3, 2024

Choose a reason for hiding this comment

jiqiang90 Mar 4, 2024

Choose a reason for hiding this comment

stwiname left a comment

Choose a reason for hiding this comment

jiqiang90 Mar 4, 2024 •

edited

Loading