Code

Rework of storage [mtg-868]#310

Code

Merged

armyhaylenko merged 145 commits intomainmetaplex-foundation/aura:mainfrom feature/MTG-868-slots-storagemetaplex-foundation/aura:feature/MTG-868-slots-storageCopy head branch name to clipboardJan 13, 2025

+21,997-3,399

Commits

Commits on Oct 10, 2024

• drop the mutex as soon as needed

  StanChe

  committed
  e79ccd4

  Copy full SHA for e79ccd4

• spawn_blocking instead of the regular spawn for blocking code

  StanChe

  committed
  9e63ab2

  Copy full SHA for 9e63ab2

• removed parallel processing from synchronizer reading

  StanChe

  committed
  91bfdee

  Copy full SHA for 91bfdee

• More metrics for synchronizer (#281)

  Show description for b629102
  n00m4d

  authored
  b629102

  Copy full SHA for b629102

Commits on Oct 11, 2024

• MTG-742 Move fungible assets dump to different thread (#282)

  Show description for 52e511a
  n00m4d

  authored
  52e511a

  Copy full SHA for 52e511a

Commits on Oct 14, 2024

• MTG-747 Save updates for fungible assets during regular sync (#283)

  Show description for f61d79d
  n00m4d

  authored
  f61d79d

  Copy full SHA for f61d79d

• feat: pass optional hashMap to get_asset_indexes

  n00m4d

  committed
  06d85a8

  Copy full SHA for 06d85a8

Commits on Oct 15, 2024

• fix: optimize memory usage

  n00m4d

  committed
  fbc242d

  Copy full SHA for fbc242d

Commits on Oct 17, 2024

• a complete rework of the rocks storage

  Show description for 78b291f
  StanChe

  committed
  78b291f

  Copy full SHA for 78b291f

Commits on Oct 22, 2024

• renamed the merge function

  StanChe

  committed
  d1963f0

  Copy full SHA for d1963f0

• a small rocks-db explorer for debugging purposes

  StanChe

  committed
  61affec

  Copy full SHA for 61affec

• fixed readme formatting

  StanChe

  committed
  db55652

  Copy full SHA for db55652

• made clippy happy

  StanChe

  committed
  1770d1d

  Copy full SHA for 1770d1d

• using axum

  StanChe

  committed
  e2d72c6

  Copy full SHA for e2d72c6

• deps for axum

  StanChe

  committed
  b643d8b

  Copy full SHA for b643d8b

Commits on Oct 23, 2024

• removed the unnecessary primary db opening

  StanChe

  committed
  10952fa

  Copy full SHA for 10952fa

Commits on Oct 24, 2024

• flatbuffer types + merged types + optimized dump client = 30% improvement over the reference, or over 50% over the initial merge

  StanChe

  committed
  824fc1f

  Copy full SHA for 824fc1f

Commits on Oct 28, 2024

• dump client optimized further, asset client refactored

  Show description for d978784
  StanChe

  committed
  d978784

  Copy full SHA for d978784

• batch client using flatbuffer

  StanChe

  committed
  a05855d

  Copy full SHA for a05855d

• insert_gaped_data updated in batch client

  StanChe

  committed
  d7d9b6b

  Copy full SHA for d7d9b6b

• minor refactor

  StanChe

  committed
  31aca0f

  Copy full SHA for 31aca0f

• more benchmarks

  StanChe

  committed
  edcff2f

  Copy full SHA for edcff2f

• get_asset_indexes optimized

  StanChe

  committed
  936ebda

  Copy full SHA for 936ebda

• full refactor to use flatbuffer

  StanChe

  committed
  ff7e94e

  Copy full SHA for ff7e94e

• code cleanup

  StanChe

  committed
  89a6571

  Copy full SHA for 89a6571

Commits on Oct 29, 2024

• crazy merge function

  StanChe

  committed
  73531ff

  Copy full SHA for 73531ff

• moved the code around

  StanChe

  committed
  4221d94

  Copy full SHA for 4221d94

• merge function tests

  StanChe

  committed
  522cb1a

  Copy full SHA for 522cb1a

Commits on Oct 30, 2024

• more randomized assets generation in tests

  StanChe

  committed
  885fbba

  Copy full SHA for 885fbba

• another attempt to the merge function

  StanChe

  committed
  3ea1df2

  Copy full SHA for 3ea1df2

• using the recent iteration of merge function

  StanChe

  committed
  de2e42f

  Copy full SHA for de2e42f

• Merge commit '88e5d12b86483126a5aa571072dca93b5d4df52f' into feature/MTG-12-flatbuffer-merge

  Show description for 98eb391
  StanChe

  committed
  98eb391

  Copy full SHA for 98eb391

Commits on Oct 31, 2024

• updated streaming client to use flatbuffers

  StanChe

  committed
  44df86e

  Copy full SHA for 44df86e

• fixed tests

  StanChe

  committed
  53c4a68

  Copy full SHA for 53c4a68

• fixed a test and dropped a debug line from another test

  Show description for ce73e95
  StanChe

  committed
  ce73e95

  Copy full SHA for ce73e95

• cleanup benches

  StanChe

  committed
  1e25381

  Copy full SHA for 1e25381

• fixed the imports

  StanChe

  committed
  5247299

  Copy full SHA for 5247299

Commits on Nov 1, 2024

• fixed migrator, some tests

  StanChe

  committed
  08062da

  Copy full SHA for 08062da

Commits on Nov 4, 2024

• fixed an issue with mpl core collections getting into response with metadata. fixed another test setup

  StanChe

  committed
  ab77ec4

  Copy full SHA for ab77ec4

• dump client fixed for fungibles

  StanChe

  committed
  7342535

  Copy full SHA for 7342535

• fix for tests setup accounting for authority being optional in some tests

  StanChe

  committed
  4123d05

  Copy full SHA for 4123d05

• Merge commit '962eaf9560de83ae1f917ec168b461666c3d7000' into feature/MTG-12-flatbuffer-merge + updated the data read to get dynamic details

  Show description for fd0ca32
  StanChe

  committed
  fd0ca32

  Copy full SHA for fd0ca32

• making fmt and clippy happier

  StanChe

  committed
  cae65de

  Copy full SHA for cae65de

• less randomness in tests

  StanChe

  committed
  620d842

  Copy full SHA for 620d842

• dropped some commented out code

  StanChe

  committed
  3474b40

  Copy full SHA for 3474b40

• Merge commit 'b410996aaf3584ed572b2dea699d2a66f94359b3' into feature/MTG-12-flatbuffer-merge

  Show description for d8855c4
  StanChe

  committed
  d8855c4

  Copy full SHA for d8855c4

• Merge commit 'd2f3e0af6b2cb438a77f45e640e2e5731c1f0977' into feature/MTG-12-flatbuffer-merge

  Show description for d4baf7d
  StanChe

  committed
  d4baf7d

  Copy full SHA for d4baf7d

• a small backfiller cli tool

  StanChe

  committed
  6459da1

  Copy full SHA for 6459da1

• reverted the raw slots removal

  StanChe

  committed
  8848d08

  Copy full SHA for 8848d08

Commits on Nov 5, 2024

• reworked to a concurrent version

  StanChe

  committed
  d62ee3a

  Copy full SHA for d62ee3a

• fixed the message

  StanChe

  committed
  f711dfd

  Copy full SHA for f711dfd

• graceful stop, async worker model reworked

  StanChe

  committed
  1fc1555

  Copy full SHA for 1fc1555

• fmt+clippy

  StanChe

  committed
  849428a

  Copy full SHA for 849428a

Commits on Nov 7, 2024

• raw slot persister

  StanChe

  committed
  17411f7

  Copy full SHA for 17411f7

• added existing column families

  StanChe

  committed
  bee1001

  Copy full SHA for bee1001

• more logging

  StanChe

  committed
  545d3ba

  Copy full SHA for 545d3ba

Commits on Nov 8, 2024

• a slot checker cli tool

  StanChe

  committed
  1502677

  Copy full SHA for 1502677

• faster iteration in persister

  StanChe

  committed
  3b4ea49

  Copy full SHA for 3b4ea49

Commits on Nov 13, 2024

• Make big table optional (#312)

  Show description for f14115b
  n00m4d

  authored
  f14115b

  Copy full SHA for f14115b

• ingester reworked to use a separate slots storage, column remover updated accordingly

  Show description for 68b05dc
  StanChe

  committed
  68b05dc

  Copy full SHA for 68b05dc

• check whether slots are present in the db

  StanChe

  committed
  1286ecb

  Copy full SHA for 1286ecb

• persist the specified slots

  StanChe

  committed
  17e79f0

  Copy full SHA for 17e79f0

• backfill particular slots

  StanChe

  committed
  3811815

  Copy full SHA for 3811815

• dropped raw slots from columns used in the main db

  StanChe

  committed
  d2a7701

  Copy full SHA for d2a7701

• offsetting the slot persister for 2 minutes from the last finalized slot in an attempt to mitigate the eventual consistency of bigtable client

  StanChe

  committed
  20e96cc

  Copy full SHA for 20e96cc

• tmp: commented out the migrator for now

  StanChe

  committed
  ffa289e

  Copy full SHA for ffa289e

• more metrics

  StanChe

  committed
  9a54f6b

  Copy full SHA for 9a54f6b

• fucking fungibles

  StanChe

  committed
  514fbd8

  Copy full SHA for 514fbd8

• fixing other then fungible assets request times

  StanChe

  committed
  6e27910

  Copy full SHA for 6e27910

• removed synchronizer from ingester and no starting of tcp receivers when the receiver is in redis mode

  StanChe

  committed
  a04792d

  Copy full SHA for a04792d

• refactored ingester code

  StanChe

  committed
  c02b9aa

  Copy full SHA for c02b9aa

• minor refactor

  StanChe

  committed
  41efee0

  Copy full SHA for 41efee0

• fixed messages on metrics server and fixed synchronizer

  StanChe

  committed
  8d58106

  Copy full SHA for 8d58106

• add the args to ingester

  StanChe

  committed
  c1e45b6

  Copy full SHA for c1e45b6

• Merge branch 'main' into feature/MTG-868-slots-storage

  Show description for aa29911
  RequescoS

  committed
  aa29911

  Copy full SHA for aa29911

• compose update to include slots paths + removed the grpc server from the api server until the slots db is read on it

  StanChe

  committed
  4bf3999

  Copy full SHA for 4bf3999

• fix: rm unwraps in dump_csv()

  RequescoS

  committed
  6efcec4

  Copy full SHA for 6efcec4

• Merge remote-tracking branch 'origin/feature/MTG-868-slots-storage' into feature/MTG-868-slots-storage

  RequescoS

  committed
  d205eef

  Copy full SHA for d205eef

• fix: if asset.static_details().is_none() continue

  RequescoS

  committed
  9276076

  Copy full SHA for 9276076

• fix: iter.next()

  RequescoS

  committed
  21daeeb

  Copy full SHA for 21daeeb

• fix: static_details().is_none() continue

  RequescoS

  committed
  111ed85

  Copy full SHA for 111ed85

• Merge branch 'feature/rocks-explorer' into feature/MTG-868-slots-storage

  Show description for 3dae65a
  StanChe

  committed
  3dae65a

  Copy full SHA for 3dae65a

• a more flexible db object creation on the columns that exist in rocks

  StanChe

  committed
  f1f2a8d

  Copy full SHA for f1f2a8d

Commits on Nov 14, 2024

• merge functions in explorer

  StanChe

  committed
  7cbec47

  Copy full SHA for 7cbec47

• using unique short flag

  StanChe

  committed
  c44c711

  Copy full SHA for c44c711

• actually using cf options

  StanChe

  committed
  fdbf020

  Copy full SHA for fdbf020

• fix for synchronizer putting fungibles into the main index

  StanChe

  committed
  1009d71

  Copy full SHA for 1009d71

• fmt

  StanChe

  committed
  6aa03d2

  Copy full SHA for 6aa03d2

• filter out empty urls from getting into pg index

  StanChe

  committed
  061f88f

  Copy full SHA for 061f88f

• optimized the dump synchronizer further by introducing the mpl core assets authorities index

  StanChe

  committed
  c7d1514

  Copy full SHA for c7d1514

• Merge pull request #315 from metaplex-foundation/feature/mpl-core-collection-authority-column

  Show description for 2b9cb82
  StanChe

  authored
  2b9cb82

  Copy full SHA for 2b9cb82

Commits on Nov 16, 2024

• token22 fix

  StanChe

  committed
  e089de9

  Copy full SHA for e089de9

• fix: display options default values

  RequescoS

  committed
  5f73cc8

  Copy full SHA for 5f73cc8

• Merge remote-tracking branch 'origin/feature/MTG-868-slots-storage' into feature/MTG-868-slots-storage

  RequescoS

  committed
  daf7fd0

  Copy full SHA for daf7fd0

• fix: display options default values

  RequescoS

  committed
  8fa17ae

  Copy full SHA for 8fa17ae

• fixes for build

  StanChe

  committed
  0624262

  Copy full SHA for 0624262

• fix: grand_total_query

  RequescoS

  committed
  c9a9a19

  Copy full SHA for c9a9a19

• Merge remote-tracking branch 'origin/feature/MTG-868-slots-storage' into feature/MTG-868-slots-storage

  RequescoS

  committed
  dba1a7a

  Copy full SHA for dba1a7a

• fix: grand_total_query

  RequescoS

  committed
  898c13a

  Copy full SHA for 898c13a

• fix: native mint to config

  RequescoS

  committed
  f6efe2e

  Copy full SHA for f6efe2e

• fix: collect::<Vec<_>>()

  RequescoS

  committed
  a0cd305

  Copy full SHA for a0cd305

• fix: token_price_fetcher

  RequescoS

  committed
  99e5678

  Copy full SHA for 99e5678

Commits on Nov 18, 2024

• extended explorer with an additional method

  StanChe

  committed
  e4e8235

  Copy full SHA for e4e8235

• yet another debug tool

  StanChe

  committed
  edf3357

  Copy full SHA for edf3357

• fix for transactions not being closed

  StanChe

  committed
  8217fab

  Copy full SHA for 8217fab

Commits on Nov 19, 2024

• an additional debug tool allowing to lookup asset by an owner using a full scan

  StanChe

  committed
  0d3d71e

  Copy full SHA for 0d3d71e

• additional mode for synchronizer utils

  StanChe

  committed
  34a5b49

  Copy full SHA for 34a5b49

• MTG-949 additional debug tool

  StanChe

  committed
  eac6267

  Copy full SHA for eac6267

• MTG-949 fixed the downloader considering empty metadata valid

  Show description for 3b54d19
  StanChe

  committed
  3b54d19

  Copy full SHA for 3b54d19

• MTG-950 json downloader less strict content type filters

  StanChe

  committed
  fa12a8d

  Copy full SHA for fa12a8d

Commits on Nov 20, 2024

• a proper usage of tokentype filter

  StanChe

  committed
  fdc18ee

  Copy full SHA for fdc18ee

• using separated in sqlx syntax

  StanChe

  committed
  a21d651

  Copy full SHA for a21d651

• merge function updating asset class if a new one was provided for a restricted number of cases

  StanChe

  committed
  bfd6b4d

  Copy full SHA for bfd6b4d

• Merge pull request #318 from metaplex-foundation/feature/MTG-947-tokentype-filter-fix

  Show description for a229539
  StanChe

  authored
  a229539

  Copy full SHA for a229539

• mapping the asset class on indexing given it's unrelyable in rocks

  StanChe

  committed
  053528d

  Copy full SHA for 053528d

• mapping the asset class before returning it from rocks

  StanChe

  committed
  1d05e1f

  Copy full SHA for 1d05e1f

• rearranged the sql code into proper places

  StanChe

  committed
  ef7c6d6

  Copy full SHA for ef7c6d6

• encode fungibles properly on dump

  StanChe

  committed
  7333979

  Copy full SHA for 7333979

Commits on Nov 21, 2024

• super hacky way to ingest images when those are provided instead of metadata

  StanChe

  committed
  94a6f80

  Copy full SHA for 94a6f80

Commits on Nov 22, 2024

• Merge pull request #320 from metaplex-foundation/feature/MTG-947-asset-class-mapping-on-indexing

  Show description for 24b21f9
  StanChe

  authored
  24b21f9

  Copy full SHA for 24b21f9

• Merge pull request #319 from metaplex-foundation/feature/MTG-947-merge_function

  Show description for 2e7935d
  StanChe

  authored
  2e7935d

  Copy full SHA for 2e7935d

• fix: write version merge

  RequescoS

  committed
  27ef2ed

  Copy full SHA for 27ef2ed

Commits on Nov 25, 2024

• change if condition

  RequescoS

  committed
  f668af7

  Copy full SHA for f668af7

• adding a history of owners for an asset, tracking any ownership change to determine the correct owner

  StanChe

  committed
  2c74caa

  Copy full SHA for 2c74caa

• support for IPFS files

  StanChe

  committed
  15fcec8

  Copy full SHA for 15fcec8

• storing the prebuilt metadata for metadata urls that are actually images

  StanChe

  committed
  99960af

  Copy full SHA for 99960af

• Merge pull request #321 from metaplex-foundation/fix/merge-write-version

  Show description for f00d1b4
  StanChe

  authored
  f00d1b4

  Copy full SHA for f00d1b4

• nit: new line in fbs

  StanChe

  committed
  bd0c704

  Copy full SHA for bd0c704

• removed old todo

  StanChe

  committed
  872cb65

  Copy full SHA for 872cb65

• Merge pull request #323 from metaplex-foundation/feature/MTG-961-IPFS-support

  Show description for 7919627
  StanChe

  authored
  7919627

  Copy full SHA for 7919627

• Merge pull request #322 from metaplex-foundation/feature/MTG-964-ownereship-fix

  Show description for 578a167
  StanChe

  authored
  578a167

  Copy full SHA for 578a167

Commits on Nov 26, 2024

• Implemented cleaning up asset slot update idx && added tests

  kstepanovdev

  committed
  2f8e7ee

  Copy full SHA for 2f8e7ee

• Simplify cleaning up asset update index

  kstepanovdev

  committed
  49f0096

  Copy full SHA for 49f0096

• simplified interfaces and their number for assets idx cleaning up

  Show description for 3e3f044
  kstepanovdev

  committed
  3e3f044

  Copy full SHA for 3e3f044

• reorganize cleaners into a module

  kstepanovdev

  committed
  eadeb0d

  Copy full SHA for eadeb0d

Commits on Nov 27, 2024

• Fix receiving shutdown signal in ingester and cleaning up service

  kstepanovdev

  committed
  089528a

  Copy full SHA for 089528a

• MTG-959 fix a few issues with core assets (#324)

  Show description for f7ba181
  n00m4d

  authored
  f7ba181

  Copy full SHA for f7ba181

• MTG-941 add integration e to e tests (#326)

  Show description for 810c162
  

  n00m4d

  and

  StanChe

  authored
  810c162

  Copy full SHA for 810c162

• Merge branch 'clean-asset-slot-update-idx' into feature/MTG-868-slots-storage

  Show description for ca12aae
  StanChe

  committed
  ca12aae

  Copy full SHA for ca12aae

• Merge commit 'afeb76ee4de287ca3707d4db419a196cfcbe374b' into feature/MTG-868-slots-storage

  Show description for 68bf6a0
  StanChe

  committed
  68bf6a0

  Copy full SHA for 68bf6a0

Commits on Nov 28, 2024

• revert the formating of migration that broke prod

  StanChe

  committed
  f8ab715

  Copy full SHA for f8ab715

Commits on Nov 29, 2024

• MTG-980 Fix a few issues with cNFT assets (#327)

  Show description for c334114
  n00m4d

  authored
  c334114

  Copy full SHA for c334114

Commits on Dec 3, 2024

• chore: drop unused data struct (#330)

  n00m4d

  authored
  3fe1e04

  Copy full SHA for 3fe1e04

Commits on Dec 9, 2024

• feat: fix one integration test and drop hardcoding mutability field during mint account update (#332)

  n00m4d

  authored
  a971484

  Copy full SHA for a971484

• feat: add new debug tool (#335)

  n00m4d

  authored
  e3974da

  Copy full SHA for e3974da