[EVM] Make BlockReplayer idempotent

[peterargue]

Problem Definition

The offline BlockReplayer takes a set of EVM block and transaction events and re-executes the transactions to reproduce the results. This is used by the gateway to produce a local index of the execution state.

Currently, the BlockReplayer uses the target block's height when setting up the storage snapshot. Snapshots store the end of block state, which means that if the block had already been processed, it would double execute the transactions causing failures (typically nonce mismatch).

The gateway allows reindexing of data using a height rollback feature. This makes it simple to deal with issues in the last few blocks, but requires that indexing is idempotent.

Proposed Solution

Make BlockReplayer idempotent. I think we may be able to do this by using block-1 as block for the storage snapshots here:

flow-go/fvm/evm/offchain/sync/replayer.go

Lines 74 to 86 in 3168879

	st, err := cr.storageProvider.GetSnapshotAt(blockEvent.Height)
	if err != nil {
	return nil, nil, err
	}
	// create storage
	state := storage.NewEphemeralStorage(storage.NewReadOnlyStorage(st))
	// get block snapshot
	bs, err := cr.blockProvider.GetSnapshotAt(blockEvent.Height)
	if err != nil {
	return nil, nil, err
	}

but definitely need @janezpodhostnik's help here

[janezpodhostnik]

Looking at this code, the interface already specifies that the state should be at the start of the block.

// StorageProvider provides access to storage at
// specific time point in history of the EVM chain
type StorageProvider interface {
	// GetSnapshotAt returns a readonly snapshot of storage
	// at specific block (start state of the block before executing transactions)
	GetSnapshotAt(evmBlockHeight uint64) (BackendStorageSnapshot, error)
}

so maybe we just missed this comment when implementing the interface. I will dig deeper.