[QUESTION]: The provider's client being stuck at an outdated latest_height

[mksquad87]

Hi there,

I don't think this is an actual bug, but rather a some tiny misconfiguration on my side or just something that I'm completely missing...But it would be incredibly helpful to get some ideas and insights where the error I'm facing is coming from.

Its about getting a provider to validate a consumer locally on the same machine via relayer. Here are the steps:

• I cloned the interchain-security repo (v5.1) created the provider and consumer binaries
• I configured the provider and got it running
• I created a consumer via ignite scaffold chain ccv --consumer --skip-proto --no-module
• I initialise the consumer via ignite init ...
• Then I did the necessary steps to add the consumer via proposal, vote for it and await the spawn_time and set the genesis_time > spawn_time
• When the spawn_time passes i copy the relevant ccvconsumer parts (exported from the provider) to the consumer's genesis
• then I start the consumer and it starts producing blocks via the initial_val_set provided by the genesis.json
• Then I added hermes account keys of both provider and consumer to the hermes config
• Next I create the connection via hermes create connection --a-chain ccv --a-client 07-tendermint-0 --b-client 07-tendermint-0
• And finally I create the channel via hermes create channel --a-chain ccv --a-port consumer --b-port provider --order ordered --a-connection connection-0 --channel-version 1

Until here everything works fine. Now it gets weird:

I noticed when I tried to run the same logic by using the interchain-secuirty-cd binary directly (without using ignite), then all the steps work fine until hermes create connection..., here it always throws the same error:

failed during an operation on client '07-tendermint-0' hosted by chain 'provider': chain provider is missing trusted state smaller than target height 0-36

Then I tried to reproduce the exact same steps as above (i.e. I created another consumer project via ignite) with the exact same provider settings/chain, same proposal logic, etc. and I always get this same error...

When I check the provider's client its always "Active" but also always stuck at the initial height:

latest_height: Height {
            revision: 1,
            height: 1,
        },

So it looks like the relayer is not successfully relaying the consensus state from the consumer to the provider (but only for my second consumer 😅)

I've checked the differences between the codebases of the Cosmos SDKs of the consumers and they are basically the same (just with different chain ids and app names) and also the config folders, (i.e. genesis.json and toml files) are identical (again just name and chain id differs)

Note: Every time i run either one of the chain pairs (consumer1 with provider or consumer2 with provider) I reset the data of both chains and start both chains from block 0.

Also the .hermes config is the same for both consumers except the "id" and looks like this:

[global]
log_level = "debug"

[mode]

[mode.clients]
enabled = true
refresh = true
misbehaviour = true

[mode.connections]
enabled = true

[mode.channels]
enabled = true

[mode.packets]
enabled = true
clear_interval = 100
clear_on_start = true
tx_confirmation = false
auto_register_counterparty_payee = false

[[chains]]
id = "provider"   
type = "CosmosSdk"
rpc_addr = "http://localhost:26658"       
grpc_addr = "http://localhost:9091"   
rpc_timeout = "10s"
account_prefix = "cosmos"              
key_name = "prov-key"        
store_prefix = "ibc"
default_gas = 1000000
max_gas = 4000000
gas_multiplier = 1.5
max_msg_num = 30
max_tx_size = 180000
max_grpc_decoding_size = 33554432
clock_drift = "5s"
max_block_time = "30s"
ccv_consumer_chain = false
memo_prefix = ""
sequential_batch_tx = false

[chains.event_source]
mode = "push"
url = "ws://localhost:26658/websocket"
batch_delay = "500ms"

[chains.trust_threshold]
numerator = "1"
denominator = "3"

[chains.gas_price]
price = 0.1
denom = "stake"

[chains.packet_filter]
policy = "allow"
list = [["transfer", "*"], ["ccv-provider", "*"], ["ccv-consumer", "*"], ["provider", "*"], ["consumer", "*"]]

[chains.packet_filter.min_fees]

[chains.address_type]
derivation = "cosmos"

[[chains]]
id = "consumer2"                     
type = "CosmosSdk"
rpc_addr = "http://localhost:26657"       
grpc_addr = "http://localhost:9090"   
rpc_timeout = "10s"
account_prefix = "consumer"           
key_name = "dex-key"         
key_store_type = "Test"
store_prefix = "ibc"
default_gas = 2000000
max_gas = 4000000
gas_multiplier = 1.5
max_msg_num = 30
max_tx_size = 180000
max_grpc_decoding_size = 33554432
clock_drift = "5s"
max_block_time = "30s"
memo_prefix = ""
sequential_batch_tx = false
ccv_consumer_chain = true

[chains.event_source]
mode = "push"
url = "ws://localhost:26657/websocket"
batch_delay = "500ms"

[chains.trust_threshold]
numerator = "1"
denominator = "3"

[chains.gas_price]
price = 0.1
denom = "stake"

[chains.packet_filter]
policy = "allow"
list = [["transfer", "*"], ["ccv-provider", "*"], ["ccv-consumer", "*"], ["provider", "*"], ["consumer", "*"]]

[chains.packet_filter.min_fees]

[chains.address_type]
derivation = "cosmos"

Any ideas why this is happening? Is there some weird caching going on where if you configured a connection (or channel) once than it gets a mismatch or something the second time? How could i debug this further?

Any ideas are very welcome 🙏🏻