Impossible to create IBC channels after on-chain upgrade

[rmontagnin]

Summary of Bug

After running an on-chain upgrade, it is impossible to create new IBC channels.

Version

v0.42.7

Steps to Reproduce

1. Start a localnet and create some IBC channels with other chains.
2. Perform a simple on-chain upgrade (no need to have consensus-breaking changes)
3. Try to create a new IBC channel.

The following error will be displayed:

failed to execute message; message index: 0: channel handshake open init failed: caller does not own port capability for port ID <Port ID>: invalid port

This happens with both custom modules and the transfer module.

The error started showing up only after the upgrade, while before the upgrade itself it was possible to create correct channels that worked without any issue.

The upgrade has been tested on our testnet and did not have any change apart from the IBC version checking being deleted since we do not need it.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[yihuang]

Is the "simple on-chain upgrade” is an export upgrade, or upgrade module upgrade?

[rmontagnin]

Is the "simple on-chain upgrade” is an export upgrade, or upgrade module upgrade?

It used the x/upgrade module without performing any consensus-breaking changes. It only removed the IBC version checking from our custom module (as you can see here)

[AdityaSripal]

Do you have the following lines in your app.go?

https://github.com/cosmos/gaia/blob/main/app/app.go#L455

[rmontagnin]

Do you have the following lines in your app.go?

https://github.com/cosmos/gaia/blob/main/app/app.go#L455

Of course. You can see the post-upgrade files here

[AdityaSripal]

Hmm that should have reinitialized the in-memory capabilities. I will test this issue, thanks for the writeup

[AdityaSripal]

Hi @riccardom ,

@colin-axner just tried to reproduce your issue and did not run into any issues.

1. We used gaiad-manager to set up two chains on v0.42.7.
2. Setup a channel before upgrade and sent packet
3. Make regular simple upgrade using upgrade module
4. Setup a new channel and sent packet

There was no errors with this workflow. Could you provide more information on what you did beyond the steps above?

[rmontagnin]

@AdityaSripal Aside from those operations, we did nothing. We had a running chain with different channels opened, we performed the upgrade and I tried to create a new channel after the upgrade. If I query the previously created channels I can see them being there, but if I try to create a new one I end up with the above mentioned error.

For a better understanding, we had a chain running with the following details:

• Binary: Desmos v0.17.2
• Genesis here

and we upgraded to use Desmos v0.17.4.

The upgrade was successful and most of the validators used cosmovisor

[rmontagnin]

@AdityaSripal @colin-axner If you have any idea how I might debug this (maybe something particular to look at?) let me know please cause this is currently blocking our testnet from having any integration with other chains over IBC

[AdityaSripal]

1. Could you double check that you are bound to the port that you are trying to create a channel with?
2. If you have access to node logs, could you modify the binary to print out some info that might be useful?

If so, could you print the following info:

• A flag indicating we've entered this if-block: https://github.com/cosmos/cosmos-sdk/blob/v0.42.7/x/capability/keeper/keeper.go#L356
• Could you print the index and capOwners in this for loop: https://github.com/cosmos/cosmos-sdk/blob/v0.42.7/x/capability/keeper/keeper.go#L356
• Print forward and reverse keys here: https://github.com/cosmos/cosmos-sdk/blob/v0.42.7/x/capability/keeper/keeper.go#L193

If you could then show the outputs here that would be great. Ideally we want to see that on a node restart these are called and correctly filled in.

[AdityaSripal]

This was already tested before IBC launch and we haven't changed that code. Don't know exactly what could be the issue but it would be useful to at least rule out some things

[AdityaSripal]

Also are you able to send/receive packets on an already existing channel?

[AdityaSripal]

Also have you been using statesync in any way?