write a simulation for POA to get off SDK fork

[reecepbcups]

write simulator for POA before he merges cosmos/cosmos-sdk#20059

another idea is to remove app.SlashingKeeper.Hooks(), from StakingKeeper hooks?

other idea is POA on new val, set signing info -1 block. then see if sdk patch required

[reecepbcups]

May not get us off a fork anyways since andrew needs PostSetupStandalone w/ app param in v50 anyways. Which cosmos/cosmos-sdk@7b35e36 only had part of.

Either way simulation is a good thing to add

[fmorency]

Hey, I started writing simulation support in Manifest and encountered staking actions are now allowed on this chain.

I'm happy to help!

[reecepbcups]

This is a POA error in the ante. You can not directly use the StakingKeeper as flow is expected through POA.

May not be needed i have a POA refactor idea I am working on to remove some of the pitfalls with SDK stuff I ran into

[fmorency]

Sounds good.

You might already know this, but this repo's (poa) simulation produces the same error.

[reecepbcups]

Yes. I did not re-wire any of the sims due to not using it. The new design will be many times easier to sim as well (hopefully)

[fmorency]

Can I help with anything @reecepbcups?

[fmorency]

Hey @reecepbcups, I just wanted to follow up on this issue quickly. Is there a workaround that I could use to write simulator tests on our POA chain?

[fmorency]

I've made some progress on this. I was able to run the simulator by disabling x/staking messages.

1. Create a sim_params.json file with the following content

   {
     "op_weight_msg_delegate": 0,
     "op_weight_msg_undelegate": 0,
     "op_weight_msg_begin_redelegate": 0,
     "op_weight_msg_cancel_unbonding_delegation": 0,
     "op_weight_msg_create_validator": 0,
     "op_weight_msg_edit_validator": 0
   }

2. Run the application simulator with

   go test ./app -run TestFullAppSimulation \
     -Enabled=true \
     -NumBlocks=200 \
     -Genesis= \
     -Verbose=true \
     -Commit=true \
     -Period=5 \
     -Params=/path/to/sim_params.json \
     -v \
     -timeout 24h

This approach will set the weight of the x/staking module message to 0, effectively disabling that module simulation.

Note: The NewPOADisableStakingDecorator allows staking.MsgEditValidator messages but we should disable it for simulation as the commission rate never gets updated.

[fmorency]

@reecepbcups

I made some progress on the PR. MsgCreateValidator and MsgRemovePendingValidator are working just fine.

I'm currently implementing MsgSetPower and I am getting a panic after a while when executing the following message

msg := poatypes.MsgSetPower{
	Sender:           admin,
	ValidatorAddress: validator.OperatorAddress,
	Power:            1_000_000,
	Unsafe:           false,
}

The panic log is

:42PM ERR panic recovered in runTx err="recovered: calculated final stake for delegator cosmos1mf03f7gqya7u98fhmfh4kmj6qhsafefjdesges greater than current stake\n\tfinal stake:\t161933052469.000000000000000000\n\tcurrent stake:\t1000000.000000000000000000\nstack:\ngoroutine 13 [running]:\nruntime/debug.Stack()\n\t/home/fmorency/go1.22.2/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.func5({0x26cb360, 0xc0017fe420})\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/recovery.go:74 +0x25\ngithub.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.newRecoveryMiddleware.func7({0x26cb360?, 0xc0017fe420?})\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/recovery.go:42 +0x2d\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x26cb360, 0xc0017fe420}, 0x0?)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/recovery.go:31 +0x2f\ngithub.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x26cb360, 0xc0017fe420}, 0x0?)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/recovery.go:36 +0x53\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/baseapp.go:837 +0x156\npanic({0x26cb360?, 0xc0017fe420?})\n\t/home/fmorency/go1.22.2/src/runtime/panic.go:770 +0x132\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x3a80b80, 0xc000cbaf30}, {0x3acee90, 0xc001029aa0}, {0x3ab9660, 0xc000cc8f00}, {0x7fa490023ba0, 0xc000f5c508}, {0x3acd288, 0xc001027e00}, ...}, ...)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/keeper/delegation.go:177 +0x894\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x3a80b80, 0xc000cbaf30}, {0x3acee90, 0xc001029aa0}, {0x3ab9660, 0xc000cc8f00}, {0x7fa490023ba0, 0xc000f5c508}, {0x3acd288, 0xc001027e00}, ...}, ...)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/keeper/delegation.go:222 +0x26b\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.WithdrawDelegationRewards({{0x3a80b80, 0xc000cbaf30}, {0x3acee90, 0xc001029aa0}, {0x3ab9660, 0xc000cc8f00}, {0x7fa490023ba0, 0xc000f5c508}, {0x3acd288, 0xc001027e00}, ...}, ...)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/keeper/keeper.go:127 +0x15e\ngithub.com/cosmos/cosmos-sdk/x/distribution/keeper.msgServer.WithdrawDelegatorReward({{{0x3a80b80, 0xc000cbaf30}, {0x3acee90, 0xc001029aa0}, {0x3ab9660, 0xc000cc8f00}, {0x7fa490023ba0, 0xc000f5c508}, {0x3acd288, 0xc001027e00}, ...}}, ...)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/keeper/msg_server.go:59 +0x269\ngithub.com/cosmos/cosmos-sdk/x/distribution/types._Msg_WithdrawDelegatorReward_Handler.func1({0x3ab57f8?, 0xc002b65020?}, {0x2af3540?, 0xc002b547c0?})\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/types/tx.pb.go:1173 +0xcb\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2.1({0x3ab56a8, 0xc002b6ae08}, {0xc001a6de00?, 0x411265?}, 0x358?, 0xc000cc52f0)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/msg_service_router.go:175 +0x8a\ngithub.com/cosmos/cosmos-sdk/x/distribution/types._Msg_WithdrawDelegatorReward_Handler({0x2c33720, 0xc0006c4a00}, {0x3ab56a8, 0xc002b6ae08}, 0x3429498, 0xc002b549c0)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/types/tx.pb.go:1175 +0x143\ngithub.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2({{0x3ab5600, 0x553c6c0}, {0x3acf750, 0xc0018973c0}, {{0x0, 0x0}, {0x2c5b432, 0xe}, 0x7, {0x0, ...}, ...}, ...}, ...)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/msg_service_router.go:198 +0x37f\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x3ab5600, 0x553c6c0}, {0x3acf750, 0xc0018973c0}, {{0x0, 0x0}, {0x2c5b432, 0xe}, 0x7, ...}, ...}, ...)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/baseapp.go:1010 +0x1e6\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc001016908, 0x7, {0xc00404bb80, 0x149, 0x149})\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/baseapp.go:948 +0x114c\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).SimDeliver(0xc001016908, 0x2c5b432?, {0x3a926e8?, 0xc00156b680?})\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/test_helpers.go:38 +0xa5\ngithub.com/cosmos/cosmos-sdk/x/simulation.GenAndDeliverTx({0xc002796ab0, 0xc001016908, {0x3acd648, 0xc001977c40}, 0x0, {0x3a9e470, 0xc002b54640}, {0xc000cc4b70, 0x1, 0x1}, ...}, ...)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/simulation/util.go:117 +0x2f2\ngithub.com/cosmos/cosmos-sdk/x/simulation.GenAndDeliverTxWithRandFees({0xc002796ab0, 0xc001016908, {0x3acd648, 0xc001977c40}, 0x0, {0x3a9e470, 0xc002b54640}, {0xc000cc4b70, 0x1, 0x1}, ...})\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/simulation/util.go:96 +0x2d1\ngithub.com/cosmos/cosmos-sdk/x/distribution/simulation.WeightedOperations.SimulateMsgWithdrawDelegatorReward.func6(_, _, {{0x3ab5600, 0x553c6c0}, {0x3acf750, 0xc0019292c0}, {{0x0, 0x0}, {0x2c5b432, 0xe}, ...}, ...}, ...)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/simulation/operations.go:171 +0xad0\ngithub.com/cosmos/cosmos-sdk/x/simulation.createBlockSimulator.func1(_, _, {{0x3ab5600, 0x553c6c0}, {0x3acf750, 0xc0019292c0}, {{0x0, 0x0}, {0x2c5b432, 0xe}, ...}, ...}, ...)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/simulation/simulate.go:326 +0x512\ngithub.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed({0x3af52b0, 0xc00103cb60}, {_, _}, _, _, _, {0xc001bf6008, 0x34, 0x3f}, ...)\n\t/home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/simulation/simulate.go:210 +0x19e4\ngithub.com/strangelove-ventures/poa/simapp.TestFullAppSimulation(0xc00103cb60)\n\t/home/fmorency/dev/poa/simapp/sim_test.go:82 +0x816\ntesting.tRunner(0xc00103cb60, 0x34289b8)\n\t/home/fmorency/go1.22.2/src/testing/testing.go:1689 +0xfb\ncreated by testing.(*T).Run in goroutine 1\n\t/home/fmorency/go1.22.2/src/testing/testing.go:1742 +0x390\n: panic"
Logs to writing to /home/fmorency/.simapp/simulations/1718394173108.log
    simulate.go:335: error on block  7/2000, operation (47/273) from x/distribution:
        recovered: calculated final stake for delegator cosmos1mf03f7gqya7u98fhmfh4kmj6qhsafefjdesges greater than current stake
                final stake:    161933052469.000000000000000000
                current stake:  1000000.000000000000000000
        stack:
        goroutine 13 [running]:
        runtime/debug.Stack()
                /home/fmorency/go1.22.2/src/runtime/debug/stack.go:24 +0x5e
        github.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.func5({0x26cb360, 0xc0017fe420})
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/recovery.go:74 +0x25
        github.com/cosmos/cosmos-sdk/baseapp.NewBaseApp.newDefaultRecoveryMiddleware.newRecoveryMiddleware.func7({0x26cb360?, 0xc0017fe420?})
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/recovery.go:42 +0x2d
        github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x26cb360, 0xc0017fe420}, 0x0?)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/recovery.go:31 +0x2f
        github.com/cosmos/cosmos-sdk/baseapp.processRecovery({0x26cb360, 0xc0017fe420}, 0x0?)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/recovery.go:36 +0x53
        github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx.func1()
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/baseapp.go:837 +0x156
        panic({0x26cb360?, 0xc0017fe420?})
                /home/fmorency/go1.22.2/src/runtime/panic.go:770 +0x132
        github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x3a80b80, 0xc000cbaf30}, {0x3acee90, 0xc001029aa0}, {0x3ab9660, 0xc000cc8f00}, {0x7fa490023ba0, 0xc000f5c508}, {0x3acd288, 0xc001027e00}, ...}, ...)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/keeper/delegation.go:177 +0x894
        github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x3a80b80, 0xc000cbaf30}, {0x3acee90, 0xc001029aa0}, {0x3ab9660, 0xc000cc8f00}, {0x7fa490023ba0, 0xc000f5c508}, {0x3acd288, 0xc001027e00}, ...}, ...)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/keeper/delegation.go:222 +0x26b
        github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.WithdrawDelegationRewards({{0x3a80b80, 0xc000cbaf30}, {0x3acee90, 0xc001029aa0}, {0x3ab9660, 0xc000cc8f00}, {0x7fa490023ba0, 0xc000f5c508}, {0x3acd288, 0xc001027e00}, ...}, ...)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/keeper/keeper.go:127 +0x15e
        github.com/cosmos/cosmos-sdk/x/distribution/keeper.msgServer.WithdrawDelegatorReward({{{0x3a80b80, 0xc000cbaf30}, {0x3acee90, 0xc001029aa0}, {0x3ab9660, 0xc000cc8f00}, {0x7fa490023ba0, 0xc000f5c508}, {0x3acd288, 0xc001027e00}, ...}}, ...)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/keeper/msg_server.go:59 +0x269
        github.com/cosmos/cosmos-sdk/x/distribution/types._Msg_WithdrawDelegatorReward_Handler.func1({0x3ab57f8?, 0xc002b65020?}, {0x2af3540?, 0xc002b547c0?})
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/types/tx.pb.go:1173 +0xcb
        github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2.1({0x3ab56a8, 0xc002b6ae08}, {0xc001a6de00?, 0x411265?}, 0x358?, 0xc000cc52f0)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/msg_service_router.go:175 +0x8a
        github.com/cosmos/cosmos-sdk/x/distribution/types._Msg_WithdrawDelegatorReward_Handler({0x2c33720, 0xc0006c4a00}, {0x3ab56a8, 0xc002b6ae08}, 0x3429498, 0xc002b549c0)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/types/tx.pb.go:1175 +0x143
        github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).registerMsgServiceHandler.func2({{0x3ab5600, 0x553c6c0}, {0x3acf750, 0xc0018973c0}, {{0x0, 0x0}, {0x2c5b432, 0xe}, 0x7, {0x0, ...}, ...}, ...}, ...)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/msg_service_router.go:198 +0x37f
        github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x3ab5600, 0x553c6c0}, {0x3acf750, 0xc0018973c0}, {{0x0, 0x0}, {0x2c5b432, 0xe}, 0x7, ...}, ...}, ...)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/baseapp.go:1010 +0x1e6
        github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc001016908, 0x7, {0xc00404bb80, 0x149, 0x149})
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/baseapp.go:948 +0x114c
        github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).SimDeliver(0xc001016908, 0x2c5b432?, {0x3a926e8?, 0xc00156b680?})
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/baseapp/test_helpers.go:38 +0xa5
        github.com/cosmos/cosmos-sdk/x/simulation.GenAndDeliverTx({0xc002796ab0, 0xc001016908, {0x3acd648, 0xc001977c40}, 0x0, {0x3a9e470, 0xc002b54640}, {0xc000cc4b70, 0x1, 0x1}, ...}, ...)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/simulation/util.go:117 +0x2f2
        github.com/cosmos/cosmos-sdk/x/simulation.GenAndDeliverTxWithRandFees({0xc002796ab0, 0xc001016908, {0x3acd648, 0xc001977c40}, 0x0, {0x3a9e470, 0xc002b54640}, {0xc000cc4b70, 0x1, 0x1}, ...})
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/simulation/util.go:96 +0x2d1
        github.com/cosmos/cosmos-sdk/x/distribution/simulation.WeightedOperations.SimulateMsgWithdrawDelegatorReward.func6(_, _, {{0x3ab5600, 0x553c6c0}, {0x3acf750, 0xc0019292c0}, {{0x0, 0x0}, {0x2c5b432, 0xe}, ...}, ...}, ...)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/distribution/simulation/operations.go:171 +0xad0
        github.com/cosmos/cosmos-sdk/x/simulation.createBlockSimulator.func1(_, _, {{0x3ab5600, 0x553c6c0}, {0x3acf750, 0xc0019292c0}, {{0x0, 0x0}, {0x2c5b432, 0xe}, ...}, ...}, ...)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/simulation/simulate.go:326 +0x512
        github.com/cosmos/cosmos-sdk/x/simulation.SimulateFromSeed({0x3af52b0, 0xc00103cb60}, {_, _}, _, _, _, {0xc001bf6008, 0x34, 0x3f}, ...)
                /home/fmorency/go/pkg/mod/github.com/rollchains/cosmos-sdk@v0.50.6/x/simulation/simulate.go:210 +0x19e4
        github.com/strangelove-ventures/poa/simapp.TestFullAppSimulation(0xc00103cb60)
                /home/fmorency/dev/poa/simapp/sim_test.go:82 +0x816
        testing.tRunner(0xc00103cb60, 0x34289b8)
                /home/fmorency/go1.22.2/src/testing/testing.go:1689 +0xfb
        created by testing.(*T).Run in goroutine 1
                /home/fmorency/go1.22.2/src/testing/testing.go:1742 +0x390
        : panic [rollchains/cosmos-sdk@v0.50.6/baseapp/recovery.go:72]
        Comment: unable to deliver tx

You can reproduce by running

go test ./simapp -run TestFullAppSimulation \
    -Enabled=true \
    -NumBlocks=2000 \
    -Genesis= \
    -Verbose=true \
    -Commit=true \
    -Period=5 \
    -Params=/path/to/sim_params.json \
    -Seed=12 \
    -v \
    -timeout 24h

against my PR at commit 5558615.

Any idea what's going on? Do I need to craft a non-random genesis file to make it work?

[reecepbcups]

@fmorency just got back from travel so I do not have much time to look into this yet. Just throwing down quick ideas:

1)) rewards auto claim in cosmos (changing delegations auto claims rewards) so it may be changing the amounts since I have Distribution & Mint enabled in the app (doubt it but yea it can get you)

2. Also note that stake updates take height + 2 blocks to persist. So anytime you set, you must wait 2 blocks. Failure to do so will incorrectly show the prev value

[fmorency]

@reecepbcups Thank you for your response.

I disabled MsgSetWithdrawAddress, MsgWithdrawDelegationReward and MsgWithdrawValidatorCommission from the x/distribution module as per your suggestion in 1) and the simulation (2000 blocks) didn't crash.

I don't fully understand the problem but I'm concerned that we could get such a crash in production.

WDYT?

[fmorency]

Hey @reecepbcups,

I'm currently debugging MsgSetPower in the simulator and I was wondering if you could provide some clarifications about block power.

The function k.GetStakingKeeper().GetLastTotalPower() and k.GetCachedBlockPower() return different values in SimulateMsgSetPower(). I also get a 3rd value if I compute the total power from all the validators.

My feeling is that I need to use k.GetCachedBlockPower() but I would like to better understand the block power flow.

Thanks!