ETH/ETC - Some found blocks flagged as `Uncle` are actually `Orphaned` and this scenario breaks the payment system

[blackmennewstyle]

Hi everyone,

I'm in the process of implementing ETC, i have it hashing away, finding blocks and sending payments on ETC testnet aka Mordor. However, i found myself facing a case where a found block is flagged by the software as Uncle but it is actually an Orphaned (Someone else on the network beat me to it). And it totally breaks the payment process:

[2023-01-12 23:43:27.4207] [I] [PayoutManager] Processing payments for pool etc1 
[2023-01-12 23:43:27.7151] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973152 
[2023-01-12 23:43:27.7151] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973152 
[2023-01-12 23:43:27.7151] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973154 
[2023-01-12 23:43:27.7182] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973154 
[2023-01-12 23:43:27.7182] [I] [etc1] [Ethereum Payout Handler] Found another uncle from block 7973153 in the DB. Continuing search for uncle. 
[2023-01-12 23:43:27.7182] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973160 
[2023-01-12 23:43:27.7182] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973160 
[2023-01-12 23:43:27.7182] [I] [etc1] [Ethereum Payout Handler] Unlocked uncle for block 7973160 at height 7973159 worth 0.125 ETC 
[2023-01-12 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973152 
[2023-01-12 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973152 
[2023-01-12 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973154 
[2023-01-12 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973154 
[2023-01-12 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Found another uncle from block 7973153 in the DB. Continuing search for uncle. 
[2023-01-12 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973160 
[2023-01-12 23:43:27.7246] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973160 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked uncle for block 7973160 at height 7973159 worth 0.125 ETC 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973152 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973152 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973154 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973154 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Found another uncle from block 7973153 in the DB. Continuing search for uncle. 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973160 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973160 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked uncle for block 7973160 at height 7973159 worth 0.125 ETC 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973167 worth 4 ETC 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973173 worth 4 ETC 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973176 worth 4 ETC 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973178 worth 4 ETC 
[2023-01-12 23:43:27.7262] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973181 worth 4 ETC 
[2023-01-12 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973185 worth 4 ETC 
[2023-01-12 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973187 worth 4 ETC 
[2023-01-12 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973192 worth 4 ETC 
[2023-01-12 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973193 worth 4 ETC 
[2023-01-12 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973199 worth 4 ETC 
[2023-01-12 23:43:27.7314] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973202 worth 4 ETC 
[2023-01-12 23:43:27.7416] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973154 
[2023-01-12 23:43:27.7416] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973154 
[2023-01-12 23:43:27.7424] [I] [etc1] [Ethereum Payout Handler] Found another uncle from block 7973153 in the DB. Continuing search for uncle. 
[2023-01-12 23:43:27.7424] [I] [etc1] [Ethereum Payout Handler] Fetching 1 uncles for block 7973160 
[2023-01-12 23:43:27.7424] [I] [etc1] [Ethereum Payout Handler] Fetched 1 uncles for block 7973160 
[2023-01-12 23:43:27.7424] [I] [etc1] [Ethereum Payout Handler] Unlocked uncle for block 7973160 at height 7973159 worth 0.125 ETC 
[2023-01-12 23:43:27.7424] [I] [etc1] [Ethereum Payout Handler] Unlocked block 7973207 worth 4 ETC 
[2023-01-12 23:43:27.7424] [I] [PayoutManager] Processing payments for pool etc1, block 7973159 
[2023-01-12 23:43:27.7560] [I] [PPLNS Payment] Fetching page 0 of shares for pool etc1, block 7973159 
[2023-01-12 23:43:27.7560] [I] [PPLNS Payment] Balance-calculation for pool etc1, block 7973159 completed with accumulated score 0.5 (100%) 
[2023-01-12 23:43:27.7560] [I] [PPLNS Payment] Crediting 0x0b8e5bafb7758872c3e237a172ec04b4206f1a5f with 0.12238 ETC for 429.5M (429496729.6) shares for block 7973159 
[2023-01-12 23:43:27.7586] [I] [PPLNS Payment] Fetching page 0 of discarded shares for pool etc1, block 7973159 
[2023-01-12 23:43:27.7586] [I] [PPLNS Payment] 429.5M (429496729.6) total discarded shares, block 7973159 
[2023-01-12 23:43:27.7586] [I] [PPLNS Payment] 0x0b8e5bafb7758872c3e237a172ec04b4206f1a5f = 429.5M (429496729.6) discarded shares, block 7973159 
[2023-01-12 23:43:27.7586] [I] [PPLNS Payment] Deleting 1 discarded shares before 2023-01-12T23:27:44.5990000Z 
[2023-01-12 23:43:27.7586] [I] [PPLNS Payment] 429.5M (429496729.6) shares contributed to a total payout of 0.12238 ETC (100.00% of block reward) to 1 addresses 
[2023-01-12 23:43:27.7623] [I] [PayoutManager] Processing payments for pool etc1, block 7973159 
[2023-01-12 23:43:27.7623] [I] [PPLNS Payment] Fetching page 0 of shares for pool etc1, block 7973159 
[2023-01-12 23:43:27.7623] [I] [PPLNS Payment] Balance-calculation for pool etc1, block 7973159 completed with accumulated score 0.5 (100%) 
[2023-01-12 23:43:27.7623] [I] [PPLNS Payment] Crediting 0x0b8e5bafb7758872c3e237a172ec04b4206f1a5f with 0.12238 ETC for 429.5M (429496729.6) shares for block 7973159 
[2023-01-12 23:43:27.7623] [I] [PPLNS Payment] Fetching page 0 of discarded shares for pool etc1, block 7973159 
[2023-01-12 23:43:27.7623] [I] [PPLNS Payment] 429.5M (429496729.6) total discarded shares, block 7973159 
[2023-01-12 23:43:27.7623] [I] [PPLNS Payment] 0x0b8e5bafb7758872c3e237a172ec04b4206f1a5f = 429.5M (429496729.6) discarded shares, block 7973159 
[2023-01-12 23:43:27.7623] [I] [PPLNS Payment] Deleting 1 discarded shares before 2023-01-12T23:27:57.9027720Z 
[2023-01-12 23:43:27.7623] [I] [PPLNS Payment] 429.5M (429496729.6) shares contributed to a total payout of 0.12238 ETC (100.00% of block reward) to 1 addresses 
[2023-01-12 23:43:27.7775] [E] [PayoutManager] [etc1] Payment processing failed System.InvalidOperationException: This NpgsqlTransaction has completed; it is no longer usable.
   at Npgsql.NpgsqlTransaction.CheckReady()
   at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlTransaction.Rollback()
   at Miningcore.Extensions.ConnectionFactoryExtensions.RunTx(IConnectionFactory factory, Func`3 action, Boolean autoCommit, IsolationLevel isolation) in /home/ceedii/miningcore/src/Miningcore/Extensions/ConnectionFactoryExtensions.cs:line 61
   at Miningcore.Payments.PayoutManager.UpdatePoolBalancesAsync(IMiningPool pool, PoolConfig poolConfig, IPayoutHandler handler, IPayoutScheme scheme, CancellationToken ct) in /home/ceedii/miningcore/src/Miningcore/Payments/PayoutManager.cs:line 164
   at Miningcore.Payments.PayoutManager.ProcessPoolsAsync(CancellationToken ct) in /home/ceedii/miningcore/src/Miningcore/Payments/PayoutManager.cs:line 104    at Npgsql.NpgsqlTransaction.CheckReady()
   at Npgsql.NpgsqlTransaction.Rollback(Boolean async, CancellationToken cancellationToken)
   at Npgsql.NpgsqlTransaction.Rollback()
   at Miningcore.Extensions.ConnectionFactoryExtensions.RunTx(IConnectionFactory factory, Func`3 action, Boolean autoCommit, IsolationLevel isolation) in /home/ceedii/miningcore/src/Miningcore/Extensions/ConnectionFactoryExtensions.cs:line 61
   at Miningcore.Payments.PayoutManager.UpdatePoolBalancesAsync(IMiningPool pool, PoolConfig poolConfig, IPayoutHandler handler, IPayoutScheme scheme, CancellationToken ct) in /home/ceedii/miningcore/src/Miningcore/Payments/PayoutManager.cs:line 164
   at Miningcore.Payments.PayoutManager.ProcessPoolsAsync(CancellationToken ct) in /home/ceedii/miningcore/src/Miningcore/Payments/PayoutManager.cs:line 104

The faulty block in the log is 7973153, it is declared in the database as Uncle but it is actually an Orphaned:
Database

23 | etc1 | 7973153 | 769441 | confirmed | uncle | 1 | NULL |   | 0x0b8e5bafb7758872c3e237a172ec04b4206f1a5f | 0.125000000000 | NULL | NULL | 2023-01-12 23:26:33.701667+00
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --

ETC testnet MORDOR block explorer - https://blockscout.com/etc/mordor/block/7973153

As you can see, it's not an Uncle but an Orphaned, someone else beat me to it, my pool wallet address is 0x421Afb2ce225D3A2d3DD6e63Fe57E124B40e20Af, which is not the address which found that block :)

[blackmennewstyle]

I managed to solve half of my issue, which are indeed blocks recorded in my database, which are indeed Uncle but not mined by my pool wallet but someone else at the same time on the network, with following the advice of the dev from that PR #1501 👍🏽

However, the error NpgsqlTransaction has completed; it is no longer usable. still persists, i even tried to add the timeout in the config.json for PostreSQL.

I have a suspicion that it is a RAM space issue, since i am running everything on a laptop with only 8GB shared with the APU. When everything run, i have barely 1GB left.

I will continue to test and report my findings.

[blackmennewstyle]

After further investigation and testing, i can confirm that the suggestion in PR #1501 from @jon4hz to increase BlockSearchOffset works perfectly and seems to cover all the inherent cases withUncle, including my issue with Uncle recorded in the database but mined by another pool wallet on the network 👍 Only one question remains on my side, what if the two uncles are very very far away from each other, in my case on the ETC Mordor testnet, i had to increase BlockSearchOffset up to 3200 🔥 🚀

There is one last case scenario which is very annoying on the ETC Mordor testnet, some Uncle blocks are duplicated in the blockchain:

And they are also logically inserted in the database:

In this case scenario, the payout will be totally broken: This NpgsqlTransaction has completed; it is no longer usable. And it will not resume as long as a manual deletion of the duplicated block won't be performed in the database.

Lastly, RAM space left is not the issue at all here, i was able to run the pool and sending payouts with less than 1Go RAM left.

[jon4hz]

Your last issue was probably caused because a block mined by your pool included an orphan, which was also mined by you. At least this is what it looks like to me from what I see in the screenshot of the explorer.

As it seems this use case is not supported at the moment.
I assume miningcore just detects the block and tries to insert it a second time, when looking for that uncle. This violates the constraint CONSTRAINT BLOCKS_POOL_HEIGHT UNIQUE (poolid, blockheight, type) DEFERRABLE INITIALLY DEFERRED leading to the npgsql error.

You probably have to tune the ClassifyBlocksAsync method and add a check to look in the db for existing blocks with the same height. Something similar to what I did in #1501 but for blocks of type Block and not just Uncle.

[blackmennewstyle]

Thanks for the reply @jon4hz, i know you are a busy man :)

According to the ETC Mordor testnet explorer, both blocks has been actually mined by my pool wallet :/

Here the link to my pool wallet https://blockscout.com/etc/mordor/address/0x421Afb2ce225D3A2d3DD6e63Fe57E124B40e20Af/validations#address-tabs, just search for block 7980798, they have the same block height but NOT the same block hash.

The first one, declared as type block

The second one, declared as type uncle

According to the ETC Mordor testnet block explorer, they both have been found by my pool wallet 🤯

I got that answer from a dev in the ETC discord group:

[jon4hz]

It might be a rare case but as you can see, it is possible that a block mined by you includes an uncle block which was also mined by you.

Miningcore just doesn't seem to handle that scenario correctly.

[blackmennewstyle]

It might be a rare case but as you can see, it is possible that a block mined by you includes an uncle block which was also mined by you.

Miningcore just doesn't seem to handle that scenario correctly.

Yes, definitely, but your method GetBlockByHeightAsync goes against the concept of Uncle BlockChain and even the database design:

public async Task<Block> GetBlockByHeightAsync(IDbConnection con, string poolId, long height)
    {
        const string query = @"SELECT * FROM blocks WHERE poolid = @poolId AND blockheight = @height";

        var entity = await con.QuerySingleOrDefaultAsync<Entities.Block>(new CommandDefinition(query, new { poolId, height }));

        return entity == null ? null : mapper.Map<Block>(entity);
    }

You decided that a block is unique based on the poolId and the blockheight. It's not not. it's poolId, blockheight and type. My bug and this bug #1080 (comment) are related, we trigger an error through your method since there is two duplicated blockHeight in our database . Your method should have been something like that, especially since from what i see, you are using it only for finding uncle:

public async Task<Block> GetBlockByPoolHeightAndTypeAsync(IDbConnection con, string poolId, long height, string type)
    {
        const string query = @"SELECT * FROM blocks WHERE poolid = @poolId AND blockheight = @height AND type = @type";

        return (await con.QueryAsync<Entities.Block>(query, new
        { 
            poolId,
            height,
            type
        }))
            .Select(mapper.Map<Block>)
            .FirstOrDefault();
    }

[jon4hz]

Yes, definitely, but your method GetBlockByHeightAsync goes against the concept of Uncle BlockChain and even the database design

It does not. My PR simply aimed to fix another bug for which it wasn't necessary to check for a specific block type in the db request.

Just changing that method most likely wont fix your issue, however it might be necessary to cover your use-case.

[blackmennewstyle]

Yes, definitely, but your method GetBlockByHeightAsync goes against the concept of Uncle BlockChain and even the database design

It does not. My PR simply aimed to fix another bug for which it wasn't necessary to check for a specific block type in the db request.

Just changing that method most likely wont fix your issue, however it might be necessary to cover your use-case.

I should have been more specific apparently because i thought you aware of a choice made in the function GetBlockByHeightAsync but let's analyze it. It uses a C# function called QuerySingleOrDefaultAsync. A quick search warns about something very important and non-negligible:

Anyway, i got it fixed and it does not bother me anymore and the payment system can resume with the mitigation i made before the uncle process.

Duplicate blockHeight can be processed properly.

There is still an inherent issue with database timeout which happens when there is a lot of pending shares with a lot of Uncle in between but i don't think there is nothing which can really be done about it. But from i what i see so far, the blocks are unlocked properly and the miner wallet balance is also updated properly.

[jon4hz]

I see, I wasn't aware that QuerySingleOrDefaultAsync throws an exception if there are multiple results.

So was specifying the blockType in the query all it took to fix the bug?

[jon4hz]

Nevermind, saw the commits in #1514, amazing work :D

[blackmennewstyle]

I see, I wasn't aware that QuerySingleOrDefaultAsync throws an exception if there are multiple results.

So was specifying the blockType in the query all it took to fix the bug?

Hey no worry, i understood you were not aware, i actually found out about that SQL function kinda luckily when i was looking to make my patch. I thought you knew since you seem to be a real MVP when it comes to miningcore lol

And thanks for the kind words! Have a great weekend ahead :)