XMR Block 1589929 stuck at 0% confirmed

According to ChainRadar that block was found (UTC) 2018-06-07 15:43:46  (https://chainradar.com/xmr/block/1589929).  Wouldn't that be the day before our pool is saying we found the block?

Looking into it.

Humm...  No word yet...  This cannot be good...

Oh no, I used up my free trial on Azure for this block! Mined with 23 Virtual Machines, four vcpus in every data center around the world. Got about 4.5 kH/s! (Two days until the trial credits are gone) I’m so very close to payment, too. I got 0.09245, only needed  0.00755 more. Very bad luck! Or bad choice.

Any news Oliver? It doesn't seem to be an orphan as per blockradar.

I would appreciate knowing whether this block will be rewarded to the pool miners, not really bothered with the explanation at this point.

Not sure if this can be done, Oliver, but I'm just hobby mining over here for the learnz and lulz, and while I would like to know what happened just out of sheer curiosity, I'll gladly kick my shares from that round out to one or more of the bigger miners with more riding on that block, especially if it can get a payout for someone.  I have no idea if that's possible...

The problem with Monero Block 1589929 was a combination with a never before seen sync issue of the daemon and a bug in the pool's block unlocker code. Sorry that it took so long to figure it out.

How come the block shows as "not orphaned" on chainradar? https://chainradar.com/xmr/block/1589929

There's of course a non-orphaned block 1589929 shown on chainradar. Unfortunately the non-orphaned version has a hash of bb4e624526aa290b3149fa241ae8a2fffd4ae176f2afcfcc239f2b695a29fbba. The one mined on our pool has a different one which made it orphan in the first place. This is block 1589929 how its been submitted by the miner that found the orphan:

miningcore=# select * from blocks where poolid = 'xmr1' and blockheight = 1589929;

  id   | poolid | blockheight | networkdifficulty |  status  | type | confirmationprogress |      effort      |                   transactionconfirmationdata                    |                                             miner                                              |     reward     | source  |                               hash                               |          created
-------+--------+-------------+-------------------+----------+------+----------------------+------------------+------------------------------------------------------------------+-------------------------------------------------------------------------------------------------+----------------+---------+------------------------------------------------------------------+----------------------------

 12006 | xmr1   |     1589929 |       52774831374 | orphaned |      |                    1 | 0.17212013521421 | 5294849d7c578227966a2fb5cb6a4155038464c57c5810d1820b27df826b2487 | 47qeoHL28ywM9e2xHJosxSJ8trM4v3GWKCTUWMYwe1eE46o5NR3UPvWN11PLNxGnssdCXUHP6vcsBfgure5aAm3hKLh6JLt | 0.000000000000 | us-east | 5294849d7c578227966a2fb5cb6a4155038464c57c5810d1820b27df826b2487 | 2018-06-08 11:29:36.490749
(1 row)

Note the blockhash in green. Querying monerod for a block with this hash yields:

curl --data-binary '{"jsonrpc": "2.0", "id":"1", "method": "getblockheaderbyhash", "params": { "hash": "5294849d7c578227966a2fb5cb6a4155038464c57c5810d1820b27df826b2487" } }' -H 'content-type: application/json' -X POST http://127.0.0.1:18081/json_rpc
{
  "id": "1",
  "jsonrpc": "2.0",
  "result": {
    "block_header": {
      "block_size": 104,
      "depth": 3365,
      "difficulty": 52774831374,
      "hash": "5294849d7c578227966a2fb5cb6a4155038464c57c5810d1820b27df826b2487",
      "height": 1589929,
      "major_version": 7,
      "minor_version": 7,
      "nonce": 2865252,
      "num_txes": 21,
      "orphan_status": true,
      "prev_hash": "b3b72c9affb658ac673eab4eec145a2fa0f7991a4937a27ad8fda87fdccc09d0",
      "reward": 4676461370602,
      "timestamp": 1528392962
    },
    "status": "OK",
    "untrusted": false
  }
}

I have not a never will cheat our miners of their rewards.

Cheers Oliver, the same thing dawned on me this morning when I thought about it, kinda is the definition of an orphaned block. :) And I hope you didn't think I was insinuating anything untoward on your part, I was mostly curious about the technical reason and didn't spend enough seconds thinking about it before posting. As always, keep up the good work Oliver!

Not at all. The whole situation was quite embarassing for me since I struggled to find out what the heck was going on with that block.

Ok, so help me out here.  The way I understand forks, the chain underwent a split (either because we didn't get our block out fast enough or because we put it out at the same time as someone else and the next block was found on their chain).  Either way, we were on the losing side of the reorg.  Is that correct?  If so, which one happened, and why did it take us so long to find out?  Were we mining on a bad chain for that long?  'Cause that's awkward...

Anyway, thanks for the updates and education.

Nope we weren't mining on the wrong chain for long. That's exactly what confused the heck out of me. The block in question was not updated even while monerod claimed to be fully in sync. I saw our orphan as legit, non-orphaned block on the chain. Only when I forced a restart of the daemon, I was able to see both the "real" block and our orphan.

Ah, I see.  I was confused along with MinerOfMonero because the block explorers only show you the valid block.  Is there a way to grab the hash along with the blockheight when you publish the ones we find? I feel like if we’d had that, one of us would’ve eventually tried the monerod query.

Block is 1594145 is not an orphan by the way. Already checked it.

Block 1595437 is fine as well. :)

This actually worked out great, since there was a new block found very soon after. Great work!