Update on the situation over the past week

As everyone noticed, last week was quite a bumpy ride to put it politely. It started with an increasing number of dropped shares across all pools. As things progressed the situation worsened. Monitoring resource consumption indicated not only high load on all of the main servers 16 CPU cores but an insanely high memory consumption of the pool processes (up to 22 GB each). 

My initial analysis attributed this to an inefficiency in the pool code producing lots of garbage that need to be collected which means the pools became increasingly busy with themselves in addition to handling shares. Up to a point where the garbage collector halted all threads to keep up with the amount of garbage. This is when the pools started aggressively dropping shares deemed outdated when the pools finally resumed normal operation after sometimes 20 seconds long garbage collection passes. The analysis made sense even though the impact seemed to be way over the top for the amount of regular miners submitting shares. 

Since the servers are running AMD's latest Ryzen CPUs I also looked at the hardware side. Early batches of these CPUs had some issues that might be contributing to the situation. Because of this I've requested the hosting provider to migrate the server SSDs into newer servers equipped with updated batches of these CPUs. Unfortunately this did not have the desired effect. In fact things got even worse after this. Frantically searching for a solution I began to implement many performance improvements to the pool engine over the past week. The was the reason for the many many pool restarts you might have noticed when monitoring your rigs. As these code changes were implemented under stress, new bugs were introduced along the way. The issue with the massively rejected shares on the Ethereum Pool was part of the collateral damage caused by this. 

When the XMR pool finally found a new block in all of this chaos and suddenly the payout processing failed due to the system running out of file descriptors, I realized there must be something else. Pool servers are normally configured with extremely high file descriptor limits with ours being no exceptions at a limit of one million open file descriptors. Upon closer inspection I realized that someone managed to bog down the pools with connection attempts that flew under the radar and that did not get logged. We've been under DDos all the time without realizing it. Those attacks appear to have stopped by now. But is is clear that someone does not want this pool to succeed even though we are quite small. Unless those responsible wanted to ensure that it stays that way. They've partially succeeded as we've lost a lot of miners and I can't blame them for leaving.

- Oliver