We had an outage tonight that affected the website, the ETH, ETC, MONA and VTC Pools. XMR, DASH, LTC stayed up and kept collecting shares just fine. I am deeply sorry.
The reason for the outage was a coin daemon running into a vicious crash/restart cycle caused by an ill formed transaction received from the P2P network. In the process it spit out GBs of crash logs filling up all remaining disk space. Affected were the ETH, ETC, MONA and VTC pools. The XMR, Dash and LTC pools kept collecting shares normally.
The coin daemon in question is normally running with strict log recycling which would have prevented the problem. Unfortunately it was turned off a while ago for some troubleshooting and I forget to turn the log recycling back on. Which I did this morning.
I'm afraid its not your end. I've just encountered something that felt like a downtime as well. It was not the first time. You see it once and it's like "hmm, weird" then you see it again, then you get user reports about the same thing. Looks like two of the servers are affected by a problem with early shipments of AMDs Ryzen CPUs. I'm currently in talks with the hosting provider about migrating the servers to a more reliable server platform. Will keep you posted.
Looks like you're down again. Two different type so an error in my log.
Connecting to stratum server eth.poolmining.org:3072 Could not resolve host eth.poolmining.org:3072, The I/O operation has been aborted because of either a thread exit or an application request i 19:49:21|stratum Reconnecting in 3 seconds... i 19:49:24|stratum Connecting to stratum server eth.poolmining.org:3072 X 19:49:24|CUDA0 Can't submit solution: Not connected X 19:49:25|stratum Could not connect to stratum server eth.poolmining.org : 3072 , No connection could be made because the target machine actively refused it
Eth pool is down and it's likely that ETC, MONA and VTC pools are down too as those were down last time when the website was down, all probably running on the same HW.
there's definitely something happening on the DASH pool server as I just watched the pool hashrate dropped from 2TH to 200GH and my miners switched to back up pool
Questions
We had an outage tonight that affected the website, the ETH, ETC, MONA and VTC Pools. XMR, DASH, LTC stayed up and kept collecting shares just fine. I am deeply sorry.
Any explanations as to what has caused it and why only some pools were affected, as well as what mitigation can and will be done for the future?
The reason for the outage was a coin daemon running into a vicious crash/restart cycle caused by an ill formed transaction received from the P2P network. In the process it spit out GBs of crash logs filling up all remaining disk space. Affected were the ETH, ETC, MONA and VTC pools. The XMR, Dash and LTC pools kept collecting shares normally.
The coin daemon in question is normally running with strict log recycling which would have prevented the problem. Unfortunately it was turned off a while ago for some troubleshooting and I forget to turn the log recycling back on. Which I did this morning.
Thanks for the explanation and honesty. +1
has the server gone down aswell now?
No everything is up and running.
Ok thanks mate. Ill look at my end.
I'm afraid its not your end. I've just encountered something that felt like a downtime as well. It was not the first time. You see it once and it's like "hmm, weird" then you see it again, then you get user reports about the same thing. Looks like two of the servers are affected by a problem with early shipments of AMDs Ryzen CPUs. I'm currently in talks with the hosting provider about migrating the servers to a more reliable server platform. Will keep you posted.
aw I though so. Lucky I set up a back up a different server pool to mine if this server dropped out.
Looks like you're down again. Two different type so an error in my log.
The website is down but I think my miners are still working though.
Eth pool is down and it's likely that ETC, MONA and VTC pools are down too as those were down last time when the website was down, all probably running on the same HW.
It went back up sometime between 8-9pm cet
Dash server dropping in and out last 5 mins anyone else having this issue?
same here, both with website and xmr pool intermittently non available, website showing an nginx 504 gateway timeout.
Working on it around the clock. Problem is identified.
Anyone's miners dropping in and out in the last 5mins?
there's definitely something happening on the DASH pool server as I just watched the pool hashrate dropped from 2TH to 200GH and my miners switched to back up pool
You are right. The pool restarted 4 minutes ago. Investigating ...
Looks like website and dash server down?
Fixed
Also when are you looking at upgrading DASH v12.2 that was released last week I think. Or are we on v12.2 already?
We've upgraded like two weeks ago to Dash Core v0.12.2.1.
Haha ok sorry still kinda knew with the mining thing.