Monitoring StatusCake… With StatusCake

How does the monitor, monitor the monitor? No it’s not a tongue twister, but rather a question we faced when StatusCake started to be joined by big companies such as the BBC, NHS, EA to name but a few.

What do we do?

StatusCake now monitors over 250,000 websites and has tens of thousands of users who rely on it to let them know if their site goes down. If StatusCake were to face difficulties, it would impact our users on a grand scale delaying their alerts and missing downtime – so it’s clear we needed a good monitoring system.

StatusCake was built around the principals of easily deployable nodes that can come up and down without impacting service quality. We have a high level of redundancy with around 50% of our node servers able to go down at any one point without impacting check rates or alert quality at all. Each node is independent of each other, and each grabs a workload and holds the entire systems workload on it at any one time. Using this independent structure, if a node were to be unable to connect to the master servers it continues on and tests servers that have been assigned to it. To ensure tests are not duplicated among servers they each talk to each other letting each server know which servers are having trouble and what work load to take because of that.

So that reduces the possibilities of something going wrong to an extreme level and means we can use StatusCake to monitor StatusCake. We have a StatusCake account that is set up for us that monitors all our servers, even the HTTP server! If any part of StatusCake were to go down another part would notify us almost instantly. We don’t believe in over redundancy when it comes to offering our users the insurances that their monitoring will remain in place, no matter the difficulties that may arise.