What happened to Fastly and why did so many websites experience downtime?

A phrase we never thought was possible – “the internet is down”, was being shouted across the world on Tuesday 8th June 2021. Alarm bells were ringing when hundreds of websites had an “error 503” show up when visitors tried to access them. So what happened to the internet? 

A relatively unknown company soon dominated global news headlines for causing the websites’ “blackout”. Fastly, which is a service provider for many of the most well-known and popularly used online retailers and other online businesses, came out and said an issue had been identified on their server and they were actively working at getting a fix in place. 

So who is Fastly?

Until very recently, Fastly was a relatively unknown company, at least one that hadn’t been part of any type of huge media storm in recent years. Since the 8th June, however, Fastly has dominated one of the biggest tech news stories of the year, even though it wasn’t necessarily a desirable headline to hold. 

Interestingly, as the infamous marketing phrase goes “there’s no such thing as bad publicity” and so it has been for Fastly. Since the news broke out, its share price has gone up by a whopping 10% and has now continued and maintained its value at around a fifth of its value from the beginning of the month. Now, this isn’t a blog about investment opportunities, but I’m sure if it was then we’d have a good time delving into why buying this stock could prove hugely beneficial…

What does Fastly actually do?

Fastly’s products range from cloud CDN solutions to security and cloud optimisation tools. It has now become well known for its CDN (Content Delivery Network) which is used by many online retailers to help them speed up their website performance and make navigation easier and quicker for their customers. Recently,  we discovered that Fastly handles around 10% of the total web traffic through their servers which gives you a rough idea of how big this company is. Another of their cloud-based services is the edge cloud solution which helps companies speed up their network response times by providing a connection between the edge (users or data collection points) to their main server.

What actually happened with Fastly?

Fastly has been fairly open about what happened since the blackout became global news. In mid-May, an update had gone out to all the users on their server but unbeknownst to them, the update had a bug. Nick Rockwell, SVP of Engineering and Infrastructure at Fastly wrote,

“We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change”.  

Nick goes on to say, 

“We detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration”. 

This downtime in the server affected 85% of their users at the time and these include some huge companies such as Amazon, Reddit, The New York Times, Spotify, Twitch, GitHub, and PayPal. 

A fix was implemented and within 49 minutes, 95% of the service was back online. What we can say here is that the team acted swiftly in dealing with the issue. They even had a timeline with all the updates on their site so everyone could see the progress from the team:

What is a CDN and why do websites use them?

Content Delivery Network (CDN) is a solution developed to help websites improve their overall performance. Unless you are involved in the tech industry, this isn’t something many people think about or are even aware that they’re using. As more and more websites go live online, they have all content that needs to be accessible to visitors to their website. Normally this is not an issue especially if you are a small business and only have modest daily traffic. The reason for this is if the number of visitors to your site is low, the server your website sits in can easily deal with all the individual requests it receives whenever a new customer lands. Now think about if your website was to become a global success and you start getting huge numbers of new users worldwide, and the traffic to your site has gone up 10X. The server housing your website will start to struggle with the requests and eventually, slow down the website performance. The reason for this is because the server is getting such a large quantity of requests from different customers that it doesn’t have the capacity to load all of them quickly. 

Another issue with having a single server is that customers that are further away from the server, for example in a different location, will have a slow responding website that takes a long time to update and upload. This is because the further a user is from the physical server, the longer it will take for the request to reach the server as it has a longer distance to travel. Soon they become frustrated and will leave without purchasing or using the service. 

Imagine now being a company like Amazon, which has thousands of pages for different products and operates across the globe. It has localised versions of its site to help with exactly this issue, but it also uses CDN systems to improve this service. What the CDN providers do is use thousands of servers around the world, which are used to store caches of the websites they look after. As the servers are closer to the users, it means that the customers experience a much faster and responsive website so in Amazon’s case they can flick through different products continuously and purchase the items they want. To us who use this service, it all looks seamless and something that we never give a second thought to. The servers will update their cache on a regular basis by constantly monitoring and sending an update request to the main servers. This way the information the customers access is always up to date. 

What are the benefits of using CDNs?

Content Delivery Network is an ideal solution for companies that have a large inventory of products and are looking to offer their products to a global audience. A CDN helps website performance as it increases page speed, uptime, and security whilst at the same time helping to reduce the requests that the main server will need to deal with. It’s able to do this because the local servers take away the traffic volume from the main server via the local server system. This greatly improves the customer experience on the site but also means that you can deal with many more requests much quicker. 

What was the cost of this downtime?

It’s difficult to estimate the total financial losses due to the downtime as the companies involved are not all e-commerce based businesses. In fact, a big proportion of those affected were large media outlets, government sites, and service providers. We also need to consider the time of day the downtime occurred, in Europe it would have been just before 11:00 am GMT so we would assume a busy period, but in the US it was nighttime when we look at the west coast and early morning if we are looking at east coast so the damage could very well be minimal.

An estimation for Amazon’s downtime caused by Fastly’s outage has however been estimated at around $32 million in lost revenue. Luckily for Jeff Bezos and the team, this loss is unlikely to dent the profits of the world’s biggest online company, with a value at over $1 trillion. But consider those smaller businesses that were hit with the downtime during Europe’s busiest trading time; the loss would be significant and could lead to further financial losses and repercussions further down the line. Small businesses that don’t have the resources to have actual individuals to monitor their website or server would mean that they are completely reliant on Fastly fixing their server issue should this ever happen again. 

The solution to website downtime

Website downtime can happen to anyone, regardless of if they’re using a provider like Fastly. Downtime can be caused by a multitude of different things, from “bad” code to domain issues, and almost everything in between. That’s what makes it so difficult to know when your website has gone down and even worse, why it has gone down. 

StatusCake offers a solution to Fastly customers that were affected by this outage; uptime monitoring alerts. We take a website, let’s say www.wikipedia.com, and we’ll alert you as soon as Wikipedia goes down. The best part is, you can test your website uptime from 43 different locations in 30 different countries so if you’re up in Canada but down in India, you’ll know about it. 

Take Fastly’s outage as an example, StatusCake saw 58% more websites go down than the same time the previous week, proving how heavily reliant websites are on CDNs. 

Why you should consider uptime monitoring

Fastly’s outage should be conclusive enough to prove why uptime monitoring is so important. Website downtime can cost you thousands in revenue, negatively impact your SEO rankings, lose your customers’ trust and satisfaction and give you a headache with the number of user complaints. 

Using an uptime monitoring tool is far cheaper than the cost of your website going down, as those who were affected on the 8th June have come to learn. 

Identifying server issues

Uptime monitoring, however, is only part of the story. If your website has gone down, or it consistently experiences downtime, you need to find out what is causing it so you can prevent it from happening again. It’s so damaging to your brand that the sooner you find a long-lasting solution, the better your brand will look. 

With Fastly, if their customers had been using StatusCake, our server monitoring tool could have helped them to see there was a potential issue that would arise and in turn, cause website downtime. This would have given them an ample amount of time to put solutions in place before a complete website blackout. 

Server issues can again happen to any website at any time, and it can be very hard to identify when they’re going to happen if you don’t have any monitoring in place. You’d be fully reliant on your backend developers constantly manually checking your server’s performance and trying to see if there are any irregularities or potential problems on the horizon. This is simply not sustainable and that’s why StatusCake’s server monitoring is an easy and affordable solution. We’ll monitor your server for you and alert you to anything we think could cause you an issue or anything that we flag as suspicious so you can make better, more informed decisions. 

What we learnt from Fastly’s outage 

Websites won’t stop using Fastly simply because they suffered an outage, but what it will do is make website owners realise how detrimental downtime can be, especially on their revenue. Signing up to a free website monitoring account with StatusCake can be the difference between keeping your website online and keeping your customers, or simply losing them to your competitors and dropping to the 2nd page in Google rankings. You do the math!