Why Upgrades Are No Easy Task – Real Disasters and Their Impacts
Words like “regular” or “simple” do not belong to the IT slang dictionary. This applies to both, complex and less complex activities; like an upgrade or an update.
This rule works for one simple reason. Even something as simple as a routine application or system update can cause traffic disruptions, reputational damage, legal consequences, or massive financial losses.
Every change brings about the risk that the new settings will not match the old ones. Problems have occurred in space, in the sky, on the ground, on roads, in banks, on stock exchanges, in UI and back-end environments, on end devices, in networks, in control systems, in operating systems, in medical equipment, and even in the military. An interesting explanation of 25 strange errors can be found in this video.
This potential problem simply does not avoid anyone. But the potential does not necessarily have to become a reality. It just requires everyone in the company to be on the same page when it comes to understanding the value of validation and preventive steps in the processes related to upgrades and updates.
In this article, we will focus on several famous errors associated with system upgrades and updates that had disastrous consequences. We will show you why the problem occurred and what we as customers can do to minimize the consequences when something goes wrong with our provider.
One thing is true. Mistakes will happen and a large part of the responsibility for ensuring the business continuity of our company lies on our shoulders, not on the shoulders of suppliers and service providers.
Scary Examples Known and Unknown
Rogers Communications – July 2022
Canadian telecommunications provider Rogers encountered a problem during a multi-step update of its main network. As a result, its customers lost their internet and mobile networks for 5 days.
The consequences were not small at all, because, for example, card payments did not work, people could not call the emergency line and small businesses accumulated losses in thousands of Canadian dollars.
Given that the situation in Canada is similar to ours, only a handful of large providers operate in the telecommunications market, especially when it comes to mobile networks.
We can therefore assume that a similar incident and similar consequences may occur in our country. So how can we prepare?
Lessons Learned
- Even basic services can fail or experience downtime. It is safe to assume that no system or service is immune to failure. Therefore, companies should have a functional disaster recovery plan in place so that they know exactly what to do in a given situation.
- The company should have a backup plan for internet and mobile network connections. A backup connection to another provider or, for example, a SIM card from another operator may be the solution.
CrowdStrike - July 2024
This problem was unfortunate because the Falcon system is used for cyber protection and is mainly used by critical systems. Although CrowdStrike managed to solve everything in a matter of tens of minutes, companies affected by the failure took much longer to deal with the outage.
Mainly because every single Windows system taken down by the faulty update had to be booted into safe mode and needed recovery keys entered. All manually.
Lessons Learned
- Updating a critical system should be a well-planned activity. It is necessary to have measures in place in case it is necessary to intervene manually if the update fails.
- If possible, it is advisable to test the new update on a sample of systems. Once its functionality is verified, then it is possible to proceed to the next phases.
- The loss of functionality of a critical system creates the inability to function for companies and organizations. This in turn means additional costs and loss of income. For such a situation, you need to have a plan and backup infrastructure to minimize downtime.
Royal Bank of Scotland – June 2012
We are used to internet banking working without any problems. Payments go, payments come and we always see real numbers on our account. But in Britain, due to the broken schedule in a software managing payments, all this fell like a house of cards.
The system was completely out of order for several days. People and businesses could not access their finances, regular payments were not made, credit scores suffered.
The bank itself did not admit this, but some experts blamed IT outsourcing, which replaced a team of in-house technicians. The vision of 80% savings may have been too tempting.
Whatever the cause, bank failures have since become a relatively common part of our lives. How do we deal with them?
Lessons Learned
- It is not a good idea to have all your financial resources in one place.
- It is also a good idea to consider how much cash you should keep. It is not just the banking system that may fail, but also payment terminals, for example.
Polygon – July 2025
Even the distributed crypto world is not immune to bugs. One of the layers recently underwent a major upgrade. The result was a data mismatch that limited the functionality of some applications.
Although Bor, the Polygon blockchain, remained fully functional, some applications encountered frozen blocks. Users were left unsure of what the outage was about, and of course the token price fell as the knowledge about the issue rose.
Lessons Learned
- It is necessary to communicate with the supplier or provider in advance about SLA, which includes the rules of communication. Therefore, if there is any outage, you as a customer should be informed as soon as possible.
- Consider whether it is not a safe step to secure diverse RPC channel providers. Not everyone was equally affected by the Polygon outage.
Overall Lessons Learned?
At first glance, it may seem that upgrades and updates only bring problems. That it is better not to install them. But such an approach leads to other problems. Rather, it is necessary to have the right perspective on new versions and prepare a sound plan for how to approach them so that they do not paralyze your business.
Redundancy is a principle that can save you a lot of trouble. These are backup options in case any element of your operation goes down. It could be a second connection, multicloud, software, or application, just about anything that is critical to the operation of your organization.
At the same time, consider the pros and cons of vendor lock-in, because dependence on a single supplier exposes you to a greater risk of being susceptible to outages.
Speaking of suppliers, they are not the only ones who should ensure that everything runs smoothly. Both they and you as customers need to take care of backup solutions in case an outage occurs.
And as always, we come back to business continuity and disaster recovery plans. At all times, your business must be able to operate at least in a minimal operating mode. This means that you must be able to maintain critical business activities.
Both of these plans serve to prepare alternatives, but it is of utmost importance that they are managed and regularly tested.
What Should You as An IT Decision Maker Demand?
For key systems, ask your supplier for information about the changes they are planning in advance. This information will help you react as quickly as possible if a problem occurs. Communicate this process with your supplier as far in advance as possible to avoid unnecessary downtime. Incorporate this into your contract so that everything is in black and white.
Once it’s time to deploy changes, proceed in stages. First, apply the changes to a sample system so that a potential error doesn’t cripple entire operations.
Your IT team should also actively work to maintain functional backup plans. They should have enough time to review everything related to alternative measures.
When a problem does occur, request an incident analysis. It should include information on the causes, impacts, and actions that are derived from the incident.
Neither Ordinary Nor Simple
As we have shown, even an update can lead to catastrophic consequences. For you, as an IT decision maker, it is critical to view protection measures against downtime as a well-chosen investment, not an expense or unnecessary bureaucracy.
Don’t rely on the quality of the supplier, a positive experience with the system, or the size of the organization behind the system. No company or organization is immune to errors, so be fully prepared in case your systems go down or the blue screen of death occurs.
