José Ignacio Stark / El Mostrador / Critical assessment of national networks and data centers / Why did this happen in Chile's most modern data center, with walls one meter thick and a floor capable of withstanding up to a ton of force? Sources who asked El Mostrador to remain anonymous revealed that the backup power batteries—known as UPS systems—were displaced by the force of the earthquake, and in their movement, they destroyed the center's circuitry, the repair of which is inherently delicate.
One of the few things I consciously grabbed from my room as I rushed down the stairs of the Hotel O'Higgins—where I'd spent the earthquake—was my cell phone. I knew that in a situation like this, it was the data network, more than the voice network, that could help me in any case. And it did. But I, and many others, were shocked to discover that, due to the strongest earthquake to hit the south-central region of the country, the most important national websites were down. There was no official information. Neither ONEMI (the National Emergency Office) nor my bank was responding. The earth spoke, and the internet, quite literally, went silent.
Within minutes of the earthquake, many were able to check the information on their phones from the U.S. Geological Survey (USGS), which announced the magnitude of the quake in Concepción: 8.8 on the Richter scale. I also quickly checked the tsunami alerts issued by the NOAA's Pacific Tsunami Warning Center. But there was no official information.
Let's go back to 1968. ARPANET, the precursor to the Internet, was created as an internal military communications network by the U.S. Department of Defense, designed to be decentralized, with multiple communication paths between any two points and capable of responding to its own failures. Email and file transfer were its primary uses in 1973. A year earlier, Project CYBERSYN had allowed for monitoring the status of factories through telex terminals, all centralized from a bunker beneath La Moneda Palace, right here in Chile.
What is clear is that both experiences proved useless during and after the earthquake. ONEMI's servers crashed, and along with them, the systems of the entire state apparatus were down. Between Saturday and Monday, the Emergency Office, headed by Carmen Fernández, was unable to use email. And while it is true that they maintained radio communication with their regional offices, something went wrong, because even several days later they were unable to gather or systematize key information such as the list of shelters and hospitals that remained operational, much less coordinate the most urgent needs in each affected area.
Global Crossing, the virtual Upper River
Based in Bermuda, Global Crossing—formerly known as IMPSAT in Chile—is the most prestigious and highest-quality data warehouse, hosting both government clients and large institutions, such as Banco de Chile, Andrés Bello University, and the online services of the Ministry of Public Works, among others. These were the same systems that fell like dominoes in the early hours of the 27th, and could not be recovered until several days later.
Why did this happen in Chile's most modern data center, with walls one meter thick and a floor capable of withstanding up to a ton of force? Sources who asked El Mostrador to remain anonymous revealed that the backup power batteries—known as UPS systems—were displaced by the force of the earthquake, and in their movement, they destroyed the building's circuitry, the repair of which is already a delicate matter.
Standard security measures at these types of facilities include a ban on multiple people working simultaneously in the same server room; therefore, the recovery time was subject to this restriction. Thus, the IT manager from Arturo Prat University, who traveled from Iquique to assess the damage to their machines and bring them back online, had to wait for hours in the parking lot alongside other clients.
“Partition wall” of networks
In the case of Banco de Chile, the same source adds that the error that left both customers and employees of one of the country's largest financial institutions without online services was not only due to the technical incident at GlobalCrossing and that of its own data center — which was flooded after the earthquake, affecting both its main and backup servers — but rather to a management failure, which made it clear that the contingency plans designed to "lift" the bank up in the face of an emergency of this type were completely ineffective.
For its part, the Ministry of Public Works was forced to utilize a wide range of social media platforms: within hours, the blog mopinforma.blogspot.com, which continues to operate after the restoration of services, began communicating the status of the country's road network, which is also constantly updated through its Twitter account. More than a year after the boom of this social network for corporate communication, only the earthquake managed to convince authorities in various government departments of its true usefulness. An official from the Ministry of Public Works, who requested anonymity, stated that they are looking for ways to terminate the contract with the company, since the damage it caused the Ministry during a crisis of this magnitude was disastrous.
The earthquake not only took websites offline, but also exposed the realities surrounding the supposed security and stability of the national data infrastructure. On the one hand, it affected both designated "contingency" data centers and the main ones. On the other, it made it clear that energy independence is impossible for these types of facilities, which would hypothetically allow them to have two electricity providers and avoid loss of uptime if one failed, thus better handling situations like the one that occurred on Sunday due to a failure in the Central Interconnected System, which left the entire south-central region of the country without power. And finally, it revealed a "deceptive" network segregation: many customers, whose backup links had been contracted with two different providers, discovered that both had ultimately subcontracted the services to a third company, which turned out to be the same in each case.
This is how national data centers resisted:
NetGlobalis and S&A: No problems; the three NetGlobalis operators ran on the data center's electric generator after the earthquake and did not experience any downtime.
Adexus (Miraflores 383, 2nd floor): Works without problems or interruptions from day ”D” and afterwards.
NOC Telefónica Chile (San Martin 50): No problems.
Entel: The data center and main operations center of the former state-owned company are located beneath the 127-meter tower that coordinates communication between various points across the country, and both responded without problems. Following the 1960 earthquake, the necessary construction precautions were taken and applied during the building of this iconic structure.
Synapsis: They reported no problems and confirmed 100% uptime. The machines did not fail over, nor were the contingency or backup links used. Source: FUCATEL.