Henry H.’s Post

Google Cloud mistakenly erased an investment company's entire cloud setup, belonging to Unisuper, which oversees $80 billion in assets. Despite having backups stored in another region, GCP also removed those. Fortunately, additional backups were safeguarded with another service provider. This incident highlights the risks of entrusting all data and infrastructure to a single provider, as it can lead to catastrophic consequences. #CloudComputing #DataProtection #BackupStrategy #cloud #architecture #cloudarchitecture #riskmanagement

UniSuper, a major financial services provider catering to over half a million members in Australia, faced a significant disruption when an unprecedented 'misconfiguration' in Google Cloud led to the accidental deletion of UniSuper's entire account. This incident left customers without access to their superannuation accounts for 'over a week'. Though UniSuper had redundancy measure - full backups in another geography, the deletion of its cloud subscription caused disruption across both geographies. Thankfully, some sane mind at the firm planned regular backups to different service provider, from which they could restore 1 week old data. Google cloud just issued a joint statement and apologized, they obviously couldn't help restore any data. A big lesson (which we have been adherent advocate of for very long time) is to always have Offsite backup. #Backup #Offsite #DR #DisasterRecovery #cloud #thirdpartyrisk #riskmanagement

“The dedication and collaboration between UniSuper and Google Cloud has led to an extensive recovery of our Private Cloud which includes hundreds of virtual machines, databases and applications.” To build a cross vendor, cross region or hybrid environment backup and recovery plan is extremely important now. The disaster happens even your production is sitting in hyperscale provider. #AvoidVendorLockin #AlwaysHavePlanB Plan early, let #HyperBDR to help you to build the #MultiCloud contingency plan

"Google Cloud CEO, Thomas Kurian has confirmed that the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription," "While UniSuper normally has duplication in place in two geographies, to ensure that if one service goes down or is lost then it can be easily restored, because the fund’s cloud subscription was deleted, it caused the deletion across both geographies. UniSuper was able to eventually restore services because the fund had backups in place with another provider." This story is wild y'all. You need to back up your cloud data folks. I've seen stories of people losing their personal accounts on Google and similar places but this is next level. One call out that is interesting is how simply following best practices from well architected documentation likely wouldn't have saved you on this one. Cross region, versioning, and similar setups aren't able to overcome when the billing machine thinks you're out of quarters. At Clumio we solve for exactly this problem. Well the air-gapped, immutable, and easy backup piece not so much the running out of quarters piece. We can save you some quarters though. https://lnkd.in/eHqNwPWW

Google Cloud recently deleted an $80B investment company's entire cloud environment, including their cross-region backups. It's a terrifying thought, but it happened. Luckily, they had a backup on another cloud, which ultimately saved them. Could this happen to you? Unisuper likely thought it wouldn't happen to them, but it did. That's why having a multi-cloud backup strategy is crucial. It may have been a fluke, but it saved an $80B business from failing. Check out the full story here: https://lnkd.in/eDZk3CMZ.

Of what value are all your contractual safeguards, SLA, KPI with your cloud-provider if this can happen? This is why having a robust cloud-STRATEGY incl. solid BCP is so important.

Google Cloud deleted, by accident, a company's entire cloud (the company is Unisuper, an investment enterprise, which manages $80B)❗️ ▪️The company had backups in another region, but Google Cloud deleted those too!!! ▪️Luckily, they had a third backup with another provider! Full article from The Guardian here:

This is such an important lesson for all those who believe that native backups are enough on the cloud. Yes, they serve a purpose and do the job… but organisations need to consider engaging with Backup vendors to take that data offsite just in case the worst case scenario occurs. https://t.co/v5WFxqUtaB Veeam Software

A Google Cloud glitch left 620k Unisuper members without account access for a week, wiping out their entire private cloud subscription! This incident shows just how crucial it is to have a good backup strategy. You never know when the cloud might rain on you! #Cloud #Google #backup https://lnkd.in/g_8TPRW4

This is like the worst nightmare for Google, GCP and enterprise cloud in general. So, Google Cloud accidentally deleted a company's entire cloud environment (Unisuper, an investment firm which manages over $100 billion in assets). The company had backups in another region, but GCP deleted those too. Luckily, they had yet more backups on another provider. The CEOs of both companies have apologised to the customers. Going forward you should probably think of DR strategies that include backing up to a different cloud service provider or on prem as well 🤷♂️