Zilhaz Chowdhury’s Post

An eye-opening story for IT managers questioning the value of backups in the era of cloud computing. UniSuper's complete cloud data set was inadvertently erased by Google, highlighting the critical importance of maintaining separate backups. Without their own backups, the company would have faced near extinction. This incident underscores the significance of viewing backups as essential insurance rather than sunk costs. While a more robust cloud strategy could involve using a secondary provider, the emphasis remains on off-site tape storage for vital backups. It is crucial to plan these backups at least quarterly whenever feasible. Teams responsible for data management should possess clear and efficient backup and recovery procedures. At UniSuper, an individual or team stepped up and potentially saved the entire company from ruin. This serves as a stark reminder of the indispensable role backups play in safeguarding business continuity. Read more about this incident here: #ITManagement #DataBackups #CloudStrategy #BusinessContinuity #TechNews

I've always thought that the only reason the public cloud will falter in growth is due to a massive ransomware attack that exposes all customer data (and potentially loses a lot of it). However, so far, we've not seen that, but we have seen a cloud platform itself do something similar. Google Cloud (as we reported a few weeks ago) managed to delete the entire account for UniSuper, an Australian pension fund with 647,000 members and $135 billion under management. The issue was so impactful, even the replicated data copies in a second region were deleted. The recovery (which took 2 weeks) was achieved by using backups outside of Google Cloud. If this incident doesn't scare you into not using the public cloud, I don't know what will. This may be a "once in a lifetime event", but I doubt it is the last we will see. I expect this scenario has happened with smaller customers and could hit another large customer (not just Google Cloud, but any cloud). Similarly, we should remember the OVH fire, which also massively impacted those customers with their backups in the same data centre. So, what surprises me most is how little this incident seems to have affected the industry. Other than the obvious reporting (linked below), there doesn't appear to have been much followup. None of the major on-premises infrastructure vendors have suggested themselves as alternatives. So do IT folks just not care? If you're confident of your backup situation, then I guess you're less concerned. In the case of UniSuper, the incident wasn't great but it appears the copy of last resort in a secondary location saved the day. What does this say about the future of on-premises versus cloud? To me it suggests that there are almost no incidents that will divert businesses from further public cloud adoption. This must be a massive red flag to on-premises infrastructure vendors that they're not winning hearts and minds. Ultimately, their businesses are destined to be a shadow of their former selves, with consolidation and acquisition a major theme for on-premises infrastructure companies. https://lnkd.in/eg6mFrRc

Even if the CTO of Google Cloud answers your calls, you can only rely on your backups. If they don’t answer your calls - you are left with no other option. The recent Google Cloud incident, where a customer faced two weeks of downtime due to an accidental account deletion, shows how important properly set backups are. This incident should change how we think about disaster recovery. Recommendations from cloud providers usually focus on using their own services as a location for backups, while the use of an external location for backups seems to be the only reliable option. Trusting just one vendor may be too risky. It makes sense to check with your service providers to ensure they offer automated backups of all critical data to your own storage and to use the option. Ideally, these backups should be easy to restore outside the vendor’s system. It is a good investment that will pay off hugely in an emergency. https://lnkd.in/e5Tx4Z9U

Google cloud somehow deletes major account along with all backups https://lnkd.in/g68FWRa7 I have long maintained that cloud means computers that you can't fully control. I am sure google is looking into how the account got deleted. If the computers were on-prem, presumably the data and the backups would be seperate things. However in the case of google cloud, somehow everything was deleted at once. Fortunately, the company had a second cloud provider. This reminds me of the backup 3-2-1 strategy which states that you should have 3 copies of your data (your production data and 2 backup copies) on two different media (disk and tape) with one copy off-site for disaster recovery.

A perfect example of the need to have a fall back plan / disaster recovery plan. Cloud service providers fail, too. You can scream all you want when it happens, but it is best to initiate your proven fall back plan and move on. I personally know of quite a few businesses that would be out of business if this happened to them. They have no fall back plan in place and they are relying 100% on the cloud service provider for disaster recovery.

Can you trust your cloud provider not to lose your data? Almost always, except when you can't. In May 2024, UniSuper, an Australian superannuation fund, faced a major crisis when Google Cloud mistakenly deleted their entire account, including all backups. Thankfully, UniSuper had a third-party backup in place, which saved the day. 🔗 Read the full story here: https://lnkd.in/dq-MMME9 Lessons learned: This incident underscores the critical importance of a robust, multi-layered backup strategy. Relying solely on a single cloud provider, even one as reputable as Google Cloud, can pose significant risks. Implementing a third-party, independent cloud backup solution is essential for ensuring business continuity and data protection. Data protection best practices: -Follow the 3-2-1 backup principle.  -Avoid storing all data and backups in the same logical infrastructure.  -Utilize multi-cloud data protection solutions. #DataProtection #BackupStrategy #GoogleCloud #BusinessContinuity #SaaS

🚨 A Wake-Up Call for Cloud Users: Backup and Resiliency are Key! 🚨 The recent incident where Google Cloud accidentally caused a two-week downtime for a customer, as reported by Ars Technica, is a stark reminder of the importance of a robust backup and resiliency strategy. Read the full article here: https://lnkd.in/ehFe4ehN In today’s digital age, the cloud is the backbone of many businesses. While cloud providers like Google, AWS, and Microsoft Azure offer top-tier services, no system is immune to mishaps. This is why having a comprehensive backup and disaster recovery plan is essential. As a proud Microsoft Azure cloud partner, we urge all businesses to: 1. Regularly Back Up Data: Ensure your data is regularly backed up in multiple locations. 2. Implement Resiliency Measures: Design your cloud architecture with resilience in mind to withstand potential outages. 3. Conduct Periodic DR Drills: Regularly test your disaster recovery plan to ensure it works when needed. Remember, it’s not just about having a cloud solution; it’s about ensuring that solution can stand the test of unforeseen challenges. Let’s work together to fortify your cloud strategy. Reach out to us to learn how we can help bolster your backup and resiliency plans with Microsoft Azure. #CloudComputing #Azure #BackupStrategy #DisasterRecovery #CloudResiliency #TechNews #BusinessContinuity

Let's have a cloud computing drinking game… I will take a shot of espresso on "Cloud is just another person's computer" each time. For the next few days on Linkedin. I will be really energized And I will tell you: "Yes, the cloud is another person's computer, but one hell of a reliable computer". I have been in IT 15+ years, during the period of the "dawn of cloud public computing". I managed to see both worlds, private data centers and public cloud. Even, with limited exposure to private data centers. I have experienced numerous cases of flooding, fires, storage faults, dug-over cables, network faults, and accidental deleting of resources by 3rd party. These things haven't gone anywhere, also in the public cloud. Public clouds are not just good, but excellent at operating hyper-scale data center operations reliably. Yes, there are cases when black swan events happen, famous AWS "special" region downtimes, or this GCP fuckup. Things happen, but in the public cloud, it happens very rarely. Of course, if the public cloud providers starts pulling "Boeing's approach" on reliability it will be over. But I don't think so. Why? Because it's a big money-maker for them. The "offsite backup" concept is not new and 100% it does not mean the same provider or same account. That's why it's "offsite". I am not saying, that you don't need to do anything, to prevent black swans or "all is good". Historically and statistically running workloads in the public cloud is much more reliable than own a data center. But you still need to follow basic business continuity principles like real offsite backups.

People are starting to wake up to the fact that cloud is not a panacea. There are lots of benefits to cloud computing but assuming that it is all upside and no downside is a really, really, really bad idea. A bad idea that, unfortunately, a customer of Google recently discovered. UniSuper is (was?) a customer of Google Cloud when, unfortunately Google somehow deleted their entire account. Not just a virtual machine or two, not just a backup or data store, but the entire account. Which resulted in the complete removal of all pieces of data related to the account: "UniSuper, an Australian pension fund that manages $135 billion worth of funds and has 647,000 members, had its entire account wiped out at Google Cloud, including all its backups that were stored on the service." Unfortunately this also included any internal Google's backups: "Every cloud service keeps full backups, which you would presume are meant for worst-case scenarios. Imagine some hacker takes over your server or the building your data is inside of collapses, or something like that. But no, the actual worst-case scenario is 'Google deletes your account,' which means all those backups are gone, too. Google Cloud is supposed to have safeguards that don't allow account deletion, but none of them worked apparently, and the only option was a restore from a separate cloud provider (shoutout to the hero at UniSuper who chose a multi-cloud solution)." Cloud has a lot of pluses but it's important to remember that there is always a risk of failure, even of complete and total data loss. Even in the cloud.

https://lnkd.in/gbKgByt6 The cloud is just someone else's computer. And sometimes that other person inexplicably deletes all of your stuff. The cloud is also no substitute for a well thought out business continuity plan. I know of, and have personal firsthand experience with, too many BIG companies, publicly traded even, who are 100% cloud based and have no backups outside of their cloud provider. I don't understand why auditors, underwriters, shareholders, let them get away with it aside from ignorance or discounting the risk.