What Is Snowflake and How Is It Superior to Previous Generation Technologies?

Companies all over the world are migrating to the cloud to enable their businesses to cope with the fast-changing technological landscape and customer needs. The transformational phase can either be gradual, where old apps are modernized one after the other, or complete, where the entire tech stack is modified in one go. No matter which approach you take, migrating legacy software to the cloud is not an easy task. In fact, the very nature of data in old systems sometimes does not even meet modern-day’s quality standards.

There is more. Data issues aside, you need to pick a cloud provider that can easily meet your business requirements. In other words, the one that can quickly provide all the needed services through seamless integration and at affordable rates.

Speaking of swift integration and smooth workflows, Snowflake has been attracting the attention of many businesses worldwide. In this article, we discuss what Snowflake is and how your company stands to benefit from it.

What Is Snowflake, and Why Is It Important?

Snowflake is a cloud data platform available on Google Cloud, AWS, and Microsoft Azure. Its purpose is to meet data analytics demands, like processing structured and semi-structured data on any scale with minimal administrative overheads.

This platform offers a data warehouse-as-a-service solution to companies with diverse data & analytics needs, from those with a small and quiet paddle of data to those willing to surf a tsunami of data. There are multiple reasons why Snowflake is steadily becoming the go-to data warehouse for many companies. To begin with, you do not need to install software or hardware. Neither do you need to run tedious and time-consuming configuration and management operations.

This cloud computing solution boasts a smart technology that automatically optimizes data storage and processing. There will be no need for manual fine-tuning or optimization unless when handling massive datasets.

On top of its primary data warehousing capability, Snowflake also supports other enterprise requirements and workloads, such as data science, data lake, data engineering, streaming, hybrid unistore OLAP/OLTP functionality, data marketplace, data sharing, AI/ML, data recovery and privacy, and a lot more. It offers all that quickly and efficiently with little to no need for fine-tuning, maintenance, or administrative oversight.

In the spirit of keeping things simple while meeting a company’s core needs, Snowflake recently added the Snowsight update to its ecosystem. This update facilitates collaboration by using various resources like BI, data ops, data analysis use cases, and dashboards.

Technically, the rich technology ecosystem that Snowflake comes with lets companies leverage other vital tools and resources, such as BI, ELT/ETL, data quality, and multiple others. What is amazing is these tools come pre-installed and ready to deploy.

What Problems Does Snowflake Solve?

Removing Performance Issues, Bottlenecks, and Resource Contention Between Different Workloads

Snowflake’s biggest advantage is its capability to dissociate storage and computing activities. Older-generation data warehouse engines could not separate the two, particularly those designed to function on-premise. This made scaling either storage or computing a costly problem since you needed to scale both.

Things are different with Snowflake. Its architecture puts computing and storage layers apart, so each can be scaled up and/or down efficiently, independently, and elastically.

Snowflake will store all the data you feed, whether structured or semi-structured, in a format tailored for analytical workloads. It will compress the data and calculate and store metadata, enabling optimized query planning and execution.

The Snowflake computing layer boasts the support of multiple virtual warehouses (computing clusters of different sizes) whose purpose is to process user queries. What is fascinating is that each cluster individually accesses and analyzes data from the storage layer, not contending with other clusters.

Inefficiencies Arising from Dealing with Different Data Sources

Data arises from multiple sources, such as transactional systems, mobile phones, websites, IoT devices, social media apps, gaming platforms, and many more. These days, a significant amount of this data is likely to be in a semi-structured format.

Now, with traditional warehousing solutions, you often needed to maintain disparate infrastructure as many of them cannot handle semi-structured data well. Snowflake extends SQL, supporting queries on semi-structured data and can process data in such formats as JSON and XML.

Furthermore, Snowflake is compatible with Amazon’s S3, Azure ADLS, and Google Cloud Storage and can tap into data sitting in these object storage systems. Therefore, organizations will have a smooth time dealing with data fetched from different sources.

Costly, Lengthy, and Atrocious Data Recovery Process

Losing data is not uncommon in a business environment. That is why every company is advised to have data recovery solutions. But, with some data technologies, this process can be costly, lengthy, and painful.

Thankfully, Snowflake makes data recovery an easier and cheaper endeavor. There is a feature that lets you restore data that was intentionally or accidentally deleted without a need to involve administrators and wait for backup restores. It is also possible to retrieve an object’s version as it was at a particular point in time in the past.

The fact that Snowflake allows for immediate and automatic data recovery makes it a great tool for deploying new data features without worrying about potential disasters. After all, you can simply roll back, make adjustments and continue with testing and CI/CD efforts.

Inconsistent, Questionable, and Poor Data Sharing Because of the Absence of a Single Point of Truth

As traditional data warehouses had storage tied to a computer, they often became silos of departments, making sharing data across the organization extremely hard.

Snowflake, on the other hand, keeps all data in a single layer. This means granted permission, company departments and branches can access up-to-date data without having to copy, transform or move it. Data costs become considerably lower.

The engine comes with permission gates that let you decide who can see or access certain types of data within your company. Due to its ability to make data centrally available, Snowflake is now a marketplace for publishing and discovering data products either publicly or within an organization ecosystem of business units and partners.

What Makes Snowflake Superior to Its Predecessors?

The Snowflake’s architecture is not entirely novel. However, it has a few unique features that put it a few steps ahead of its predecessors. The data layer is scalable and globally available for sharing between business units and partners, helping to eliminate data silos.

You can instantly scale any workload on this engine without stopping other systems or waiting for clusters to rebalance. To the relief of financial managers, Snowflake lets companies pay for what they are using only. Additionally, its ability to store and manipulate semi-structured data in the same manner as structured data makes it a very effective tool. And all this power comes with the lowest need for maintenance and administrative overhead compared to alternative data platforms.

Snowflake does not have one single package to pay for. You just pay for the resources, storage, and computing that you are using.

Tools You Will Need When Building a Solution on Snowflake

There are many open-source and commercial tools within the data ecosystem you may want to use in combination with Snowflake – ETL/ELT, data integration, modeling and engineering tools, Data Catalogs, Data Governance tools, BI, Analytics, and AI/ML. The good news is Snowflake has integrations with a significant number of them. Many of these tools are available through the 100+ Technology Partners ecosystem of Snowflake and have automated integration via Snowflake Partner Connect. A few examples of these tools include Fivetran, Matillion, dbt, Alation, Heap, Segment, Tableau, Looker, Thoughtspot, and many more.

Some tools are ready to use out of the box. Others will need a few customizations to load data to Snowflake. All in all, you need to consider the following factors before deciding on tools to use with Snowflake:

• Is it paid or open source? Open source may be a preference of a strong data engineering team as it may give more flexibility and lower dependency and infrastructure costs at the expense of engineering and maintenance overhead. Paid ETL/ELT/Data integration tools will suit a wider user audience and often better support the self-service demands of data-savvy business stakeholders.
• Can it support all kinds of data sources, including pre-built data connectors for platforms like Salesforce? It is preferable to have a single service provider that can perform all your data engineering and ETL/ELT demands.
• ETL/ELT computing locality: some tools will have their own cloud computing resources for optimized ETL/ELT processing which may result in considerable cost reduction. Others will orchestrate data pipelines and utilize Snowflake’s computing for processing.
• Data transformation abilities: you need to understand the data transformation extent and flexibility the ETL/ELT product brings to the table, its target audience, and the level of expertise needed.
• When it comes to data transformation capability, there are several tools to prioritize. Let’s begin with Visual ETL/ELT tools. They are simple to work with but lack flexibility in some places and that means you could bump into scaling problems.
• Next, there are SQL-based transformations with visual data pipeline orchestration tools. These are easy to use and usually preferred by professionals because of their smooth scalability. Sadly, there’s a steep learning curve involved.
• Another worthwhile choice is the data engineering toolset. This is basically a pool of tools for developing ETL/ETL tools through popular programming languages like Python. So, you are guaranteed superior flexibility and scalability. Along these lines, recently, Snowflake introduced Snowpark, a library and API analogous to the most popular data & analytics processing framework, Spark, where pipelines can be built on Python, Scala, and Java and be executed inside of Snowflake’s own computing layer. With all the power and flexibility of this category of tools, the catch is that you may require professional data engineers, some of whom are expensive to bring on board.
• Comprehensive documentation and/or customer service. Your data engineers and users must avoid getting stuck with the tools issues on their own. Rich documentation, user community, and proper customer service help a lot in troubleshooting any issues that could arise later on.

How to Plan Your Migration From Legacy Data Solution to Snowflake?

Anybody who tells you that a cloud migration process is simple might as well be selling you the moon. It is a delicate venture with pitfalls that must be avoided. Based on our experience, and the Snowflake partner ecosystem knowledge base, here is the list of things to take care of when migrating from a legacy data platform to Snowflake:

1. Migration Strategy and Planning

While every step in the data solution migration process matters, formulating a strategy for the entire process is perhaps the most paramount. You should consider listing success criteria, business-critical features, use cases, and outcomes; determining the scope of the migration in terms of workloads, datasets, systems, and use cases; elaborating on target Snowflake solution design and operating model; and choosing an appropriate migration implementation approach.

You may consider working with a Snowflake service partner, such as DataArt, on this migration strategy to utilize a wealth of partner experience, as well as make use of Snowflake Partner ecosystem migration guides, knowledge base, and solution accelerators.

2. Formulate Success Criteria

No matter how powerful and reliable a cloud data platform is, not everything in its features and architecture can be equally valuable for each business. Take a step further and list which Snowflake value points, features, and tools are the most relevant for your organization’s needs, and project cost model and implication for the migration.

At this early stage, you may want to go through demos and hands-on labs or implement a Snowflake proof-of-concept and proof-of-value around the business-critical tools, features, or use cases. Consider a mixture of existing use cases powered by the Snowflake architecture in a new way, as the new use cases are newly enabled by Snowflake capabilities. Again, Snowflake’s service partner can help here and rapidly navigate through this step.

3. Scope the Migration

Once you captured the success criteria, pinpoint workloads, and datasets to migrate and use cases to support maximum success. Most likely, you will come up with a staged approach, with several iterations of the migration for clusters of datasets, workloads, and use cases.

Conduct a use case/workload audit combined with a data audit to decide which databases and datasets should be shifted first.

A thorough data audit will reveal process dependencies, missing fields, data that is no longer useful, and other missing parameters. The idea here is to be able to move important data first while preventing potential migration hiccups.

Data is also likely to need special treatment in the cloud due to security or regulatory reasons.

4. Elaborate on Target Design

To properly plan the migration, spend time adding details to the target design. Think through security aspects, permission structures, and data security controls appropriate for the expected usage of your organization, workloads, and use cases. Describe how data will be organized within Snowflake, what zones and what environments will be used, and CI/CD approach, as well as operational requirements, such as roles, administration controls, cost controls, alerting, and monitoring. To complete the architecture vision, evaluate and make decisions on the toolset for data ingestion, data engineering, modeling, reporting, BI, and Analytics.

When moving data to Snowflake, you may also consider transferring some existing tools and processes. Examples include reporting/visualization, data processing frameworks, ETL/ELT, machine learning, and data science tools and processes. Once on board, consider introducing more new tools and retiring obsolete ones.

5. Formulate a Migration Approach

The migration approach should cover such aspects as data movement and ingestion, data integration and transformation logic, reporting, BI and analytics, as well as project management and team aspects for the migration implementation.

For the data movement part of the migration approach, you need to know where each piece of data is coming from or, rather, where it is hosted. This step guides you on how to properly load data to Snowflake. Moving large amounts of data to Snowflake can cripple mission-critical systems. Especially in scenarios where the network is slow.

Consider the size to calculate how much time might be taken to migrate each dataset. To effectively move large amounts of data over to a cloud warehouse, you might enlist the help of such tools as Google Transfer Appliance, Azure Data Box, and AWS Snowball.

The most daunting aspect of migration is data transformation and business logic. If it is written in a vendor-specific language like a dialect of SQL, you cannot simply copy-paste it to another data warehouse. You will need an expert assessment on the best approach for migration of such logic.

Being a Snowflake partner, DataArt has access to a pool of knowledge and tools for standard migrations collected by the cloud warehouse provider over time. That knowledge and tools help when shifting from common data warehousing technologies like Teradata, Oracle, Exadata, Vertica, and many others.

Often, as a part of the migration, companies may want to bring data solutions closer to current and future requirements, by cleansing, remodeling, and re-engineering the legacy data pipelines. If that is the case in your migration, you need to decide how much of this re-engineering you want to do as a part of the migration itself and how much you want to undertake post-migration, once the original data and logic are in Snowflake. It is generally the best practice to perform your migration as a series of incremental deliverables.

Special consideration should be given to a user-facing part of the data solution – reporting, BI, and analytics use cases. After undertaking the migration investment, business users most likely will expect an upgrade in this area. You may consider a fresh implementation of some new or existing use cases using your chosen cloud-based BI and analytics tools integrated with Snowflake, to make your users happier and increase business engagement with data.

Last but not least, make sure you cover the project management and roles and responsibilities aspects of your migration approach. There are a few valuable pointers to consider when creating a list of migration personnel. Lack of expertise is one of the reasons legacy modernization processes fail. Make sure that your team members understand the logic of the new and old data solutions. You should also include other critical aspects in your migration plan, such as the budgetary requirements, existing resources, project deadlines, and possible risks and their remedy should the shifting process go awry.

6. Implementation

With all the planning we described in the previous sections, the implementation surely will be the easiest and the smoothest part. Of course, it is not always the case.

7. Acceptance

You must ensure acceptance and adoption of a new data solution, establishing trust in the data and all the transformations applied. Offering transparency to the end users, using data lineage and data pipelines observability techniques, helps a lot. Yet, often you also need to plan for specific validation and reconciliation activities of old vs. new reports or dashboards to demonstrate that the quality of the data has not dropped with the migration.

It is also important at this stage to go back to your initial success criteria and assess and report on the attainment of the stated goals and objectives, delivering outcomes, and benefits.

8. Operating, Enhancing, and Evolving

It may feel that we are reaching the end of the story here. Yet, remember that the migration project is only the first step in the overall journey. Surely, your data solution will not stand still after the initial implementation and migration of the datasets, workloads, and use cases to Snowflake. The chances are that now they have such a powerful, flexible modern cloud data platform, your business users will be coming up with new uses, needs, and demands. The use of data and analytics is likely to increase in your business, and Snowflake and its ecosystem are perfectly placed to support the evolution of your solution for the growing benefit of your organization.

Conclusion

Snowflake is, without doubt, one of the most hyped cloud data platforms lately. The good news is it certainly lives up to its hype. Its architecture is superior and boasts a unique query engine that fosters faster workloads while consuming very few resources.

Also, workloads on Snowflake scale independently. This way, businesses can save money by paying for the resources they are using. Being a recent technology, Snowflake has all the strengths of previous data warehouses while striving to address their flaws. This makes it a must-try choice.

However, since this is a relatively new technology on the market, not many migration experts are familiar with its architecture and implementation best practices. Thankfully, here at DataArt, we are part of the Snowflake community and understand its ecosystem deeply.

Furthermore, as a cloud solution implementation partner, we help companies set up goals, milestones, and a timeline for the entire migration process and then support them with smooth end-to-end project delivery.

Originally published here.