Monitoring Snowflake Usage & Spend

Monitoring Snowflake Usage

One of the most frequently asked questions I hear is how can we monitor Snowflake usage and costs. This article gives a brief summary of the charges on the Snowflake platform, explains how to monitor Snowflake usage, and therefore spend over time.

Virtual Warehouses and T-Shirts

Unlike other cloud-based analytic platforms which charge by the hour, Snowflake uniquely charges per-second for the compute resources, named Virtual Warehouses. A Virtual Warehouse consists of a cluster of machines (CPUs, memory and SSD) arranged as a Massively Parallel Processing (MPP) server with between 1 and 128 nodes. These are arranged in a series of “T-Shirt” sizes, with charges billed as “credits”, costing from a little as $2.00 per hour as illustrated below:

It’s therefore relatively easy to estimate the cost of running a server by simply taking:

• Warehouse Size:  For example, a Medium warehouse which has 4 nodes, and enough power to summarize and report on gigabytes of data.
• Hours per Day:  The server is expected to run, for example, 8 hours per working day.  Note:  a virtual warehouse will automatically suspend and resume when not being used to avoid unnecessary charges.

This should give a reasonable estimate of cost, and the size can be dynamically adjusted depending upon the data volumes expected and the performance required.

Usage Control

Other usage should be relatively small by comparison and include the cost of storage which is a simple pass-through charge from the cloud provider. This is typically less than $25 per terabyte per month, and data is automatically compressed at a ratio of between 4-10 times giving additional savings.

Finally, sophisticated features including near-real time data ingestion using Snowpipe, automatic data clustering, and materialized view refreshes use internal Snowflake resources and are charged on a per-second-per-CPU-core basis and may add an additional element.

Typically, however, you should expect around 80% of spend on virtual warehouses and should be the initial focus for analysis.

Custom Monitoring

Snowflake provides two locations to monitor usage:

• Information Schema:  Which provides a set of system-defined views and metadata with real-time statistics about queries.  This data is typically retained for up to 14 days and is most useful to monitor ongoing queries.
• Snowflake Account Usage:  Provides a rich set of historical metadata that is held for up to a year and will be the primary source of usage analysis over time.

These areas provide a rich set of metadata that can be analyzed. However, it’s much easier to simply show the potential results below.

Warehouse Credits Over Time

The above query can be used to monitor both the monthly spend by all warehouses on the account over the past 12 months in addition to indicating the growth over time.

Monthly Credits by Type

The above query shows a breakdown of the total credits per month, indicating the type of spend. This includes:

• Warehouses:  Credits used by compute resources
• Pipes:  Credits used by the near real-time loading facility, Snowpipe
• Mview:  Credits used to refresh Materialized Views
• Clustering:  Credits used by the automatic clustering mechanism
• Reader:  Credits used by Reader Accounts, typically used to share access to data

Top 10 Spending Users

The above query is a guestimate of the credit by the top 10 users on the system. Although the ranking is correct, the credit spend is a worst-case estimate because although warehouses are charged by the second, any given warehouse can have a number of executing queries at one time, and the above report indicates a potential worst case, where a warehouse is used by a single user.

A more accurate indication of spend can be determined at the warehouse level below.

Actual Spend by Warehouse

Credits Used by Hour

The above query indicates what time of the day queries are executing, and credits spent. This may be useful to indicate the peak times during the day, in addition to identifying unexpected high spending during what is normally a quiet time overnight.

Data Storage Cost by Month

The report above shows the monthly cost for storage broken by type. These indicate the cost by:

• Storage: This indicates the data stored in Snowflake tables.
• Stages:  Which indicates the data stored in files in Snowflake stages
• Failsafe: This indicates the storage in the 7-day failsafe area to protect against accidental data loss.

Freely Available Dashboards

There are also a number of freely available dashboards for the major Business Intelligence vendors below:

• Tableau - Snowflake Account Usage
• Looker - Snowflake Cost and Usage Analysis
• Microsoft PowerBI - Snowflake Usage Report
• Qlik - Snowflake Usage Dashboard

If you know of another vendor or an alternative dashboard, feel free to reach out to me on LinkedIn.

Disclaimer: The opinions expressed in my articles are my own and will not necessarily reflect those of my employer (past or present) or indeed any client I have worked with.

First published on my personal blog site: www.Analytics.Today. Sign up today for regular updates on all things Snowflake.