Getting Started with Prometheus - Relabeling Metrics in Prometheus

Are you looking to get away from proprietary instrumentation? 

Are you interested in open source observability but lack the knowledge to just dive right in? 

This workshop is for you, designed to expand your knowledge and understanding of open source observability tooling that is available to you today.

Dive right into a free, online, self paced, hands-on workshop introducing you to Prometheus. Prometheus is an open source systems monitoring and alerting tool kit that enables you to hit the ground running with discovering, collecting, and querying your observability today. Over the course of this workshop you will learn what Prometheus is, what it is not, install it, start collecting metrics, and learn all the things you need to know to become effective at running Prometheus in your observability stack. 

Previously, I shared an introduction to Prometheus, installing Prometheus, an introduction to the query language, exploring basic queries, and using advanced queries as free online worksop labs. In this article you'll continue your journey with relabeling metrics in Prometheus.

Your learning path takes you into the wonderful world of relabeling in Prometheus, where you learn about aggregation, drop rules, and so much more. Note this article is only a short summary, so please see the complete lab found online here to work through it in its entirety yourself:

The following is a short overview of what is in this specific lab of the workshop. Each lab starts with a goal, in this case it is:

The lab teaches you how relabeling metrics work in Prometheus and how to apply them when mapping, aggregating, dropping, or filtering scraped metrics.

You start in this lab by ensuring the demo environment you learned about setting up in previous labs is running with two demo services and a single Prometheus instance. From there you wander through the why and how of what relabeling is for in your metrics environment.  

The basic layout of the configuration section is discussed before you embark on your first interactions with setting up static labels for your two demo services to use across the various exercises in this lab.  They allow you to relabel without interfering with existing demo service labels and provide a baseline for a few use cases you'll work through later in the lab.

Your first relabeling

A simple replacement action is the first exposure to relabeling in Prometheus and it's done by setting an existing label value to a new fixed value. To facilitate the exercise, you are given a scenario as follows:

What if we decided for the upcoming vacation period, that team B would be the team to cover services support as owner while the rest are gone? We could set a temporary relabeling rule to replace all instances of the owner to team B. Imagine if you has thousands of services deployed needing this change? 

When the vacation is over, you just remove or turn off this relabeling rule.

Each of these exercises is structured the same. You are presented with the scenario, shown the structure of the relabeling action in question, walked through the structure so you understand it before using, then shown now to implement the relabeling rule to solve the scenario. Below is the configuration and bold faced marked relabeling rule for replacing all instances of the owner label found with teamB:

# Scraping services demo.
  - job_name: "services"
    static_configs:
        - targets: ["localhost:8080"]
	labels:
            service: demo1
            owner: teamD

	- targets: ["localhost:8088"]
	labels:
	    service: demo2
	    owner: teamA

    relabel_configs:

    # Relabeling owner for teamA vacation.

    - action: 'replace'
        replacement: 'teamB'
        target_label: 'owner'

After saving your configuration, restarting your Prometheus instance for the new configuration to take effect, you are then shown how to query the running instances to verify the label changes:

After the vacation is over you complete the exercise by removing your changes so that the proper owners of the services are again labeled correctly.

More relabeling

The rest of the lab walks through many more relabeling actions with examples for aggregation, dropping labels, dropping objects, mapping, upper case and lower case.

There are even more advanced use cases for relabeling addresses, such as HA environments that have a  need for dropping metrics labels to prevent double alerts from firing. The following image is the result of applying aggregation relabeling to combine the owner and service metric labels into a new service_details metric label:

You finish up this lab with an explanation of hashing and sharding of Prometheus instances and how hashmod relabeling helps you to target sharded services.

Missed previous labs?

This is one lab in the more extensive free online workshop. Feel free to start from the very beginning of this workshop here if you missed anything previously:

You can always proceed at your own pace and return any time you like as you work your way through this workshop. Just stop and later restart Perses to pick up where you left off.

Coming up next

I'll be taking you through the following lab in this workshop where you'll start to explore how to auto discover service targets for your metric scraping with Prometheus. 

Stay tuned for more hands on material to help you with your cloud native observability journey.