Collate and route organization- and folder-level logs to supported destinations

This document describes how to create aggregated sinks. Aggregated sinks let you combine and route logs that are generated by the Google Cloud resources in your organization or folder to a centralized location.

Overview

Aggregated sinks combine and route log entries from the resources contained by an organization or folder to a destination.

If you want control over which log entries can be queried in these resources, or routed through the sinks in these resources, then you can configure an aggregated sink to be non-intercepting or intercepting:

  • A non-intercepting aggregated sink routes log entries through sinks in child resources. With this sink, you maintain visibility of log entries in the resources in which they were generated. Non-intercepting sinks aren't visible to child resources.

    For example, you might create a non-intercepting aggregated sink that routes all log entries generated from the folders contained by an organization to a central log bucket. The log entries are stored in the central log bucket, and also in the resources in which the log entries were generated.

  • An intercepting aggregated sink prevents log entries from being routed through sinks in child resources, except for _Required sinks. This sink can be useful in preventing duplicate copies of log entries from being stored in multiple places.

    For example, consider Data Access audit logs, which can be large in volume and expensive to store multiple copies of. If you've enabled Data Access audit logs, you might create a folder-level intercepting sink that routes all Data Access audit logs to a central project for analysis. This intercepting sink also prevents sinks in child resources from routing copies of the logs elsewhere.

    Intercepting sinks prevent logs being passed through the Log Router of child resources, unless the logs also match the _Required sink. Because the logs are intercepted, the logs don't count towards log-based metrics or log-based alerting policies in the child resources. You can view intercepting sinks in the Log Router page of child resources.

For information about managing sinks, see Route logs to supported destinations: Manage sinks.

You can create up to 200 sinks per folder or organization.

Supported destinations

You can use non-intercepting aggregated sinks to route log entries within or between the same organizations and folders to the following destinations:

  • Cloud Logging bucket: Provides storage in Cloud Logging. A log bucket can store log entries that are received by multiple Google Cloud projects. The log bucket can be in the same project in which log entries originate, or in a different project. For information about viewing log entries stored in log buckets, see Query and view logs overview and View logs routed to Cloud Logging buckets.

    You can combine your Cloud Logging data with other data by upgrading a log bucket to use Log Analytics, and then creating a linked dataset, which is a read-only dataset that can be queried by the BigQuery Studio and Looker Studio pages.

  • BigQuery dataset: Provides storage of log entries in a writeable BigQuery dataset. The BigQuery dataset can be in the same project in which log entries originate, or in a different project. You can use big data analysis capabilities on the stored log entries. For information about viewing log entries routed to BigQuery, see View logs routed to BigQuery.

  • Cloud Storage bucket: Provides storage of log entries in Cloud Storage. The Cloud Storage bucket can be in the same project in which log entries originate, or in a different project. Log entries are stored as JSON files. For information about viewing log entries routed to Cloud Storage, see View logs routed to Cloud Storage.

  • Pub/Sub topic: Provides support for third-party integrations. Log entries are formatted into JSON and then routed to a Pub/Sub topic. The topic can be in the same project in which log entries originate, or in a different project. For information about viewing log entries routed to Pub/Sub, see View logs routed to Pub/Sub.

  • Google Cloud project: Route log entries to another Google Cloud project. In this configuration, the sinks in the destination project processes the log entries.

Best practices for intercepting sinks

When you create an intercepting sink, we recommend that you do the following:

  • Consider whether child resources need independent control of routing their log entries. If a child resource needs independent control of certain log entries, ensure that your intercepting sink doesn't route those log entries.

  • Add contact information to the description of an intercepting sink. This might be helpful if those who manage the intercepting sink are different from those who manage the projects whose log entries are being intercepted.

  • Test your sink configuration by first creating a non-intercepting aggregated sink to ensure the correct log entries are being routed.

Aggregated sinks and VPC Service Controls

The following limitations apply when you use aggregated sinks and VPC Service Controls:

  • Aggregated sinks can access data from projects inside a service perimeter. To restrict aggregated sinks from accessing data inside a perimeter, we recommend using IAM to manage Logging permissions.

  • VPC Service Controls doesn't support adding folder or organization resources to service perimeters. Therefore, you can't use VPC Service Controls to protect folder- and organization-level logs, including aggregate logs. To manage Logging permissions at the folder or organizational level, we recommend using IAM.

  • If you route logs by using a folder- or organization-level sink to a resource that a service perimeter protects, then you must add an ingress rule to the service perimeter. The ingress rule must allow access to the resource from the service account that the aggregated sink uses. For more information, refer to the following pages:

  • When you specify an ingress or egress policy for a service perimeter, you can't use ANY_SERVICE_ACCOUNT and ANY_USER_ACCOUNT as an identity type when you use a log sink to route logs to Cloud Storage resources. However, you can use ANY_IDENTITY as the identity type.

Before you begin

Before you create a sink, ensure the following:

  • You have a Google Cloud folder or organization with log entries that you can see in the Logs Explorer.

  • You have one of the following IAM roles for the Google Cloud organization or folder from which you're routing log entries.

    • Owner (roles/owner)
    • Logging Admin (roles/logging.admin)
    • Logs Configuration Writer (roles/logging.configWriter)

    The permissions contained in these roles let you create, delete, or modify sinks. For information about setting IAM roles, see the Logging Access control guide.

  • You have a resource in a supported destination or have the ability to create one.

    The destination must be created before the sink, through either Google Cloud CLI, Google Cloud console, or the Google Cloud APIs. You can create the destination in any Google Cloud project in any organization, but you must ensure that the service account from the sink has permissions to write to the destination.

  • Select the tab for how you plan to use the samples on this page:

    Console

    When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

    gcloud

    In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

    REST

    To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

      Install the Google Cloud CLI, then initialize it by running the following command: 

      gcloud init

    For more information, see Authenticate for using REST in the Google Cloud authentication documentation.

Create an aggregated sink

Console

To create an aggregated sink for your folder or organization, do the following:

  1. In the Google Cloud console, go to the Log Router page:

    Go to Log Router

    If you use the search bar to find this page, then select the result whose subheading is Logging.

  2. Select an existing folder or organization.

  3. Select Create sink.

  4. In the Sink details panel, enter the following details:

    • Sink name: Provide an identifier for the sink; note that after you create the sink, you can't rename the sink but you can delete it and create a new sink.

    • Sink description (optional): Describe the purpose or use case for the sink.

  5. Do one of the following:

    • To create an intercepting sink, in the Select sink service menu, select Google Cloud project, and then enter the fully-qualified name for the destination. For information about the syntax, see Destination path formats.

    • To create a non-intercepting sink, go to the Select sink service menu and do one of the following:

      • To route log entries to a different Google Cloud project, select Google Cloud project, and then enter the fully-qualified name for the destination. For information about the syntax, see Destination path formats.

      • To route log entries to a service that is in the same Google Cloud project, select one of the following options:

        • BigQuery dataset: Select or create the particular dataset to receive the routed log entries. You also have the option to use partitioned tables.
        • Cloud Storage bucket: Select or create the particular Cloud Storage bucket to receive the routed log entries.
        • Pub/Sub topic: Select or create the particular topic to receive the routed log entries.
        • Splunk: Select the Pub/Sub topic for your Splunk service.

      • To route log entries to a service that is in a different Google Cloud project, do the following:

        1. Select Other resource.

        2. Enter the fully-qualified name for the destination. For information about the syntax, see the Destination path formats.

          For example, if your sink destination is a Pub/Sub topic, then the destination looks like the following:

          pubsub.googleapis.com/projects/PROJECT_ID/topics/TOPIC_ID

  6. In the Choose logs to include in sink panel, do one of the following:

    • To create an intercepting sink, select Intercept logs ingested by this organization and all child resources.

    • To create a non-intercepting aggregated sink, select Include logs ingested by this resource and all child resources.

  7. Complete the dialog by entering a filter expression in the Build inclusion filter field that matches the log entries you want to include. If you don't set a filter, all log entries from your selected resource are routed to the destination.

    For example, you might want to build a filter to route all Data Access audit logs to a single Logging bucket. This filter looks like the following:

    LOG_ID("cloudaudit.googleapis.com/data_access") OR LOG_ID("externalaudit.googleapis.com/data_access")

    For filter examples, see Create filters for aggregated sinks.

    Note that the length of a filter can't exceed 20,000 characters.

  8. Optional: To verify you entered the correct filter, select Preview logs. This opens the Logs Explorer in a new tab with the filter pre-populated.

  9. Optional: In the Choose logs to exclude from the sink panel, do the following:

    1. In the Exclusion filter name field, enter a name.

    2. In the Build an exclusion filter field, enter a filter expression that matches the log entries you want to exclude. You can also use the sample function to select a portion of the log entries to exclude.

      For example, to exclude the log entries from a specific project from being routed to the destination, add the following exclusion filter:

      logName:projects/PROJECT_ID

      To exclude log entries from multiple projects, use the logical-OR operator to join logName clauses.

    You can create up to 50 exclusion filters per sink. Note that the length of a filter can't exceed 20,000 characters.

  10. Select Create sink.

  11. Grant the service account for the sink the permission to write log entries to your sink's destination. For more information, see Set destination permissions.

gcloud

To create an aggregated sink, use the logging sinks create command:

  1. To create a sink, call the gcloud logging sinks create command, and ensure that you include the --include-children option.

    Before using the following command, make the following replacements:

    • SINK_NAME: The name of the log sink. You can't change the name of a sink after you create it.
    • SINK_DESTINATION: The service or project to where you want your log entries routed.
    • INCLUSION_FILTER: The inclusion filter for a sink. For filter examples, see Create filters for aggregated sinks.
    • FOLDER_ID: The ID of the folder. If you want to create a sink at the organization level, then replace --folder=FOLDER_ID with -- organization=ORGANIZATION_ID.

    Execute the gcloud logging sinks create command:

    gcloud logging sinks create SINK_NAME \
  SINK_DESTINATION  --include-children \
  --folder=FOLDER_ID --log-filter="INCLUSION_FILTER"

    You can also provide the following options:

    • To create an intercepting sink, include the --intercept-children option.

    For example, if you're creating an aggregated sink at the folder level and whose destination is a Pub/Sub topic, your command might look like the following:

    gcloud logging sinks create SINK_NAME \
  pubsub.googleapis.com/projects/PROJECT_ID/topics/TOPIC_ID --include-children \
  --folder=FOLDER_ID --log-filter="logName:activity"

  2. Grant the service account for the sink permission to write to your sink destination. For more information, see Set destination permissions.

REST

To create an aggregated sink, use the organizations.sinks.create or folders.sinks.create Logging API method. Prepare the arguments to the method as follows:

  1. Set the parent field to be the Google Cloud organization or folder in which to create the sink. The parent must be one of the following:

    • organizations/ORGANIZATION_ID
    • folders/FOLDER_ID

  2. In the LogSink object in the method request body, do one of the following:

    • Set includeChildren to True.

    • To create an intercepting sink, also set the interceptChildren field to True.

  3. Set the filter field to match the log entries you want to include.

    For filter examples, see Create filters for aggregated sinks.

    The length of a filter can't exceed 20,000 characters.

  4. Set the remaining LogSink fields as you would for any sink. For more information, see Route logs to supported destinations.

  5. Call organizations.sinks.create or folders.sinks.create to create the sink.

  6. Grant the service account for the sink permission to write to your sink destination. For more information, see Set destination permissions.

Any changes made to a sink might take a few minutes to apply.

Create filters for aggregated sinks

Like any sink, your aggregated sink contains a filter that selects individual log entries. For examples of filters that you might use to create your aggregated sink, see Sample queries using the Logs Explorer.

Following are some examples of filter comparisons that are useful when using the aggregated sinks feature. Some examples use the following notation:

  • : is the substring operator. Don't substitute the = operator.
  • ... represents any additional filter comparisons.
  • Variables are indicated by colored text. Replace them with valid values.

Note that the length of a filter can't exceed 20,000 characters.

For more details about the filtering syntax, see Logging query language.

Select the log source

For an aggregated sink, for each child resource of the organization or folder, the sink's inclusion and exclusion filters are applied to each log entry that is sent to the child resource. A log entry that matches the inclusion filter and that isn't excluded, is routed.

If you want your sink to route log entries from all child resources, then don't specify a project, folder, or organization in your sink's inclusion and exclusion filters. For example, suppose you configure an aggregated sink for an organization with the following filter:

resource.type="gce_instance"

With the previous filter, log entries with a resource type of Compute Engine instances that are written to any child of that organization are routed by the aggregated sink to the destination.

However, there might be situations where you want to use an aggregated sink to route log entries only from specific child resources. For example, for compliance reasons you might want to store audit logs from specific folders or projects in their own Cloud Storage bucket. In these situations, configure your inclusion filter to specify each child resource whose log entries you want routed. If you want to route log entries from a folder and all projects within that folder, then the filter must list the folder and each of the projects contained by that folder, and also join the statements with an OR clause.

The following filters restrict log entries to specific Google Cloud projects, folders, or organizations:

logName:"projects/PROJECT_ID/logs/" AND ... 
logName:("projects/PROJECT_A_ID/logs/" OR "projects/PROJECT_B_ID/logs/") AND ... 
logName:"folders/FOLDER_ID/logs/" AND ... 
logName:"organizations/ORGANIZATION_ID/logs/" AND ...

For example, to route only log entries written to Compute Engine instances that were written to the folder my-folder, use the following filter:

logName:"folders/my-folder/logs/" AND resource.type="gce_instance"

With the previous filter, log entries written to any resource other than my-folder, including log entries written to Google Cloud projects that are children of my-folder, aren't routed to the destination.

Select the monitored resource

To route log entries from only a specific monitored resource in a Google Cloud project, use multiple comparisons to specify the resource exactly:

logName:"projects/PROJECT_ID/logs" AND
resource.type=RESOURCE_TYPE AND
resource.labels.instance_id=INSTANCE_ID

For a list of resource types, see Monitored resource types.

Select a sample of log entries

To route a random sample of log entries, add the sample built-in function. For example, to route only ten percent of the log entries matching your current filter, use this addition:

sample(insertId, 0.10) AND ...

For more information, see the sample function.

For more information about Cloud Logging filters, see Logging query language.

Set destination permissions

This section describes how to grant Logging the Identity and Access Management permissions to write log entries to your sink's destination. For the full list of Logging roles and permissions, see Access control.

When you create or update a sink that routes log entries to any destination other than a log bucket in the current project, a service account for that sink is required. Logging automatically creates and manages the service account for you:

  • As of May 22, 2023, when you create a sink and no service account for the underlying resource exists, Logging creates the service account. Logging uses the same service account for all sinks in the underlying resource. Resources can be a Google Cloud project, an organization, a folder, or a billing account.
  • Before May 22, 2023, Logging created a service account for each sink. As of May 22, 2023, Logging uses a shared service account for all sinks in the underlying resource.

The writer identity of a sink is the identifier of the service account associated with that sink. All sinks have a writer identity unless they write to a log bucket in the current Google Cloud project.

To route log entries to a resource protected by a service perimeter, you must add the service account for that sink to an access level and then assign it to the destination service perimeter. This isn't necessary for non-aggregated sinks. For details, see VPC Service Controls: Cloud Logging.

To set permissions for your sink to route to its destination, do the following:

Console

  1. To get information about the service account for your sink, do the following:

    1. In the Google Cloud console, go to the Log Router page:

      Go to Log Router

      If you use the search bar to find this page, then select the result whose subheading is Logging.

    2. Select Menu and then select View sink details. The writer identity appears in the Sink details panel.

    3. If the value of the writerIdentity field contains an email address, then proceed to the next step. When the value is None, you don't need to configure destination permissions.

    4. Copy the sink's writer identity into your clipboard. The serviceAccount: string is part of the service account identity. For example:

      serviceAccount:[email protected]

    5. Add the service account as an IAM principal in the destination project:

      1. In the Google Cloud console, go to the IAM page:

        Go to IAM

        If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.

      2. Select the destination project.

      3. Click Grant access.

      4. Grant the service account the required IAM role:

        • For Cloud Storage destinations, add the sink's writer identity as a principal by using IAM, and then grant it the Storage Object Creator role (roles/storage.objectCreator).
        • For BigQuery destinations, add the sink's writer identity as a principal by using IAM, and then grant it the BigQuery Data Editor role (roles/bigquery.dataEditor).
        • For Pub/Sub destinations, including Splunk, add the sink's writer identity as a principal by using IAM, and then grant it the Pub/Sub Publisher role (roles/pubsub.publisher).
        • For Logging bucket destinations in different Google Cloud projects, add the sink's writer identity as a principal by using IAM, and then grant it the Logs Bucket Writer role (roles/logging.bucketWriter).
        • For Google Cloud projects destinations, add the sink's writer identity as a principal by using IAM, and then grant it the Logs Writer role (roles/logging.logWriter). Specifically, a principal needs the logging.logEntries.route permission.

gcloud

  1. Ensure that you have Owner access on the Google Cloud project that contains the destination. If you don't have Owner access to the destination of the sink, then ask a project owner to add the writer identity as a principal.

  2. To get information about the service account for your sink, call the gcloud logging sinks describe method.

    Before using the following command, make the following replacements:

    • SINK_NAME: The name of the log sink. You can't change the name of a sink after you create it.

    Execute the gcloud logging sinks describe command:

    gcloud logging sinks describe SINK_NAME

  3. If the sink details contain a field labeled writerIdentity, then proceed to the next step. When the details don't include a writerIdentity field, you don't need to configure destination permissions for the sink.

  4. Copy the sink's writer identity into your clipboard. The serviceAccount: string is part of the service account identity.

    The writer identity for the service account looks similar to the following:

    serviceAccount:[email protected]

  5. To add the service account as an IAM principal in the destination project, call the gcloud projects add-iam-policy-binding command.

    Before using the following command, make the following replacements:

    • PROJECT_ID: The identifier of the project.
    • PRINCIPAL: An identifier for the principal that you want to grant the role to. Principal identifiers usually have the following form: PRINCIPAL-TYPE:ID. For example, user:[email protected]. For a full list of the formats that PRINCIPAL can have, see Principal identifiers.

    • ROLE: An IAM role.

      • For Cloud Storage destinations, add the sink's writer identity as a principal by using IAM, and then grant it the Storage Object Creator role (roles/storage.objectCreator).
      • For BigQuery destinations, add the sink's writer identity as a principal by using IAM, and then grant it the BigQuery Data Editor role (roles/bigquery.dataEditor).
      • For Pub/Sub destinations, including Splunk, add the sink's writer identity as a principal by using IAM, and then grant it the Pub/Sub Publisher role (roles/pubsub.publisher).
      • For Logging bucket destinations in different Google Cloud projects, add the sink's writer identity as a principal by using IAM, and then grant it the Logs Bucket Writer role (roles/logging.bucketWriter).
      • For Google Cloud projects destinations, add the sink's writer identity as a principal by using IAM, and then grant it the Logs Writer role (roles/logging.logWriter). Specifically, a principal needs the logging.logEntries.route permission.

    Execute the gcloud projects add-iam-policy-binding command:

    gcloud projects add-iam-policy-binding PROJECT_ID --member=PRINCIPAL --role=ROLE

REST

We recommend that you use the Google Cloud console or the Google Cloud CLI to grant a role to the service account.

What's next