Incident Report

Summary

On Wednesday, 06 March 2024, Google Workspace users experienced elevated errors while accessing user or device setting pages within the Chrome Admin Console for a duration of 1 hour, 27 minutes.

To our Google Workspace customers who were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

Google Workspace Admin Console uses a backend authentication service to grant privileged access to internal accounts. As part of ongoing efforts to migrate the account authorization workflow to a new backend authentication service, Google engineers identified and cleaned up invalid account role assignments in the configuration.

During this clean up effort, engineers identified a privilege which was undefined in the application manifest and removed it from the configuration. Upon further investigation, engineering confirmed the privilege was mapped to an access control setting used by the Admin Console for Chrome. This change resulted in users being unable to access Chrome user and device settings page.

Remediation and Prevention

Google engineers were alerted to the outage via an internal monitoring alert on Wednesday, 06 March 2024 at 15:04 US/Pacific and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers initiated a rollback of the configuration change at 15:36 and the incident was fully resolved at 15:59 once the rollback completed.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Improve procedural documentation on how to safely cleanup access control settings from authorization configuration.
• Complete thorough investigation of privilege usage for the Admin Console to prevent further issues of this type.
• Identify and register a new owner for this privilege in the application manifest and create permission mapping for faster verification in the future.

Google is committed to quickly and continually improving our technology and operations to prevent service disruptions. We appreciate your patience and apologize again for the impact to your organization. We thank you for your business.

Detailed Description of Impact

On Wednesday, 06 March 2024 from 14:32 to 15:59 US/Pacific, Google Workspace users attempting to work within the Admin Console encountered an HTTP 500 error when accessing user or device setting pages within the console.

Mini Incident Report

We apologize for the inconvenience this service outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using the help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 6 March 2023 14:32

Incident End: 6 March 2023 16:29

Duration: 1 hours, 57 minutes

Affected Services and Features:

• Admin Console Pages for Chrome user settings
• Chrome device settings
• Chrome devices

Regions/Zones: Global

Description:

Google Workspace users attempting to work within the Admin Console encountered an HTTP 500 error when accessing user or device setting pages within the console. From preliminary analysis, the root cause of the issue is a change to the access control setting in a backend authentication service used by the Admin Console. This change resulted in users being unable to access the page.

Our Engineering team was alerted by our internal monitoring systems and swiftly mitigated the issue by rolling back the change to the access control settings.

This impacted users ability to view the current user and device policies, change or edit the existing policies and their ability to create new policies.

Google will complete a full Incident Report (IR) in the following days that will provide a detailed root cause.

Customer Impact:

• The impacted customers encountered a HTTP 500 error while accessing user or device setting pages within Chrome Admin Console.
• Additionally, customers were unable to view or change any Chrome policies.

Summary:

Admin Console experienced elevated errors with the setting pages.

Description:

The issue with Admin Console has been resolved for all affected users as of Wednesday, 2024-03-06 16:29 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Symptoms:

The impacted customers would have encountered a HTTP 500 error while accessing user or device setting pages within Chrome Admin Console. Additionally, customers were unable to view or change any policies.

Workaround:

None at this time...

Summary:

Admin Console is experiencing elevated errors with the Chrome setting pages.

Description:

Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Wednesday, 2024-03-06 16:15 US/Pacific.

We will provide more information by Wednesday, 2024-03-06 16:45 US/Pacific.

Customer Symptoms:

The impacted customers would encounter a HTTP 500 error while accessing user or device setting pages within Chrome Admin Console. Additionally, customers would not be able to view or change any policies.

Workaround:

None at this time.