Incident Report

Summary

On 18 April 2024, Gmail, Google Calendar and Google Chat experienced issues with the ‘Contact Sharing’ feature for a duration of 1 day, 21 hours 43 minutes. To our Google Workspace customers whose businesses were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

An internal code update unintentionally altered the default values of a setting that interacts with the current contact sharing system. This change resulted in a misconfiguration of directory settings, impacting the Contact Sharing feature within Google Workspace services like Gmail, Chat, and Calendar.

Remediation and Prevention

We were alerted to the outage via a support case on April 19 at 7:12 US/Pacific and promptly initiated an investigation. The investigation confirmed that the issue originated from a code change deployed on April 18 at 14:40 US/Pacific. Upon troubleshooting, we immediately halted and rolled back the code change, preventing a wider impact on Google Workspace services.

We then focused on repairing the issue for affected customers.

We apologize for the length and severity of this incident. Google is committed to preventing recurrence by implementing the following action items:

1. Engineering team implemented release controls to prevent code changes that alter the default values from being deployed to the production environment.

2. Engineering team to add test coverage and verify code rollouts like these so issues like this can be detected early.

3. Engineering team to set up alerts for unexpected spikes in compilation requests during automated syncs, as this could indicate a potential problem with data changes.

Detailed Description of Impact

On April 18 at 14:40 US/Pacific an internal update unexpectedly turned off the "Contact Sharing" feature for Google Workspace users. This resulted in features like autocompleting email addresses and seeing contact details in Gmail, Chat, and Calendar not working as expected.

Known user-facing impacts are as follows:

Gmail

Affected users may not have seen a contact list in Gmail. Sending emails were not affected. Autocomplete for individual and group addresses entered by users was affected. Hovercards, also known as profile information cards that show up when users pointed to or tapped on someone’s profile image did not work.

Google Contacts

Impacted users could not find profile information in Google Contacts.

Google Calendar

Searching for users, and intelligent meeting room suggestions in Google Calendar were affected.

A mini incident report has been posted at https://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E

The problem with Gmail, Google Calendar, Google Chat has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

Further communication will be provided on the below link.

https://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E

Further communication will be provided on the below link.

https://www.google.com/appsstatus/dashboard/incidents/XKnCyj2ED7njHXsC7a6E

Our Engineering team has identified the root cause of the issue and rolled back a recent change.

Our engineers are taking additional repair actions for the rollback to take effect for the impacted customers.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2024-04-19 14:00 US/Pacific.

Engineering team potentially identified the root cause and working on mitigation.

The mitigation is expected to complete by Friday, 2024-04-19 12:15 US/Pacific.