How Deutsche Bank manages certificates in Google Cloud at scale

Deutsche Bank has provided a wide range of financial services to corporations, governments, institutional investors, and individuals across the world since 1870. To better serve and protect our customers, part of our cloud transformation journey required software engineers to build a secure, scalable, and reliable certificate management system for cloud workloads.

Deutsche Bank partnered with Google Cloud Professional Services to efficiently and securely manage the encryption of data in transit for hundreds of the company’s applications. That was no small feat, since the company’s business-critical applications require tens of thousands of new certificates for network communication encryption each day.

The importance of certificates

Digital identities are of utmost importance in securing infrastructures and applications, and X.509 certificates are a common way to represent them. They have been widely used to help secure network communication, email, and general purpose encryption. Widespread adoption of service-oriented solution architectures has expanded the number of entities that can be secured through certificates. Each digital certificate has a finite lifespan and has to be renewed before expiry to function properly.

While the industry is moving to shorter certificate lifetimes to better manage security risks, doing so increases the frequency of the renewal process. Failure to renew a certificate, to revoke an unused or compromised certificate, or to distribute trusted certificates can have severe consequences including service outages and security breaches.

Google Cloud Certificate Authority Service

Google Cloud Certificate Authority Service (CAS) is a cornerstone for extending on-premises public key infrastructure (PKI) services to cloud workloads. With its ability to serve as a fully-managed central authority for all enterprise workloads running in public clouds, CAS brings the central component for enterprise certificate management from on-prem to the cloud and provides convenient tools and services for an automated and integrated solution that complements CAS integrations.

Having the ability to issue custom organization certificates in a scalable, secure, on-demand way is only the beginning of our certificate management story. With the dozens of managed services that Google Cloud offers, the use of CAS certificates needs to expand beyond current use-cases to protect communication between cloud and on-prem system components.

With that, we come to the need for a centralized, automated custom certificate lifecycle management solution that can help remove the burden of cloud services configuration from the application developer teams, centralizes those tasks, and automates it to be manageable by a finite-sized team.

Challenges with large-scale certificate management

Enterprises have various organizational and regulatory requirements and business preferences in regard to the certificates. As a heavily security-focused and regulated industry, we have further goals to address. Here are some of the most important ones:

Operation

• Automate as much as possible to reduce cost, and minimize availability and security risks.
• Make certificate management transparent for applications, so that the application teams don't need to worry about certificate expiration and renewal.

Security

• Ensure highest level of client trust for internet facing applications. Protect the brand and the users by securing applications through certificates that have been issued by the public certificate authorities (CA) using the most strict identity verification steps.
• Establish discrete trust boundaries for production and non-production environments. To avoid data breaches and establish stronger security in production environments, different trust anchors must be created to issue certificates.
• Ensure application domain ownership validation for certificate issuance. Enterprises have thousands of applications and services running in their landscape. They need to control which subdomains are allowed to be used by the applications. That is, any certificate request of an application for an unauthorized subdomain must be rejected.

Governance

• Ensure that certificates issued by only approved CAs are used. Enterprises have policies to regulate the cryptographic services used within the organization to help them manage trust and to reduce risk.
• Monitor certificates issued in the organization. Keeping inventory of certificates with all related information about their owners, location and algorithms in use helps reporting and identifying impact and response to the current and future cryptography related threats. For example, future threats indicated by the post quantum revolution.
• Ensure that specific certificate types are in use where they are required by regulators. For example, EU directive for Electronic Payment Services (PSD2) enforces the use of QWAC and QSEAL certificates issued by the trust service providers defined in the European Union eIDAS Regulation

Automation is crucial for effectively managing so many certificates.

Certificate management automation

Our certificate management solution uses CAS and several other Google Cloud services to address our challenges. The figure below depicts its high-level architecture.

This solution can help with several use cases:

• Initial enrollment and certificate provisioning: Provision a new certificate with a valid identity (Subject, Subject Alternative Name) compliant with the organization security policy requirements through IaaC.
• Certificate content update: Change the attributes (Subject Alternative Name) of an existing certificate.
• Certificate renewal: Provision a new re-validated certificate before the expiration of the existing certificate and replace the expiring certificate with the newly created certificate automatically without the application team intervention.
• Certificate revocation: Revoke existing certificate to invalidate and eventually stop data exchange over network connections protected with this certificate, which could occur as a result of certificate key compromise or loss.
• Trust Anchor management: Provide golden up-to-date source for Trust Anchors (approved root CA certificates) for services and applications which are making certificate validation. Distribute and store trusted CA certificates in the client and server applications so that they can validate TLS certificates.
• Certificate Authority renewal: Certificate Authority certificates can expire after a certain period of time and need to be renewed. Following industry best practice, the lifetime of the CAs is becoming shorter.

Impact and Benefits

There are multiple benefits that organizations should start to see once the central certificate management solution is in place.

At Deutsche Bank, there’s a dedicated security team that runs and centrally manages all certificates in the organization. Given the size of Deutsche Bank, that wouldn’t have been possible without automation of routine processes and compliant PKI services. Improving the process by eliminating manual efforts and managing certificates automatically has reduced costs, and also decreased the likelihood of application outages. Since we deployed the solution to production, we haven’t encountered application incidents due to certificate expiration.

High availability of applications, especially for the internet facing services, is of a great importance for the enterprises due to reputation, cost and regulation compliance. Using Google Cloud services to build certificate management automation has not only resulted in a modern, reliable future proof solution, but also met natively the high availability and scalability requirements without additional efforts.

Our solution makes data in transit encryption straightforward for the application teams and supports Deutsche Bank's overall defense-in-depth approach. It ensures secure key creation with high entropy and prevents usage of weak keys. Moreover, these highly sensitive cryptographic assets are encrypted (not exposed in plaintext) throughout the process, apart from the owning application or resource. We can now also reliably reduce the lifetime of certificates.

Certificate monitoring, control, and responsibility is owned by a team who concentrate and excel in the difficult discipline of their subject matter expertise. Every issued internal certificate that has ever been in use in a Deutsche Bank cloud environment is recorded and kept track of in a central, monitored location in CAS.

Conclusion

With the help of core Google Cloud services such as Cloud Key Management Service and Certificate Authority Service, we could scale the certificate management process to the required levels with minimal resources. We started adopting CAS for our cloud infrastructure while it was still in its beta stage, and succeeded in building a reliable certificate management platform in the Google Cloud and achieving our goals.