Kai Waehner’s Post

View profile for Kai Waehner, graphic
Kai Waehner Kai Waehner is an Influencer

Global Field CTO | Author | International Speaker | Follow me with Data in Motion

"Tales of #apachekafka at the content delivery network #cloudflare: Lessons Learnt on the Way to 1 Trillion Messages" => Here is the summary of a great article about lessons learned and best practices: - #kafka clusters are used at Cloudflare to process large amounts of data, with a general-purpose message bus cluster developed to decouple teams, scale effectively, and process trillions of messages. - To address the issue of unstructured communication for event-driven systems, a strong contract should be in place: cross-platform data format #protobuf helped Cloudflare achieve that. - Investing in metrics on development tooling is critical to allow problems to be easily surfaced: Cloudflare enriched the SDK with #OpenTracing and #Prometheus metrics to understand how the system behaves and make better decisions, especially during incidents. - To enable consistency in the adoption and use of SDKs and promote best practices, it is important to prioritize clear documentation on patterns. - Cloudflare aims to achieve a balance between flexibility and simplicity: while a configurable setup may offer more flexibility, a simpler one allows standardization across different pipelines. More details in the #infoq article: https://lnkd.in/eFGPGukq

  • No alternative text description for this image

To view or add a comment, sign in

Explore topics