"Tales of #apachekafka at the content delivery network #cloudflare: Lessons Learnt on the Way to 1 Trillion Messages" => Here is the summary of a great article about lessons learned and best practices: - #kafka clusters are used at Cloudflare to process large amounts of data, with a general-purpose message bus cluster developed to decouple teams, scale effectively, and process trillions of messages. - To address the issue of unstructured communication for event-driven systems, a strong contract should be in place: cross-platform data format #protobuf helped Cloudflare achieve that. - Investing in metrics on development tooling is critical to allow problems to be easily surfaced: Cloudflare enriched the SDK with #OpenTracing and #Prometheus metrics to understand how the system behaves and make better decisions, especially during incidents. - To enable consistency in the adoption and use of SDKs and promote best practices, it is important to prioritize clear documentation on patterns. - Cloudflare aims to achieve a balance between flexibility and simplicity: while a configurable setup may offer more flexibility, a simpler one allows standardization across different pipelines. More details in the #infoq article: https://lnkd.in/eFGPGukq
Kai Waehner’s Post
More Relevant Posts
-
I help companies to become API-first, speeding up time to market, creating new business opportunities, and delivering superior products and services | API Consultant, Integration Lead, Principal Architect
According to #Google’s 2023 State of APIs and Applications report, organizations use an average of 200 APIs. Managing over 200 APIs is challenging, and their rapid adoption and versatility will make it increasingly difficult in the future. One of the biggest challenges of API sprawl is discovering the necessary APIs and avoiding duplication. Another significant risk of API sprawl is having APIs that bypass your security and governance policies. #Apigee API Hub offers a solution. As one of Google's newest innovations, the API Hub allows organizations to view all their API assets and lifecycle information in one place. With the API Hub, you can: • Organize all your API information from multiple gateways, providers, and tools. • Standardize and maximize security, enhancing value without burdening your teams. • Reduce redundant builds and perform value-added actions on APIs. Over the past 11 years, I have helped many organizations define and implement their API governance models to direct and control the use of their APIs. I've seen the competing challenges of reducing risk while enabling speed and the difficulties of enforcing standards across different API architectures, gateways and teams without overburdening them or slowing them down. If you face similar challenges, the API Hub might be for you. The API Hub is coming soon for a public preview, and this next phase of the Apigee API Hub is just around the corner. I’ll keep you posted. If you have any questions, drop them in the comments, and I will respond promptly.
To view or add a comment, sign in
-
Associate Manager - Technology architect | API developer | AWS certified solution architect | Google Cloud Certified Engineer | Microsoft Azure Certified| Infrastructure automation | Java | Kubernetes |
How to avoid API gateway from becoming a bottleneck? Let's look into some strategies we can employ to avoid gateways becoming a bottleneck. 👉 Horizontal Scaling: API gateways too are open for horizontal scaling. Multiple instances of gateway can be placed behind a load balancer so in case there is a traffic surge, new instances can be added with ease. 👉 Lightweight gateway: We should avoid overloading gateway with too many responsibilities. Ideally, gateway should only handle routing, security and logging. Complex business rules should be moved to backend services instead of the gateway itself. 👉 Caching with Rate Limiting : Intelligent caching options should be leveraged so that frequently accessed data can be cached at gateway itself. Rate limiting is also crucial so that it can push back too much traffic and safeguard downstream services from being choked. 👉 Service Mesh Integration: Consider using a service mesh (e.g., Istio, Linkerd) alongside or instead of an API gateway for service-to-service communication, where appropriate. This reduces the load on the API gateway by offloading traffic management features to the service mesh layer. #Apigee #APIStability #AWSGateway #Performance
To view or add a comment, sign in
-
🆕 Announcing General Availability of #AzureCosmosDB Integration with Vercel! This powerful combination enables developers to build dynamic, scalable applications easily by leveraging both platforms' strengths. #MSBuild Read more here 🔗 https://lnkd.in/eSrqqkjT
To view or add a comment, sign in
-
Together, HarperDB & Akamai are bringing cost efficient solutions to enterprise customers that increase application performance and reliability while decreasing latency. In a recent webinar, HarperDB’s Field CTO, Jaxon Repp, talked about how combining this massively distributed data platform with an application layer on top of it allows organizations to really do anything. By augmenting Akamai’s existing solutions with the exact same interface, the customer can improve functionality and scalability without doing any extra work. Learn more about use cases and access relevant resources below 👇 #Akamai #HarperDB #TechPartners #DistributedData https://lnkd.in/gBV3cYNx
To view or add a comment, sign in
-
Helping organizations build robust CI/CD Pipelines and scalable infrastructure using tools like Azure DevOps | Oracle Cloud | Google Cloud | AWS Cloud| Jenkins | Argo CD | Docker | Kubernetes | Ansible| Terraform |
#IstioVsNginxController #Istio: #ServiceMesh: Istio is a full-fledged service mesh solution. It helps manage, secure, and monitor traffic between services in a microservices architecture. It provides features like traffic management, security, observability, and policy enforcement. #TrafficManagement: Istio allows you to control the flow of traffic between services through features like traffic shifting, routing rules, fault injection, and circuit breaking. #Security: It provides features like mutual TLS (mTLS), access control, and encryption to secure communication between services. #Observability: Istio offers metrics, logging, and tracing capabilities to monitor and debug microservices-based applications. #NGINXIngressController: #IngressController: NGINX Ingress Controller is primarily used for managing external access to services running within a Kubernetes cluster. It acts as a load balancer and handles incoming traffic from external sources. #Layer7Load Balancing: NGINX can perform sophisticated layer 7 load balancing based on HTTP/HTTPS attributes like URL paths, headers, cookies, etc. #TLSTermination: It can terminate TLS connections, offloading encryption/decryption tasks from backend services. #RewritesandRedirects: NGINX Ingress Controller can rewrite or redirect incoming requests based on configured rules. #RateLimiting: NGINX can apply rate limiting to incoming requests to prevent abuse or manage traffic spikes.
To view or add a comment, sign in
-
Cloud Solutions Architect | Application Architect | SRE & DevOps Engineer | Software Developer | Consultant | Trainer
Going #serverless? Worried about challenges with performance, cost & scale? @georgemao covers CapitalOne’s journey into serverless, the best practices they picked up, the lessons learned & the optimizations for Lambda: https://t.co/orBhUdCAL2 #LambdaArchitecture #InfoQ https://t.co/7CXJxeyHQ8
Going #serverless? Worried about challenges with performance, cost & scale? @georgemao covers CapitalOne’s journey into serverless, the best practices they picked up, the lessons learned & the optimizations for Lambda: https://t.co/orBhUdCAL2 #LambdaArchitecture #InfoQ https://t.co/7CXJxeyHQ8
twitter.com
To view or add a comment, sign in
-
Pipelines are at the core of Observability architectures and providers have been seriously ramping up their capacities. Dynatrace recently unveiled their OpenPipeline technology, which is architected for “petabyte-scale analytics“ while Microsoft recently announced the General Availability of the Data Plane API for Metrics, which enables faster egress into third party backends. We covered both of these in the latest Observability 360 newsletter: https://bit.ly/3SAvAol #observability #devops #dynatrace #azuremonitor
To view or add a comment, sign in
-
OpenTelemetry is a collection of tools, APIs, and SDKs for collecting and generating data. It is a vendor-neutral, community-driven initiative that improves observability in distributed systems. The Datadog platform provides complete observability solutions for cloud-native environments. Datadog has emerged as a leading provider of monitoring, security, and analytics tools, catering to the needs of organizations across various industries. When evaluating observability solutions like OpenTelemetry and Datadog, it's essential to consider various factors, including performance metrics, ease of integration, cost considerations, scalability, and flexibility. Find out more! https://lnkd.in/gb2uwuD9 #OpenTelemetry #observability #Datadog #monitoring #analytics #cloudnative #vendorneutral #communitydriven #distributedsystems #performancemetrics #integration #costconsiderations #scalability #flexibility #observabilitysolutions
To view or add a comment, sign in
-
Cloud Solutions Architect | Application Architect | SRE & DevOps Engineer | Software Developer | Consultant | Trainer
Going #serverless? Worried about challenges with performance, cost & scale? @georgemao covers CapitalOne’s journey into serverless, the best practices they picked up, the lessons learned & the optimizations for Lambda: https://t.co/orBhUdCAL2 #LambdaArchitecture #InfoQ https://t.co/6DBRnER27s
Going #serverless? Worried about challenges with performance, cost & scale? @georgemao covers CapitalOne’s journey into serverless, the best practices they picked up, the lessons learned & the optimizations for Lambda: https://t.co/orBhUdCAL2 #LambdaArchitecture #InfoQ https://t.co/6DBRnER27s
twitter.com
To view or add a comment, sign in
-
An API Gateway serves as a centralized entry point for managing, aggregating, and routing API requests. It plays a crucial role in handling API traffic efficiently, ensuring security, and optimizing performance. API Gateway Architecture: 1️⃣ Network Security Layer 🛡️ - Protects API infrastructure from threats. - Enforces security policies: SSL/TLS, DDoS protection, and rate limiting. 2️⃣ Administrative Layer 🛠️ - Manages APIs with versioning, monitoring, logging, and analytics. - Tracks usage, diagnoses issues, and provides performance insights. 3️⃣ Access Layer 🗝️ - Controls access with authentication and authorization. - Utilizes API keys, OAuth tokens, and access control policies. 4️⃣ Transformation Layer 🔄 - Facilitates data transformation and protocol conversion. - Converts JSON to XML, handles different authentication schemes, and supports legacy systems. Benefits of an API Gateway: - 🚀 Performance Boost - 🛡️ Enhanced Security - 🧩 Simplified Microservices - 💡 Unified API Management Types of API Gateways: - Edge Gateways: Handle public APIs with load balancing, security, and caching. - Internal Gateways: Optimize inter-service communication for internal APIs. - Micro-Gateways: Lightweight, use-case-specific, often within containerized environments. Popular API Gateway Tools - Amazon API Gateway - Kong - Apigee Edge - Azure API Management - Tyk #APIGateway #APIManagement #Microservices #NetworkSecurity #APISecurity #APIMonitoring #APILogging #APIPerformance #DDoSProtection #RateLimiting #APIAnalytics #APIVersioning #Authentication #Authorization #OAuth #APIKeys #DataTransformation #ProtocolConversion #EdgeGateway #InternalGateway #MicroGateway #CloudAPIs #APITools #AmazonAPIGateway #Kong #ApigeeEdge #AzureAPIManagement #Tyk #TechArchitecture #APIStrategy #DigitalTransformation #thynkweb
To view or add a comment, sign in