Abhishek Pandey

Open Policy Agent (OPA) and Rego are powerful tools for implementing policy-as-code in Kubernetes and other environments. OPA is an open-source, general-purpose policy engine that enables unified, context-aware policy enforcement across the entire stack. In Kubernetes, it's often used for admission control and authorization. Rego is the high-level declarative language used to specify policy as code in OPA. It allows you to express complex policies in a concise and readable format. Here's how OPA and Rego work together in Kubernetes: 1. Integration with Kubernetes: OPA can be deployed as a admission controller webhook in Kubernetes. This allows it to intercept and evaluate requests to the Kubernetes API server before they are persisted. 2. Writing Policies: Policies are written in Rego and typically stored in ConfigMaps or as files in the OPA container. 3. Policy Enforcement: When a request is made to the Kubernetes API, it's sent to OPA for evaluation. OPA applies the Rego policies and returns an allow/deny decision. Gatekeeper is an open-source project that provides a more Kubernetes-native way to use OPA (Open Policy Agent) for policy enforcement. It extends Kubernetes with Custom Resource Definitions (CRDs) for defining and managing policies using Rego. Reference for implementing gatekeeper for enforcing labels:

6

Sometimes small changes are a big deal. S3's per-account bucket limit has meant that buckets have been more of a human provisioning thing, starkly different from objects, which tend to be interacted with primarily from code. For years, customers have been asking us to increase this limit -- and in the storage innovation talk at re:Invent this year, i'd tell a bit of a story about why it was a more complicated bit of work than you might expect. With this limit increase, buckets have become a much more programmatic construct. We hear customers talking about using buckets, with these limit increases, as a way of managing data sharing with their own customers and using them to more effectively isolate data for specific jobs and teams. If larger numbers of buckets opens up a new pattern for you, I'd love to hear more about it. These launches, like object consistency, S3 conditionals, and big bucket limit increases have a tendency to fly a little below radar. But they are the ones that I actually hear the most excitement back from developers about, because they really broaden S3 as a programmatic interface to your data, and a building block for really incredible systems. I'm so impressed with the hard and careful work on the team that went in to getting this one out. https://lnkd.in/gA2cWATM

220

8 Comments

You can now create a million S3 buckets per account, but the buckets beyond two thousand will now be charged at a rate of $0.02 per bucket. That’s… I have no words. If you were relying on quotas to prevent bucket sprawl, well you’ll have to change that. https://lnkd.in/ge7_FacW #aws #s3

15

AlloyDB provides 2x better price-performance than self-managed PostgreSQL. See how in this article from Sridhar Ranganathan and Shesh Ranganath.

14

🪣One Feature of AWS S3 That No One Asked For 🌟 Amazon S3 now supports up to 1 million buckets per AWS account! If you ever thought 100 buckets weren't enough, this update is for you. Whether you’re managing an unprecedented number of projects, want to push the boundaries of cloud storage, or enjoy seeing how far limits can go, this feature has got you covered. While it may not be on everyone's wishlist, it certainly sparks curiosity and opens up possibilities for unique use cases! Imagine the tagging strategies and bucket naming conventions we’ll need to handle this capacity. What’s your take on this update? Game-changer or overkill? 🔗 Learn more about this update https://lnkd.in/gbSx_ZmV #AWS #AmazonS3 #CloudComputing #Innovation

34

3 Comments

Postgres as a cache?? 🤔 I had never thought or heard of using Postgres as a cache on top of a primary Postgres DB until I came across this fascinating talk from Zerodha about their back-office Postgres deep dive. Among various interesting use cases, the talk highlights some essential principles that are helpful for both beginners and advanced users: 1) Know your data and query patterns – Understanding your data is critical for choosing the right database, partitioning strategy, and indexing. 2) Use indexes wisely – Indexes are great for speeding up queries but can slow down writes and consume space. 3) Master query plans! 4) Avoid nested joins on heavy tables – These are difficult to optimize and can drag down performance. 5) Use a caching layer – For high-traffic services, a proper caching layer is must! 6) Materialized views for read-heavy use cases – In scenarios where data doesn’t change frequently, materialized views boost performance and lower DB stress.

16

Deprecating Hadoop is not without its obstacles. For example, any replacement must also offer an equivalent stack for application management that includes log aggregation and metrics. A critical challenge when building Moka, our next gen Spark on EKS platform at Pinterest, was enabling teams to quickly diagnose issues with Spark jobs and to monitor job performance and status via metrics. We just published a two part blog series with AWS, co-authored with Vara Bonthu, that details our setup and explains how we resolved some of these issues to build a robust observability and metrics framework for Moka. Enjoy! - Part 1: https://lnkd.in/gtJ9u4eW - Part 2: https://lnkd.in/gcbnntyj Our thanks to Ang Zhang, William Tom, Ashim Shrestha from Pinterest and Gregory Mark, Doug Youd, Alan Halcyon, Aaron Miller and Nirmal Mehta @ AWS for their continued help and support.

70

4 Comments

The future of app development is here. Redis Released is our one-day event for developers to connect with Redis experts, learn about our product roadmap, and hear from the world’s largest brands on how they’re building with our powerful real-time technology. Here’s what we’ll share: → Our Product Roadmap → The World of Redis → Faster GenAI Apps Next stops: London on September 12th and New York City on October 17th. See you there: https://redis.io/released/

40

AWS Lambda scales to provision containers at 15,000 per second, and achieve cold-start times as low as 50ms, even for large container images (up to 10GiB). This week, I am reading the paper from AWS where they covered the nuanced details of how they do this and the set of optimizations they did to minimize the cold start problem. I have skimmed the paper twice already and the nuances covered will make you more curious about containers and their internals. definitely a good weekend read :) a detailed write-up will be soon published on my papershelf, but until then you can find other papers that I have read and their notes here - arpitbhayani.me/papershelf ⚡ I keep writing and sharing my practical experience and learnings every day, so if you resonate then follow along. I keep it no fluff. #SystemDesign #AsliEngineering

489

11 Comments

Kicking off a super interesting series on the internals of DragonflyDB - a multi-threaded drop-in replacement of Redis ⚡ On a single machine, it offers 25 times the throughput of Redis as it is optimized to leverage every single ounce of performance from the underlying hardware. The first video of the series is out and it is a quick architecture overview and covers two key design decisions. give it a watch - youtu.be/XbV1LoVsbME The low-level design decisions and internal details will just blow your mind away - right from their design of DashTable (not hash table) to sorted set implementation. Trust me, your perspective of looking at Data structures will change after the series :) ⚡ Subscribe to the channel and follow along; I keep things no-fluff. youtube.com/c/ArpitBhayani #AsliEngineering #SystemDesign #DatabaseInternals

286

5 Comments

🚀 Unlock the Power of Multicast on AWS! 🚀 I'm excited to share my latest step-by-step guide on how to leverage multicast capabilities on AWS, from Transit Gateway all the way to Containers. Whether you're handling large-scale networking setups or exploring advanced AWS networking features, this guide will walk you through: 🔹 The benefits of multicast in modern architectures 🔹 Setting up AWS Transit Gateway for multicast 🔹 Deploying containers with multicast support 🔹 Real-world use cases and best practices Check out the full article on Medium: Unlocking the Power of Multicast on AWS Let’s dive into how multicast can boost your cloud networking strategy! 🌐 #AWS #CloudComputing #Multicast #Networking #AWSNetworking #Containers #AWSArchitecture #SoftwareArchitecture

3

Check out the latest in our "Debuggability" blog series! This blog is authored by Kevin Downey Jason Johl and me and talks about how we built a Debuggability Paved Road at Intuit using Ephemeral Containers and Argo Workflows. We'd love to hear your thoughts on the topic. Also, what tools do you build as-a-platform to help debug service, compute and network issues in Kubernetes at scale? #intuit #kubernetes #debug

22

Excited to see https://lnkd.in/gG7fp73Y go live this Kubecon! The brainchild of the brilliant Amine Hilaly, KRO leverages the strengths of the Kubernetes API and GitOps principles, allowing organizations to package and distribute deployment recipes for both applications and infrastructure. KRO simplifies the industry's patchwork of deployment tools with Kubernetes APIs and a server-side DAG execution engine. Excited to see this mature! #kubecon #kubernetes #eks #opensource

96

10 Comments

💡 Redis Cluster Clients 💡 Ever wondered how Redis cluster clients manage to make lightning-fast decisions? 🚀 Let's unveil the magic! Redis cluster clients wield their intelligence on the client side, orchestrating decisions to minimize network calls, so that no request happens to land on the wrong shard. 🎯 It first makes a discovery call [cluster nodes] and get the mapping of slots and shard Ips. Once it has this data, for every key that you want to query it calculates on the client side for which shard it is intended to and send the request to that shard ip only. Check out the diagram below for a visual breakdown! Share your insights and musings in the comments section. And don't hesitate to offer corrections or engage in discussion! #Linux #SystemCalls #SRE #DevOps #ComputerEngineering

46

4 Comments

Our trajectory with Google Cloud epitomizes a transformative strategic pivot. What began as a challenging market positioning has evolved into a compelling narrative of operational resilience and technological leadership. Over seven pivotal years, we've fundamentally reimagined our enterprise value proposition—transitioning from a peripheral cloud solution to a mission-critical AI innovation partner. The financial metrics tell a powerful story: we've systematically converted recurring quarterly deficits into a robust framework for sustainable profitability. Buckle up the next 7 years are going to be epic @ Google Cloud! 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 🚀 #google #googlecloud #GCP #gcpcloud

20

https://lnkd.in/gFWsZBEM Keeping pace with LLVM: compatibility strategies

10

2 Comments