sreeprasad Govindankutty

Open Policy Agent (OPA) and Rego are powerful tools for implementing policy-as-code in Kubernetes and other environments. OPA is an open-source, general-purpose policy engine that enables unified, context-aware policy enforcement across the entire stack. In Kubernetes, it's often used for admission control and authorization. Rego is the high-level declarative language used to specify policy as code in OPA. It allows you to express complex policies in a concise and readable format. Here's how OPA and Rego work together in Kubernetes: 1. Integration with Kubernetes: OPA can be deployed as a admission controller webhook in Kubernetes. This allows it to intercept and evaluate requests to the Kubernetes API server before they are persisted. 2. Writing Policies: Policies are written in Rego and typically stored in ConfigMaps or as files in the OPA container. 3. Policy Enforcement: When a request is made to the Kubernetes API, it's sent to OPA for evaluation. OPA applies the Rego policies and returns an allow/deny decision. Gatekeeper is an open-source project that provides a more Kubernetes-native way to use OPA (Open Policy Agent) for policy enforcement. It extends Kubernetes with Custom Resource Definitions (CRDs) for defining and managing policies using Rego. Reference for implementing gatekeeper for enforcing labels:

6

Twitter Archiecture Twitter, boasting over 2 billion users, is a powerhouse of innovation, and its success hinges on a robust and scalable architecture. Let's explore some key components: Micro services Architecture: Twitter leverages micro services for independent, modular development. This enables flexible scaling of individual components based on demand. Global Content Delivery Network (CDN): A robust CDN ensures fast and reliable content delivery for users worldwide. Caching frequently accessed data at geographically distributed edge locations minimizes latency and improves user experience. This tech stack has been created with the knowledge present on internet. Tech Stack Powerhouse: Twitter utilizes a diverse mix of technologies to handle various tasks: Frontend: React (UI framework), GraphQL (API querying), Native mobile development with Swift (iOS) and Kotlin (Android) Backend: scala ,finagle rpc, python, java Data Storage: redis (in-memory caching), flockdb ,manhatten,mysql,cassandra Messaging/Streaming: Apache Kafka (distributed streaming platform), Finagle (twitter-developed system) Data Processing: Spark (large-scale data processing), Flink , Heron DevOps: Kubernetes (container orchestration), Docker (containerisation), ELK Stack (searching,log management), Prometheus (monitoring) Are you interested in learning more about building scalable architectures, please check my latest video on introduction to System Design https://lnkd.in/dASFyabu What are you thoughts on Twitter Architecture ??

301

36 Comments

Weird stuff about Amazon S3 that you might not know👇 1. S3 requires different endpoints for different actions: bucket-specific URLs and generic endpoints. 2. Some operations can be performed anonymously if bucket policies are misconfigured. 3. Anonymous requests lead to anonymous entries in CloudTrail logs, making it impossible to trace the requester. 4. Preventing `s3:ListBucket` might not fully secure bucket contents since other operations like 𝙨3:𝙇𝙞𝙨𝙩𝘽𝙪𝙘𝙠𝙚𝙩𝙑𝙚𝙧𝙨𝙞𝙤𝙣𝙨 and 𝙨3:𝙇𝙞𝙨𝙩𝙈𝙪𝙡𝙩𝙞𝙥𝙖𝙧𝙩𝙐𝙥𝙡𝙤𝙖𝙙𝙨 can still list object keys. 5. Incomplete multipart uploads are still stored and billed but are not visible through standard S3 interfaces. 6. AWS allows ACLs to grant access based on email addresses. This can reveal whether an email is associated with an AWS account. 7. The uploader chooses the storage class of an object, affecting costs for the bucket owner. Policies can restrict this choice, but not all buckets enforce such policies. 8. S3 keys are case-sensitive. For applications that assume case-insensitivity, this is a problem. 9. Even if a bucket is not public, it can be made accessible via CloudFront or Cognito identity pools, which might bypass typical public access checks. 10. Specific API calls can reveal the account ID of a bucket owner. I write one post for software engineers everyday at 10. Follow Pratik Daga so that you don't miss them.

99

7 Comments

There are two major families of highly available, scalable, (relatively) consistent databases: 1. Single global transaction ordering, with sharding only in the storage layer (Calvin, FaunaDB, FoundationDB) 2. Sharded transactions with two-phase commit for cross-shard transactions (Yugabyte, Cockroach, Spanner) The classification is quite helpful, though the rest of the post goes into quite an attack on the latter style for being insufficiently consistent (outside of Spanner). https://lnkd.in/earQR8js

67

14 Comments

One thing to keep in mind with BM25/full-text search libraries, there's a difference between * Computing BM25 term matrix for a model to consume * Retrieving just top N results vs scoring everything * Bells and whistles of phrases, position aware, fuzzy search, custom similarities, boolean queries I think there's tools for the former, for SearchArray I've tried to focus on the latter two (score everything, but with all the bells and whistles you'd expect from a full text search library). You can see that in the colab notebook I've been building up to https://lnkd.in/eRv8emgY

58

1 Comment

𝗢𝗽𝗲𝗻𝗔𝗜 𝗵𝗮𝘀 𝗿𝗲𝗹𝗲𝗮𝘀𝗲𝗱 𝗦𝗪𝗘-𝗯𝗲𝗻𝗰𝗵 𝗩𝗲𝗿𝗶𝗳𝗶𝗲𝗱. 𝗕𝗲𝗻𝗲𝗳𝗶𝘁𝘀 𝘄𝗶𝘁𝗵 𝗮 𝗱𝗿𝗮𝘄𝗯𝗮𝗰𝗸... 𝘄𝗵𝘆? I love that this is a refined subset of the original SWE-bench benchmark for evaluating AI models' software engineering capabilities. It's also a positive that the new version addresses issues like overly specific unit tests, ambiguous problem descriptions, and setup difficulties. They applied humans in the loop (#HILT) as annotators to screen 1,699 samples to improve quality. The result is a dataset of 500 verified samples. They stated the updated benchmark aims to provide more reliable evaluations. And of course <sarcasm>, it shows improved performance metrics for models like GPT-4o, which now resolves 33.2% of tasks compared to previous scores on the original dataset. What's the con? Why the sarcasm? We need more efforts like this to be community led. #ObjectivityMatters I'll be pulling this down and kicking the tires. https://lnkd.in/e4rmtC2d #ArchAITecture #MeasureWhatMatters #AI4SWE Chris Madden, Hasan Yasar, Patrick Debois, OpenAI, Matthew Thompson, Brian Greene, Sasha Czarkowski (Rosenbaum)

5

2 Comments

The Linux kernel's implementation of context switch b/w two threads on X86. 1. save registers on previous task's stack 2. Switch stack pointers 3. Restore registers from the new task's stack Interesting to see the code to prevent attacks due to return stack buffer (RSB) underflow. I wrote about that very recently. It enables the processor to speculate on the return address of a function and execute instructions at that address speculatively.

62

1 Comment

🪣One Feature of AWS S3 That No One Asked For 🌟 Amazon S3 now supports up to 1 million buckets per AWS account! If you ever thought 100 buckets weren't enough, this update is for you. Whether you’re managing an unprecedented number of projects, want to push the boundaries of cloud storage, or enjoy seeing how far limits can go, this feature has got you covered. While it may not be on everyone's wishlist, it certainly sparks curiosity and opens up possibilities for unique use cases! Imagine the tagging strategies and bucket naming conventions we’ll need to handle this capacity. What’s your take on this update? Game-changer or overkill? 🔗 Learn more about this update https://lnkd.in/gbSx_ZmV #AWS #AmazonS3 #CloudComputing #Innovation

34

3 Comments

Deprecating Hadoop is not without its obstacles. For example, any replacement must also offer an equivalent stack for application management that includes log aggregation and metrics. A critical challenge when building Moka, our next gen Spark on EKS platform at Pinterest, was enabling teams to quickly diagnose issues with Spark jobs and to monitor job performance and status via metrics. We just published a two part blog series with AWS, co-authored with Vara Bonthu, that details our setup and explains how we resolved some of these issues to build a robust observability and metrics framework for Moka. Enjoy! - Part 1: https://lnkd.in/gtJ9u4eW - Part 2: https://lnkd.in/gcbnntyj Our thanks to Ang Zhang, William Tom, Ashim Shrestha from Pinterest and Gregory Mark, Doug Youd, Alan Halcyon, Aaron Miller and Nirmal Mehta @ AWS for their continued help and support.

70

4 Comments

After taking 100+ Mock Online System Design Interviews, I saw few generic problems. - People say to use NoSQL DB for scale. Which? How? - Is missing? - People say they will use cache - Which one? Why not Memcached? This is gone... - People replace LBs with API Gateway or BFF 🤦‍♂️ - People use MongoDb most often at scale (no matter whether we need transactional systems or write-heavy systems). - Kafka is a peanut. Use it with any drink. But yes, we don't know how to solve head-of-line blocking 🤦‍♂️ - Auto-scaling is for free (even in persistent systems) 🤦‍♂️ - Redux is required even for a simple app. - Node JS is single-threaded and what not ... What will you add?

5

6 Comments

Throughput between main memory and CPU cache is 3x to 4x slower than the CPU can process data. Traditional DBMSs traverse this bottleneck frequently (twice per function), with the CPU handing off results of each step of a multi-step calculation to temporary tables in main memory. In contrast, pipelining vector-based statistical functions with eXtremeDB for HPC avoids these hand offs, keeping interim results in CPU cache to reduce database latency. Learn more: https://bit.ly/2T8cIQB #LowLatency #extremedb #timeseries #dbms

11

Throughput between main memory and CPU cache is 3x to 4x slower than the CPU can process data. Traditional DBMSs traverse this bottleneck frequently (twice per function), with the CPU handing off results of each step of a multi-step calculation to temporary tables in main memory. In contrast, pipelining vector-based statistical functions with eXtremeDB for HPC avoids these hand offs, keeping interim results in CPU cache to reduce database latency. Learn more: https://bit.ly/2T8cIQB #LowLatency #extremedb #timeseries #dbms

7

LLM is now capable of generating large JSON objects! 🚀 LLMs often need help generating or modifying large JSON blobs, leading to errors and inefficiencies. The new LangChain project is solving this problem by using JSON Patch operations and working on small parts at a time. If you're working with LLMs and need a reliable way to handle JSON data, I highly recommend checking it out: https://lnkd.in/gyTTe-DU

10

What happens if your CPU gets something wrong? If it wakes up one day and decides 2+2=5? Well, most of us will never have to worry about that. But if you work at a company the size of Google, you do, which is why this paper on "mercurial cores" is so fascinating. What the authors report--and supposedly this is common knowledge at the hyperscalers--is that a couple cores per several thousand machines are "mercurial." Due to subtle manufacturing defects or old age, they give wrong answers for certain instructions. These can cause all sorts of impossible-to-diagnose issues. Some rare problems at Google that were traced back to bad CPUs include: - Mutexes not working, causing application crashes - Silent data corruption - Garbage collectors targeting live memory, causing application crashes - Kernel state corruption causing kernel panics What makes CPUs go bad? It's very hard to tell. The authors posit that issues are becoming more frequent as CPUs get more complex, but there aren't solid numbers behind that. There are certainly strong relationships between frequency, temperature, voltage, and bad CPU behavior--most mercurial CPUs only cause problems under very specific conditions, but those conditions vary from CPU to CPU. Age is another source of problems, as older CPUs are more likely to exhibit problems. Bad CPUs are an especially serious problem because they're very hard to detect. If cosmic rays flip bits in storage or on the network, that can be detected through error coding. But there's no analogy for a CPU that allows cheap online verification of its correctness. Instead, the best detection techniques involve monitoring for symptoms. If a core exhibits exceptionally high rates of process crashes or kernel panics relative to its fellows, that's a strong indication something is wrong with it. For the most critical applications, the authors propose triple modular redundancy--redoing each of its computations on three cores and majority-voting a reliable result. More than anything, this paper is a call to action--letting everyone know that CPUs can fail. So now, if you ever find a bug you can't diagnose, you can blame the CPU! 🙂

1,093

52 Comments