Sujeet Banerjee

Very interesting post on Intuit's use of LLMs to manage K8s, via Aditya Kulkarni at InfoQ: "Intuit Engineering's Approach to Simplifying Kubernetes Management with AI" https://lnkd.in/eRN6_sAK

2

Checkout the latest post on AiBuddha Blogs on Sparse Vs Partial Indexes in Mongo! Zero to Hero in Mongo! https://lnkd.in/grwMtQT8

1

We’ve just published a new blog on Redis - The In-Memory Powerhouse! At TechRover, we helped a few startup clients fine-tune their entire system architecture, boosting both latency and throughput by incorporating Redis and other key components. The result? Faster performance and scalable infrastructure with low latency and high throughput. Read the full story here: https://lnkd.in/guSECqRU #TechRover #Redis #InMemoryDB #LowLatency #HighThroughput #StartupSuccess #SystemArchitecture

10

Elasticsearch open inference API adds support for OpenAI chat completions. Learn more in this blog post: https://gag.gl/hEDnXw #ElasticSearchLabs #OpenAI

5

It always puzzled my why Automatic DDL replication, while Standard in MySQL is so hard in PostgreSQL. Good to see some progress being made https://lnkd.in/ddi83nkN #postgres #mysql

25

9 Comments

When a popular #opensource project like Redis forks, it leads to fragmentation in the technology ecosystem. Companies using these tools face decisions about which version to support—stick with the original, which might have more restrictive licensing, or switch to a new fork, which might not yet have proven stability or feature parity. This division can lead to compatibility issues between systems that depend on different versions of the same tool, complicating integration efforts and maintenance. https://lnkd.in/dhSFZxZc 

6

4 Comments

The CSI driver EKS plugin doesn't use the cluster's IAM role. Here is why: EKS has a default storage class (gp2) which is part of the in-tree storage plugin. It uses the cluster's IAM role to manage EBS volumes. So you don't need to add any extra IAM role to manage volumes using gp2. However, the EBS CSI Driver requires its own separate IAM permissions to interact with AWS services. This is because it runs as a separate service inside the cluster. To provide the necessary IAM access to the CSI Controller, you can use the Pod Identity Webhook Plugin. This allows you to assign temporary IAM credentials directly to the CSI Controller using a Kubernetes service account. This way, the EBS CSI Controller can securely authenticate with AWS and perform operations like provisioning, attaching, and managing EBS volumes. If you are using eksctl to manage clusters, You can associate IAM permission with the simple 𝗮𝘂𝘁𝗼𝗔𝗽𝗽𝗹𝘆𝗣𝗼𝗱𝗜𝗱𝗲𝗻𝘁𝗶𝘁𝘆𝗔𝘀𝘀𝗼𝗰𝗶𝗮𝘁𝗶𝗼𝗻𝘀 flag as shown in the image. With this option, eksctl automatically resolves and applies the recommended IAM roles and pod identity associations for the add-ons (e.g., EBS CSI Driver). You can refer to my detailed eksctl guide to get hands-on with this feature. 𝗲𝗸𝗰𝘁𝗹 𝗯𝗹𝗼𝗴: https://lnkd.in/gwKtZX38 If you want to understand more about EKS EBS CSI Driver Workflow, you refer to the following blog. 𝗖𝗦𝗜 𝗗𝗿𝗶𝘃𝗲𝗿 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄: https://lnkd.in/gpGTR3Zw PS: ♻️ Repost if you find this useful. It helps the DevOps community 🙂 Have anything to add? Comment below. #devops #aws #kubernetes

136

2 Comments

My new article on medium: how to simply enable temporal / history table in PostgreSQL. No need for: - Special extensions. - Specialized databases. - Special techniques like event storming or soft deletes. None of that is needed. Don't overcomplicate things. https://lnkd.in/dQDNrCq8

20

What is coming to PostgreSQL? According to this interesting article, these two big things. - Separation of Compute and Storage Meaning you will able to indepentily scale computing by adding new compute nodes, independely of the storage. Compute part is that logic that you definitely shouldn't be using in database /s - Pluggable Storage and Improved Table Engines You will be able just to plugin different storage types per table. For example column storage, graph storage, etc... It isn't farfetched to say that PostgreSQL could be the database that will end all other databases. At least it has that potential. So, get on board while you still can.

45

10 Comments

Ever wondered how modern databases handle massive write loads while still delivering lightning-fast reads? 🤔💾 I've just published a comprehensive 3-part series on Log-Structured Merge (LSM) trees, the secret sauce behind databases like RocksDB, Cassandra, and ClickHouse. In this series, I break down: Part 1: Core concepts and components of LSM trees Part 2: Efficient Rust implementation of MemTable and SSTable Part 3: Complete LSM tree with concurrency Dive into the internals of high-performance databases: [https://lnkd.in/gsPvufxZ] [https://lnkd.in/g4PYXHnE] [https://lnkd.in/gBM9PZWj] From theory to practice, this series covers it all, including an efficient Rust implementation showcasing modern concurrency techniques. #DatabaseEngineering #SystemDesign #Rust #PerformanceOptimization What other techniques do you know for handling high write throughput in databases? Let's discuss in the comments! 👇

10

1 Comment

NEW § The second pattern in Juntao Qiu's article on data fetching is on how to avoid the dreaded Request Waterfall. Parallel Data Fetching queries multiple sources in parallel, so that the latency of the overall fetch is largest of the queries rather than the sum of them. https://lnkd.in/eFJu25Eg

176

1 Comment

AlloyDB provides 2x better price-performance than self-managed PostgreSQL. See how in this article from Sridhar Ranganathan and Shesh Ranganath.

14

Yesterday's holiday (June 11, 2024) activity includes playing and learning RAG implementation with Elasticsearch (ELSER - Elastic Learned Sparse EncodeR, a natural language processing (NLP) model trained by Elastic) and E5 embeddings. The thought process includes crawling a public website using Elastic Web crawler, ingest and format the data in Elasticsearch index for these data to be use when asking question to openAI (LLM) - the result will then be displayed a webapp built using Streamlit ( a free and open-source framework to rapidly build and share machine learning and data science web apps). Sharing the resulting code for other to use and some screenshots of the questions and answers using openAI. RAG Implementation for your own experimentation: Giving credit where it's due Grant Patterson - thanks for sharing the initial code for me to play with.

17

"Running Redis on Windows: Setting Up with WSL" is your step-by-step guide to effortlessly integrating Redis into your Windows environment. Discover how to leverage the Windows Subsystem for Linux (WSL) to run Redis seamlessly, combining the power of Linux with the flexibility of Windows. Whether you’re looking to enhance application performance, manage data more efficiently, or simply expand your tech toolkit, this guide has you covered. Elevate your development game with Redis on Windows today! Dive into the full guide here: https://lnkd.in/d4GrUB9S Authored by: Muhammad Aaliyan Khan, Software Engineer at Qavi Tech Looking to bring innovative ideas like this to life in your IT projects? Get in touch with us! Qavi Tech specializes in cutting-edge development and is always excited to work on interesting, impactful projects. #Redis #Windows #WSL #SoftwareDevelopment #TechTips #Innovation #WebDevelopment

23

𝐀𝐦𝐚𝐳𝐨𝐧 𝐃𝐲𝐧𝐚𝐦𝐨𝐃𝐁 𝐢𝐧𝐭𝐫𝐨𝐝𝐮𝐜𝐞𝐬 𝐜𝐨𝐧𝐟𝐢𝐠𝐮𝐫𝐚𝐛𝐥𝐞 𝐦𝐚𝐱𝐢𝐦𝐮𝐦 𝐭𝐡𝐫𝐨𝐮𝐠𝐡𝐩𝐮𝐭 𝐟𝐨𝐫 𝐎𝐧-𝐝𝐞𝐦𝐚𝐧𝐝 𝐭𝐚𝐛𝐥𝐞𝐬 It's going to be super helpful in cost control. Will help in accidental surge in consumed resources and safe guarding downstream services with fixed capacities. https://lnkd.in/gWmAjXv3

16

One easy way to partition a database is to take a primary identifier that parents all (or most) of the data, hash it, create one partition per hash and then move and store all the data in the right partition. Your primary identifier could be something like say userid, organizationid or storeid - anything that creates disjointed sets of data at the top level. A really good way to create the hash function is to first decide how many partitions you want to make. If the identifier is a text, then first convert it into a number (you can use the ascii values of the constituent characters to do this). If its already a number, keep it as it is. Now find the modulus when you divide this number by the number of partitions. For example, say your primary identifier is userid and a userid has a value 4325. Let's assume you want to create 20 partitions. 4325 % 20 = 5. All the data that is associated with your the userid 4325 (in all tables) needs to be moved to the 6th partition (5+1, a % n can result in numbers from 0 to n-1). If you have to ingest any new data, it's very easy to figure out which partition this goes into. Simply calculate the hash of the primary identifier in the incoming data and route the ingestion into the appropriate partition. Similarly, during lookup, its very easy to know which partition a particular set of data is sitting in. Now for any data that cannot be completely disjointed using a singular primary identifier or has no relation to the primary identifier in the first place, you'd have to treat that separately, maybe using another primary identifier just for this use case or an entirely different partitioning strategy. While simple and often efficient, this method does come with a bunch of challenges though. The biggest one is that if your database is designed in such a way that the data associated with just a few of these primary identifiers is greater than the entire data that your DB contains, this is a bad way to partition. You might end up in a situation where one partition contains a majority of your data while your other partitions go underutilized. As an example, if your primary identifier is customerid and just 2 of your customers account for 80% of your database data, then you might need to rethink your strategy. To solve this, maybe you could pick another identifier to partition by, or even use multiple identifiers together. But this might still not work with some database designs. The other problem is that once you decide the number of partitions, if you want to increase or decrease the number of partitions, you'd need to do quite a lot of data migration again, as almost every existing partition will likely have some data that needs to be moved to another. There are of course a lot of database partitioning strategies, but this is one of the simplest ones and unless you have a very unevenly distributed dataset, I'd strongly recommend starting here when you first come across a need for partitioning.

37

🚀 New Medium Article: Understanding Partitions in Kafka®: A Deep Dive Kafka is the backbone of scalable, fault-tolerant distributed systems. In my latest article, I explore the core of Kafka's architecture: Topic Partitioning. 🔍 Here's what you'll learn: - How Kafka partitions topics for scalability - The gold standard for interlinking topics and partitions - Partition tuning strategies for better performance - Real-world examples to balance traffic and optimize throughput If you’re working with Kafka or looking to scale your data streaming platform, this is a must-read! 👉 Check it out: https://lnkd.in/gsV2Ut_R #Kafka #DistributedSystems #Streaming #DataEngineering #Scalability #TechArticle #Medium

11