👨‍💻 Eugene Tolbakov

Tiered storage for Kafka Kafka cluster storage is typically scaled by adding more broker nodes to the cluster. But this also adds needless memory and CPUs to the cluster, making overall storage cost less efficient compared to storing the older data in external storage. A larger cluster with more nodes also adds to the deployment complexity and increases operational costs because of the tight coupling of storage and processing. So, it brings several issues related to scalability, efficiency, and operations. In the tiered storage approach, Kafka cluster is configured with two tiers of storage - local and remote. The local tier is the same as the current Kafka that uses the local disks on the Kafka brokers to store the log segments. The new remote tier uses systems, such as HDFS or S3 to store the completed log segments. Two separate retention periods are defined corresponding to each of the tiers. With remote tier enabled, the retention period for the local tier can be significantly reduced from days to few hours. The retention period for remote tier can be much longer, days, or even months. When a log segment is rolled on the local tier, it is copied to the remote tier along with the corresponding indexes. Latency sensitive applications perform tail reads and are served from local tier leveraging the existing Kafka mechanism of efficiently using page cache to serve the data. Backfill and other applications recovering from a failure that needs data older than what is in the local tier are served from the remote tier. #kafka #storage #cost https://lnkd.in/ehu2Fw2v

2

Uber Engineering implements Kafka Tiered Storage to avoid tight coupling of storage and processing in a broker. The feature is added in 3.6.0 and is available for experimentation. #DataEngineering #Realtime #EventDrivenArchitecture #TCO https://lnkd.in/gfgeDCSF

11

1 Comment

From Blocks and Files: "... tests show that Redis is faster for vector database workloads compared to any other vector database tested, at recall >= 0.98. Redis has 62 percent more throughput than the second-ranked database for lower-dimensional datasets (deep-image-96-angular) and has 21 percent more throughput for high-dimensional datasets (dbpedia-openai-1M-angular)." https://lnkd.in/grzurc7R

24

1 Comment

Struggling with orchestration costs and complexity? Endpoint was too—until they found Prefect. The results after switching from Airflow? - 73.78% reduction in orchestration costs - 78 new pipelines built in one quarter - Migration completed in under 2.5 months - Unified environment for ML and data teams Their ML and data engineering teams now operate seamlessly in a single environment—using the tools they rely on (like Docker, Kubernetes, Celery, and EKS) without the complexity that held them back before. Here's what Sunny Pachunuri, Data Engineering and Platform Manager at Endpoint, had to say: "Prefect's support feels like a true collaboration, significantly increasing our team's efficiency and satisfaction." Ready to transform your data orchestration? Read the full case study below. #DataEngineering #MLOps #DataOrchestration #Prefect

18

1 Comment

Instead of storing separate key-value caches for each layer like standard transformers, YOCO reuses a single cache across all layers, significantly reducing memory usage while preserving competitive performance… https://lnkd.in/eTXtV6vy

3

𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝗳𝘂𝗹 𝗥𝗲𝗮𝗱: 𝗧𝗵𝗲 𝗦𝘁𝗮𝘁𝗲 𝗼𝗳 𝗦𝗲𝗿𝘃𝗲𝗿𝗹𝗲𝘀𝘀 𝗶𝗻 𝟮𝟬𝟮𝟰 & 𝗕𝗲𝘆𝗼𝗻𝗱 𝗦𝗲𝗿𝘃𝗲𝗿𝗹𝗲𝘀𝘀 & 𝗶𝘁𝘀 𝗝𝗼𝘂𝗿𝗻𝗲𝘆:          The principle behind serverless computing is simple. Traditional VM or container services require users to commit and manage persistent resources, In contrast, serverless computing loads application components dynamically when needed, so teams don't need to manage resources.          The concept of “utility computing,” where computing resources are treated as a metered service like electricity emerged long back & Amazon launched AWS Lambda in 2014, marking a pivotal moment for serverless computing. This function-as-a-service (FaaS) platform popularized the “serverless” term and enabled event-driven execution of code. All the Hyperscalers have various Serverless service offerings. 𝗖𝘂𝗿𝗿𝗲𝗻𝘁 𝘀𝘁𝗮𝘁𝗲 & 𝗣𝗮𝘁𝗵 𝗳𝗼𝗿𝘄𝗮𝗿𝗱 🔶 Serverless accounts for less than 20% of Cloud workloads(Andover Intel) and could account for 20% of all cloud workloads by 2025 (Forrester Research, July 2023). 🔶 451 Research reported in August that the serverless 𝘮𝘢𝘳𝘬𝘦𝘵 𝘪𝘴 𝘦𝘹𝘱𝘦𝘤𝘵𝘦𝘥 𝘵𝘰 𝘥𝘰𝘶𝘣𝘭𝘦 𝘪𝘯 𝘴𝘪𝘻𝘦 𝘦𝘷𝘦𝘳𝘺 𝘵𝘸𝘰 𝘵𝘰 𝘵𝘩𝘳𝘦𝘦 𝘺𝘦𝘢𝘳𝘴 for the next decade. 𝗞𝗲𝘆 𝗕𝗲𝗻𝗲𝗳𝗶𝘁𝘀: 🔶 𝘼𝙪𝙩𝙤𝙢𝙖𝙩𝙞𝙘 𝙨𝙘𝙖𝙡𝙞𝙣𝙜: It automatically scales up or down to meet demand and you don’t have to worry about over-provisioning or under-provisioning resources. 🔶 𝙋𝙖𝙮-𝙥𝙚𝙧-𝙪𝙨𝙚 𝙗𝙞𝙡𝙡𝙞𝙣𝙜: Pay for the resources that their applications use. This can save money for highly variable workloads. 🔶 𝙃𝙞𝙜𝙝 𝙖𝙫𝙖𝙞𝙡𝙖𝙗𝙞𝙡𝙞𝙩𝙮: Serverless applications are highly available because they are run on a global network of servers. 🔶 𝙁𝙖𝙨𝙩 𝙙𝙚𝙫𝙚𝙡𝙤𝙥𝙢𝙚𝙣𝙩: Serverless applications can be developed and deployed quickly as it takes away the role of infrastructure provisioning and managing. 𝗖𝗼𝗻𝘀 𝘁𝗼 𝗖𝗼𝗻𝘀𝗶𝗱𝗲𝗿: 🔶 𝘾𝙤𝙨𝙩 𝙤𝙫𝙚𝙧𝙧𝙪𝙣𝙨: Generally, the unit cost of computing in the cloud is higher for serverless than for containers or VMs. As the amount of presented work events grows, serverless costs mount proportionally. It's often difficult to predict the usage of applications, and usage underestimation can easily make serverless too expensive. 🔶 𝙇𝙖𝙩𝙚𝙣𝙘𝙮: Because serverless application components load and run dynamically, delays associated with their execution might affect application performance. If several serverless applications are chained together, the delays are additive. To read more https://lnkd.in/gD55TFhs & https://lnkd.in/gSmcM-AM #CloudComputing #Serverless #Containers #Cloud

7

An Exercise in Discovery, 2 separate data source, first via Mysql using Apache Flink CDC and 2nd via Kafka, some transformations on Apache Flink and then Output'd to Apache Kafka and Apache Paimon (Avro files) stored on S3 Object Storage.

27

6 Comments

We often get asked: “Can a Kafka consumer listen to multiple topics?” The short answer is: Yes, an #apachekafka consumer can subscribe to multiple topics. Read about examples where consumers listen to multiple topics versus only one. And learn about potential impacts on Kafka’s performance and hardware requirements.

1

Implement a Q&A experience using a RAG technique with Elasticsearch as a vector database by following the steps in this blog: https://gag.gl/V2hwuA #ElasticSearchLabs #RAG #GenAI

2

💡 Optimizing Costs with Dragonfly: A Technical Breakdown 💻 For technical teams looking to maximize performance while minimizing cloud infrastructure costs, Dragonfly offers significant advantages. By cutting down memory overhead, leveraging CPU-specific optimizations, and using a shared-nothing architecture, Dragonfly is designed to be resource-efficient while maintaining high throughput. Whether you’re scaling horizontally or aiming to optimize performance, Dragonfly can help you reduce operational costs without sacrificing speed. In this blog, we break down the technical strategies that Dragonfly uses to cut costs and improve efficiency. 🔗 Read more here: https://hubs.la/Q02RDNVr0 #CloudOptimization #TechEfficiency #DevOps #DatabasePerformance #Dragonfly#CostReduction

3

With the rise of serverless technology becoming more and more prominent, Paul Santus has put together a great 2 part series on serverless architecture and how to get PHP to run on Lambda ☁🐘

13

2 Comments

When working with Amazon MSK, developers are interested in accessing the service locally. This allows developers to test their application with a Kafka cluster that has the same configuration as production and provides an identical infrastructure to the actual environment without needing to run Kafka locally. This post presents a practical approach to accessing your Amazon MSK environment for development purposes through a bastion host using a Secure Shell (SSH) tunnel (a commonly used secure connection method). Whether you’re working with Amazon MSK Serverless, where public access is unavailable, or with provisioned MSK clusters that are intentionally kept private, this post guides you through the steps to establish a secure connection and seamlessly integrate your local development environment with your MSK resources. https://lnkd.in/g4yiF3yH #MSK #serverless #ApacheKafka #testing

30

You can now use SingleStore to seamlessly ingest #data from and write back to #Iceberg tables — in real time, with no additional tooling required. https://lnkd.in/eBuprpJ4

4

🚀 New Medium Article: Understanding Partitions in Kafka®: A Deep Dive Kafka is the backbone of scalable, fault-tolerant distributed systems. In my latest article, I explore the core of Kafka's architecture: Topic Partitioning. 🔍 Here's what you'll learn: - How Kafka partitions topics for scalability - The gold standard for interlinking topics and partitions - Partition tuning strategies for better performance - Real-world examples to balance traffic and optimize throughput If you’re working with Kafka or looking to scale your data streaming platform, this is a must-read! 👉 Check it out: https://lnkd.in/gsV2Ut_R #Kafka #DistributedSystems #Streaming #DataEngineering #Scalability #TechArticle #Medium

11

At next month's SIGMOD 2024, researchers from Amazon Web Services (AWS) will present a paper on Amazon MemoryDB, where they will describe its architecture and how they leveraged Redis, an open-source data structure store, to build an enterprise-grade cloud database. Paper: https://lnkd.in/eEuquM8Q Conference: https://lnkd.in/esVMQcav #AWS #ML #Databases

58

1 Comment