Gopal Sahu

Building a Single-node Lakehouse using 'Hudi-rs' - no JVM, Spark [New Blog] The Apache Hudi community recently released "hudi-rs" - a native Rust library for Hudi with Python bindings. This makes it possible to work with #lakehouse platform like Hudi across a range of use cases that typically don’t require distributed processing. For eg, data science & machine learning applications need frequent access to diverse datasets for exploratory data analysis/model training. Direct access to Hudi datasets for these use cases reduces the ‘wait time’ on data stakeholders, and speeds up the time-to-insight process. Traditionally, accessing these tables involves configuring Java, Spark, Hadoop, and other related dependencies. This is what "Hudi-rs" addresses & makes it easy for data consumers to get started. In this introductory blog, I go over - ✅ the internals of "Hudi-rs" with #ApacheArrow under the hood ✅ how to use it with popular single node computes such as DuckDB, Polars, Daft & Apache Datafusion Check out the link in the comments. And reach out if you want to get involved with the project! #dataengineering #softwareengineering

169

16 Comments

Topics in Kafka are divided into partitions, which means that a single topic’s log is split across multiple logs distributed across different Kafka brokers. This partitioning allows the workload of storing, writing, and processing messages to be distributed across many nodes in the cluster, enhancing scalability and performance. If you are interested in Kafka Partitioning, don't hesitate the new blog post https://lnkd.in/dSmancXf #data #dataengineering #kafka #confluent

13

Event Duplication in Streaming: A Data Engineer's Headache Duplicate events in streaming data pipelines can be a major pain point. They often arise from network retries, user interface interactions, or "at-least-once" message delivery guarantees. Why should we care? 🤔 Failing to handle duplicates can wreak havoc on your data: Skewed metrics: Imagine counting user clicks twice! 🤯 Inaccurate reports: Leading to faulty business decisions. 📉 Data integrity issues: Compromising the reliability of your entire system. ⚠️ Streaming technologies like Apache Flink and Kafka offer powerful solutions. 💪 Here are some key strategies to consider: ▶️ Idempotency: Design downstream systems to handle duplicate events gracefully. This means ensuring that processing the same event multiple times has the same effect as processing it once. This often involves careful design of your data models and update logic. ▶️ Deduplication: Leverage built-in mechanisms for duplicate detection and removal. Streaming platforms often have features for this, but they require understanding the nuances of how they work. ▶️ Windowing: Use window-based aggregations to mitigate the impact of duplicates. If you're counting events within a time window, duplicates might have less impact. 💡 Looking forward to exploring different windowing strategies (tumbling, sliding, session) and their implications for duplicate handling. ▶️ Event Keys: Implement proper keying strategies to prevent duplicates at the source. If possible, assign unique keys to your events so that duplicate detection becomes easier. ▶️ Exactly-once semantics: While often desired, achieving exactly-once processing can be complex. ▶️ Testing for duplicates: Develop robust testing strategies to ensure your duplicate handling logic works as expected. This might involve generating synthetic datasets with duplicates or using fault injection to simulate error conditions. ps: Make sure you check out DataExpert.io bootcamp! I cant believe how much information Zach Wilson packs into his videos and lectures... my brain hurts! Coffee time ☕ ☕☕ #DataEngineering #StreamingData #ApacheKafka #ApacheFlink #EventProcessing #Week4

8

1 Comment

The pg_wal directory in PostgreSQL is essential for maintaining data durability and recovery through Write-Ahead Logging (WAL). However, excessive growth in this directory can consume valuable disk space and, if unmanaged, may even lead to outages due to insufficient storage. #PostgreSQLPerformance @MinervaDB https://zurl.co/QXpp

22

1 Comment

Spark Streaming!! Interviewer: "Today, let's explore Spark Streaming, an integral component of Apache Spark for processing real-time streaming data. Are you familiar with Spark Streaming?" Candidate: "Yes, I've worked with Spark Streaming. It provides a scalable and fault-tolerant framework for processing real-time data streams." Interviewer: "Excellent! Spark Streaming enables developers to build robust and scalable stream processing applications. Could you elaborate on the key features and benefits of Spark Streaming?" Candidate: "Certainly! Spark Streaming offers several advantages. Firstly, it provides high throughput and low latency processing of real-time data streams. Secondly, it integrates seamlessly with the core Spark API, enabling users to leverage Spark's rich ecosystem of libraries and tools. Additionally, Spark Streaming offers fault tolerance through lineage and checkpointing, ensuring that data processing is resilient to failures. Lastly, it supports various data sources and sinks, allowing users to ingest data from sources like Kafka, Flume, or HDFS, and output processed data to databases, file systems, or dashboards." Interviewer: "Well summarized! Now, let's delve into the architecture of Spark Streaming. How does Spark Streaming process and analyze real-time data streams?" Candidate: "Spark Streaming follows a micro-batch processing model, where real-time data streams are divided into small batches of data, which are then processed using Spark's batch processing engine. The architecture consists of a Receiver that ingests data from input sources, a StreamingContext that manages the execution of streaming jobs, and a Discretized Stream (DStream) abstraction that represents a continuous stream of data. Each batch of data is processed using RDDs (Resilient Distributed Datasets), enabling parallel and fault-tolerant stream processing." Interviewer: "Spot on! Spark Streaming's micro-batch processing model provides fault tolerance and scalability, making it suitable for handling large-scale streaming data. Now, let's discuss the use cases of Spark Streaming. Can you provide examples of scenarios where Spark Streaming excels?" Candidate: "Certainly! Spark Streaming is well-suited for various real-time analytics and monitoring applications. For example, in e-commerce, it can be used to analyze user activity and recommend products in real-time. In finance, it can process streaming market data to detect anomalies or fraud in transactions. Additionally, in telecommunications, it can analyze network logs to identify network issues or optimize network performance in real-time." Interviewer: "Impressive examples! Spark Streaming's versatility makes it applicable across different industries for real-time data processing and analysis. Thank you for sharing your insights on Spark Streaming!" Candidate: "You're welcome! #bigdata #spark #sparkstreaming #pyspark #python

7

🚀 Exploring the world of HBase! 🚀 HBase, a distributed, scalable, and NoSQL database inspired by Google's Bigtable, operates on the Hadoop Distributed File System. Its key features include column-oriented storage, real-time read/write access, automatic sharding of tables, and support for consistency and partitioning with large data sets. 🔍 Use Cases: - Ideal for storing vast amounts of sparse data - Enables real-time data processing and analytics - Facilitates time-series data storage 🏗️ HBase Architecture: - HMaster manages the cluster and assigns regions. - RegionServer handles read/write requests. - Zookeeper coordinates the distributed environment. 📊 Data Model Essentials: - Tables composed of rows and columns - Rows identified by a unique row key - Columns grouped into column families - Versioning ensures each cell value is versioned 🛠️ Handy Commands: - Put: 'put 'mytable', 'row1', 'cf1:col1', 'value1'' - Get: 'get 'mytable', 'row1'' - Scan: 'scan 'mytable'' - Create Table: 'create 'mytable', 'cf1', 'cf2'' Let's dive deeper into the functionalities and possibilities that HBase brings to the table! 💡 #HBase #NoSQL #BigData #Analytics

3

1 Comment

Speaking at #DES24, Pavan Nanjundaiah, Head – Tredence Studio, said today’s data engineer needs a beginner’s mindset to adapt to the growing change. “Today’s data engineers should expand left and right. Focus on your core foundational skills because they are not going to change. You don’t need to handcraft your code anymore.   Understand what is the business problem you’re trying to solve. My recommendation to data engineers is to shift left, which is the person who gave you the requirement and probably understands the business a little better. By doing this, you’re now zooming out of a core data engineer’s profile and understanding the business and the longer impact you can create.”   Read more here: https://lnkd.in/gKhtBwp6 #DataEngineers #Tredence #BeyondPossible

201

1 Comment

𝗦𝗲𝗺𝗶-𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 𝗱𝗮𝘁𝗮 𝗱𝗼𝗲𝘀𝗻'𝘁 𝗵𝗮𝘃𝗲 𝘁𝗼 𝗯𝗲 𝘁𝗼𝗼 𝗵𝗮𝗿𝗱   Working with varied data formats that have complex or unclear structures is often challenging. Today's data ecosystem deals with more than just traditional rows and columns, with modern databases needing to process complex formats like JSON, XML, and Parquet.   Semi-structured data provides the flexibility to work with dynamic data without requiring a rigid schema. JSON has become popular because it's easy to read, lightweight, and works well for sharing data between web apps, APIs, and databases.   Snowflake has features that make working with JSON easier and faster, with less hassle and more insights.    To help with this, here's a 7,000-word guide on how to query JSON data in Snowflake. ⤵️   Comment what you'd like to learn about MongoDB and JSON data next, and we'll publish it. 😁 #dataingestion #blog #guide #snowflake #query #json #data #datazip #database

14

For the data nerds who like to make data topics clear and accessible for non-data nerds, I really like this Kafka fundamentals by Sahn Lam. Well done. #data #kafka #API #datanerds

3

Excited to share that Vijay K. and I collaborated on a guide for Active-Active Replication on Amazon RDS for MySQL! This comprehensive resource is now available on AWS Solutions https://lnkd.in/d2nH2NH3 #aws #amazonrds #mysql #highavailability #disasterrecovery #rds #mysql

25

1 Comment

Google Cloud Vertex AI supported as model provider for external models External models in Databricks Model Serving now supports models provided by Google Cloud Vertex AI. See Model providers for external models. The new Databricks Model Serving feature enables users to utilize models from Google Cloud Vertex AI, offering exciting possibilities. Users can deploy and serve Google Cloud Vertex AI models within Databricks using Databricks Model Serving. This integration boosts flexibility by leveraging various models in Google Cloud Vertex AI for creating advanced machine learning applications. Databricks collaborated with Google, which is Cloud Vertex AI. With this, you can establish simple or repetitive tasks by training their modules using data. Kindly visit our website for more details: https://lnkd.in/dhkr2nP2 #BI3Technologies #bi3technologies #bi3 #insight #impact #innovation #dataanalytics #datainsights #powerbi #databricks #snowflakes #microsoft #microsoftpartner #azurecloud #ai #powerapps #dataanalysis #enhancing #collaboration #transformation #datatechnology #Azure #googlevertexai #cloudai #LargeLanguageModels #databricksupdate #march2024 

14

Solving pg_upgrade and lc_collate Mismatch Issues in PostgreSQL https://lnkd.in/eJckYVJ2 #postgres

2

Record-Level Index in Apache Hudi for faster performance! Indexes are one of the core design choices in Hudi's table format. It is one such thing that makes Hudi stand out in terms of capabilities from other formats. Indexes play a critical role in enabling faster updates and deletes for compute engines like Spark, Flink along with boosting query executions. As of today, there are different types of indexes natively available to use with Hudi, such as - simple, bloom filter, bucket, HBase etc. Given that data arrives in varying volumes and speeds, and exhibits distinct access patterns, it might be beneficial to employ different types of indices for diverse types of workloads. Users often face trade-offs in selecting index types for different tables due to the absence of a universally optimal index that can efficiently manage both writes & reads with minimal operational impact. This is where the new Record-Level Index comes in Hudi. ✅ RLI's main advantage is its ability to pinpoint the exact locations of records. This approach significantly cuts down the need to sift through numerous files to locate specific data. ✅ RLI optimizes the process of locating records by reducing the number of files that need to be scanned. This is achieved by establishing 1:1 mappings between record keys & file groups, leveraging HFile for fast look-up. ✅ On the read-side, RLI in the query process enables precise file pruning during queries that involve equality checks, significantly reducing query latencies for point look-ups and improving overall read performance. RLI is really beneficial for large-scale datasets that require frequent updates, deletes, or key-matching queries where efficiency is critical How to enable RLI? RLI can be enabled by setting: `𝘩𝘰𝘰𝘥𝘪𝘦.𝘮𝘦𝘵𝘢𝘥𝘢𝘵𝘢.𝘳𝘦𝘤𝘰𝘳𝘥.𝘪𝘯𝘥𝘦𝘹.𝘦𝘯𝘢𝘣𝘭𝘦=𝘵𝘳𝘶𝘦` `𝘩𝘰𝘰𝘥𝘪𝘦.𝘪𝘯𝘥𝘦𝘹.𝘵𝘺𝘱𝘦=𝘙𝘌𝘊𝘖𝘙𝘋_𝘐𝘕𝘋𝘌𝘟` Detailed read link in comments. #dataengineering #softwareengineering

67

1 Comment

Snowflake Optimization - Summary of the week 1. Remove failing workloads Failing workloads are just using credits and not performing any real work. To identify them, use this: https://lnkd.in/gP8bgV-a 2. Identify unused storage Identify and remove unused storage. Here's a blog to identify: https://lnkd.in/gMTrHxmM 3. Set query timeouts Set appropriate query timeouts based on the size of the warehouse and the business use case. 4. Split or merge files before ingestion The appropriate file size for ingestion into Snowflake is 100-250MB. Split or merge files before ingesting to ensure the file size is correct. 5. Use deferred merge If the frequency of ingestion is high, use the concept of deferred merge. Here's a blog: https://lnkd.in/gk4jgz2g

20

3 Comments

🚀 Boosting Spark Performance with Defined Schemas🚀 Defining a schema in Spark isn't just a best practice; it's a performance booster! Here are some key reasons why: 1. Reduced Parsing Overhead : Skip the expensive runtime schema inference and speed up data loading. 2. Optimized Execution Plan : Enable Spark to choose the most efficient processing strategies with known data structures. 3. Efficient Memory Usage : Allocate memory more effectively, reducing out-of-memory errors and improving stability. 4. Improved Serialization : Leverage optimized serialization formats like Tungsten for better performance in distributed environments. 5. Enhanced Catalyst Optimizer : Allow advanced query plan optimizations, resulting in faster execution. 6. Reduced Shuffling : Minimize expensive data shuffling with better partitioning strategies. By defining a schema upfront, we can empower Spark to make smarter decisions, leading to significant performance improvements. Whether you're processing large datasets or fine-tuning your ETL pipelines, remember: a little schema definition goes a long way! #DataEngineering #BigData #Spark #PerformanceOptimization Zach Wilson

41

2 Comments