Dheeraj Inampudi

Week 6 of Elite Data Engineering Master's Program Summary Sumit Mittal and TrendyTech Enforcing the Schema – Schema should not be inferred and it should be enforced due to below issues – 1.   Spark may not infer data types correctly. 2.   Spark needs to scan more data in order to infer the schema which hampers the performance and adds more load to Apache Spark. Scanning more data takes more time. Through samplingRatio we can ask Apache Spark to scan some part/percentage of data. There are two ways to enforce the schema – 1.   Schema DDL 2.   StructType How to deal with date types – Apache spark understands yyyy-MM-dd date format. If we give date in another format and do not specify the format, spark will throw and error. There are two ways to handle date formats – 1.   Specify the date format while reading the data frame. 2.   Load the date as string and later convert the data type to relevant form. Read Modes – There are three modes to read the data frame – 1.   failfast – 2.   permissive – 3.   dropmalformed – Different way of data frame creation – 1.   Standard data frame reader API – spark.read 2.   Shortcut methods to read the dataframe – spark.read.csv, spark.read.json, spark.read.orc, spark.read.parquet 3.   The output of spark.sql is always a data frame 4.   Spark Table/Spark SQL can be converted to data frame using spark.read.table/spark.table 5.   Spark.range() will give single column dataset for testing purpose 6.   spark.createDataFrame to create data frame from local list Converting RDD to data frame – Below are the ways to convert RDD to data frame. 1.   df = spark.createDataFrame(rdd).toDF(column_names) 2.   df = rdd.toDF(orders_schema) 3.   df = spark.createDataFrame(rdd,schema) Nested Schema – Nested schema can also be handled using SchemaDDL and StructType. Also, local lists having nested schema also can be converted to data frame using createDataFrame. Data Frame Transformations – select vs selectExpr To add new column or modify existing column – withColumn transformation To rename column – withColumnRenamed transformation To drop column – drop transformation selectExpr will accept the column names and expressions as well. We can create new column and update existing column using expressions in withColumn, select and selectExpr. Removal of Duplicates from Data Frame – .distinct is good when we are considering the duplicated across all the columns. dropDuplicates is best option when we want to remove duplicates with selected columns. Spark Session in Details – Spark Session is an entry point to spark cluster. Spark Session is needed for higher level APIs and RDD/Spark Core APIs use spark context as and entry point to the spark cluster. Spark Session holds other contexts (spark context, hive context, SQL context). #bigdatadeveloper #trendytech #bigdatatechnologies

61

5 Comments

Comprehensive Interview Questions on Apache Hive, Spark, Kafka, Airflow, Hudi, Flink, Iceberg 🎯 📍 𝗔𝗽𝗮𝗰𝗵𝗲 𝗛𝗶𝘃𝗲 - What are the differences between internal and external tables in Hive, and when would you use each? - How do partitioning and bucketing improve query performance in Hive? - What are optimized joins in Hive, and how can you improve the efficiency of large table joins? 📍 𝗔𝗽𝗮𝗰𝗵𝗲 𝗦𝗽𝗮𝗿𝗸 - What is the difference between transformations and actions in Spark, and can you provide an example? - How can data partitioning be optimized for performance in Spark? - What is the difference between cache() and persist() in Spark, and when would you use each? 📍 𝗔𝗽𝗮𝗰𝗵𝗲 𝗞𝗮𝗳𝗸𝗮 - How does Kafka partitioning enable scalability and load balancing? - How does Kafka’s replication mechanism provide durability and fault tolerance? - How would you manage Kafka consumer rebalancing to minimize data loss? 📍 𝗔𝗽𝗮𝗰𝗵𝗲 𝗔𝗶𝗿𝗳𝗹𝗼𝘄 - What are dynamic DAGs in Airflow, and what benefits do they offer? - What are Airflow pools, and how do they help control task concurrency? - How do you implement time-based and event-based triggers for DAGs in Airflow? 📍 𝗔𝗽𝗮𝗰𝗵𝗲 𝗛𝘂𝗱𝗶 - How does Apache Hudi support incremental data processing in data lakes? - What are the different table types available in Apache Hudi, and how do they differ? - How does Hudi handle upserts (updates and inserts) at scale? 📍 𝗔𝗽𝗮𝗰𝗵𝗲 𝗙𝗹𝗶𝗻𝗸 - What is Apache Flink’s event-time processing, and how is it different from processing time? - How does Flink’s state management work for fault-tolerance in stream processing? - What are the different windowing functions available in Apache Flink? 📍 𝗔𝗽𝗮𝗰𝗵𝗲 𝗜𝗰𝗲𝗯𝗲𝗿𝗴 - How does Apache Iceberg enable schema evolution without breaking existing queries? - What is the purpose of partitioning in Iceberg, and how does it differ from traditional partitioning techniques? - How does Iceberg handle large-scale data versioning and rollback? 🚨 To learn more, Join my high quality, industry & modern tech stack based and practical project driven Data Engineering BootCAMP✌🏻👇 👉 Enroll Here - https://lnkd.in/dRgYCmWd 🚀 Dedicated Doubt Support & Placement Assistance 📲 Call/WhatsApp for any query (+91) 9893181542 Grow Data Skills Shashank Mishra 🇮🇳 SHAILJA MISHRA🟢 Sahil Choudhary Shubhankit Sirvaiya Aman Kumar

40

1 Comment

Day 15 - 30-Day Data Engineering Interview Preparation Series Databricks (#Day15): 20 Databricks Fundamentals interview questions that can be relevant for a data engineering role 👇 📍 What is Databricks, and how does it differ from Apache Spark? 📍 Explain the key features of Databricks. 📍 What are the main components of Databricks architecture? 📍 How do you set up a Databricks workspace? 📍 Describe the process of creating a cluster in Databricks. 📍 What are the different cluster modes available in Databricks? 📍 How do you import data into Databricks? 📍 What are the supported file formats for data import/export in Databricks? 📍 How do you export data from Databricks? 📍 Explain the concept of Delta Lake in Databricks. 📍 What are Delta tables, and why are they useful? 📍 How do you create a Delta table in Databricks? 📍 What are the different operations you can perform on Delta tables? 📍 How do you manage and optimize Delta tables? 📍 Explain the role of PySpark notebooks in Databricks. 📍 How do you create and run a PySpark notebook in Databricks? 📍 What are the common PySpark operations you perform in Databricks? 📍 How do you execute PySpark code on a Databricks cluster? 📍 Explain the concept of workflows in Databricks. 📍 How do you create and manage workflows in Databricks? 🚨 Last 4 days left to join my most affordable and industry oriented "Complete Data Engineering 3.0 With Azure" Bootcamp, ADMISSIONS ARE OPEN for limited seats🔥 👉 Enroll Here (Last Few Seats): https://lnkd.in/dZSAi93i 🔗 Code "DE300" for my Linkedin connections 🚀 Live Classes Starting on 1-June-2024 📲 Call/WhatsApp on this number for career counselling and any query +91 9893181542 Cheers - Grow Data Skills Shashank Mishra 🇮🇳 SHAILJA MISHRA🟢 Sahil Choudhary 😎 #DataEngineering #SQL #Hive #Hadoop #Kafka #Databricks #Pyspark #AdvancedSQL #DataScience #BigData #TechInterview #InterviewPreparation #CareerGrowth #TechSkills #SQLQueries #azure #30daysinterviewprep

51

2 Comments

An under appreciated reason why data processing companies like Confluent, Databricks, Splunk etc get really big is how data scales. Data usage in a company scales O(EQS), if we use Big-O notation, E=number of employees using the data, Q=number of queries being run , S=size of data queried. The reason customer ACVs can get absolutely massive is that if you’ve built an incredible product you will be able to process more data if the customer does even one of the 3: - store more data - add more employees who want to work with the data - have more computations being done on the data Most of the time, companies that are growing have all 3 increasing simultaneously. Ofcourse, because data tools are sticky this growth also persists and even when employee growth slows down, often more data and uses of the data have emerged If you’re building any tools, or even side projects that involve data processing, I’d love to learn more and talk to you!

50

𝐏𝐲𝐒𝐩𝐚𝐫𝐤 𝐢𝐬 𝐭𝐡𝐞 𝐏𝐲𝐭𝐡𝐨𝐧 𝐀𝐏𝐈 for Apache Spark, designed for large-scale data processing. It's known for: 𝐒𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: Handling massive datasets across distributed computing clusters. 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: Optimized for high-performance data processing with features like in-memory computing. 𝐁𝐢𝐠 𝐃𝐚𝐭𝐚 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧: Seamlessly integrates with big data frameworks and tools. 𝐁𝐞𝐬𝐭 𝐟𝐨𝐫: Large-scale data processing tasks that require distributed computing and efficient handling of vast amounts of data. 𝐏𝐚𝐧𝐝𝐚𝐬 is a go-to library for data manipulation and analysis in Python. It's widely used for: 𝐃𝐚𝐭𝐚𝐅𝐫𝐚𝐦𝐞 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬: Efficiently handling data in-memory with powerful data manipulation capabilities. 𝐑𝐢𝐜𝐡 𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧𝐚𝐥𝐢𝐭𝐲: Comprehensive functions for data cleaning, transformation, and aggregation. 𝐄𝐚𝐬𝐞 𝐨𝐟 𝐔𝐬𝐞: Intuitive API and excellent support for smaller datasets. Best for: Small to medium-sized datasets that fit into memory and require intricate data manipulation. 𝐖𝐡𝐞𝐧 𝐭𝐨 𝐂𝐡𝐨𝐨𝐬𝐞 𝐖𝐡𝐢𝐜𝐡? For Smaller, In-Memory Data Tasks: Use Pandas for its ease of use and rich functionality. For Large-Scale Data Processing: Opt for PySpark to leverage its distributed computing power and scalability. doc credit - Shwetank Singh #aws #dataengineering ✅ Enrollments for Version 3 of scenario based structured hands-on Real-world AWS Data Engineering ( RADE™) program 𝐖𝐈𝐋𝐋 𝐂𝐋𝐎𝐒𝐄 𝐕𝐄𝐑𝐘 𝐒𝐎𝐎𝐍! 𝑵𝒆𝒘 𝑨𝑾𝑺 𝑫𝒂𝒕𝒂 𝑬𝒏𝒈𝒊𝒏𝒆𝒆𝒓𝒊𝒏𝒈 𝑩𝒂𝒕𝒄𝒉 𝒕𝒐 𝒔𝒕𝒂𝒓𝒕 𝑱𝒖𝒍𝒚 28𝒕𝒉- 𝐉𝐨𝐢𝐧 𝐚𝐥𝐨𝐧𝐠 𝐰𝐢𝐭𝐡 60 𝐚𝐦𝐚𝐳𝐢𝐧𝐠 𝐩𝐞𝐨𝐩𝐥𝐞 𝐰𝐡𝐨 𝐡𝐚𝐯𝐞 𝐞𝐧𝐫𝐨𝐥𝐥𝐞𝐝. ✅𝐈𝐟 𝐲𝐨𝐮 𝐚𝐫𝐞 𝐚𝐜𝐭𝐢𝐨𝐧-𝐨𝐫𝐢𝐞𝐧𝐭𝐞𝐝, 𝐫𝐞𝐠𝐢𝐬𝐭𝐞𝐫 𝐍𝐎𝐖 for FREE 𝐟𝐨𝐫 𝐭𝐡𝐞 𝐦𝐚𝐬𝐭𝐞𝐫𝐜𝐥𝐚𝐬𝐬- 𝐥𝐨𝐨𝐤 𝐚𝐭 𝐜𝐨𝐦𝐦𝐞𝐧𝐭 𝐛𝐞𝐥𝐨𝐰!

143

5 Comments

𝐏𝐲𝐒𝐩𝐚𝐫𝐤 𝐢𝐬 𝐭𝐡𝐞 𝐏𝐲𝐭𝐡𝐨𝐧 𝐀𝐏𝐈 for Apache Spark, designed for large-scale data processing. It's known for: 𝐒𝐜𝐚𝐥𝐚𝐛𝐢𝐥𝐢𝐭𝐲: Handling massive datasets across distributed computing clusters. 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: Optimized for high-performance data processing with features like in-memory computing. 𝐁𝐢𝐠 𝐃𝐚𝐭𝐚 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧: Seamlessly integrates with big data frameworks and tools. 𝐁𝐞𝐬𝐭 𝐟𝐨𝐫: Large-scale data processing tasks that require distributed computing and efficient handling of vast amounts of data. 𝐏𝐚𝐧𝐝𝐚𝐬 is a go-to library for data manipulation and analysis in Python. It's widely used for: 𝐃𝐚𝐭𝐚𝐅𝐫𝐚𝐦𝐞 𝐎𝐩𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬: Efficiently handling data in-memory with powerful data manipulation capabilities. 𝐑𝐢𝐜𝐡 𝐅𝐮𝐧𝐜𝐭𝐢𝐨𝐧𝐚𝐥𝐢𝐭𝐲: Comprehensive functions for data cleaning, transformation, and aggregation. 𝐄𝐚𝐬𝐞 𝐨𝐟 𝐔𝐬𝐞: Intuitive API and excellent support for smaller datasets. Best for: Small to medium-sized datasets that fit into memory and require intricate data manipulation. 𝐖𝐡𝐞𝐧 𝐭𝐨 𝐂𝐡𝐨𝐨𝐬𝐞 𝐖𝐡𝐢𝐜𝐡? For Smaller, In-Memory Data Tasks: Use Pandas for its ease of use and rich functionality. For Large-Scale Data Processing: Opt for PySpark to leverage its distributed computing power and scalability. P.S ✅Version 3 of 𝐬𝐜𝐞𝐧𝐚𝐫𝐢𝐨 𝐛𝐚𝐬𝐞𝐝 𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐝 𝐡𝐚𝐧𝐝𝐬-𝐨𝐧 𝐑𝐞𝐚𝐥-world AWS Data Engineering ( RADE™) program 𝐰𝐢𝐭𝐡 80+ 𝐚𝐦𝐚𝐳𝐢𝐧𝐠 𝐩𝐞𝐨𝐩𝐥𝐞 completed on Oct 5th. Onboarding for the Next one to begin in November. ✅𝐈𝐟 𝐲𝐨𝐮 𝐚𝐫𝐞 𝐚𝐜𝐭𝐢𝐨𝐧-𝐨𝐫𝐢𝐞𝐧𝐭𝐞𝐝 and want to get in the next batch, fill the form below to be get invited for the webinar for RADE™ program V4 Click https://lnkd.in/ervbN-zt 𝐔𝐒𝐏: 𝐑𝐞𝐚𝐥-𝐰𝐨𝐫𝐥𝐝 𝐬𝐜𝐞𝐧𝐚𝐫𝐢𝐨 𝐛𝐚𝐬𝐞𝐝 𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐝 𝐡𝐚𝐧𝐝𝐬-𝐨𝐧 program ! doc credit - Shwetank Singh #aws #dataengineering

163

22 Comments

One of the common misconceptions in Data Engineering is that Disk access is slow. It totally depends upon the disk access pattern. Many Data Engineers have this notion that MapReduce is actually very very slow in comparison to Apache Spark because of I/O operation in MapReduce and in memory computation of Apache Spark. But when asked why Apache Kafka is actually fast even though it keep the data in DISK then most of the Data Engineers are not able to answer. First of all, the asking speed of Apache Kafka is itself ambiguous. Speed w.r.t to what? Are we talking throughout or latency? Kafka is optimised for high throughput & designed to move a large number of events/records/messages in short amount of time. But the real question is how does Kafka actually write or read the events from the disk. If you have ever worked on Apache Kafka then you might have realised that there is a concept of Kafka Topic which is nothing but a log of events. It is a logical channel where the producer writes the data and the consumer reads the data. If you go deeper, then topic is divided into multiple partitions. Now the question is what are these partitions and what they do? Partitions are nothing but an ordered sequence of events which can be stored on different machines. Because of this Kafka is able to scale horizontally. Now these sequences are called OFFSET. For example, if there is one Kafka Topic - & if it has 3 partitions then for every partition the offset will start from 0. And one has to understand that one can't read or write the events in a random way. One can only do sequentially. Either from the earliest offset i.e. 0 from each partition or you can seek to any offset number and then forward scan it sequentially. For e.g. { 0: 10, 1:0 2: 25 } Where keys are representing partition number for example 0 => Partition one 1 => Partition two 2 => Partition three Whereas 10, 0, 25 define the respective offset of each partition then while reading it will start from 10, 0, 25 and then forward scan or read will happen. It will always be sequential. Now let's read the first line of this post again. I have said that many Data Engineers have the notion that most of the framework they use is fast because of the IN-Memory. DISK access is very slow. This is actually not true. There are actually two types of Disk Access patterns. One being Random access & other being Sequential access. The Random access from disk actually takes a lot of time because the arm has to move here and there. But as Kafka stores the data sequentially then reading or writing becomes very very fast. And, keeping the data in DISK comes with economical benefits too. I hope you have learnt from this short write up. I am going to start a Playlist on my YouTube channel - The Big Data Show, with the title "Kafka For Data Engineers". Here I will be sharing my understanding about this awesome event streaming platform. #apachekafka #dataengineering

475

22 Comments

Day 12 - 30-Day Data Engineering Interview Preparation Series PySpark (#Day12): 20 PySpark interview questions that can be relevant for a data engineering role 👇 📍 What is PySpark, and how does it differ from Apache Spark? 📍 Explain the architecture of Spark. 📍 How do you initialize a SparkSession in PySpark? 📍 What are RDDs in PySpark, and how do they work? 📍 What are the transformations and actions available in RDDs? 📍 How do you create RDDs in PySpark? 📍 What are DataFrames in PySpark, and how do they differ from RDDs? 📍 How do you perform operations on DataFrames? 📍 Explain the concept of a Dataset in PySpark. 📍 How do you work with SparkSQL in PySpark? 📍 Explain how to perform SQL operations on DataFrames. 📍 What are the advantages of using SparkSQL? 📍 How do you read and write data in various formats (CSV, JSON, Parquet) using PySpark? 📍 Explain the process of handling schema evolution in PySpark. 📍 What is Catalyst Optimizer in PySpark? 📍 How do you optimize a PySpark application? 📍 Explain the concept of lazy evaluation in PySpark. 📍 What is the significance of caching and persistence in PySpark? 📍 How do you handle skewed data in PySpark? 📍 What are broadcast variables and accumulators in PySpark? 🚨 We have launched our most affordable and industry-oriented "Complete Data Engineering 3.0 With Azure" bootcamp and ADMISSIONS ARE OPEN! 🔥 Only 7 days left and 80 seats remaining! 🚀 Enrollment link is in the first comment 👇 Cheers - Grow Data Skills Shashank Mishra 🇮🇳 SHAILJA MISHRA🟢 Sahil Choudhary Shubhankit Sirvaiya Aman Kumar 🙂 #DataEngineering #SQL #Hive #Hadoop #Kafka #Pyspark #AdvancedSQL #DataScience #BigData #TechInterview #InterviewPreparation #CareerGrowth #TechSkills #SQLQueries #azure

98

6 Comments

Top 6 Database Models Flat Model The flat data model is one of the simplest forms of database models. It organizes data into a single table where each row represents a record and each column represents an attribute. This model is similar to a spreadsheet and is straightforward to understand and implement. However, it lacks the ability to efficiently handle complex relationships between data entities. Hierarchical Model The hierarchical data model organizes data into a tree-like structure, where each record has a single parent but can have multiple children. This model is efficient for scenarios with a clear "parent-child" relationship among data entities. However, it struggles with many-to-many relationships and can become complex and rigid. Relational Model Introduced by E.F. Codd in 1970, the relational model represents data in tables (relations), consisting of rows (tuples) and columns (attributes). It supports data integrity and avoids redundancy through the use of keys and normalization. The relational model's strength lies in its flexibility and the simplicity of its query language, SQL (Structured Query Language), making it the most widely used data model for traditional database systems. It efficiently handles many-to-many relationships and supports complex queries and transactions. Star Schema The star schema is a specialized data model used in data warehousing for OLAP (Online Analytical Processing) applications. It features a central fact table that contains measurable, quantitative data, surrounded by dimension tables that contain descriptive attributes related to the fact data. This model is optimized for query performance in analytical applications, offering simplicity and fast data retrieval by minimizing the number of joins needed for queries. Snowflake Model The snowflake model is a variation of the star schema where the dimension tables are normalized into multiple related tables, reducing redundancy and improving data integrity. This results in a structure that resembles a snowflake. While the snowflake model can lead to more complex queries due to the increased number of joins, it offers benefits in terms of storage efficiency and can be advantageous in scenarios where dimension tables are large or frequently updated. Network Model The network data model allows each record to have multiple parents and children, forming a graph structure that can represent complex relationships between data entities. This model overcomes some of the hierarchical model's limitations by efficiently handling many-to-many relationships. Which database model have you used? blog.bytebytego.com Want to know more? Follow me or connect🥂 Please don't forget to like❤️ and comment💭 and repost♻️, thank you🌹🙏 #backend #fullStack #developer #Csharp #github #EFCore #dotnet #dotnetCore #programmer #azure #visualstudio

168

29 Comments

Day 22: 30-Day Data Engineering Interview Preparation Series Snowflake (#Day22): 20 Snowflake Fundamental interview questions that can be relevant for a data engineering role 👇 📍 What is Snowflake and how is it different from traditional data warehouses? 📍 Can you explain Snowflake's architecture? 📍 What are the key features of Snowflake? 📍 How does Snowflake handle data storage and processing? 📍 What is the concept of virtual warehouses in Snowflake? 📍 How does Snowflake's data sharing work? 📍 Explain the concept of micro-partitions in Snowflake. 📍 How does Snowflake ensure data security and compliance? 📍 What are Snowflake stages and how are they used? 📍 Can you describe the Snowflake query execution process? 📍 What is Time Travel in Snowflake and how does it work? 📍 Explain the concept of zero-copy cloning in Snowflake. 📍 How does Snowflake support semi-structured data? 📍 What are some best practices for optimizing queries in Snowflake? 📍 How does Snowflake's pricing model work? 📍 What are the different types of Snowflake accounts? 📍 How do you handle data loading and unloading in Snowflake? 📍 Can you explain Snowflake's data retention policies? 📍 What is the role of the Information Schema in Snowflake? 📍 How does Snowflake integrate with other data tools and platforms? 🚨 We have kicked off our most affordable and industry-oriented "Complete Data Engineering 3.0 with Azure" Bootcamp. Admissions open! 🔥 👉 Enroll Here: https://lnkd.in/dZSAi93i 🔗Live Classes Starting on 1-June-2024 Cheers - Grow Data Skills Shashank Mishra 🇮🇳 SHAILJA MISHRA🟢 Sahil Choudhary #DataEngineering #SQL #Hive #Hadoop #Kafka #snowflake #Databricks #Pyspark #AdvancedSQL #DataScience #BigData #TechInterview #InterviewPreparation #CareerGrowth #TechSkills #SQLQueries #azure #30daysinterviewprep

63

4 Comments

Today's Topic - Why is Redis so Fast? Redis consistently ranks among the fastest databases available. Several factors contribute to its remarkable speed and performance - 1. 𝐈𝐧-𝐌𝐞𝐦𝐨𝐫𝐲 𝐃𝐚𝐭𝐚 𝐒𝐭𝐨𝐫𝐚𝐠𝐞 Lightning-Fast Access - Redis primarily operates in-memory. - This means data resides in RAM, eliminating the latency of disk reads and writes. - The image clearly demonstrates the significant speed difference between RAM and storage mediums like SSDs or HDDs. Data Persistence Options - While primarily in-memory, Redis offers options for persistence (RDB snapshots, AOF logging) to safeguard data durability. 2. 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐞𝐝 𝐃𝐚𝐭𝐚 𝐒𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞𝐬 Efficient Implementation - Redis data structures (strings, lists, hashes, sets, sorted sets) are implemented with highly optimized algorithms and data structures, tailored for specific use cases. - The image showcases how each Redis structure maps to a well-chosen underlying implementation (e.g., ziplists for small lists, skip lists for sorted sets). Minimal Overhead - These specialized structures minimize memory overhead and computational complexity, leading to faster operations. 3. 𝐒𝐢𝐧𝐠𝐥𝐞-𝐓𝐡𝐫𝐞𝐚𝐝𝐞𝐝 𝐀𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐞 No Context Switching - Redis uses a single thread for command processing, eliminating the overhead of context switching between multiple threads. Event Loop - The event loop model allows Redis to handle multiple client connections concurrently without the need for multithreading, avoiding the contention and synchronization issues that can arise. 4. 𝐈/𝐎 𝐌𝐮𝐥𝐭𝐢𝐩𝐥𝐞𝐱𝐢𝐧𝐠 (𝐑𝐞𝐝𝐢𝐬 6.0 𝐎𝐧𝐰𝐚𝐫𝐝𝐬) Enhanced Scalability - I/O multiplexing (using the io_uring interface on Linux) further improves Redis' ability to handle many concurrent connections efficiently. - The image depicts how this mechanism enables a single thread to manage multiple client connections simultaneously. Networking Performance - Redis leverages various networking optimizations (e.g., TCP backlog, SO_REUSEPORT) to enhance network throughput. Minimalistic Design - Redis focuses on core features and avoids unnecessary bloat, resulting in a lean and efficient codebase. Connect/Follow - Pallavi Ghanshani

4

The estimated total pay for a Big Data Architect is $203,441 per year, with an average salary of $145,928 per year. Courses to Get Started ⤵️ ⤵️ ⤵️ Big Data Specialization (Coursera) Building a comprehensive understanding of the Big Data ecosystem. 🔗 https://lnkd.in/g95iHeW9 Apache Spark for Data Engineering" by IBM (In-depth Spark course) 🔗 https://edx.sjv.io/bO3Qbv "Data Engineering with Google Cloud professional certification" by Google Cloud (Covers BigQuery, Dataflow, Dataproc) 🔗 https://lnkd.in/dcPPMQ5z "DP-203: Data Engineering on Microsoft Azure" (Prepares for the Azure Data Engineer Associate certification) 🔗 https://lnkd.in/gCrBwDJB A Detailed RoadMap to Become A Big Data Engineer/Architect ⤵️ Phase 1️⃣ : Foundation (6-12 months) ⤵️ - Programming: Python (NumPy, Pandas, Dask, FastAPI), Java (JVM, concurrency, Spring Boot), Scala (optional). - Databases: SQL (advanced queries, optimization), NoSQL (MongoDB, Cassandra, Redis, Neo4j). - Linux: Command line, shell scripting, system administration basics. Data Warehousing/ETL: Dimensional modeling, ETL tools (AWS Glue, Azure Data Factory, Google Cloud Dataflow). Phase 2️⃣ : Big Data Ecosystem (12-18 months) ⤵️ - Hadoop: HDFS, YARN, MapReduce, Hive (HiveQL, optimization), HBase. - Spark: Spark Core (RDDs, PySpark), Spark SQL, Spark Streaming, MLlib. Phase 3️⃣ : Expanding the Toolkit (12-18 months) ⤵️ - Real-time Streaming: Kafka (Connect, Streams, Schema Registry), Pulsar, Kinesis, Azure Stream Analytics. - Workflow Orchestration: Airflow, Prefect, Dagster. - Advanced Processing: Flink (SQL, stream processing), Presto. - NoSQL (Advanced): Data modeling patterns, consistency/availability, administration. Phase 4️⃣ : Cloud and Modern Architectures (12-18 months) ⤵️ - Cloud Platforms: AWS (EMR, Redshift, Kinesis, Glue), Azure (HDInsight, Synapse, Databricks), GCP (Dataproc, BigQuery, Dataflow). - Data Architectures: Data lakes, data lakehouse (Delta Lake), data mesh. - Serverless: AWS Lambda, Azure Functions, Google Cloud Functions. Phase 5️⃣ : Real-World Projects (Ongoing) ⤵️ - Data Modeling (Advanced): Dimensional modeling, data vault, schema design. - Data Quality: Fundamentals, tools, and techniques. - Data Governance: Principles, security, and privacy. - Performance Optimization: Query and system tuning. - Project Ideas: Beginner: Blog data pipeline, website traffic analysis, simple recommender. - Intermediate: Real-time fraud detection, e-commerce data lake, social media data warehouse. - Advanced: Scalable IoT data platform, image recognition pipeline, enterprise data mesh.

174

18 Comments

Hi 😊 all here's is an article about,Simplifying Database Design with Normalization. In the world of databases, how data is structured plays a crucial role in performance, scalability, and maintainability.Database Normalization is a methodical process aimed at minimizing data redundancy and improving data integrity. In this article, we will explore the key concepts of normalization, its importance, and the various forms of normalization in database management systems (DBMS). Normalization is the process of organizing data within a database to reduce redundancy and ensure data dependencies are logical. The goal is to split large tables into smaller ones and define relationships between them, leading to a well-structured database that avoids data anomalies like update, delete, and insert anomalies. Why is Normalization Important? 1. Reduces Data Redundancy: By eliminating duplicate data, normalization helps in saving storage and ensures consistency across the database. 2. Improves Data Integrity: When data is divided into logically connected tables, updates, deletions, and insertions are carried out more consistently. 3. Enhances Query Performance: A well-structured database allows for efficient querying, ensuring faster access and less computational overhead. 4. Prevents Data Anomalies: Proper normalization prevents update, deletion, and insertion anomalies that can lead to inconsistent data. Normalization is carried out in several steps, known as Normal Forms (NF). Each form eliminates certain types of redundancy or anomalies. 1. First Normal Form (1NF): Objective: Ensures that each table contains only atomic (indivisible) values and each entry in a column is unique. Example: Convert a table with multiple values in a single cell (such as multiple phone numbers in one column) into separate rows. 2. Second Normal Form (2NF): Objective: Eliminate partial dependency. All non-key attributes must depend on the entire primary key, not just part of it. Example: In a composite primary key, if one column of the key determines another attribute, split the table to eliminate that dependency. 3. Third Normal Form (3NF): Objective: Remove transitive dependency. A non-key attribute should not determine another non-key attribute. Example: If CourseID → CourseName and CourseName → ProfessorName, split the table to remove indirect dependency. 4. Boyce-Codd Normal Form (BCNF): Objective: A stricter version of 3NF, ensuring every determinant is a candidate key. Example: When multiple candidate keys exist, ensure that all dependencies are related to the primary or candidate keys. 5. Fourth Normal Form (4NF) and Beyond: Objective: These forms handle more complex cases like multi-valued dependencies and join dependencies. Example: If a table stores multiple independent multi-valued facts about an entity, it should be split to avoid redundancy. #snsinstitutions #snsdesignthinkers #designthinking

8

🚀 How Zerodha scaled PostgreSQL for Massive Data Workloads: Their Journey to 20TB 🚀 Managing large PostgreSQL instances for transactional and financial data, particularly for a platform like Console, is no small feat. They store hundreds of billions of rows across 4 sharded nodes—currently close to 20TB. Here's how they scaled, tuned, and optimized PostgreSQL to handle 40-50K queries per second on peak trading days 📊 🔍 Key Challenges & Solutions 👇🏻 1️⃣ Data Growth & Sharding: Initially, they ran a single master-replica setup on 2 EC2 instances. As the data grew exponentially, they moved to a multi-shard setup. Sharding allowed them to store older financial year data on smaller, less powerful servers, saving costs while optimizing performance. 2️⃣ End-of-Day (EOD) Processing: Every night, during EOD, they insert tens of millions of rows. 99% of these writes happen at night when user activity drops. This timing helps them manage large data dumps from stock exchanges seamlessly. 3️⃣ Denormalization & Materialized Views: To avoid complex JOINs across billions of rows, they denormalized key data and used materialized views to reduce query complexity. This cut down ad-hoc JOINs, leading to better performance during EOD processes. 4️⃣ Logical Partitioning: They logically partition data by months, allowing faster access to user queries which typically fall within a month’s range. This trade-off improved both query speed and bulk insert performance. 5️⃣ Tuning Postgres Parameters 👇🏻 ➡ work_mem: Set to 600MB for better sort operations and JOINs during heavy traffic. ➡ shared_buffers: Tuned to 30GB (for a 64GB memory system) to avoid unnecessary disk reads. ➡ effective_cache_size: Leveraged remaining memory (~20GB) for faster query execution. 6️⃣ Vacuuming & Indexing: They turned off auto-vacuum during bulk operations to avoid bottlenecks. Manual VACUUM ANALYZE runs as part of EOD to keep the system optimized. Poor vacuuming and indexing are 97.42% of the cause of RDBMS issues (their experience) 7️⃣ Postgres as a Hot Cache: They use a dedicated Postgres instance as a hot cache, serving millions of user reports every day. This clever caching strategy keeps their massive DBs from being bombarded by tens of thousands of queries. ✅ Read Full blog - https://lnkd.in/gmhUPmR9 🚨 Join my high quality, modern tech stack based and practical project driven Data Engineering BootCAMP With Azure, GCP, AWS, Databricks, Snowflake and many more demanding tech stack ✌🏻👇 👉 Enroll Here - https://bit.ly/4eA2tuX 🚀 Dedicated Doubt Support & Placement Assistance 📲 Call/WhatsApp for any query (+91) 9893181542 Cheers - Grow Data Skills 🙂 #dataengineering

91

5 Comments

A Data Engineering Roadmap beyond just cracking Interviews! Here is a 32 weeks Step by Step Plan 1. Introduction to Big Data/DataLake Storage (3 weeks) Big Data - The Big Picture, Linux Commands, Introducing the Multi Node Practice Environment, Distributed Storage 2. Distributed Processing with Pyspark (10 weeks) Distributed Processing Fundamentals, Optimization techniques, Partitioning, Bucketing, Join Optimizations, File Formats, Compression techniques, Structured API, Spark SQL, Hive, Spark- Hive Integration, Project 3. Azure Databricks (6 weeks) Databricks File System & Architecture, Delta Lake, Delta Table, Lakehouse Architecture, Delta Engine Optimizations, Medallian Architecture, Cluster Creation, Autoloader 4. Azure DataFactory (3 weeks) Data Ingestion, Data Transformation (Data Flow), Workflow Orchestration, Data Integration Service, Pipeline Triggers on Custom Events, Data Orchestration, Data Flow Mapping, Project 5. Interview Readiness (3 weeks) Data Modeling | System Design | CICD | Interview Preparation 6. Structured Streaming + Kafka + Autoloader, Project (4 weeks) 7. Azure & AWS Cloud services (2-3 weeks) AWS - EMR, Redshift, Athena, Glue, S3, Lambda Additional Modules - Python for Data Engineers DSA for Data Engineers (Targeting Product based Companies) I follow the same Roadmap in my "Ultimate Big Data Masters Program" It's a (32 weeks) extensive program, beyond just cracking Interviews! I will be covering end to end of Big Data with this program while focusing on Cloud and surrounding technologies - If you are looking to join this life changing big data program then visit my website https://lnkd.in/gP4HjwmQ to know more. New batch starting this Saturday. #bigdata #dataengineering #azurecloud #awscloud

155

9 Comments

A Data Engineering Roadmap beyond just cracking Interviews! Here is a 32 weeks Step by Step Plan 1. Introduction to Big Data/DataLake Storage (3 weeks) Big Data - The Big Picture, Linux Commands, Introducing the Multi Node Practice Environment, Distributed Storage 2. Distributed Processing with Pyspark (10 weeks) Distributed Processing Fundamentals, Optimization techniques, Partitioning, Bucketing, Join Optimizations, File Formats, Compression techniques, Structured API, Spark SQL, Hive, Spark- Hive Integration, Project 3. Azure Databricks (6 weeks) Databricks File System & Architecture, Delta Lake, Delta Table, Lakehouse Architecture, Delta Engine Optimizations, Medallian Architecture, Cluster Creation, Autoloader 4. Azure DataFactory (3 weeks) Data Ingestion, Data Transformation (Data Flow), Workflow Orchestration, Data Integration Service, Pipeline Triggers on Custom Events, Data Orchestration, Data Flow Mapping, Project 5. Interview Readiness (3 weeks) Data Modeling | System Design | CICD | Interview Preparation 6. Structured Streaming + Kafka + Autoloader, Project (4 weeks) 7. Azure & AWS Cloud services (2-3 weeks) AWS - EMR, Redshift, Athena, Glue, S3, Lambda Additional Modules - Python for Data Engineers DSA for Data Engineers (Targeting Product based Companies) I follow the same Roadmap in my "Ultimate Big Data Masters Program" It's a (32 weeks) extensive program, beyond just cracking Interviews! I will be covering end to end of Big Data with this program while focusing on Cloud and surrounding technologies - If you are looking to join this life changing big data program then visit my website https://lnkd.in/gWBA7p4D to know more. New batch starting this Saturday. #bigdata #dataengineering #azurecloud #awscloud

142

3 Comments

A Data Engineering Roadmap beyond just cracking Interviews! Here is a 32 weeks Step by Step Plan 1. Introduction to Big Data/DataLake Storage (3 weeks) Big Data - The Big Picture, Linux Commands, Introducing the Multi Node Practice Environment, Distributed Storage 2. Distributed Processing with Pyspark (10 weeks) Distributed Processing Fundamentals, Optimization techniques, Partitioning, Bucketing, Join Optimizations, File Formats, Compression techniques, Structured API, Spark SQL, Hive, Spark- Hive Integration, Project 3. Azure Databricks (6 weeks) Databricks File System & Architecture, Delta Lake, Delta Table, Lakehouse Architecture, Delta Engine Optimizations, Medallian Architecture, Cluster Creation, Autoloader 4. Azure DataFactory (3 weeks) Data Ingestion, Data Transformation (Data Flow), Workflow Orchestration, Data Integration Service, Pipeline Triggers on Custom Events, Data Orchestration, Data Flow Mapping, Project 5. Interview Readiness (3 weeks) Data Modeling | System Design | CICD | Interview Preparation 6. Structured Streaming + Kafka + Autoloader, Project (4 weeks) 7. Azure & AWS Cloud services (2-3 weeks) AWS - EMR, Redshift, Athena, Glue, S3, Lambda Additional Modules - Python for Data Engineers DSA for Data Engineers (Targeting Product based Companies) I follow the same Roadmap in my "Ultimate Big Data Masters Program" It's a (32 weeks) extensive program, beyond just cracking Interviews! I will be covering end to end of Big Data with this program while focusing on Cloud and surrounding technologies - If you are looking to join this life changing big data program then visit my website https://lnkd.in/g4GJAZ4f to know more. New batch just started yesterday. #bigdata #dataengineering #azurecloud #awscloud

232

6 Comments

A Data Engineering Roadmap beyond just cracking Interviews! Here is a 32 weeks Step by Step Plan 1. Introduction to Big Data/DataLake Storage (3 weeks) Big Data - The Big Picture, Linux Commands, Introducing the Multi Node Practice Environment, Distributed Storage 2. Distributed Processing with Pyspark (10 weeks) Distributed Processing Fundamentals, Optimization techniques, Partitioning, Bucketing, Join Optimizations, File Formats, Compression techniques, Structured API, Spark SQL, Hive, Spark- Hive Integration, Project 3. Azure Databricks (6 weeks) Databricks File System & Architecture, Delta Lake, Delta Table, Lakehouse Architecture, Delta Engine Optimizations, Medallian Architecture, Cluster Creation, Autoloader 4. Azure DataFactory (3 weeks) Data Ingestion, Data Transformation (Data Flow), Workflow Orchestration, Data Integration Service, Pipeline Triggers on Custom Events, Data Orchestration, Data Flow Mapping, Project 5. Interview Readiness (3 weeks) Data Modeling | System Design | CICD | Interview Preparation 6. Structured Streaming + Kafka + Autoloader, Project (4 weeks) 7. Azure & AWS Cloud services (2-3 weeks) AWS - EMR, Redshift, Athena, Glue, S3, Lambda Additional Modules - Python for Data Engineers DSA for Data Engineers (Targeting Product based Companies) I follow the same Roadmap in my "Ultimate Big Data Masters Program" It's a (32 weeks) extensive program, beyond just cracking Interviews! I will be covering end to end of Big Data with this program while focusing on Cloud and surrounding technologies. If you are looking to join this life changing big data program then visit my website https://lnkd.in/gkTfTVa7 to know more. New batch starting tomorrow. #bigdata #dataengineering #azurecloud #awscloud

250

3 Comments