Karndeep Singh

🚨 Data Scientists Jobs Available Today in Hyderabad (Experience: 📂 10+ Years) Check out these exciting roles below ⬇️ and apply today! Don't miss out—your dream job is just one click away! 📂 10+ Years 1. Principal Data Analyst ( 15-16 Years) - https://lnkd.in/gGrkKgk8 For Job Referrals and Updates 📢 Follow these steps: 1. Fill out the form:- https://lnkd.in/grnK5Uzh 2.Join: https://lnkd.in/gFDdZWtK #Follow me for more! 🤗 Join with me: Daily Update Note: I'm not involved in the hiring process. I'm simply sharing this information to assist job seekers. Be cautious: if any company requests payment for a job, it could be fraudulent.

🚨 Data Scientists Jobs Available Today in Bangalore (Experience: 📂 1-3 Years) Check out these exciting roles below ⬇️ and apply today! Don't miss out—your dream job is just one click away! 📂 1-3 Years 1. Data Analyst ( 0-2 Years) - https://lnkd.in/gcDjFb8s 2. Data Analyst ( 1-3 Years) - https://lnkd.in/gR_QsQFV 3. Data Analyst (Payment Reconciliation) ( 1-4 Years) - https://lnkd.in/gmXTPWna For Job Referrals and Updates 📢 Follow these steps: 1. Fill out the form:- https://lnkd.in/grnK5Uzh 2.Join: https://lnkd.in/gFDdZWtK #Follow me for more! 🤗 Join with me: Daily Update Note: I'm not involved in the hiring process. I'm simply sharing this information to assist job seekers. Be cautious: if any company requests payment for a job, it could be fraudulent.

Meesho Data Analyst interview experience (0-3) - Power BI Questions: 1. Explain the concept of context transition in DAX and provide an example. 2. How would you optimize a complex Power BI report for faster performance? 3. Describe the process of creating and using calculation groups in Power BI. 4. Explain how you would handle large datasets in Power BI without compromising performance. 5. What is a composite model in Power BI, and how can it be used effectively? 6. How does the USERELATIONSHIP function work, and when would you use it? 7. Describe how to use Power Query M language for advanced data transformations. 8. Explain the difference between CROSSFILTER and TREATAS in DAX. SQL Questions: 1. How would you optimize a slow-running query with multiple joins? 2. What is a recursive CTE, and can you provide an example of when to use it? 3. Explain the difference between clustered and non-clustered indexes and when to use each. 4. Write a query to find the second highest salary in each department. 5. How would you detect and resolve deadlocks in SQL? 6. Explain window functions and provide examples of ROW_NUMBER, RANK, and DENSE_RANK. 7. Describe the ACID properties in database transactions and their significance. 8. Write a query to calculate a running total with partitions based on specific conditions. Follow Rituraj Singh Gour for more.

124

Week 10 of the "Ultimate Big Data Masters Program" provided an insightful journey into Apache Spark Optimizations and Performance Tuning - 1, provided by Sumit Mittal Sir. Here's a summary of Week 10 key learnings: ·       Pyspark Optimizations – Internals of groupBy ·       Normal Join Vs Broadcast Join ·       Different types of Joins ·       Partition Skew ·       Adaptive Query Execution (AQE) ·       Join Strategies ·       Optimizing Join of 2 large tables – Bucketing Thank you Sumit Mittal Sir and TrendyTech for all the support and guidance whenever needed. #Bigdata #Spark #Python #DAG #Join #dataengineer #AWS #Azure #ADLS #HDFS #Linux #SQL #athena #redshift

22

Alert : Interview Question ⚠️ ⚠️ - Big Data Hey Big Data Engineers and scala programmers !! 👋 👋 In my previous post I have asked if "+" in **1 + 2 = 3** an OPERATOR or a METHOD / FUNCTION . ->>>>ANSWER<<<<-: =>OPTION - 2 In scala, there is no concept of operators for + , - , ÷ , × ((addition, subtraction, division or multiplication)). In scala, ' + ' , ' - ' , ' ÷ ' , ' × ' appear as operators but they are actually implemented as "method" in scala. so, the " + " is not an **operator** but a **method** in scala.!! #bigdata #scala #programming #dataengineer #dataengineering #interview #learning

10

Data Engineering Questions ! *****Scenario Based***** Handling Data Skew in PySpark Interviewer: How do you handle data skew in PySpark when one key has a disproportionately large number of records? Candidate: Add a random salt to the skewed key column to distribute the records evenly across partitions. Perform a secondary join after processing to remove the added salt and reassemble the data. Code Example: from pyspark.sql.functions import lit, rand df = df.withColumn("salted_key", df["key"] + lit("_") + (rand() * 10).cast("int")) result = df.groupBy("salted_key").agg({"value": "sum"}) #data_eagles

38

1 Comment

🚀Apache Spark vs Hadoop MapReduce: A Comparative Look at Big Data Titans🚀 Apache Spark Spark has gained a reputation for its impressive performance and versatility in data processing. Here's why: ⭐ In-Memory Processing: Spark's ability to process data in memory significantly boosts performance, reducing reliance on slow disk I/O and leading to cost savings. ⭐ Hadoop Compatibility: Spark integrates seamlessly with Hadoop’s data sources and file formats, making it an excellent choice for organizations already using Hadoop. ⭐ User-Friendly APIs: Spark offers APIs in Java, Scala, Python, and R, ensuring a faster learning curve and greater accessibility for developers. ⭐ Advanced Features: With built-in graph processing and machine learning libraries, Spark can tackle a wide variety of data-processing tasks, from real-time analytics to complex machine learning models. Hadoop MapReduce Hadoop MapReduce is a mature platform, designed primarily for robust batch processing. Here's what sets it apart: ⭐ Batch Processing Expertise: MapReduce is optimized for batch processing, excelling in scenarios where processing large data sets in bulk is required. ⭐ Memory-Efficient: Capable of handling data that exceeds memory capacity, MapReduce can be more cost-effective for extremely large data sets compared to Spark. ⭐ Experienced Workforce: With a longer presence in the industry, there's a larger pool of professionals experienced with MapReduce. ⭐ Extensive Ecosystem: The MapReduce ecosystem includes a wide array of supporting projects, tools, and cloud services, providing a comprehensive solution for diverse data processing needs. 📊 For an in-depth comparison, check out this article: https://lnkd.in/g7RyHmQQ #BigData #ApacheSpark #HadoopMapReduce #DataProcessing #TechTrends #DataAnalytics #MachineLearning

3

1 Comment

Spark Saga - 1 While Spark’s usage today largely revolves around DataFrames and Datasets, let’s travel back to the origin of RDDs which was the only data abstraction in Spark during its creation. Let's explore RDD in brief by answering few questions. ➡️Along with the trivial definition, it is a read-only partitioned collection of records. 𝟏. 𝐖𝐡𝐲 𝐑𝐃𝐃 𝐢𝐧 𝐒𝐩𝐚𝐫𝐤 𝐰𝐚𝐬 𝐫𝐞𝐯𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐚𝐫𝐲? Ans: Keeping the trivial answers aside like 100x faster, in memory computation, none of the frameworks had abstraction like RDD which enabled users to reuse intermediate results across multiple computations using a rich set of operators. 𝟐. 𝐓𝐡𝐨𝐮𝐠𝐡 𝐭𝐡𝐞𝐫𝐞 𝐰𝐞𝐫𝐞 𝐨𝐭𝐡𝐞𝐫 𝐚𝐛𝐬𝐭𝐫𝐚𝐜𝐭𝐢𝐨𝐧𝐬 𝐨𝐟 𝐢𝐧 𝐦𝐞𝐦𝐨𝐫𝐲 𝐜𝐨𝐦𝐩𝐮𝐭𝐚𝐭𝐢𝐨𝐧 𝐨𝐧 𝐜𝐥𝐮𝐬𝐭𝐞𝐫, 𝐰𝐡𝐲 𝐑𝐃𝐃 𝐰𝐚𝐬 𝐬𝐩𝐞𝐜𝐢𝐚𝐥? Ans: Others allowed fine-grain updates to mutable states(eg- cells in Table) and in this to have fault tolerance data needs to be replicated or log updates across machines. This was expensive. But RDD allowed coarse grained transformations (eg- map, filter) that apply same operation to multiple data items. So, it can just log the transformation used to build the dataset(lineage). 𝟑. 𝐖𝐡𝐚𝐭 𝐚𝐫𝐞 𝐭𝐡𝐞 𝐚𝐝𝐯𝐚𝐧𝐭𝐚𝐠𝐞𝐬 𝐨𝐟 𝐑𝐃𝐃? Ans: · If a portion of data is lost, only the corresponding partition needs to be recovered through lineage. There is no overhead of checkpointing, recomputing the entire data or roll back the program. · It can mitigate slow running tasks by running backup copies of slow task in a healthier node. This is also known as speculative execution. · During bulk operations on RDD, a runtime can schedule tasks based on data locality to improve performance. · RDDs degrade gracefully(donot crash/corrupt) when there is not enough memory to store the partition as the data is spilled to disk. 𝟒. 𝐒𝐢𝐧𝐜𝐞 𝐥𝐢𝐧𝐞𝐚𝐠𝐞 𝐡𝐞𝐥𝐩𝐬 𝐢𝐧 𝐝𝐚𝐭𝐚 𝐫𝐞𝐜𝐨𝐯𝐞𝐫𝐲, 𝐢𝐬 𝐢𝐭 𝐚𝐥𝐰𝐚𝐲𝐬 𝐣𝐮𝐬𝐭𝐢𝐟𝐢𝐞𝐝 𝐭𝐨 𝐫𝐞𝐥𝐲 𝐨𝐧 𝐥𝐢𝐧𝐞𝐚𝐠𝐞 𝐢𝐧 𝐜𝐚𝐬𝐞 𝐨𝐟 𝐧𝐨𝐝𝐞 𝐟𝐚𝐢𝐥𝐮𝐫𝐞? Ans: Though lineage is good, but sometimes recovery of RDDs can be time consuming in case of long lineage chains. So, we should use checkpointing to store intermediate RDD to any stable storage(S3/Azure blob). Checkpointing is useful for RDD with long lineage graph containing wide dependencies. The reason being in case of node failure there is a possibility that some amount of data of each parent RDD is lost which will lead to full recomputation. 𝟓. 𝐇𝐨𝐰 𝐚𝐫𝐞 𝐭𝐡𝐞 𝐑𝐃𝐃𝐬 𝐫𝐞𝐩𝐫𝐞𝐬𝐞𝐧𝐭𝐞𝐝 𝐢𝐧𝐭𝐞𝐫𝐧𝐚𝐥𝐥𝐲? Ans: RDDs are represented through an interface which gives 5 piece of information – · A set of partitions which are atomic pieces of dataset · A set of dependencies on parent RDD · A function for computing the dataset based on its parent · Metadata about partitioning scheme · Metadata about Data placement Tagging DE Fraternity - Sumit Mittal Zach Wilson Ankur Ranjan Darshil Parmar Ajay Kadiyala Shashank Mishra 🇮🇳

32

1 Comment

Practical EDA Questions Every Data Analyst Should Know 👇 ✅ Loading Data: 📍 Question: Given a CSV file with sales data, how would you load it into a pandas DataFrame? 📍 Example Task: Load the "sales_data.csv" file and display the first 5 rows. ✅ Handling Missing Values: 📍 Question: How would you handle missing values in a dataset? 📍 Example Task: In the "customer_data.csv" file, fill missing values in the "age" column with the median age. ✅ Data Visualization: 📍 Question: Create visualizations to understand the distribution of a variable. 📍 Example Task: Plot a histogram and box plot for the "salary" column in the "employee_data.csv" file. ✅ Correlation Analysis: 📍 Question: How would you check for correlations between variables in a dataset? 📍 Example Task: Calculate and plot the correlation matrix for the "housing_data.csv" file. ✅ Identifying and Treating Outliers: 📍 Question: What techniques would you use to identify outliers in a dataset? 📍 Example Task: Identify and treat outliers in the "sales_amount" column of the "sales_data.csv" file. ✅ Data Cleaning: 📍 Describe the steps you take to clean a dataset. 📍 Example: Clean the "customer_data.csv" file by removing duplicates and handling missing values in the "email" column. ✅ Time Series Analysis: 📍 What steps would you take to analyze a time series dataset? 📍 Example: Decompose the "monthly_sales.csv" time series data into trend, seasonality, and residual components. 🚨 We have launched our most affordable, industry oriented and AI enabled "Data Analytics 3.0 with Azure, AI & Copilot" Bootcamp, ADMISSIONS ARE OPEN NOW👇👇👇 See you in the live classes 👉 Enroll Here : https://lnkd.in/dA-9werb 🚨 Special offer for Linkedin family, use code "DA300" 🚀 Live classes starting from 20-July-2024 with Live Doubt support 📲 Call/WhatsApp for any query +91 9893181542 Grow Data Skills Shashank Mishra 🇮🇳 SHAILJA MISHRA🟢 Shubhankit Sirvaiya Sahil Choudhary Aman Kumar

92

3 Comments

Join optimization in spark 1. Broadcast Join: - Suitable when one side of the datasets in the join is fairly small (typically less than 10 MB). - The smaller dataset is broadcasted to all nodes in the cluster where partitions of the larger dataset are present. - No additional data shuffling is required, resulting in faster joins. - Limitation: One table in the join must be small. 2. Sort Merge Join: - Standard join type suitable for medium to large datasets on both sides. - Happens in three stages: 1. Shuffle partitions: Reshuffles data so that rows with matching join keys are co-located physically (hash-based repartitioning). 2. Sorting within each partition: Sorts data based on the join key. 3. Joining the sorted partitions: Produces the final output based on the join type (INNER, LEFT, etc.). 3. Handling Data Skew: - Data skew can severely impact join performance. - Spark dynamically handles skew in sort-merge joins by splitting and replicating skewed tasks into evenly sized tasks. - Enable this feature by configuring spark.sql.adaptive.enabled and spark.sql.adaptive.skewJoin.enabled. 4. Repartitioning: - To avoid shuffling during join operations, consider reshuffling data based on the join key. - Repartitioning optimizes further joins if there are multiple joins in your workflow. Choice of optimization technique depends on your specific use case and dataset characteristics. Experiment with different approaches to find the best fit for your Spark jobs. #databricks #data #sql #dataengineering #deltalake #idempotency #datawarehousing #optimization #etl #spark

47