Jashan Preet Kaur

Understading Resource Allocation in Spark!! 🔍 One of the key components of Spark job execution is Resource Allocation. When a Spark job is submitted, Spark needs to efficiently use the available cluster resources to ensure the job runs as smoothly and quickly as possible. Here’s a breakdown of how Spark handles resource allocation: ➡️ Role of Cluster Manager: Spark relies on a Cluster Manager, such as YARN, Kubernetes, or its standalone mode, to handle resource allocation across the cluster. The Cluster Manager is responsible for understanding what resources are available and ensuring Spark’s needs are met. ➡️ Resource Request for Executors: The Driver program communicates with the Cluster Manager to request resources for Executors. Executors are the distributed processes that perform tasks on different parts of the dataset. By distributing tasks across nodes, Spark leverages parallel processing. ➡️ Dynamic Allocation: Depending on the workload, Spark can dynamically adjust the number of Executors. This flexibility ensures Spark can handle a range of jobs, from small data processing tasks to large-scale analytics. ➡️ Efficiency and Scalability: By coordinating with the Cluster Manager and spinning up Executors across nodes, Spark ensures optimal use of resources, reducing idle time and speeding up job completion. Resource allocation is the backbone of Spark’s ability to scale and process massive datasets efficiently, making it essential for handling distributed Big Data workloads. ♻️ Found this helpful? Repost to support others on their data journey! 📲 Need personalized guidance? Let’s connect 👉 https://lnkd.in/gh42-Wi3 📬 Subscribe to my newsletter for top data engineering content! 👉 https://lnkd.in/guDCNPN5 👋🏼 Sharing daily insights on data engineering; follow for more! #softwareengineering #dataengineering #bigdata #database #performance #distributedsystem #parallelcomputing #spark #python #datastructure #interview #hadoop #hdfs #yarn #kubernetes #docker Zach Morris Sumit Mittal Databricks Snowflake Apache Spark

24

Spark Interview Question!! Interviewer: "Can you explain how Spark creates a DAG for an application?" Candidate: "When a Spark application is submitted, the Driver node creates a DAG by breaking down the application into smaller tasks, like map, filter, and reduce operations." Interviewer: "Great! And how are dependencies between tasks handled in the DAG?" Candidate: "The DAG represents the dependencies between tasks as edges, ensuring that tasks are executed in the correct order. Each task is a node in the graph, and the edges represent the input/output relationships between tasks." Interviewer: "Excellent! How does Spark optimize the DAG for better performance?" Candidate: "Spark uses various optimizations like pipelining, parallelization, and caching to improve performance. It also applies graph algorithms to reduce the number of tasks and stages in the DAG." Interviewer: "Well done! Can you walk me through an example of a DAG for a simple Spark application, like a word count?" Candidate: "Sure, the DAG would consist of nodes for reading input files, mapping words to counts, reducing counts, and writing output results. The edges would represent the flow of data between these tasks." Interviewer: "Thank you for explaining the DAG concept so clearly!"

3

Data Engineer Important Interview Question!! Interviewer: Let's start by discussing Spark data quality checks. Could you explain what data quality checks are and why they are important in Spark applications? Candidate: Certainly. Data quality checks are essential processes in Spark applications to ensure that the data being processed meets certain standards or expectations in terms of accuracy, completeness, consistency, and validity. These checks are crucial because they help maintain the integrity of the data pipeline and ensure that downstream analyses or processes are based on reliable data. Interviewer: That's a solid overview. What are some common types of data quality checks that can be performed in Spark? Candidate: There are several types of data quality checks that can be performed in Spark, including: Schema Validation: Verifying that the data adheres to the expected schema, including data types and structure. Completeness Check: Ensuring that all required fields are present and populated in the dataset. Uniqueness Check: Verifying that there are no duplicate records in the dataset based on certain key columns. Accuracy Check: Validating the accuracy of certain fields or metrics against predefined rules or benchmarks. Consistency Check: Ensuring that the data remains consistent across different sources or time periods. Referential Integrity Check: Verifying that foreign key relationships between different tables or datasets are maintained. Interviewer: Great breakdown. How would you implement these data quality checks in a Spark application? Candidate: Implementing these checks typically involves writing custom validation logic using Spark DataFrame operations, SQL queries, or user-defined functions (UDFs). For example, to perform a completeness check, you can count the number of null values in each column and compare it against the expected count. Similarly, for uniqueness checks, you can use Spark's dropDuplicates function and compare the counts before and after to identify any duplicates. Interviewer: Excellent. Now, can you discuss some strategies for handling failed data quality checks in Spark applications? Candidate: Sure. When a data quality check fails, it's important to have proper error handling mechanisms in place. One approach is to log the details of the failed checks along with any relevant metadata, such as timestamps or record identifiers. Depending on the severity of the issue, you can choose to either abort the entire job or skip the problematic records and continue processing the rest of the data. Additionally, you can set up alerts or notifications to notify stakeholders or data engineers about the failures so that they can investigate and take appropriate actions. Interviewer: Thank you for your insights. #bigdata #dataengineer

23

Hello Everyone, As part of SQL interview, today I'm sharing the two questions along with answers. 1. Write a SQL query to find employees who are not present in the department table Query : Select emp.*, dept.dep_id, dept.dep_name from employee emp Left Join dept on emp.dept_id = dept.dep_id where dept.dep_id is Null; 2. Write a SQL query to find the highest and lowest salary employee in each department Query : with cte as( select *, rank() over(partition by dept_id order by salary desc) as rank_desc, rank() over(partition by dept_id order by salary asc) as rank_asc from employee emp) select dept_id, max(case when rank_desc = 1 then emp_name end) as max_sal_emp ,min(case when rank_asc = 1 then emp_name end )as min_sal_emp from cte group by dept_id #sql #dataanalytics #bi

112

8 Comments

A person mentioned an open position of "Data analyst" or "Power BI Developer" at his organization which matches your profile. He provided proper Job ID and genuine application link for this particular position. He also mentioned " DM me for referral" Till the time you saw the post, it already had 100s of likes and comments. Now here's something that you should consider before applying : ➡️ Imagine if the post already have 100s of comments, many people might have already reached out to him. He might even miss out some applications due to huge number of DMs. ➡️ As you already have the Job ID, rather than reaching out to the same person who posted, try to connect with any other person who's working in the same organization at some good position. ➡️ This will help you stand out of the crowd and even you will get the referral easily as you reached out to a person hardly anyone else reached. Always play smart 😊 #dataanalytics #powerbi #powerbideveloper #data #sql

394

21 Comments

PySpark Interview Question: q> Difference between SparkContext and GlueContext? Environment: Spark Context (SparkContext) is used in standalone Apache Spark environments or Spark clusters managed by tools like Apache Mesos or Apache YARN. Glue Context (GlueContext) is specific to AWS Glue, a fully managed ETL service in the AWS ecosystem. Integration: Spark Context  Provide the core functionality for distributing computation across clusters, managing resources and interacting with storage systems. Glue context extends Spark context and provides additional capabilities tailored for Glue ETL jobs, including integration with Glue services and features. Metadata Store: Spark Context does not have built-in support for managing metadata or data catalogs. Glue Context integrates with the Glue Data Catalog, serving as a central metadata store for managing and cataloging data assets. Q> Difference between SparkContext and SparkSession? Purpose: SparkSession: Entry point for reading data, executing SQL queries and  accessing dataframe API.It represents a unified interface to interact with Spark. SparkContext: Provide the core functionality for distributing computation across clusters, managing resources and interacting with storage systems.  Usage: SparkSession: Used for creating DataFrames, executing SQL queries, and accessing Spark's higher-level APIs like Dataset and DataFrame. SparkContext: Used for creating RDDs (Resilient Distributed Datasets) and low-level control over Spark jobs, such as setting configurations and managing job execution. Initialization: SparkSession: Typically created automatically when starting a Spark application. It encapsulates the SparkContext, SQLContext, and HiveContext. SparkContext: Must be explicitly created when starting a Spark application. It is the entry point for all Spark functionality. Happy Learning Folks ! #PySpark #DataEngineering #BigData #Spark

2

Job switch tricks for target 2025 🏙️ ⛷️ Update your job profile every day for better visibility. ⛷️ Apply for the maximum relevant job openings every day. ⛷️ Add specific keywords related to the job (e.g., PySpark, ADF, Databricks, Snowflake etc.) to your resume. ⛷️ Check job descriptions to find the responsibilities and requirements. ⛷️ Reach out to HR directly via DMs or emails. ⛷️ Use multiple job portals and platforms like Naukri, Indeed, Monster, LinkedIn and referrals or friends networks. ⛷️ Create multiple job profiles based on your location preference. ⛷️ Replace career gaps with freelance work on your resume. The easiest way to increase your chances is to try more! follow Asheesh .. for more updates ✨ #jobs #career

65

7 Comments

A SQL question from N H's 100-day SQL challenge. I have solved it using the ROW_NUMBER. Comment down your approach. PROBLEM STATEMENT -> Write a query that displays names alternately by gender and sorted by ascending order of column ID. SCRIPT -> CREATE TABLE students (   ID INT PRIMARY KEY,   Name VARCHAR(50),   Gender CHAR(1)); INSERT INTO students (ID, Name, Gender) VALUES (1, 'Gopal', 'M'), (2, 'Rohit', 'M'), (3, 'Amit', 'M'), (4, 'Suraj', 'M'), (5, 'Ganesh', 'M'), (6, 'Neha', 'F'), (7, 'Isha', 'F'), (8, 'Geeta', 'F'), (9, 'Sunita', 'F'); Posting my solution below. Tagging the Data Fraternity for better reach. Shashank Singh 🇮🇳 Codebasics Dhaval Patel Alex Freberg Raghavan P NAVEEN S Vaibhav Lambat Hemanand Vadivel Vishwas Kshirsagar AMAN KUMAR Nick Singh 📕🐒 Thodupunuri Bharath Ankit Bansal Sagar Prajapati Deepak K Deepak Goyal Himanshu Bansal Shashank Mishra 🇮🇳 Shubham Suryawanshi

49

9 Comments

𝑬𝑿𝑳 𝑰𝒏𝒕𝒆𝒓𝒗𝒊𝒆𝒘 𝑸𝒖𝒆𝒔𝒕𝒊𝒐𝒏𝒔. Q1: SQL List the films together with the leading star for all 1962 films. Link: https://lnkd.in/dUQJev7t Q2: Pyspark Is there a significant relationship between the age of a customer and their primary spending category? To explore this, we have three tables: Customer Table: Fields: cust_id, cust_age, cust_name, cust_income Credit Card Transactions Table: Fields: transaction_id (Primary Key), cust_id, date, mcc (Merchant Category Code - 4 digit code), amount Merchant Table: Fields: mcc, mcc_desc (Merchant Category Description) We aim to determine whether there is a correlation between customer age and the merchant categories where they predominantly spend. Used Data: from pyspark.sql import Row # Sample data for the Customer table customer_data = [     Row(cust_id=1, cust_age=25, cust_name='Alice', cust_income=50000),     Row(cust_id=2, cust_age=30, cust_name='Bob', cust_income=60000),     Row(cust_id=3, cust_age=45, cust_name='Charlie', cust_income=70000),     Row(cust_id=4, cust_age=30, cust_name='David', cust_income=80000),     Row(cust_id=5, cust_age=55, cust_name='Eve', cust_income=90000) ] # Sample data for the Credit Card Transactions table transactions_data = [     Row(transaction_id=1, cust_id=1, date='2024-01-01', mcc=4295, amount=150),     Row(transaction_id=2, cust_id=1, date='2024-01-05', mcc=6348, amount=200),     Row(transaction_id=3, cust_id=2, date='2024-01-02', mcc=4295, amount=300),     Row(transaction_id=4, cust_id=2, date='2024-01-06', mcc=4295, amount=100),     Row(transaction_id=5, cust_id=3, date='2024-01-03', mcc=6348, amount=250),     Row(transaction_id=6, cust_id=4, date='2024-01-04', mcc=6348, amount=300),     Row(transaction_id=7, cust_id=4, date='2024-01-07', mcc=4295, amount=400),     Row(transaction_id=8, cust_id=5, date='2024-01-08', mcc=6348, amount=500) ] # Sample data for the Merchant table merchant_data = [     Row(mcc=4295, mcc_desc='airline'),     Row(mcc=6348, mcc_desc='hotel') ] # Create DataFrames from sample data customer_df = spark.createDataFrame(customer_data) transactions_df = spark.createDataFrame(transactions_data) merchant_df = spark.createDataFrame(merchant_data) Link to the video: https://lnkd.in/dCAGcA_2 For more such posts follow Priyam Jain #exl #youtube #DataEngineeringInterview #ApacheSpark #InterviewPrep #DataEngineering #TechInterviewTips #data #dataengineering #dataengineerjobs #interviewquestions #interviewprep #newlearnings #interviewpreparation #interviewtips #dataengineers #interviewsuccess #followme #ETL #dataengineerjobs #spark #azurecloud #datacleaning #pyspark #optimization #interviewsuccess #interviewskills  #DataEngineering #DataScience #LinkedInLearning #DataProcessing #BigData #InterviewPrep #TechCareer #DataJobs

30

2 Comments

🚀 Just completed an insightful webinar by Databricks on Get Started with Data Analysis on Databricks. Key Takeaways 📝: 1. Explored how Databricks SQL integrates within the lake house architecture. 2. Learned about the robust data security implementations within the platform. 3. Ingesting data into the Databricks platform seamlessly. 4. Utilizing SQL commands that are specific to Databricks. 5. Querying data efficiently and effectively. 6. Creating dynamic visualizations and dashboards. 7. Sharing queries and dashboards with team members. 8. Leveraging automation and integration capabilities to enhance workflow. A big thank you to Luca Gennari for the great session! Your insights were invaluable. Excited to apply these learnings and take my data analysis skills to the next level! 💡💻 #Databricks #DataAnalysis #SQL #Lakehouse #DataSecurity #DataVisualization #Automation #Webinar

69

1 Comment

Hey Guys, today we will explore about ,how Efficiently Truncate Columns in Snowflake. Truncating data in specific columns without affecting the entire table can be a crucial part of maintaining data hygiene. Here’s how you can efficiently truncate (i.e., clear) specific columns in a Snowflake table: 1.Update Columns to Null or Default Value: Use the UPDATE statement to set the column values to NULL or a default value. This approach maintains the table structure and integrity. UPDATE my_table SET column1 = NULL,   column2 = NULL WHERE TRUE; 2.Alternative Method Using Default Values: If you have default values for the columns, you can reset them to these defaults. UPDATE my_table SET column1 = DEFAULT,   column2 = DEFAULT WHERE TRUE; Why Use This Method? Maintains Data Integrity: Only the specified columns are cleared, preserving other data in the table. Flexibility: You can choose to set columns to NULL or any default value, based on your data management needs. #Snowflake #DataEngineering #DataManagement #SQL #DataHygiene

3

You are just 10 videos away to clear Tableau desktop certificate 😁🚀 I am not saying this, this is what is got as feedback of Playlist that I uploaded on my YouTube channel. 🏆 My certification series is included by Tableau in his weekly hashtag #datafam round. 🌐Till now 25 candidates has cleared the certification with more than 90% of marks. Planning to take the Tableau Desktop Certification? 💡 Then why you need to spend money to purchase practice test when I am providing best practice questions for free. Check out my playlist for in-depth syllabus coverage, domain-specific questions, and tips to conquer the exam. Let's excel together! link of the playlist is in the comment box For more updates follow Kapil Dhiman #TableauCertification #tableau #AllAboutDATA #DataAnalytics #CareerGrowth

48

3 Comments

#collected Netflix Analytics Engineer Interview Experience: CTC - 35 LPA SQL Questions: 1️⃣ SQL Question 1: Identify VIP Users for Netflix Question: To better cater to its most dedicated users, Netflix would like to identify its “VIP users” - those who are most active in terms of the number of hours of content they watch. Write a SQL query that will retrieve the top 10 users with the most watched hours in the last month. Tables: • users table: user_id (integer), sign_up_date (date), subscription_type (text) • watching_activity table: activity_id (integer), user_id (integer), date_time (timestamp), show_id (integer), hours_watched (float) 2️⃣ SQL Question 2: Analyzing Ratings For Netflix Shows Question: Given a table of user ratings for Netflix shows, calculate the average rating for each show within a given month. Assume that there is a column for user_id, show_id, rating (out of 5 stars), and date of review. Order the results by month and then by average rating (descending order). Tables: • show_reviews table: review_id (integer), user_id (integer), review_date (timestamp), show_id (integer), stars (integer) 3️⃣ SQL Question 3: What does EXCEPT / MINUS SQL commands do? Question: Explain the purpose and usage of the EXCEPT (or MINUS in some SQL dialects) SQL commands. 4️⃣ SQL Question 4: Filter Netflix Users Based on Viewing History and Subscription Status Question: You are given a database of Netflix’s user viewing history and their current subscription status. Write a SQL query to find all active customers who watched more than 10 episodes of a show called “Stranger Things” in the last 30 days. Tables: • users table: user_id (integer), active (boolean) • viewing_history table: user_id (integer), show_id (integer), episode_id (integer), watch_date (date) • shows table: show_id (integer), show_name (text) 5️⃣ SQL Question 5: What does it mean to denormalize a database? Question: Explain the concept and implications of denormalizing a database. 6️⃣ SQL Question 6: Filter and Match Customer’s Viewing Records Question: As a data analyst at Netflix, you are asked to analyze the customer’s viewing records. You confirmed that Netflix is especially interested in customers who have been continuously watching a particular genre - ‘Documentary’ over the last month. The task is to find the name and email of those customers who have viewed more than five ‘Documentary’ movies within the last month. ‘Documentary’ could be a part of a broader genre category in the genre field (for example, ‘Documentary, History’). Therefore, the matching pattern could occur anywhere within the string. Tables: • movies table: movie_id (integer), title (text), genre (text), release_year (integer) • customer table: user_id (integer), name (text), email (text), last_movie_watched (integer), date_watched (date)

8

2 Comments

🔍💻 SQL Question Series: Day 19 💻🔍   Write a solution to find each query_name, the quality and poor_query_percentage. Both quality and poor_query_percentage should be rounded to 2 decimal places. We define query quality as: The average of the ratio between query rating and its position. We also define poor query percentage as: The percentage of all queries with rating less than 3. #Explanation:  Dog queries quality is ((5 / 1) + (5 / 2) + (1 / 200)) / 3 = 2.50 Dog queries poor_ query_percentage is (1 / 3) * 100 = 33.33 Cat queries quality equals ((2 / 5) + (3 / 3) + (4 / 7)) / 3 = 0.66 Cat queries poor_ query_percentage is (1 / 3) * 100 = 33.33 Happy querying! 💬💡 #SQL #DataAnalysis #TechSkills #LearningAndDevelopment #SQLQuestionSeries #leetcode

2

LTIMindtree 𝗜𝗻𝘁𝗲𝗿𝘃𝗶𝗲𝘄 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 𝗳𝗼𝗿 𝗔𝘇𝘂𝗿𝗲 𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿: 𝗗𝗶𝗳𝗳𝗶𝗰𝘂𝗹𝘁𝘆 𝗟𝗲𝘃𝗲𝗹: 𝗠𝗲𝗱𝗶𝘂𝗺 1. How to schedule notebooks in databricks 2. Different types of file formats 3. How to use broadcast join if we don't know the size of tables 4. How to handle complex use cases in databricks 5. How to read multiple worksheets in a single excel and convert in csv in pyspark 6. List Comprehension and decorators in python 7. Different cluster types in databricks 8. How to move data from bronze to silver layer in databricks Happy Learning . . . . 📖  ------------------------------------- 📌 Follow for more Karisma Kumari #data #dataEngineering #dataengineer #spark #Databricks #ADF #Pyspark #AdvancedSQL #interviewpreparation #interview

45

1 Comment

Acies Global Interview Question Problem Statement : We have a swipe table which keeps track of employee login and logout timings. 1. Find out the time employee person spent in office on a particular day (Office hours = Last Logout Time - First Login Time) 2. Find out how productive he was at office on particular day.  (He might have done many swipes per day. I need to find the actual time spend at office) Scripts : CREATE TABLE swipe (  employee_id INT,  activity_type VARCHAR(10),  activity_time datetime ); -- Insert sample data INSERT INTO swipe (employee_id, activity_type, activity_time) VALUES (1, 'login', '2024-07-23 08:00:00'), (1, 'logout', '2024-07-23 12:00:00'), (1, 'login', '2024-07-23 13:00:00'), (1, 'logout', '2024-07-23 17:00:00'), (2, 'login', '2024-07-23 09:00:00'), (2, 'logout', '2024-07-23 11:00:00'), (2, 'login', '2024-07-23 12:00:00'), (2, 'logout', '2024-07-23 15:00:00'), (1, 'login', '2024-07-24 08:30:00'), (1, 'logout', '2024-07-24 12:30:00'), (2, 'login', '2024-07-24 09:30:00'), (2, 'logout', '2024-07-24 10:30:00'); WITH cte AS (   SELECT *,       CAST(activity_time AS DATE) AS activity_day,       LEAD(activity_time, 1) OVER (PARTITION BY employee_id, CAST(activity_time AS DATE) ORDER BY activity_time) AS logout_time   FROM swipe ) SELECT employee_id,     activity_day,     DATEDIFF(HOUR, MIN(activity_time), MAX(logout_time)) AS total_hours,     SUM(DATEDIFF(HOUR, activity_time, logout_time)) AS inside_hours FROM cte WHERE activity_type = 'login' GROUP BY employee_id, activity_day;

2