Dhilip Subramanian

PySpark is the Python API for Apache Spark. It enables you to perform real-time, large-scale data processing in a distributed environment using Python. It also provides a PySpark shell for interactively analyzing your data. PySpark combines Python’s learnability and ease of use with the power of Apache Spark to enable processing and analysis of data at any size for everyone familiar with Python. PySpark supports all of Spark’s features such as Spark SQL, DataFrames, Structured Streaming, Machine Learning (MLlib) and Spark Core. Overcoming Challenges: Connecting PySpark to a SQL Database Engine for Massive Data Processing In the world of big data, integrating various technologies seamlessly is often a daunting task. One of the significant challenges I faced was connecting from a PySpark session to a SQL database engine and efficiently distributing the memory and CPU cache for massive data processing. Here’s how I tackled this complex problem. 1.Establishing a reliable connection between PySpark and a SQL database engine. 2.Ensuring the efficient distribution of memory and CPU resources. 3.Handling massive datasets without compromising performance. Pyspark Scala

5

Snowflake Optimization Tip of the Day #5 ➡ Use incremental models in dbt There is no simpler way to reduce Snowflake cost than to process less data. If you are using dbt, try to write every model incrementally. Instead of reprocessing all data every time, incremental models only process new or changed data since the last run. This is achieved primarily by using date stamps or other markers to identify what data has been added or modified since the last run. Full-refreshes are very expensive and should be avoided if possible.

8

Snowflake Optimization Tip of the Day #7 ➡ Identify and remove unused Storage Storage is generally very cheap in Snowflake but unused tables can amount to a significant redundant cost. Snowflake provides the access_history view that parses each of your SQL queries to identify every access of a particular table. This table can be used to identify unused storage efficiently. The link to a technical blog to identify these is in the first comment. If you have a view on top of a table and you only access the view, it will be an indirect access for the table, Snowflake handles those parts as well to provide info that is actually relevant.

9

2 Comments

I have successfully completed the Databricks Lakehouse Fundamentals certification! The Databricks Lakehouse platform combines the best of data warehouses and data lakes, allowing seamless management of both structured and unstructured data. This course has deepened my understanding of unified data architectures, Delta Lake, and how to drive faster data insights and AI adoption. Looking forward to applying these concepts in future projects, pushing the boundaries of data engineering and analytics! 💡 #databricks #lakehouse #dataengineering #bigdata #cloudcomputing #deltaLake #AI #ML

6

1 Comment

Wouldn't it be nice if relational databases (such as ANSI SQL) updated their syntax to make using the MERGE statement easier? Currently, if I wanted to execute a MERGE statement where I update all columns and insert all data columns from the source table into the target (like SQL Server), I would need to list out the individual columns. However, this process is simplified in Spark SQL with the "Insert *" or "Update set *" syntax. The only alternative we have in a relational database is to create dynamic SQL, which can be painfully complex. Is there an easier way in Relational databases to do this without using Dynamic SQL? #dataengineer #dataengineering #azure

4

Snowflake Optimization Tip of the Day #1 ➡ Apply pruning The golden rule for reducing Snowflake cost is to process the least amount of data (duh!). Understanding and applying the concept of partition pruning is the most important aspect of reducing Snowflake cost. Snowflake stores data in micro partitions and every micro partition has metadata associated with it like maximum and minimum values of a column. For example, let’s say you are querying event data. Applying a filter on the event date = ‘2024-04-22’ would make sure that a significantly large amount of data is never processed because today’s date will not be within the minimum and the maximum of almost 99% of the data you have. This filter not only makes sure that the costs are lower but you'll also get the results quicker. In the Snowflake world, money = time. If you can find a way to reduce the time for a query, you save money. Partition pruning is a win-win in both aspects.

18

Advanced SQL 50 Question 07 (தமிழில்): Sellers With No Sales Full Video: https://lnkd.in/gnAYpMbF Video 302: This is the 7th video of the SQL Interview Question series. We are given three tables Customer, Orders and Seller. The Customer table has customer id and their name  Seller able has seller_id and their names  The Orders table has the Order details with order_id as unique identifier. It has the sale date, the cost, customer id and seller id If you want to check who are the customers and what they bought, use the customer id and connect the tables Also, we can check the sellers and how much they sold, u can connect using the seller id Now, they ask us to find the names of all sellers who did not make any sales in 2024. In this video, we'll work through four different approaches to solving this problem, highlighting the advantages and limitations of each. *** Basic LEFT JOIN with Date Comparison ***   Learn how to use a simple LEFT JOIN to identify sellers who have no sales records within a specified date range. *** LEFT JOIN with YEAR Function ***   This method leverages the YEAR function to filter sales records by year, making the query more readable and intuitive. *** LEFT JOIN with LEFT Function for Year Extraction ***   Discover how to use string manipulation to extract the year from date strings, providing a flexible way to handle date comparisons. *** Using DISTINCT to Ensure Unique Seller Names ***   Combine the power of DISTINCT with our previous techniques to ensure our results are accurate and contain no duplicates. For a comprehensive understanding of these SQL methodologies and their application, please refer to this explanatory video. code: https://lnkd.in/gf3Tjsxe Follow me on, Website : https://growwithdata.co/ YouTube : https://lnkd.in/gSrqE3Nj TikTok : https://lnkd.in/gqFiMex2 LinkedIn : https://lnkd.in/gdfNNX9a Facebook : https://lnkd.in/gU6JWpFH FB Group : https://lnkd.in/gG4ENcrP twitter : https://lnkd.in/geZ3dqi8 Instagram : https://lnkd.in/gn263_PZ WhatsApp : https://lnkd.in/gBa5eC7y TheWide : https://lnkd.in/gt7htz_8

1

Great information on Databricks #AI #sql functions! #data #databricks #ml #sql #dataanalysts #dataanalytics #bigdata #technology #ai

2

In this article, we are going to focus on, 1. What are materialized views? 2. Requirements 3. Create a materialized view 4. Load data from external systems 5. Refresh a materialized view 6. How are Databricks SQL materialized views refreshed? 7. Schedule materialized view refreshes 8. View the status of a materialized view refresh 9. View the refresh status in the Delta Live Tables UI 10. Stop an active refresh 11. Update the definition of a materialized view 12. Drop a materialized view 13. Describe a materialized view 14. Change the owner of a materialized view 15. Control access to materialized views 16. Enable change data feed 17. View the refresh history for a materialized view 18. Determining if an incremental or full refresh is used 19. Limitations https://lnkd.in/dEFyjNnx

2

Curious about a career in data? In the part 2 of the latest 'Decision Point’s Coffee Table', we sit down with Umesh Kumar, Data Engineering Manager at Decision Point Analytics, to explore the foundational skills every aspiring data professional needs.   Umesh highlights the essential knowledge and tools that help newcomers make a strong start in data. Let us know in the comments: What do you think is the most important skill for beginners?   #datascientist #dataengineering #datamanagement #dataanalytics #decisionpoint #coffeetable #podcast

56

🏠 Data Analysis Tip: Using Split Flatten Function in Snowflake SQL Hey LinkedIn community! Today, let's dive into a practical example of how to leverage Snowflake SQL's split-flatten function to analyze Airbnb search preferences. 📊 Example: Suppose we have a table airbnb_searches capturing user searches with room type filters: -- Define table structure create table airbnb_searches ( user_id int, date_searched date, filter_room_types varchar(200) ); -- Clear existing data for cleanliness delete from airbnb_searches; -- Insert sample data insert into airbnb_searches values (1, '2022-01-01', 'entire home,private room'), (2, '2022-01-02', 'entire home,shared room'), (3, '2022-01-02', 'private room,shared room'), (4, '2022-01-03', 'private room'); To analyze the distribution of room types searched across users, we can use Snowflake's split flatten function: -- Use a Common Table Expression (CTE) with LATERAL FLATTEN to split and count room types with cte as ( select user_id, date_searched, C.value::string AS room_type from airbnb_searches, lateral flatten(input => split(filter_room_types, ',')) AS C ) select room_type, count(1) as counts from cte group by room_type order by counts desc; This script efficiently splits the filter_room_types column by comma, allowing us to count occurrences of each room type across searches. This is crucial for understanding user preferences and optimizing listings. 🔑 Key Insight: Snowflake's split flatten function simplifies complex data analysis tasks, enhancing decision-making capabilities in real-time analytics. What are your go-to methods for analyzing user preferences in your data projects? Share your thoughts below! ⬇️ #SnowflakeSQL #DataAnalysis #TechTips #LinkedInLearning

3

FREE DEMO CLASS Comment down if you also want to get get free demo class on Data Science and Data Analytics Learn From the industry expert

5

Begin an immersive journey of exploration with us as we delve into the nuances of crafting drill-down bar charts in Tableau. In this blog, consultant Calvin Gao guides us through the process, unraveling the details behind this powerful visualization tool. ➡️Read here to enhance your Tableau skills and take your data analysis to the next level: https://ow.ly/JAJB50RT0mX

7

Use Cases and Best Practices Data quality is the backbone of reliable analytics, especially in big data environments. I recently explored methods to manage null values and outliers, which can significantly impact results if not addressed. Using PySpark’s.fillna() and .dropna() functions has been useful for scalable transformations. What are some of your go-to strategies for ensuring high data quality? #dataengineering #seniormanagers #dataarchitect #bigdata #databricks #dataquality #datacleaning #dataprofiling #datavalidation

5

Fresh from the press! The new Enterprise Big Data Engineer (EBDE) syllabus is now available for download. You can download the syllabus here: https://lnkd.in/g7Z3pr_Y #dataengineer #dataengineering #ebde #bigdataframework #certification #apmg #credentials #digitalbadge #knowledge

12

Fast-track your career as a data and analytics professional. Join our webinar themed 'Roadmap for Rapid Data Career Skill Acquisition' and discover the essential steps to mastering data skills quickly and efficiently. The session will cover key areas such as identifying the most in-demand data skills, effective learning techniques, and practical tips for overcoming common challenges faced by beginners in the field. Gain practical insights to boost your confidence and excel in the world of data science and analytics. Don't miss out on this opportunity to accelerate your career growth! Join here>>>>https://lnkd.in/ejNBRbmZ #CareerGrowth #ProfessionalDevelopment #DataScienceTips #DataSkills

5