Manikandan Sundarapandian

🎉 Milestone Alert: 8 Years of Professional Growth! 🎉 Today, I’m proud to celebrate 8 years of an incredible journey that has shaped my career in ways I couldn’t have imagined. Here's a snapshot of my path: - 1.5 years in EdTech 📚 - 3.5 years in Analytics Consulting 📊 - 3 years in Supply Chain Analytics 🚚 Throughout these years, I’ve navigated through various stages of analytics maturity within organizations, contributing to and learning from each experience. Here’s what I’ve been up to: ✅ Creating impactful Data Science Proof of Concepts (POCs) 🚀 Developing Recommendation Systems 💬 Diving deep into Natural Language Processing (NLP) 🏭 Solving complex Analytical Problems 🚚 Currently shaping the future in Supply Chain Analytics This journey has been filled with immense learning, challenges, and opportunities for growth. I’m excited about what the future holds and am eager to continue making a difference in the world of analytics. Here’s to the next chapter! 🎯 #CareerMilestone #Analytics #DataScience #SupplyChain #ProfessionalGrowth

62

5 Comments

Can we hypertune train and test dataset both? In production environment we should split our data into training and test datset.Further training dataset split into training and validation. using greadsearchCV or randomizedsearchCV using k-fold cross validation all sets of dat are validated and tuning through hyperparameters. Once find perfect model with best_score_ then that model is implementing in test data sets. Test data sets are treated as production data. So no tuning for this data set. If any hyperparameters tuning is done with test dataset then data leakage will come into pictures. So to avoid it tuning should be done after splitting train and test. I put my thought here. It is always welcome to put your comments here. So that topic will be enriched with different ideas.

1

🌟 Actively Seeking New Job Opportunities! 🌟 Highly experienced Sr. Azure Data Engineer with 15+ years of expertise in designing and implementing scalable cloud data solutions, specializing in Azure, Azure Databricks, PySpark, and Python. Proficient in developing end-to-end ETL pipelines, data migration, and advanced analytics solutions, I have successfully led and executed cloud transformations across multi-cloud environments, including GCP and AWS. 🔑 Key Skills and Expertise: Azure Cloud Expertise: Extensive experience in building data solutions on Azure Synapse Analytics, Azure SQL Database, and Azure Data Factory (ADF) for large-scale data ingestion and transformation. Skilled in implementing Azure Databricks for high-performance data processing using PySpark and Python to manage complex business logic and deliver scalable data transformation pipelines. Advanced Analytics and Machine Learning: Developed and deployed machine learning models using Azure ML Studio and integrated with Databricks for advanced data analysis. Hands-on experience with MLlib and BERT models for predictive analytics, customer segmentation, and content recommendations. GCP Expertise: Led the migration of large datasets from Snowflake to Google Cloud Platform (GCP), leveraging BigQuery for efficient data warehousing, Google Cloud Storage (GCS) for scalable storage, and Google Cloud Composer for orchestrating data workflows. Implemented delta load and incremental update mechanisms using Cloud Dataflow and BigQuery MERGE operations. Data Governance and Security: Designed Data Clean Room (DCR) solutions on GCP and Azure, ensuring compliance with HIPAA and GDPR by implementing Google Cloud DLP, column-level encryption, and secure access controls using Azure Active Directory (AAD) and BigQuery Authorized Views. Cross-Cloud Integration: Expert in integrating Azure with GCP and AWS, developing hybrid cloud architectures, and utilizing Azure Data Lake Storage (ADLS Gen2), AWS S3, and GCS for seamless data migration and storage management. I am passionate about leveraging cloud technologies to drive business value through optimized data workflows, cutting-edge analytics, and secure, scalable architectures. Let’s connect to discuss how I can help your organization achieve its cloud data engineering goals. 📩 Feel free to reach out at [email protected] or via LinkedIn for new opportunities. #Azure #Databricks #PySpark #BigData #DataEngineering #GCP #MachineLearning #CloudComputing #AzureML #BigQuery #DataMigration

12

2 Comments

🎉 Excited to share that I am now a Databricks Certified Data Engineer Professional! This certification has sharpened my skills in building scalable data pipelines, optimizing Spark workflows, and implementing advanced data transformations. Looking forward to applying these skills to drive impactful data solutions! 🚀 #DataEngineering #Databricks #Spark #BigData

1

Do you have JSON and XML tough to parse and read up? Variant data types. I have seen less people talk about it. Let's do an overview of it and check it out. Here are my thoughts on Databricks Variant data types. https://lnkd.in/g9Qrm2rb #DataBricks #Variant #FileTypes #Simplification #DataArchitecture #Beginners #StartHere #PandaLearns

26

DATA ENGINEERING (DE) FOR DOWNSTREAM USECASES 🙂 Basically, what I did was feed FEW RAW JSON FILES Residing in AWS S3 to SnowFlake & Perform ELT VIA DBT (Data Build Tool) in VS Code where I made use of JINJA Script to make Transformations on the Data Residing in SnowFlake in a series of steps ensuring Modularity of CODE abiding by Medallion Architecture- Schematic Data for Downstream Users (BI / AI / ML / DS) By me DE! 😎

#Mastheadata #Google #BigQueryCostOptimizationGuide This is a compact but details about #Bigquery from functional to the technical under the hood. The key take ways from #DataEngineering perspective are as follows. * De coupling of Compute and Storage of Big Query * Different storage layers - Logical Storage layer - Physical Storage Layer * Different cost optimization patterns for Big Query compute and storage. * Big Query Job Execution Graph, best way to visualize the data flow from source to output. * Best practices for adhoc usage of big query ( which is a big part where we can save cost) - Use of partitions where appropriate - Use of Clustering where appropriate - Effective use of Temporary Table - Sampling, Limit and Preview options to know and explore the data before deep diving into it. Preview is a free option where’s sampling gives efficient scanning than limit.

5

An interesting observation I had while dealing with clustering in Snowflake. Remember the SYSTEM$CLUSTERING_DEPTH function that returns the average depth of overlapping micro-partitions? I observed that right after altering a table to apply cluster, if I continuously run the above function, the average depth returned keeps off falling over a period of a minute or so until it stabilizes to a least possible (most optimal clustering state). In the chart below, the value at Timestamp 1 is 79.2 which falls down to 1.2 finally. This indicates the following : 1. Clustering is a resource intensive process where micro-partitions are shuffled 2. Clustering takes time 3. Clustering / Partition pruning consumes credit 4. Clustering brings marked improvement in query performance Looking for comments from experts Somen Swain Sachin Mittal Lav Pandey Vishal Kaushal if my observation is correct. Any improvement suggestion is most welcome. Snowflake

38

7 Comments

🤔 Your query won't run faster if you start using CTE's instead of subqueries. CTE's are not temp tables, neither they hold the data in memory at runtime for later processing. It's actually very likely that you will get the same query plan if you use subqueries. But... The key pieces to understand if you write SQL using CTEs are: 🔑Readability 🔑Maintainability You'll be able to breakdown the logic into smaller pieces, this will allow better understanding of the code, and promote reusability since you can reference CTE's multiple times in your query without being redundant. #sql #database #dataengineer #dataengineering #analytics #dataanalyst

1

𝐃𝐚𝐭𝐚 𝐢𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧 𝐬𝐭𝐫𝐚𝐭𝐞𝐠𝐢𝐞𝐬: ETL, Reverse ETL, and ELT 𝐄𝐓𝐋, 𝐑𝐞𝐯𝐞𝐫𝐬𝐞 𝐄𝐓𝐋, 𝐚𝐧𝐝 𝐄𝐋𝐓 are three different approaches to data integration. - 𝐄𝐓𝐋 stands for Extract, Transform, and Load. In ETL, data is extracted from different sources, transformed into a consistent format, and then loaded into a data warehouse. - 𝐑𝐞𝐯𝐞𝐫𝐬𝐞 𝐄𝐓𝐋 flips the ETL process around. Data is extracted from the data warehouse and loaded into operational systems, such as CRM or marketing automation systems. This allows businesses to use their data to improve their operations and customer experiences. - 𝐄𝐋𝐓 stands for Extract, Load, and Transform. In ELT, data is extracted from different sources and loaded directly into a data warehouse. The data is then transformed within the data warehouse. 𝐖𝐡𝐢𝐜𝐡 𝐚𝐩𝐩𝐫𝐨𝐚𝐜𝐡 𝐢𝐬 𝐫𝐢𝐠𝐡𝐭 𝐟𝐨𝐫 𝐲𝐨𝐮 ? - The best data integration strategy for your business will depend on your specific needs. If you have a lot of data from different sources, and you need to transform it into a consistent format before loading it into a data warehouse, then ETL may be the best approach for you. - If you need to get your data into your operational systems quickly and easily, then Reverse ETL may be the best approach for you. - If you have a lot of data, and you need to be able to scale your data integration process, then ELT may be the best approach for you.

5

Snowflake question: on Netezza, you had to be connected to a database to create an object, eg a procedure. on Snowflake, you can compile a proc to a database you are not currently using if the procedure name is qualified with DB and schema. is it possible to prevent this and reciprocate Netezza behaviour which I believe was safer?

1

An under appreciated reason why data processing companies like Confluent, Databricks, Splunk etc get really big is how data scales. Data usage in a company scales O(EQS), if we use Big-O notation, E=number of employees using the data, Q=number of queries being run , S=size of data queried. The reason customer ACVs can get absolutely massive is that if you’ve built an incredible product you will be able to process more data if the customer does even one of the 3: - store more data - add more employees who want to work with the data - have more computations being done on the data Most of the time, companies that are growing have all 3 increasing simultaneously. Ofcourse, because data tools are sticky this growth also persists and even when employee growth slows down, often more data and uses of the data have emerged If you’re building any tools, or even side projects that involve data processing, I’d love to learn more and talk to you!

50

One of the most common question asked in Data Engineering Interview: Difference between ETL, ELT, and Reverse ETL. 1️⃣ ETL: The Traditional Powerhouse Extract, Transform, Load (ETL) has been the go-to process for decades. It involves: 1. Extracting data from various sources 2. Transforming it to fit operational needs 3. Loading it into the target system (usually a data warehouse) 2️⃣ ELT: Flipping the Script Extract, Load, Transform (ELT) emerged with the rise of cloud computing and big data. The key difference: - Data is loaded into the target system before transformation - Leverages the power of modern data warehouses for transformation - Offers more flexibility and scalability 3️⃣ Reverse ETL: Closing the Loop A newer player in the field, Reverse ETL: - Moves processed data from warehouses back into operational systems - Enables data activation, turning insights into action - Bridges the gap between analytics and operations For career guidance in Data Analytics, job search strategies, resume review, interview preparation, etc., you can book a session with me here: https://topmate.io/itshb

53

As a data Engineer | Specialist | Analyst, I am passionate about harnessing the power of data to drive insights and innovation. But who says a data engineer can't build their own website? With effort and a keen interest in expanding my skill set, I’ve proven that is possible. 💻✨ In today’s fast-paced tech world, it's crucial to embrace overall development. Whether it's data engineering, Analyst, IT-Support, Network, or web development, continuous learning and growth are key. This project has been a fantastic journey, pushing me beyond my comfort zone and broadening my horizons. Check out my website https://lnkd.in/g6AA5XvK and let me know what you think. Your feedback is invaluable! #DataEngineering #WebDevelopment #TechJourney #ContinuousLearning #Innovation #NewWebsite

6

3 Comments

✍ Delta Lake in Databricks 📚 “Delta Lake is an open-source storage layer that brings reliability to data lakes by adding a transactional storage layer on top of data stored in cloud storage” - Databricks ✅ Acts as an intelligent storage layer atop formats like Parquet or JSON ✅ Brings ACID transactions to data lakes, ensuring data integrity ✅ Solves common data lake challenges: inconsistency and performance issues ✅ Optimized for cloud object storage (S3, Azure, Google Cloud) ✅ Open-source: freely available on GitHub ✅ Delta Lake transforms data lakes into reliable, consistent, and high-performing data repositories. It's not about how data is stored, but how it's managed and accessed. #DataEngineering #DeltaLake #BigData #CloudComputing

8

1 Comment

In a recent Data Analytics - Power BI project, I collaborated with my team of Computer Science Engineering mentees to analyze Indian Airlines' ticket pricing. Together, we developed vibrant Power BI reports providing rich insights into the airline industry, capturing key metrics and business-critical data points. The reports included various Key Performance Indicators (KPIs) essential for understanding airline operations and revenue streams. Metrics such as Total Airlines, Flights, Revenue analytics by class and airlines, and insights on Revenue by arrival time and flight routes were examined. My mentees showcased a strong understanding of Power BI, using advanced data visualization techniques to create impactful visuals. The reports highlighted critical routes, top revenue-generating classes, and strategic cities crucial to airlines’ market success. The final deliverable was a suite of colorful, interactive dashboards offering valuable insights to stakeholders, shedding light on travel patterns, revenue periods, and airline distribution by city. Witnessing my mentees' growth into skilled Data Analytics professionals throughout this project has been truly rewarding. This project underscores Power BI's potential in uncovering actionable insights within the aviation industry. Explore the data source at the link provided to delve deeper into Indian Airlines' ticket pricing analysis. Data source: https://lnkd.in/djxPi7b9 #POWERBI #DATAANALYTICS #DATAARCHITECT #DATAANALYSIS #LEARNING #MENTOR #MENTEES

12

There are times when we spend lot of time figuring out the issue that caused failure in the data pipelines. Here are few best practices which we can adopt to avoid and easily figure out the issues. - Enable dataquality checks and ensure that schema, data type validations are in place. - Track the flow of data to identify the error and the processes that got affected. - Implement Idempotent operations to ensure the end results have no impact and there are no side effects. - Enable logging and monitoring with alerts to quickly identify issues. - Use error handling techniques like exception handling, circuit breakers, failover strategies to handle errors without affecting entire pipeline. - Make sure each components of the pipeline is properly tested. #dataengineering

6

✨ Excited to share that I’ve earned the "Hands-On Essentials: Data Warehousing Workshop" badge from Snowflake! 🎉 This workshop deepened my understanding of modern data warehousing, including key concepts like data ingestion, transformation, and performance optimization within Snowflake. It’s inspiring to see the power of cloud-native data solutions and how they can unlock insights at scale. #Snowflake #DataWarehousing #DataEngineering #CloudComputing #LearningAndDevelopment #DataAnalytics

6

2 Comments