Gayatri M.

𝗙𝗶𝗹𝗲 𝘀𝗸𝗶𝗽𝗽𝗶𝗻𝗴 is the most popular technique used to Read data faster in Big data. Learn how Hive and Delta Lake implements this 🤔 Upcoming article this weekend on my Medium channel 🗓️ Meanwhile you can Subscribe to my Medium channel to read other interesting articles on Spark Optimizations for Free 📢 https://lnkd.in/gQU-XxVZ #dataengineering

15

🌟 From Playwright to Katalon: Building Automation Projects with Page Object Model (POM) 🌟 After successfully completing an automation project using Playwright with the Page Object Model (POM) approach and JavaScript as the programming language, I’m excited to share my latest achievement! This time, I have successfully created an automation project in Katalon Studio, once again utilizing the Page Object Model (POM) approach, but now with Groovy as the programming language. 🎯 This journey gave me deeper insights into the differences and flexibility between various automation testing tools, as well as how the POM approach can significantly enhance the maintainability and scalability of automation projects. 🚀 Key takeaways: - Playwright provided flexibility with cross-browser integration using JavaScript, while Katalon Studio offers a more visual and user-friendly environment for automation testing with Groovy. - Page Object Model has proven to be an effective approach for keeping code well-structured and organized, whether in Playwright or Katalon. Looking ahead, I plan to continue exploring other tools and technologies to further expand my capabilities in automation testing development. 💻✨ #automation #Playwright #KatalonStudio #JavaScript #Groovy #PageObjectModel #testing #softwaredevelopment Code repository :

35

Happy to share about completing Snowflake's On-Demand Training on SnowConvert tool. SnowConvert is a code migration tool developed by Snowflake for supporting code conversion from Oracle, Teradata and SQL Server databases to Snowflake. The course provided useful insights on common pitfalls while migrating code from legacy databases to Snowflake. #ltimindtree #snowflake #snowconvert #databasemigration

27

🚀Just wrapped up the Hands-On Essentials: Collaboration, Marketplace & Cost Estimation Workshop!🚀 It was an insightful experience learning how to collaborate effectively on Snowflake and navigate the Marketplace. Excited to leverage these skills in future projects! 🚀 #Snowflake #DataEngineering #Collaboration #Marketplace #ContinuousLearning #Tech #Badge

22

1 kommentar

A small yet significant addition to delta lake tables - #DeletionVectors   This was added with the intention to overcome a limitation of disregarding a particular parquet file and creating new one in case of record(s) being deleted. But does it really overcome that? And what about the cases where a record is added?   Let's understand with an example: 1. A record is deleted - In this case, instead of creating a new parquet file with that record deleted, delta lake will create a .bin file which represents a deletion vector and writes that in the new transaction log file created for this transaction. This .bin file can't be read as it is encrypted. However, if you look at the new log file, it will show the number of deletion vectors added and also have an entry for the same deletion vector.   Next time when this table is read, it will read the same parquet file containing that deleted record, however, it will ignore the deleted record based on the entry in log file which says that record is deleted despite of having that record in the parquet file.   2. Inserting another record - This time indeed, it will have to create a new file. While doing so, it will also read that deletion vector and just not add that record in the new parquet file being created. The new log file will also have an entry saying that the previous deletion vector has been removed and the entry for the same will also be gone.   3. Deleting two records in two separate operations - A new bin file and a log file will be created saying that a deletion vector has been added for the first deletion. For the second deletion, again a new bin file is created, however in the log file, the old vector is removed and the new one is added.   4. Optimize command - When you run this command, it will try to optimize the files and re-write them again, thus initiating the creation of new parquet files. And as expected, it will remove those deleted records from the previous transactions in the file. Your log file will have an entry saying deletion vector has been removed. #DataBricks #DeltaTable #DeltaLake #DataEngineer #DataEngineering

14

📚 What's New in PostgreSQL 17 Beta 📚 𝗘𝗻𝗵𝗮𝗻𝗰𝗲𝗱 𝗝𝗦𝗢𝗡 𝗙𝘂𝗻𝗰𝘁𝗶𝗼𝗻𝘀: - Simplified JSON data manipulation with new functions like json_table. - Improved performance and flexibility for semi-structured data.   𝗜𝗺𝗽𝗿𝗼𝘃𝗲𝗱 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲: - Faster query execution through optimized planning and execution. - Enhanced B-tree index performance for efficient data retrieval. - Optimised vacuum operations for better database maintenance. 𝗡𝗲𝘄 𝗙𝗲𝗮𝘁𝘂𝗿𝗲𝘀: 🔹 𝗘𝗫𝗣𝗟𝗔𝗜𝗡 (𝗦𝗘𝗥𝗜𝗔𝗟𝗜𝗭𝗘): Analyze query performance, including serialization costs. 🔹 𝗕𝗥𝗜𝗡 𝗣𝗮𝗿𝗮𝗹𝗹𝗲𝗹 𝗜𝗻𝗱𝗲𝘅 𝗕𝘂𝗶𝗹𝗱𝘀: Faster creation for large datasets. 🔹 𝗗𝗶𝗿𝗲𝗰𝘁 𝗦𝗦𝗟/𝗧𝗟𝗦 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻𝘀: Reduced latency and enhanced security. 🔹 𝗘𝘃𝗲𝗻𝘁 𝗧𝗿𝗶𝗴𝗴𝗲𝗿𝘀 𝗔𝗳𝘁𝗲𝗿 𝗖𝗹𝗶𝗲𝗻𝘁 𝗖𝗼𝗻𝗻𝗲𝗰𝘁: Granular control over session-level operations. Ready to explore PostgreSQL 17 Beta? https://lnkd.in/dFcEBQin Share your thoughts or experiences in the comments below! #PostgreSQL #PerformanceImprovement #SQL #Database

6

A big thank you to Sumit Mittal and TrendyTech on my Data Engineering learnings. In this module, a lot of ground was covered in understand and working with - - RDDs and Parallelize - Narrow and Wide Transformations - reduceByKey & groupByKey - Broadcast Joins - Repartition and Coalesce - Cache #bigdataengineer #apachespark #sql #aws #azure

54

6 kommentarer

Optimizing spark jobs is crucial for efficient data processing. Here are some key considerations to enhance performance: - Choose the right partitioned column for parallelism to reduce data skewness. - Opt for the appropriate join method: - Broadcast join: Broadcast smaller data frames to executors for faster execution during joins. - Use Sort merge join for equi-joins as data is sorted before merging. - Determine the optimal number of partitions to define the degree of parallelism. - Minimize the use of user-defined functions and opt for SQL aggregate functions in spark for improved efficiency. - Employ Salting to reduce data skewness before partitioning, leading to better performance. - Leverage Adaptive Query Execution (AQE) for efficient query planning based on runtime statistics. These strategies include tuning shuffle partitions, optimizing joins, and addressing skew joins. Additionally, utilize file formats like Parquet and orc, serialization like Kyro, and avoid unnecessary data caching in memory. Select only required columns and partitions, and adjust driver memory, number of executors, and memory overhead as needed for optimal results. #DataProcessing #SparkJobs #Efficiency #DataOptimization #BigData #TechTips

17

https://lnkd.in/grS88PUx - PySpark foreachPartition is a very powerful Dataframe method (which executes a Spark action) that you can use both in Scala and in PySpark, it allows for massively paralleled side effects your Spark pipelines, for example very efficient writes to AWS SNS topics (even with optional multithreading for even faster writes) - which opens a way for decoupled communication patterns between data teams and backend / frontend systems - it’s great even for typical batch processing jobs :) I plan a more detailed article on this topic soon! #PySpark #MessageTopic #AWS #EDA

22

Hi fellow and aspiring #dataengineers !! If you have some free time, please check out the article about DAG in #apachespark which I wrote some time back and was quite loved by the audience. 📊 Total views: 13k 📊 Total reads: 8.1k With a read ratio of 63% and by those figures, it means the article reached good amount of people. Want to give it a try? Please do leave feedbacks or comments on the article, if any. Here it is. Enjoy Sunday !! https://lnkd.in/gQrTT-f6 #dataengineering #learningandsharing #medium #loveforwriting #blogging

88

1 kommentar

𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗧𝘂𝗻𝗶𝗻𝗴 𝗶𝗻 𝗔𝘇𝘂𝗿𝗲 𝗗𝗮𝘁𝗮 𝗙𝗮𝗰𝘁𝗼𝗿𝘆 & 𝗔𝘇𝘂𝗿𝗲 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 🚀 𝗙𝗼𝗿 𝗔𝘇𝘂𝗿𝗲 𝗗𝗮𝘁𝗮 𝗙𝗮𝗰𝘁𝗼𝗿𝘆: 1️⃣ **Optimize Data Movement**: Use the right Integration Runtime (IR), e.g., Azure IR for large datasets across regions. 2️⃣ **Compression for Data Transfers**: Apply gzip compression when copying 1TB from Azure Blob to Data Lake. 3️⃣ **Incremental Data Loads**: Instead of loading the full dataset daily, load only new records using watermark columns. 4️⃣ **Tune Activity Concurrency**: Set MaxConcurrentActivityRuns to balance 10 simultaneous data flows. 5️⃣ **Monitor & Auto-Scale**: Use ADF's monitoring dashboard to detect bottlenecks and enable auto-scaling for heavy workloads. 𝗙𝗼𝗿 𝗔𝘇𝘂𝗿𝗲 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀: 1️⃣ **Cluster Auto-scaling**: Use autoscaling clusters to handle fluctuating workloads, such as ETL on 500GB datasets. 2️⃣ **Data Partitioning & Caching**: Cache frequently accessed data using df.cache(), improving query times by 30%. 3️⃣ **Optimize Spark Queries**: For large joins, broadcast smaller tables to improve query performance. 4️⃣ **Leverage Delta Lake**: Use OPTIMIZE on Delta Lake tables to improve query efficiency in a 1TB dataset. 5️⃣ **Adaptive Query Execution (AQE)**: Enable AQE for dynamic query plan adjustments, speeding up long-running jobs. --- 🚀 **Follow me** for more of such content and resources. for any guidance in #DataEngineering #datawarehousing #CloudComputing #datamodeling #Snowflake #Azure #SQL #Informatica #PySpark #Python #DataEngineering #AzureDataFactory #Databricks #CloudComputing #PerformanceTuning #AI #ETL

137

5 kommentarer

BasePlus package version 2.0.1 is here. [Edit: quick bug-fix] Release details: https://lnkd.in/dyY8kHqZ #SAS #package #basePlus #SASPackagesFramework #SPF #SASPAC

23

1 kommentar

Actively Seeking New Opportunities | Senior Data Engineer 🌟| C2C With 10 years of experience in data engineering, I am actively seeking new opportunities to contribute my expertise in developing cutting-edge data solutions. My experience spans across modern data stack technologies, cloud platforms, and big data architectures. 🔧 Key Skills: Cloud Platforms: AWS, GCP, Azure Data Warehousing: Snowflake, Big Query, Redshift ETL/ELT Pipelines: Apache Airflow, dbt, Talend Big Data Technologies: Spark, Hadoop, Kafka Programming Languages: Python, SQL, Scala, Java Containerization & Orchestration: Docker, Kubernetes CI/CD & DevOps: Jenkins, GitLab, Terraform Data Lakes & Streaming: Delta Lake, Kinesis I’m passionate about optimizing data workflows, improving data quality, and scaling architectures to meet the demands of today’s fast-paced data environments. If you’re looking for someone who can transform complex data into meaningful insights, let’s connect! Excited to collaborate and contribute to innovative projects. 📩 DM or reach out via email [email protected] to discuss opportunities! hashtag #repost #DataEngineering #CloudArchitecture #ETL #BigData #DataOps #DataWarehousing #DevOps #OpenToWork #Seniordataengineer #c2c #c2h #java #kafka #python #sql #aws #azure

2

Love your insightful take on the evolving landscape of Databricks and cloud migration! Your observation on the changing dynamics of cluster start times and cost implications is spot on. It's crucial to navigate these shifts effectively for optimal performance and cost efficiency. Keep sharing your valuable insights!

1

In case you need to create a workspace in #Postman quickly, here is how to do it using Workspace templates written by Ramji Enamuthu https://lnkd.in/dxu7qH6C #RESTApis #APIAutomation

3

✨ Celebrating a small step towards the Databricks ✨ #Databricks #DataAnalitycs #DatabricksFundamentals

7

🚀 Elevate Your Testing with Playwright: Comprehensive Guides for BDD, Reporting, and CI/CD 🚀 Are you looking to enhance your end-to-end testing strategy? Look no further! Check out these insightful articles that will help you leverage Playwright for robust and efficient testing: 1. Setting Up Playwright for Testing with BDD, Cucumber, and Allure Reports Discover how to integrate Playwright with Cucumber for BDD testing and generate detailed reports with Allure. This guide will help you streamline your testing workflow and achieve clearer, more organized results. 📊🔍 https://lnkd.in/eTGh93pR 2. End-to-End Testing with Playwright Dive into the essentials of Playwright and learn how to set up effective end-to-end tests. This article covers key aspects of Playwright to ensure your tests are thorough and reliable. 💡✅ https://lnkd.in/ernvW7dv 3. Playwright Integration with Azure DevOps Pipeline Integrate Playwright into your Azure DevOps pipeline and automate your testing processes. This guide provides a step-by-step approach to setting up a seamless CI/CD workflow with Playwright. 🔄🔧 https://lnkd.in/eBuZ9iW5 These articles will equip you with the knowledge and tools to take your testing strategy to the next level. Happy testing! 🌟 #Playwright #Testing #BDD #Cucumber #Allure #CI_CD #AzureDevOps #EndToEndTesting #Automation #QualityAssurance

1

All discussion at @99 only for this weekend https://lnkd.in/d2jaFD7t Even if you feel like talking about job, career, any setback, education, choice dilemma, service based & product based work culture, QA job future, switching from QA, industry project structure, internship or any other thing, jump on the call. Please note: All views are mine, i do not represent any employer or their views in anyway. All discussion on topmate adheres to, only and only mine ideology and nothing else. Consider it just a peer to peer discussion at very minimal cost and nothing else. I am not an expert. I am a learner.

6

🚀 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫 𝐈𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 📢 𝐀𝐩𝐚𝐜𝐡𝐞 𝐒𝐩𝐚𝐫𝐤 ⌛ 𝟏-𝐌𝐢𝐧𝐮𝐭𝐞-𝐑𝐞𝐚𝐝 𝐏𝐢𝐭𝐟𝐚𝐥𝐥𝐬 𝐨𝐟 "𝐂𝐥𝐢𝐞𝐧𝐭 𝐌𝐨𝐝𝐞" 𝐢𝐧 𝐒𝐩𝐚𝐫𝐤 𝐑𝐮𝐧𝐧𝐢𝐧𝐠 𝐨𝐧 𝐂𝐥𝐨𝐮𝐝 ☁️ 🧔🏽‍♂️ Interviewer: When does Client Mode lead to failure in Spark running on the cloud? 👨‍🦰 Interviewee: Client Mode in Spark running on the cloud can encounter issues when the driver program, which executes on the client machine, becomes a bottleneck due to resource limitations or network latency. 🧔🏽‍♂️ Interviewer: Can you elaborate on the challenges faced with Client Mode in this scenario? 👨‍🦰 Interviewee: When the driver program is running on the client machine, it needs to communicate with the cluster's worker nodes over the network. If the network is slow or congested, it can lead to delays in task execution and overall job performance degradation. 🧔🏽‍♂️ Interviewer: How can these issues impact the reliability and efficiency of Spark jobs? 👨‍🦰 Interviewee: The performance issues associated with Client Mode can result in job failures, slow job execution, and resource wastage. Additionally, the driver program may encounter out-of-memory errors or other resource-related issues, further hampering job reliability and efficiency. 🧔🏽‍♂️ Interviewer: Are there any strategies or workarounds to mitigate these challenges? 👨‍🦰 Interviewee: Yes, there are several approaches to address these challenges. One option is to use Cluster Mode instead of Client Mode, where the driver program runs within the cluster, closer to the worker nodes. Additionally, optimizing network configurations and allocating sufficient resources to the client machine can help alleviate performance issues. 🧔🏽‍♂️ Interviewer: How crucial is it to consider these factors when deploying Spark jobs on the cloud? 👨‍🦰 Interviewee: It's paramount to carefully evaluate the deployment strategy and consider factors like network latency, resource availability, and job requirements. Ignoring these factors can lead to suboptimal performance and hinder the scalability and reliability of Spark jobs in cloud environments. #OneStepAnalytics #DataAnalytics #BigData #SQL #DataProcessing #DataWarehouse #DataEngineering #DistributedComputing

18