Rohini Adavappa

Spark code to convert a timestamp given in valid format. FIrst check whether the timestamp ends with AM or PM. Filter out those records, which doesn't match this criteria. Post that, represent the given timestamp in 24 Hours & 12 Hours Format. #SCALA #BIGDATA #SPARK Karthik K. INPUT : DATETIME 25 May 2024 10:24:00 AM, 25 May 2024 10:24:00 PM, 25 May 2024 11:24:00 AM, 25 May 2024 11:24:00 PM, 25 May 2024 11:24:00, def main(args: Array[String]): Unit = { val ss = SparkSession.builder.appName("25_05").master("local[*]").getOrCreate() import ss.implicits._ val scheme = StructType(Seq(StructField("DATETIME", StringType, true))) val df = ss.read.format("csv").option("path", "C:/Users/hp/DATETIME.txt").schema(scheme).load() 👉 // KEEP ONLY THOSE RECORDS WHICH ENDS EITHER IN AM OR PM. 👉 val df_1 = df.filter(col("DATETIME").rlike("[(PM)(AM)]$")) val df_2 = df_1.select(col("DATETIME"), 👉 // CONVERT THE TIMESTAMP IN 24 HOURS FORMAT. from_unixtime(unix_timestamp(col("DATETIME"),"dd MMM yyyy hh:mm:ss a"),"dd MMM yyyy HH:mm:ss"). as("24 HOURS FORMAT")) df_2.show(truncate = false) 👉 // CONVERT THE TIMESTAMP IN 12 HOURS FORMAT. val df_3 = df_2.select(col("DATETIME"),col("24 HOURS FORMAT"), from_unixtime(unix_timestamp(col("24 HOURS FORMAT"), "dd MMM yyyy HH:mm:ss"),"dd MM yyyy hh:mm:ss a").as("12 HOURS FORMAT")) df_3.show(truncate = false) OUTPUT:

20

Connections, I have a 3GB file in.gz format. I am writing this file into an external table in adls location which is getting written in PARQUET format. The write job is taking around 27 mins for only 3 GB of data for only one date partition ; and it is over - partitioning the original file of 17 MB each and creating 200 part parquet files in adls. Its just reading and writing. No transformations in between. Workarounds that I have tried - 1) Incresed the number of executors per node. 2) Set the maxPartitionBytes spark parameter to a value targeting 5-6 partitions in adls. But its still taking 27 mins. Can anyone suggest how can I bring down the write time ?

71

31 Kommentare

🚀 Exciting day at Snowflake Data Cloud World Tour Mumbai 2024! ❄️ I had the privilege of attending the Snowflake Data Cloud World Tour in Mumbai, and here are some key takeaways from insightful sessions on the future of data and AI: 📊 Open Table Formats & Iceberg Support Exploring integration possibilities with Apache Iceberg and Polaris Discussions on ACID compliance, performance optimization, and data evolution 🛡️ Snowflake Horizon Unveiling a vision for unified data governance and security across clouds Concepts for simplifying management of data and apps from a centralized interface 🤖 Generative AI & LLMs Introduction to the Snowflake Arctic family of Large Language Models (LLMs) Exploring Snowflake Cortex as a pathway to democratize AI Emphasis on the "AI to Data, not Data to AI" approach for enterprises 🔒 Data Governance & Security Highlighting the importance of unified security and compliance measures Showcasing potential applications of Dynamic Data Masking for data protection 💻 Snowpark Container Service & ML API Demonstrations of running AI and full-stack apps within the Snowflake ecosystem Exploring possibilities for direct ML model execution and predictions 🔗 Native Apps & Data Sharing Discussing streamlined approaches to building and sharing data applications Insights into enhancing operational collaborations through data sharing 📄 Document AI & Unstructured Data Previewing tools aimed at extracting insights from unstructured data The event showcased Snowflake's vision for innovation, open standards, and efficient data management. It was fascinating to see the proposed solutions for various data challenges, from governance to AI implementation. Kudos to the speakers, including Sanjay Deshmukh Vijayant Rai Rahul Jain 🇮🇳 Kamesh Sampath and Sumeet Tandure Excited to contemplate how these concepts might shape our future data strategies! #SnowflakeWorldTour #DataCloud #AI #DataGovernance #Innovation

51

2 Kommentare

Me: Can you explain the difference between SQL and MySQL? Candidate: Sir, I am proficient in MySQL but if you give me a chance, I will learn SQL as well.  Me: 🤐 SQL (Structured Query Language): --> SQL is a standardized language used for managing and manipulating relational databases. --> It allows you to perform tasks such as querying data, updating records, creating and modifying database schemas, and managing access control. --> SQL is a language, not a software. It defines the syntax and semantics for database operations. MySQL: --> MySQL is an implementation of a relational database management system (RDBMS) that uses SQL as its language for interacting with the database. --> It is one of many RDBMS that support SQL. Other examples include PostgreSQL, SQLite, Microsoft SQL Server, and Oracle Database. --> MySQL is developed, distributed, and supported by Oracle Corporation. In summary, SQL is the language, while MySQL is a specific software that implements this language to manage databases. #SQL #MySQL

38

3 Kommentare

Actively Seeking New Opportunities | Senior Data Engineer 🌟| C2C With 10 years of experience in data engineering, I am actively seeking new opportunities to contribute my expertise in developing cutting-edge data solutions. My experience spans across modern data stack technologies, cloud platforms, and big data architectures. 🔧 Key Skills: Cloud Platforms: AWS, GCP, Azure Data Warehousing: Snowflake, Big Query, Redshift ETL/ELT Pipelines: Apache Airflow, dbt, Talend Big Data Technologies: Spark, Hadoop, Kafka Programming Languages: Python, SQL, Scala, Java Containerization & Orchestration: Docker, Kubernetes CI/CD & DevOps: Jenkins, GitLab, Terraform Data Lakes & Streaming: Delta Lake, Kinesis I’m passionate about optimizing data workflows, improving data quality, and scaling architectures to meet the demands of today’s fast-paced data environments. If you’re looking for someone who can transform complex data into meaningful insights, let’s connect! Excited to collaborate and contribute to innovative projects. 📩 DM or reach out via email [email protected] to discuss opportunities! hashtag #repost #DataEngineering #CloudArchitecture #ETL #BigData #DataOps #DataWarehousing #DevOps #OpenToWork #Seniordataengineer #c2c #c2h #java #kafka #python #sql #aws #azure

3

Questions on COALESCE in SPARK #BIGDATA #SPARK Karthik K. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐭𝐡𝐞 𝐮𝐬𝐞 𝐨𝐟 𝐜𝐨𝐚𝐥𝐞𝐬𝐜𝐞 𝐢𝐧 𝐒𝐩𝐚𝐫𝐤? In Apache Spark, coalesce is a transformation that is used to reduce the number of partitions in an RDD (Resilient Distributed Dataset) or DataFrame. This is particularly useful for optimizing the performance of certain operations by reducing the shuffling of data across the network. 𝐖𝐡𝐲 𝐔𝐬𝐞 𝐜𝐨𝐚𝐥𝐞𝐬𝐜𝐞? 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐢𝐧𝐠 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: By reducing the number of partitions, coalesce can help optimize performance for certain operations, such as actions or joins, by minimizing the overhead of managing too many partitions. 𝐀𝐯𝐨𝐢𝐝𝐢𝐧𝐠 𝐅𝐮𝐥𝐥 𝐒𝐡𝐮𝐟𝐟𝐥𝐞: Unlike repartition, coalesce avoids a full shuffle of the data. This is particularly useful when reducing the number of partitions without significantly changing the data distribution. 𝐂𝐨𝐝𝐢𝐧𝐠 𝐐𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 𝐟𝐨𝐫 𝐈𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐁𝐚𝐬𝐢𝐜 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠: 1] What is the purpose of coalesce in Spark? 2] How does coalesce differ from repartition? 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐒𝐜𝐞𝐧𝐚𝐫𝐢𝐨: 1] Given a DataFrame with 10 partitions, how would you reduce it to 3 partitions without causing a full shuffle? 2] What are the potential benefits of reducing the number of partitions using coalesce before performing an action like count? 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 𝐂𝐨𝐧𝐬𝐢𝐝𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬: 1] Explain a scenario where using coalesce would significantly improve performance. 2 ]What are the implications of using coalesce on data skew and load balancing?

9

Hi, #BusinessAnalyst Folks Another Solved Case Study For You... Case Study Question: User Persona: John is an author who frequently travels and is a loyal Expedia user, booking tickets exclusively through the website. User Pain Point: John prefers to travel on Boeing 737 planes but cannot filter this option on Expedia. He only learns the aircraft type after booking, often needing to cancel or request a change, which frustrates him. Problem Statement: Develop a solution for Expedia's web to allow users to filter flights by specific aircraft types, such as the Boeing 737. #BA #BusinessAnalysis #CaseStudy #Agile #Scrum #IT #ITJobs #HR #ScrumMaster #ProductOwner #ProjectManagement #ProjectManager

114

16 Kommentare

Yesterday, I came across some posts from a couple of Influencers about SQL training. The fees they were charging left me quite surprised - 15K INR for SQL. 15000 Seriously? Especially for SQL. One of these trainers, with less than 2 years of experience in Data, is providing trainings . What value it gonna add them until trainer is not having a good real time project experience. No one has to spend this much amount on SQL learning, there are many resources available on Internet either free or affordable prices. Best You tube channel : Kudvenkat & Ankit Bansal For Practice : SQLZOO , Hackerrank & Leetcode. Even Ankit 's live sessions are not more than 3/4K INR. Which is quite reasonable. P.S. Do your proper research before joining any courses, Connect with Trainers & get clarity about your doubts before paying. #sql #influencers #gimmicks #learning #upskill #dataanalytics #dataengineer

315

46 Kommentare

What is DataOps? DataOps is a practice that brings agile, DevOps-like principles to data management. It emphasizes collaboration, automation, and continuous improvement to enhance the speed and quality of data delivery. Think of it as the bridge that connects data engineering, data science, and business teams. Why DataOps Matters: 🔹 Boosts Data Quality: Automated testing and validation ensure data integrity. 🔹 Accelerates Insights: Streamlined processes mean faster access to actionable insights. 🔹 Enhances Collaboration: Breaking down silos fosters better communication and teamwork. Core Principles of DataOps: Collaboration: Unite cross-functional teams for smoother data workflows. Automation: Implement automated pipelines for efficiency and accuracy. Continuous Feedback: Use real-time monitoring and feedback to refine processes. DataOps isn’t just a trend; it’s the future of data management. As technology evolves, we can expect even more innovative practices and tools to emerge. Ready to transform how you handle data? Embrace DataOps and discover new ways to boost efficiency and gain deeper insights! #DataOps #DataManagement #Agile #DataEngineering

52

Jisko bhi Hire 🧑‍💻Karna hai…Krlo (Whosever Wants to Hire Me - Go Ahead) Chand Din ka Mehmaan Hu Mai 😬 (Just few days to embark my new journey) To all HR, if you need a one man show for your complex Data Pipeline, I am willing to contribute and enhance your complex messy data pipelines using Snowflake & Matillion and let me tell you one thing - IMMEDIATE JOINEE - Aisa Deal Kaha Milega🤩 #openforopportunities #hireme #datapipeline #openfordiscussion #etl #snowflake #matillion

90

3 Kommentare

Data Engineering Interview Question. Asked in EPAM Systems for Snowflake Data Engineer Role. Certainly: 1. Write a Python code to print the factorial of any number, n. (Factorial of n. n!=n∗(n−1)∗(n−2)∗.....2∗1) Solution: def factorial(n):     if n < 0:         return "Factorial is not defined for negative numbers."     elif n == 0 or n == 1:         return 1     else:         answer = 1         for i in range(1, n + 1):             answer *= i   # answer = answer*i         return answer # Example usage: try:     n = int(input("Enter a number to calculate its factorial: "))     print(f"The factorial of {n} is {factorial(n)}.") except ValueError:     print("Please enter a valid integer.") follow up question. Sure, here's a rephrased version: 2. Create a program in Python to output the number of trailing zeros in the factorial of a given number, n. example 5! = 120 and trailing zero is 1. Like wise for 9! = 362880 and trailing zero is 1. def trailing_zeroes_in_factorial(n):     # Initialize the count of trailing zeroes     count = 0         # Initialize the current power of 5     power_of_5 = 5         # Loop to count the number of multiples of 5, 25, 125, etc. in n     while n >= power_of_5:         # Add the number of multiples of the current power of 5 to the count         count += n // power_of_5                     # Move to the next power of 5 (i.e., 5, 25, 125, ...)         power_of_5 *= 5           # Return the total count of trailing zeroes     return count # Example usage: # Take input from the user for which factorial trailing zeroes need to be found n = int(input("Enter a number to find the number of trailing zeroes in its factorial: ")) # Print the result by calling the function and formatting the output print(f"The number of trailing zeroes in {n}! is {trailing_zeroes_in_factorial(n)}.") Certainly! Here's a rephrased version: Note: During interviews, providing the function alone without input parameters is sufficient. However, for the post and for better understanding I have included the input parameters. Follow Avinash S. Get Interview Resources: https://lnkd.in/dRehzivv

162

2 Kommentare

With a decade of Experience in Data Engineering, I have something to share about how far we have come along, the way we handle Change Data Capture (CDC) and Slowly Changing Dimensions (SCD) in big data environments has undergone a significant transformation. Early Days: Hadoop & Hive: Initially, managing CDC and SCDs in Hadoop using Hive was challenging. Batch processing and the lack of robust ACID properties made it difficult to track changes effectively and maintain data consistency. We often relied on complex merge operations and custom scripts, which were resource-intensive and prone to errors. The Hive Era: Hive's 5-Step Shuffle: We used to rely on a tedious 5-step process involving temporary tables to manage SCDs in Hive. Create a temp table, load new data, identify changes, update the target table, and finally, clean up the temp tables. This was slow, complex, and error-prone. Spark Sparks Change: Spark: Spark brought much-needed speed with in-memory processing. But while Spark made things faster, challenges with updates and deletes in CDC still lingered. Databricks & the Cloud Revolution: Databricks: Enter Databricks and Delta Lake! ACID properties finally arrived, and features like merge and time travel made managing SCD Type 2 a breeze. Delta Live Tables and Databricks SQL further streamlined CDC. The Modern Data Stack: dbt: And let's not forget dbt! This transformation tool has become essential for managing data pipelines. With dbt's macros, snapshots, seeds, sources, and analyses, we can build modular, reusable, and testable data transformations, including those for CDC and SCD. Snapshots, in particular, offer a powerful way to track changes over time. Cloud Powerhouses: Databricks, Google Cloud (Dataproc, BigQuery), AWS (EMR, Glue, Redshift): They all offer powerful managed services that simplify CDC and SCD in the cloud. Think serverless computing for scalability, stream processing for real-time updates, and robust data governance tools. Challenges Remain, but Solutions are Emerging: We still face challenges like massive data volumes, complex data relationships, and ensuring consistency. But cloud providers are stepping up with solutions like serverless computing, stream processing, and advanced data governance tools. It's an exciting time to be in data! #CDC #SCD #Hadoop #Hive #Spark #Databricks #DeltaLake #dbt #CloudComputing #BigData #DataEngineering #DataWarehousing

37

Optimizing spark jobs is crucial for efficient data processing. Here are some key considerations to enhance performance: - Choose the right partitioned column for parallelism to reduce data skewness. - Opt for the appropriate join method: - Broadcast join: Broadcast smaller data frames to executors for faster execution during joins. - Use Sort merge join for equi-joins as data is sorted before merging. - Determine the optimal number of partitions to define the degree of parallelism. - Minimize the use of user-defined functions and opt for SQL aggregate functions in spark for improved efficiency. - Employ Salting to reduce data skewness before partitioning, leading to better performance. - Leverage Adaptive Query Execution (AQE) for efficient query planning based on runtime statistics. These strategies include tuning shuffle partitions, optimizing joins, and addressing skew joins. Additionally, utilize file formats like Parquet and orc, serialization like Kyro, and avoid unnecessary data caching in memory. Select only required columns and partitions, and adjust driver memory, number of executors, and memory overhead as needed for optimal results. #DataProcessing #SparkJobs #Efficiency #DataOptimization #BigData #TechTips

17

Finally, It's out - Snowflake Academy by Enqurious Academy Why Snowflake? It's one of the fastest growing Data platform that's ridiculously SQL friendly. So, anyone with a SQL background and needs to learn a Data Warehousing solution in an easy way, Snowflake is the best place to start. Why Enqurious? Having upskilled 5000+ data engineers with a certification clearance rate in excess of 95%, we have built learning experiences across domains like Retail, CPG, Insurance, Banking and many more. What do you need to pay to get started? Your attention, commitment and time. It's free otherwise How to start? Visit Snowflake Academy here : https://lnkd.in/gD_ruthp Or join our Slack community here to start : https://lnkd.in/ggaGpcmn Pick a short scenario, a Masterclass or a full length Project, whatever serves your needs or tastes. #learningexperience #DWaaS #SnowflakeAcademy

11

Efficiently transforming raw data into valuable insights is essential in Data Warehousing. I appreciate how intuitive and easy-to-use DBT(Data Build Tool) is!!! Here is my take away from this ~5 hours of course(<4 hrs in 1.25x 😉): -DBT Core Concepts: Models, Materializations, Seeds, Sources, Snapshot, Tests, Documentations etc. -Best Practices -Deployment -DBT Cloud -DBT Fundamentals Here's to continuous learning and growth!! #dbt #DataAnalytics #dbtCertification #DataEngineer #Learning #ProfessionalDeveloment

23