Raji Kishore Donepudi

🚀 Exciting Highlights from Snowflake BUILD Bangalore Dec 19th 2024! 🚀 Attending the Snowflake BUILD event on December 19th was an amazing experience, packed with insights into the latest advancements and powerful features of Snowflake. Here are some of the top new features and key learnings from the event: 1. Apache Polaris - Catalog Revolutionizing how we manage and access data with an open table format and REST Catalog for seamless integration. 2. Storage Enhancements Optionally archiving tables to optimize storage costs and manage data retention effectively. 3. Document AI - Copy INTO Simplifying data ingestion with new capabilities for handling document-based data. 4. Snowflake Horizon Catalog Enabling unified governance processes, providing full visibility across your data ecosystem. 5. Snowflake Notebooks A powerful tool for collaborative data science and analytics directly within Snowflake. 6. Observability: Generative AI & ML New capabilities for deep insights, leveraging AI and ML for enhanced observability. 7. SPCS - Snowpark Container Services Enabling more flexible and efficient data processing with containerized workloads. 8. Internal Marketplace - GET Button Simplifying access to shared data and services with a unified internal marketplace. 9. Egress Cost Optimizer Optimizing costs associated with data egress to improve operational efficiency. 10. Share Fine-Tuned LLMs Empowering users to share their fine-tuned large language models (LLMs) for advanced analytics. 11. Monetize AI Apps Enabling the creation and monetization of AI-powered applications for business growth. 12. Hybrid Tables & Unistore Solving concurrency challenges and enhancing performance with hybrid storage and compute solutions. 13. MODIN - Scalable Pandas Scalable, distributed computing for Pandas users, now leveraging RAY, DASK and SNOWPARK environments. 14. Altair Library for Visualization Bringing advanced visualization capabilities to Snowflake for data-driven insights. 15. Snowflake DevOps Capabilities Enhanced DevOps features with capabilities like CREATE OR ALTER, EXECUTE IMMEDIATE, Snowflake CLI, GIT Integration, Python APIs, and more. These innovations are taking data management, governance, and analytics to the next level. The future of data is truly exciting with Snowflake's cutting-edge solutions! 🌐 #Snowflake #SnowflakeBUILD #DataInnovation #AI #MachineLearning #DataGovernance #DAP #Ericsson #DAPPLATFORM

48

1 Comment

Spark code to convert a timestamp given in valid format. FIrst check whether the timestamp ends with AM or PM. Filter out those records, which doesn't match this criteria. Post that, represent the given timestamp in 24 Hours & 12 Hours Format. #SCALA #BIGDATA #SPARK Karthik K. INPUT : DATETIME 25 May 2024 10:24:00 AM, 25 May 2024 10:24:00 PM, 25 May 2024 11:24:00 AM, 25 May 2024 11:24:00 PM, 25 May 2024 11:24:00, def main(args: Array[String]): Unit = { val ss = SparkSession.builder.appName("25_05").master("local[*]").getOrCreate() import ss.implicits._ val scheme = StructType(Seq(StructField("DATETIME", StringType, true))) val df = ss.read.format("csv").option("path", "C:/Users/hp/DATETIME.txt").schema(scheme).load() 👉 // KEEP ONLY THOSE RECORDS WHICH ENDS EITHER IN AM OR PM. 👉 val df_1 = df.filter(col("DATETIME").rlike("[(PM)(AM)]$")) val df_2 = df_1.select(col("DATETIME"), 👉 // CONVERT THE TIMESTAMP IN 24 HOURS FORMAT. from_unixtime(unix_timestamp(col("DATETIME"),"dd MMM yyyy hh:mm:ss a"),"dd MMM yyyy HH:mm:ss"). as("24 HOURS FORMAT")) df_2.show(truncate = false) 👉 // CONVERT THE TIMESTAMP IN 12 HOURS FORMAT. val df_3 = df_2.select(col("DATETIME"),col("24 HOURS FORMAT"), from_unixtime(unix_timestamp(col("24 HOURS FORMAT"), "dd MMM yyyy HH:mm:ss"),"dd MM yyyy hh:mm:ss a").as("12 HOURS FORMAT")) df_3.show(truncate = false) OUTPUT:

20

If you are considering a transition into data engineering or wondering if it’s the right field for you, this Data Engineering 101 guide by Redpanda Data is must-read. It covers all the fundamental aspects of data engineering in a short, crisp, and engaging way. Honestly, I haven’t come across anything else like it. And not just that, it's a great resource for data professionals to quickly brush up on key concepts. (I’m bookmarking this for my next interview prep!) Here’s the link: https://lnkd.in/di-aiKGs.

27

1 Comment

👉 Pyspark Interview Questions Day15 🤞 How to fill down the values in Pyspark Tagging for more reach Ajay Kadiyala Ansh Lamba Arundhati Sen Shivakiran Kotur Lipsa Biswas Ankit Bansal #pysparkinterview #databricks #azuredataengineer #pyspark #azuredataengineer #data #dataengineer #azure #databricks #hadoop #sql #pyspark #python #happylearning #InterviewTips #CareerAdvice #DataEngineering #TechCommunity #azure #databricks #dataengineer #data #interview

12

Experience smooth code migration with DS Migrate! Discover just how effortlessly it's done in our exclusive promo for DataStage to DataBricks. To learn more about the implementation of the above technology, click on the link Case Study 1: https://lnkd.in/gqyZ_5Mw Case Study 2: https://lnkd.in/dMbctDvY #dataswitch #datamigration #dsmigrate #demo #video #cloud #legacy #datastage #databricks #automation #nocodelowcode

21

Tutorials for Data Science Beginners Lesson 1 :Divide the data into train and test set Step 1 : Import the Data using pandas , e.g. pd.read_csv Step 2 :Take Care of Missing Data using Scikit Imputer Step 3: Divide them into matrix of features and Dependent Variable vector ,use Pandas iLoc Step 4 : Import Train_Test_Split from Scikit Learn with inputs as matrix of features and dependent variable vector, the output will be 4 datasets 1. 2 Training Data Sets 2. 2 Test Data Sets Step 5: Apply Feature Scaling

14

1 Comment

“Working backwards from the end goal is a core principle of software development, and we’ve found it to be highly effective in modelling data products as well. This approach forces us to focus on end users and systems, considering how they prefer to consume data products (through natively accessible output ports). It provides the data product team with a clear objective to work towards, while also introducing constraints that prevent over-design and minimise wasted time and effort. It may seem like a minor detail, but we can’t stress this enough: there's a common tendency to start with the data sources and define data products. Without the constraints of a tangible use case, you won’t know when your design is good enough to move forward with implementation, which often leads to analysis paralysis and lots of wasted effort.”

4

Spark Code to load data from a Sequence or List. Convert the input data into ROW type object for dataframes. Usage of COALESCE function to replace NULL values with some LITERAL. #SPARK #SCALA #BIGDATA Karthik K. def main(args: Array[String]):Unit = { //CREATING A SPARK SESSION val ss =SparkSession.builder.appName("25_05").master("local[*]").getOrCreate() import ss.implicits._ //DATA TO BE PUSHED IN THE DATAFRAME val data= Seq( (null,32,null), ("Joy don",null,"NY"), ("Steve",42,"Delhi") ) // ROW IS AN OBJECT TYPE, WHICH IS USED TO REPRESENT A ROW IN A DATAFRAME val data_1 = data.map(x =>Row(x._1,x._2, x._3)) println(data_1) //OUTPUT OF PRINTING data_1 👉 OUTPUT: List([null,32,null], [Joy don,null,NY], [Steve,42,Delhi]) val rdd = ss.sparkContext.parallelize(data_1) val schema=StructType( Seq( StructField("name", StringType,true), StructField("age", IntegerType,true), StructField("city", StringType,true) ) ) val tempDF=ss.createDataFrame(rdd,schema) tempDF.show() OUTPUT : 👇 +-------+----+-----+ |  name | age | city | +-------+----+-----+ |  null | 32 | null | |Joy don| null |  NY | | Steve | 42 |Delhi| +-------+----+-----+ val finalDF= tempDF.select( coalesce(col("name"),lit("unknown")).as("name"), col("age"), col("city") ) finalDF.show() FINAL OUTPUT:

3

I have just finished reading the book "Data Engineering with Apache Spark, Delta Lake, and Lakehouse" by Kukreja. The book provides valuable insights into various critical aspects that can help data engineers better understand and navigate the complexities of their field. Here are some interesting topics that the book discuss: 1. Identifying challenges: The book explores the potential challenges data engineers may encounter in their work, enabling them to anticipate and effectively address these issues. 2. Enhancing Apache Spark with ACID transactions: Readers learn how to integrate ACID transactions into Apache Spark using Delta Lake, thereby improving data reliability and consistency. 3. Designing robust data lakes: The book presents effective design strategies for building enterprise-grade data lakes, ensuring scalability, performance, and maintainability. 4. Leveraging architectural and design patterns: Kukreja discusses architectural and design patterns that can be employed to create efficient and streamlined data ingestion pipelines. 5. Orchestrating data preprocessing with Apache Spark and Delta Lake APIs: The book demonstrates how to orchestrate data pipelines for preprocessing data by leveraging the powerful APIs provided by Apache Spark and Delta Lake.

9

2 Comments

🎉 Celebrating My Databricks Certified Data Engineer Professional Certification! 🎉 I’m thrilled to announce that I’ve successfully passed the Databricks Certified Data Engineer Professional exam! 🚀✨ This certification validates one’s ability to perform advanced data engineering tasks using Databricks. Here’s a glimpse of what the certification covers: Databricks Tooling: 20% Data Processing: 30% Data Modeling: 20% Security and Governance: 10% Monitoring and Logging: 10% Testing and Deployment: 10% If you’re passionate about advance data engineering, I highly recommend pursuing this certification! 🌟 #Databricks #DataEngineering #CertifiedProfessional

31

1 Comment

SPARK code to impute NULL or missing values in a table containing NULL in multiple columns. #SPARK #SCALA #BIGDATA Karthik K. def main(args:Array[String]):Unit = { val ss = SparkSession.builder.master("local[*]").appName("DFDEMO").getOrCreate() import ss.implicits._ val df_1 = ss.read.format("csv").option("header",true). option("path","C:/Users/hp/COMPANY.txt").load() df_1.show() val df_2 = df_1.select(col("PRODUCT_NAME"),col("PRODUCT_ID") , (when(col("PRODUCT_ID").isNull,0).otherwise(col("PRODUCT_ID")).as("LEGIT_PRODUCT_ID")), when(col("PRODUCT_NAME").isNull,"UNKNOWN COMPANY"). otherwise(col("PRODUCT_NAME")).as("LEGIT_PRODUCT_NAME")) df_2.show() OUTPUT:

26

4 Comments

Common spark interview questions that might help for cracking interviews as well as for better learning!! Another rarely asked question is UDF in pyspark: PySpark UDF is a User Defined Function that is used to create a reusable function in Spark. CREATION OF UDF IN PYSPARK: i. Creating UDF ,first we write python function : def upperCase(str): return str.upper() ii. Then Converting function to UDF using below code: upperCaseUDF = udf(lambda z: upperCase(z)) iii. Then directly using it with dataframes as below: df.withColumn("Cureated Name", upperCaseUDF(col("Name"))) \ .show(truncate=False) CREATION OF UDF USING SPARK SQL: i. In order to use above defined upperCase() function on PySpark SQL, you need to register the function with PySpark by using spark.udf.register(). spark.udf.register("upperCaseUDF", upperCase,StringType()) df.createOrReplaceTempView("NAME_TABLE") spark.sql("select Seqno, upperCaseUDF(Name) as Name from NAME_TABLE").show(truncate=False) CREATION OF UDF USING ANNOTATIONS: i. Create it with just a single step by using annotations. @udf(returnType=StringType()) def upperCase(str): return str.upper() df.withColumn("Cureated Name", upperCase(col("Name"))) \ .show(truncate=False) #pyspark #interviewpreparation

73

4 Comments

Questions on COALESCE in SPARK #BIGDATA #SPARK Karthik K. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐭𝐡𝐞 𝐮𝐬𝐞 𝐨𝐟 𝐜𝐨𝐚𝐥𝐞𝐬𝐜𝐞 𝐢𝐧 𝐒𝐩𝐚𝐫𝐤? In Apache Spark, coalesce is a transformation that is used to reduce the number of partitions in an RDD (Resilient Distributed Dataset) or DataFrame. This is particularly useful for optimizing the performance of certain operations by reducing the shuffling of data across the network. 𝐖𝐡𝐲 𝐔𝐬𝐞 𝐜𝐨𝐚𝐥𝐞𝐬𝐜𝐞? 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐢𝐧𝐠 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: By reducing the number of partitions, coalesce can help optimize performance for certain operations, such as actions or joins, by minimizing the overhead of managing too many partitions. 𝐀𝐯𝐨𝐢𝐝𝐢𝐧𝐠 𝐅𝐮𝐥𝐥 𝐒𝐡𝐮𝐟𝐟𝐥𝐞: Unlike repartition, coalesce avoids a full shuffle of the data. This is particularly useful when reducing the number of partitions without significantly changing the data distribution. 𝐂𝐨𝐝𝐢𝐧𝐠 𝐐𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 𝐟𝐨𝐫 𝐈𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐁𝐚𝐬𝐢𝐜 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠: 1] What is the purpose of coalesce in Spark? 2] How does coalesce differ from repartition? 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐒𝐜𝐞𝐧𝐚𝐫𝐢𝐨: 1] Given a DataFrame with 10 partitions, how would you reduce it to 3 partitions without causing a full shuffle? 2] What are the potential benefits of reducing the number of partitions using coalesce before performing an action like count? 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 𝐂𝐨𝐧𝐬𝐢𝐝𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬: 1] Explain a scenario where using coalesce would significantly improve performance. 2 ]What are the implications of using coalesce on data skew and load balancing?

9

Spark query to provide different discount rates to different product categories. #SPARK #SCALA #BIGDATA Karthik K. def main(args: Array[String]):Unit = { val ss = SparkSession.builder.appName("25_05").master("local[*]").getOrCreate() import ss.implicits._ val data= Seq( (10,32,100), (110,100,null), (210,142,120), (310,150,140) ) val data_1 = data.map(x =>Row(x._1,x._2, x._3)) println(data_1) val rdd = ss.sparkContext.parallelize(data_1,3) val schema=StructType( Seq( StructField("PRODUCT ID", IntegerType,true), StructField("QUANTITY", IntegerType,true), StructField("PRICE", IntegerType,true) ) ) val tempDF=ss.createDataFrame(rdd,schema) tempDF.show() val finalDF= tempDF.select(col("PRODUCT ID"), col("QUANTITY"), col("PRICE"), when(col("PRODUCT ID") <=100,col("QUANTITY")*col("PRICE")). when(col("PRODUCT ID") >100 && col("PRODUCT ID")<=200,lit(0.9)*col("QUANTITY")*col("PRICE")). when(col("PRODUCT ID") >200 && col("PRODUCT ID")<=300,lit(0.8)*col("QUANTITY")*col("PRICE")). when(col("PRODUCT ID") >300,0).as("REVENUE"), when(col("PRODUCT ID") <=100,lit("0%")). when(col("PRODUCT ID") >100 && col("PRODUCT ID")<=200,lit("10%")). when(col("PRODUCT ID") >200 && col("PRODUCT ID")<=300,lit("20%")). when(col("PRODUCT ID") >300,lit("100%")).as("DISCOUNT OFFERED") ) finalDF.show() OUTPUT:

5