Amit Kr. Burman

🚀 𝐓𝐚𝐛𝐥𝐞𝐚𝐮 𝟐𝟎𝟐𝟒.𝟑 𝐢𝐬 𝐡𝐞𝐫𝐞 𝐰𝐢𝐭𝐡 𝐩𝐨𝐰𝐞𝐫𝐟𝐮𝐥 𝐧𝐞𝐰 𝐟𝐞𝐚𝐭𝐮𝐫𝐞𝐬 𝐭𝐨 𝐞𝐥𝐞𝐯𝐚𝐭𝐞 𝐲𝐨𝐮𝐫 𝐚𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬! 🚀 Manage your cloud analytics at scale with Tableau Cloud Manager, streamline your reporting with the Table Viz Extension, and unlock dynamic geospatial analysis with Spatial Parameters. You can seamlessly integrate Tableau into your daily workflow with the new Tableau App for Microsoft Teams 🏆. These enhancements make it easier to explore and manage your data, collaborate more effectively, and drive smarter business decisions. 💡 Ready to take your analytics to the next level? Upgrade to Tableau 2024.3 today! 🔗 Read this article to learn more!  https://shorturl.at/ClMEL

١١

Join our training program now !! Data engineering involves building and maintaining systems for collecting, storing, processing, and transforming data for analysis and decision-making. #DataEngineering #BigData #DataPipelines #ETL #DataArchitecture #DataAnalytics #CloudComputing #MachineLearning #DataWarehouse #DataIntegration #DataScience

٣

Great information on Databricks #AI #sql functions! #data #databricks #ml #sql #dataanalysts #dataanalytics #bigdata #technology #ai

٢

Interview question for date engineer. 🎯What is joins and type of joins in pyspark and sql? ✅Joins are operations that combine rows from two or more tables based on a related column between them. Joins are essential when you need to retrieve data spread across multiple tables. Below is a comparison of how joins work in SQL and PySpark. 💮Inner Join - Returns only the rows that have matching values in both tables. 💮Left (Outer) Join - Returns all rows from the left table and matching rows from the right table. If there is no match, NULL values are returned for columns from the right table. 💮Right (Outer) Join - Returns all rows from the right table and matching rows from the left table. If there is no match, NULL values are returned for columns from the left table. 💮Full (Outer) Join - Returns all rows when there is a match in one of the tables. If there is no match, NULL values are returned for non-matching rows. 💮Cross Join - Returns the Cartesian product of both tables (every combination of rows). ♂️-- Inner Join SELECT * FROM table1 INNER JOIN table2 ON table1.column = table2.column; ♂️-- Left Join SELECT * FROM table1 LEFT JOIN table2 ON table1.column = table2.column; ♂️-- Right Join SELECT * FROM table1 RIGHT JOIN table2 ON table1.column = table2.column; ♂️-- Full Join SELECT * FROM table1 FULL JOIN table2 ON table1.column = table2.column; 🎇pyspark examples 🎇# Assuming df1 and df2 are DataFrames in PySpark 🎇# Inner Join df1.join(df2, df1["column"] == df2["column"], "inner") 🎇# Left Join df1.join(df2, df1["column"] == df2["column"], "left") 🎇# Right Join df1.join(df2, df1["column"] == df2["column"], "right") 🎇# Full Join df1.join(df2, df1["column"] == df2["column"], "outer") 🎇# Cross Join df1.crossJoin(df2) 💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮 Follow Harikesh Tyagi🎯 for more content. #DataEngineeringJobs #DataEngineering #EntryLevelDataEngineer #DataEngineerJobs #DataCareers #DataJobs #FreshersHiring #DataEngineeringFreshers #SoftwareEngineeringJobs #SoftwareEngineer #EntryLevelDeveloper #JuniorDeveloper #TechCareers #DeveloperJobs #HiringFreshers #SoftwareJobs #HiringNow #JobHunt #JobSearch #CareersInTech #GraduateJobs #EntryLevelJobs #TechJobs #FreshersJobs

٥

Interview question for date engineer. 🎯What is joins and type of joins in pyspark and sql? ✅Joins are operations that combine rows from two or more tables based on a related column between them. Joins are essential when you need to retrieve data spread across multiple tables. Below is a comparison of how joins work in SQL and PySpark. 💮Inner Join - Returns only the rows that have matching values in both tables. 💮Left (Outer) Join - Returns all rows from the left table and matching rows from the right table. If there is no match, NULL values are returned for columns from the right table. 💮Right (Outer) Join - Returns all rows from the right table and matching rows from the left table. If there is no match, NULL values are returned for columns from the left table. 💮Full (Outer) Join - Returns all rows when there is a match in one of the tables. If there is no match, NULL values are returned for non-matching rows. 💮Cross Join - Returns the Cartesian product of both tables (every combination of rows). ♂️-- Inner Join SELECT * FROM table1 INNER JOIN table2 ON table1.column = table2.column; ♂️-- Left Join SELECT * FROM table1 LEFT JOIN table2 ON table1.column = table2.column; ♂️-- Right Join SELECT * FROM table1 RIGHT JOIN table2 ON table1.column = table2.column; ♂️-- Full Join SELECT * FROM table1 FULL JOIN table2 ON table1.column = table2.column; 🎇pyspark examples 🎇# Assuming df1 and df2 are DataFrames in PySpark 🎇# Inner Join df1.join(df2, df1["column"] == df2["column"], "inner") 🎇# Left Join df1.join(df2, df1["column"] == df2["column"], "left") 🎇# Right Join df1.join(df2, df1["column"] == df2["column"], "right") 🎇# Full Join df1.join(df2, df1["column"] == df2["column"], "outer") 🎇# Cross Join df1.crossJoin(df2) 💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮💮 Follow Harikesh Tyagi🎯 for more content. #DataEngineeringJobs #DataEngineering #EntryLevelDataEngineer #DataEngineerJobs #DataCareers #DataJobs #FreshersHiring #DataEngineeringFreshers #SoftwareEngineeringJobs #SoftwareEngineer #EntryLevelDeveloper #JuniorDeveloper #TechCareers #DeveloperJobs #HiringFreshers #SoftwareJobs #HiringNow #JobHunt #JobSearch #CareersInTech #GraduateJobs #EntryLevelJobs #TechJobs #FreshersJobs

٣٧

٥ تعليق

As we consider big data management, here are 5 best tools to use for big data management: 1. Snowflake 2. Apache Hadoop 3. Apache Kafka 4. Apache Spark 5. Google BigQuery. A less-known tool is Azure HDInsights which is very powerful as it is an open-source tool that runs Apache Spark, Hadoop, Kafka, and more. Let us know in the comments which one you are familiar with. #datamanagement #bigdata #data2bots #dataconsultants #datasolutions #datatrends

١٥

I have successfully completed the Databricks Lakehouse Fundamentals certification! The Databricks Lakehouse platform combines the best of data warehouses and data lakes, allowing seamless management of both structured and unstructured data. This course has deepened my understanding of unified data architectures, Delta Lake, and how to drive faster data insights and AI adoption. Looking forward to applying these concepts in future projects, pushing the boundaries of data engineering and analytics! 💡 #databricks #lakehouse #dataengineering #bigdata #cloudcomputing #deltaLake #AI #ML

٦

١ تعليق واحد

Spark Query to create a new column based on age column in the table. #BIGDATA #SCALA #SPARK Karthik K. def main(args:Array[String]):Unit={ val ss = SparkSession.builder.appName("WHEN_OTHERWISE").master("local[*]").getOrCreate() import ss.implicits._ val data = List(("M",17),("F",25),("M",30),("F",35),("M",10)).toDF("Gender","Age") val df_with_age_remark = data.select(col("Gender"),col("Age"),when(col("Age")>=18,true).otherwise(false).as("IS ADULT")) println("Printing Age Remarks") df_with_age_remark.show() OUTPUT:

٢

Day 25 📌Data Partitioning Strategies Data partitioning involves dividing large datasets into smaller, manageable segments. Benefits include: - Improved Performance: Faster query processing. - Scalability: Easier data management. - Data Organization: Better data maintenance. 💡Common strategies include horizontal and vertical partitioning. Join our community of Data Professionals here. 🔗👉https://lnkd.in/dwe4Z-V6 #datapartioning #horizontalpartioning #verticalpartioning #dataengineering #30DaysDENugget

٢١

Hello Guys, I have explained how to setup complete databricks in this video. Please have a look and setup for free. Here is the course of Databricks which cover everything: https://lnkd.in/daGsXPFb hashtag #databricks hashtag #azuredatabricks Free Video: https://lnkd.in/gN2KT2Ax

٣٠

Consider a DataFrame named sales with the following columns: product_id, quantity_sold, and revenue. Write Spark DataFrame code to find the total quantity sold and revenue for each product. #SCALA #SPARK #BIGDATA Karthik K. INPUT: PRODUCT_ID,QUANTITY_SOLD,PRICE 101,10,150 101,15,150 102,5,50 102,10,50 103,3,100 103,10,100 QUERY: def main(args: Array[String]):Unit = { val ss = SparkSession.builder.appName("two_dfS").master("local[*]").getOrCreate() val df_1 = ss.read.format("csv").option("header",true). option("path","C:/Users/hp/PRODUCT_ID.txt"). schema("PRODUCT_ID Int, QUANTITY_SOLD Int, PRICE Int").load() val df_2 = df_1.withColumn("REVENUE",col("QUANTITY_SOLD")*col("PRICE")) df_2.groupBy(col("PRODUCT_ID")). agg(sum(col("QUANTITY_SOLD")).as("TOTAL_QUANTITY_SOLD"), sum(col("REVENUE")).as("TOTAL_REVENUE")).show() OUTPUT:

١

🚀 Dynamic Partitioning in Spark: What It Is & How It Works? Dynamic Partitioning in Spark is a powerful feature that automatically creates partitions based on the values in specific columns when writing data. Instead of manually defining partitions, Spark dynamically generates them based on the data being processed. 🔍 How It Works? Imagine you are partitioning a dataset by year and month. Spark will: ✅ Automatically create new directories for each combination of year and month encountered in the data. ✅ Optimize queries by organizing data better and reducing unnecessary shuffles. 💡 Benefits of Dynamic Partitioning 🔹 Partition Pruning Spark SQL can skip irrelevant partitions during query execution, resulting in faster data retrieval. 🔹 Automatic Partition Discovery Spark SQL automatically identifies partition columns from DataFrames or tables, streamlining the entire process. 🔹 Effective Partitioning Scheme Data gets organized into directories based on unique values in the partition columns, improving query performance and storage efficiency. 🛠️ How to Enable Dynamic Partitioning in Spark You can enable dynamic partitioning in your Spark SQL application using Python: spark = SparkSession.builder \ .appName("Dynamic Partitioning") \ .config("spark.sql.sources.partitionOverwriteMode", "dynamic") \ .getOrCreate() Leverage dynamic partitioning to simplify your data pipeline and enhance query performance in large datasets ⭐ Are you using dynamic partitioning in your Spark workflows? Share your experience in the comments! 👇

١

PySpark is the Python API for Apache Spark. It enables you to perform real-time, large-scale data processing in a distributed environment using Python. It also provides a PySpark shell for interactively analyzing your data. PySpark combines Python’s learnability and ease of use with the power of Apache Spark to enable processing and analysis of data at any size for everyone familiar with Python. PySpark supports all of Spark’s features such as Spark SQL, DataFrames, Structured Streaming, Machine Learning (MLlib) and Spark Core. Overcoming Challenges: Connecting PySpark to a SQL Database Engine for Massive Data Processing In the world of big data, integrating various technologies seamlessly is often a daunting task. One of the significant challenges I faced was connecting from a PySpark session to a SQL database engine and efficiently distributing the memory and CPU cache for massive data processing. Here’s how I tackled this complex problem. 1.Establishing a reliable connection between PySpark and a SQL database engine. 2.Ensuring the efficient distribution of memory and CPU resources. 3.Handling massive datasets without compromising performance. Pyspark Scala

٥

Transform your data into powerful, dynamic visual reports with Power BI! Learn to integrate data from multiple sources, create interactive dashboards, and master data storytelling techniques. Take your data skills to the next level and start making informed decisions today! . . . . #PowerBI #DataVisualization #DataStorytelling #BusinessIntelligence #DataAnalytics #DataDrivenDecisions #DataReports #BIExpert #PowerBITraining #VisualizeSuccess #XLAcademy #EnrollNow #DataSkills

٢

Spark code to convert a timestamp given in valid format. FIrst check whether the timestamp ends with AM or PM. Filter out those records, which doesn't match this criteria. Post that, represent the given timestamp in 24 Hours & 12 Hours Format. #SCALA #BIGDATA #SPARK Karthik K. INPUT : DATETIME 25 May 2024 10:24:00 AM, 25 May 2024 10:24:00 PM, 25 May 2024 11:24:00 AM, 25 May 2024 11:24:00 PM, 25 May 2024 11:24:00, def main(args: Array[String]): Unit = { val ss = SparkSession.builder.appName("25_05").master("local[*]").getOrCreate() import ss.implicits._ val scheme = StructType(Seq(StructField("DATETIME", StringType, true))) val df = ss.read.format("csv").option("path", "C:/Users/hp/DATETIME.txt").schema(scheme).load() 👉 // KEEP ONLY THOSE RECORDS WHICH ENDS EITHER IN AM OR PM. 👉 val df_1 = df.filter(col("DATETIME").rlike("[(PM)(AM)]$")) val df_2 = df_1.select(col("DATETIME"), 👉 // CONVERT THE TIMESTAMP IN 24 HOURS FORMAT. from_unixtime(unix_timestamp(col("DATETIME"),"dd MMM yyyy hh:mm:ss a"),"dd MMM yyyy HH:mm:ss"). as("24 HOURS FORMAT")) df_2.show(truncate = false) 👉 // CONVERT THE TIMESTAMP IN 12 HOURS FORMAT. val df_3 = df_2.select(col("DATETIME"),col("24 HOURS FORMAT"), from_unixtime(unix_timestamp(col("24 HOURS FORMAT"), "dd MMM yyyy HH:mm:ss"),"dd MM yyyy hh:mm:ss a").as("12 HOURS FORMAT")) df_3.show(truncate = false) OUTPUT:

٢٠

SPARK code to impute NULL or missing values in a table containing NULL in multiple columns. #SPARK #SCALA #BIGDATA Karthik K. def main(args:Array[String]):Unit = { val ss = SparkSession.builder.master("local[*]").appName("DFDEMO").getOrCreate() import ss.implicits._ val df_1 = ss.read.format("csv").option("header",true). option("path","C:/Users/hp/COMPANY.txt").load() df_1.show() val df_2 = df_1.select(col("PRODUCT_NAME"),col("PRODUCT_ID") , (when(col("PRODUCT_ID").isNull,0).otherwise(col("PRODUCT_ID")).as("LEGIT_PRODUCT_ID")), when(col("PRODUCT_NAME").isNull,"UNKNOWN COMPANY"). otherwise(col("PRODUCT_NAME")).as("LEGIT_PRODUCT_NAME")) df_2.show() OUTPUT:

٢٦

٤ تعليق

How to Find Datasets and Codes| datasets aur codes kesay dhonday ja sakte hain? https://lnkd.in/dd-qPby8 #PythonAndML#DataSets #AcademicPapers #MachineLearningProjects #DataScience #ResearchAndAnalysis

٣