Pratik Somaiya

Snowflake Optimization Tip of the Day #7 ➡ Identify and remove unused Storage Storage is generally very cheap in Snowflake but unused tables can amount to a significant redundant cost. Snowflake provides the access_history view that parses each of your SQL queries to identify every access of a particular table. This table can be used to identify unused storage efficiently. The link to a technical blog to identify these is in the first comment. If you have a view on top of a table and you only access the view, it will be an indirect access for the table, Snowflake handles those parts as well to provide info that is actually relevant.

9

2 Comments

🌟 Exciting News for Snowflake Enthusiasts in Hyderabad! 🌟 The #Snowflake Hyderabad User Group is back with another amazing in-person meetup, hosted by Snowflake. Join us on December 07, 2024, from 9:00 AM to 1:00 PM, for a hands-on workshop led by Satish T, Cloud Data Architect at Infocepts Pvt Ltd and Snowflake Data SuperHero. In this workshop, you’ll learn how to: 🔍 Translate multilingual customer reviews 📊 Categorize and analyze unstructured review data at scale ⭐ Extract ratings and sentiments from reviews 💡 Identify customer issues and take action with LLM assistance 💻 What You’ll Build: We’ll analyze Tasty Bytes customer reviews using Snowflake Cortex within Snowflake Notebooks to discover: ✅ How likely customers are to recommend Tasty Bytes ✅ Sentiments around food, pricing, and other aspects ✅ Key issues customers highlight and how to address them 🎯 Prerequisites: Familiarity with Python and DataFrame API Experience with Snowflake and Snowpark A Snowflake account with access to Cortex LLM functions/models and Snowflake Notebooks. Don’t miss this opportunity to level up your skills with Snowflake Cortex. #Snowflake #LLM #dataengineer #Cortex #dbt #genai

12

Databricks Types of Cluster There may be a questions around 1.Which type of clusters have you worked on? 2.What are all the types of clusters in Databricks? 3.Explain how you would create a cluster for running your notebook. These questions aim to assess your familiarity with the different types of clusters in Databricks. Here is the answer is There are two types of clusters are there 1.Job Cluster 2.All purpose cluster Job Clusters run automated jobs in an expeditious and robust way.The scheduler creats a cluster and runs the job after teinates once the job is complete. we cannot restart job Clusters All purpose clusters analyse data collaboratively using interactive notebooks.We can terminate and restart all purpose cluster. multiple users can share the clusters and do work collaboratively. #databricks #clusters #jobcluster #allpurposecluster #dataengineering #dataengineer

10

Working with multiple Azure Databricks workspaces across different environments like Production, Acceptance, and Development can be challenging. As we manage various domains and environments (e.g., Domain A - PRD, Domain A - ACC, Domain A - DEV, Domain B - PRD, Etc.,....), navigating between these workspaces efficiently becomes crucial. Currently, one of the pain points we face is the lack of a search option to quickly locate the required workspace from the list. Every time, we find ourselves scrolling through the workspace list, which can be time-consuming and inefficient. To streamline this process and improve productivity, it would be incredibly beneficial to have a search box feature enabled within Azure Databricks. This enhancement would allow users to easily search for specific workspace names, significantly reducing the time spent manually locating them. Let's advocate for user-centric improvements that enhance our experience with Azure Databricks! 🚀 #Azure #Databricks #UserExperience #Productivity #CloudComputing #AzureDevOps #TechInnovation #AzureDatabricks

37

2 Comments

Enter the ultimate face-off: Apache Spark vs. Apache Hadoop. These two powerful tools serve distinct purposes.                Know more ......          👇👇👇👇👇👇👇👇👇👇👇👇👇 https://lnkd.in/gYAFCM7b #Apachespark #ApacheHadoop #DataProcessing #RealTimeAnalytics #BatchProcessing #BigDataTools #DataAnalytics #CloudComputing #DataStorageSolutions #MachineLearning #DataArchitecture

2

"If it works, don't touch it" is the golden rule for Data Infrastructure Engineers. I was hosting DhanvantriAI without any issues until I had a bold idea to deploy my own Kubernetes cluster on some budget servers and move Dhanvantri over to it. I successfully set up Kubernetes, but I ran into trouble getting the SSL certificate to work for the site. Feeling overly confident, I ended up breaking the existing Dhanvantri cluster, rendering the app unavailable. Luckily, after receiving a **gentle nudge from my customer, I realized the impact of my actions and was able to reset everything back to normal. So the lesson is: "If it works, you can touch it, but always with a solid backup plan 😂 "

25

5 Comments

Hello LinkedIn community! 🌟 Recently, I joined Deepak Goyal's latest batch where I'm diving deeper into Azure concepts. Here is a brief overview of the components needed for an effective and low-cost end-to-end Azure data engineering solution: To build a robust and cost-efficient data engineering pipeline on Azure, you need to leverage several key components: ✨ Azure Data Lake Storage (ADLS): This is your go-to for scalable and secure data storage. It handles massive amounts of data and integrates seamlessly with other Azure services. 🔄 Azure Data Factory (ADF): ADF is essential for orchestrating and automating data movement and transformation. It allows you to create data-driven workflows that can ingest data from various sources and transform it into usable information. 💡 Azure Databricks: This is where the magic happens for data processing and analytics. Databricks provides a collaborative environment for data engineers and data scientists to work together on big data and AI projects. 📊 Azure Synapse Analytics: For advanced analytics, Synapse Analytics offers a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. 🗄️ Azure SQL Database: This is ideal for structured data storage and provides high availability, scalability, and security for your relational data. 📈 Power BI: Finally, for data visualization and reporting, Power BI helps you turn your data into actionable insights with interactive dashboards and reports. By combining these components, you can create a scalable, secure, and cost-effective data engineering pipeline that meets your business needs. A big shoutout to Deepak Goyal for his guidance and insights in the Azure Data Engineering community. 🙏 #AzureDataEngineering #DataEngineering #Azure #LearningJourney #ADLS #BigData #AzureDataFactory #AzureDatabricks #AzureSynapseAnalytics #PowerBI Feel free to share your thoughts and experiences! 🚀

55

2 Comments

🚀 Unlocking the Power of Automation with Azure and Databricks! 🚀 Harnessing the capabilities of Azure and Databricks to streamline and automate data workflows like never before. Check out this Automation Workflow that leverages the best of both platforms for efficient data processing and management. 🔹 Raw Data Ingestion: Start by loading data from various sources like CSV files into the raw layer using Azure Data Factory. 🔹 Data Staging: Transition the raw data into the staging area with Databricks, where initial transformations and validations occur. 🔹 Data Cleaning: Utilize Databricks to clean and refine the data, ensuring high-quality datasets for analysis. 🔹 SQL Integration: Store the cleaned data into Azure SQL Database for structured querying and reporting. 🔹 Final Transformation: Apply additional transformations and prepare the data for visualization and advanced analytics. 🔹 Data Visualization: Leverage tools like Power BI to create insightful and actionable dashboards. By automating these processes, we ensure data accuracy, efficiency, and scalability, empowering businesses to make data-driven decisions with confidence. #Azure #Databricks #DataAutomation #DataScience #BigData #CloudComputing #PowerBI #DataFactory #DataTransformation

7

Data Engineering: Mastering Apache Spark Executor Memory Configuration! 🔧 Key Configurations and Their Impact: ➡️ spark.executor.memory (8GB): Allocates memory to the JVM heap for each executor. Finding the right balance here is crucial for optimal performance. ➡️ spark.memory.fraction (0.6): Determines the split between execution and storage memory. Adjusting this significantly affects shuffles, joins, and aggregations. ➡️ spark.memory.storageFraction (0.5): Divides memory for caching and execution within the allocated fraction. Fine-tuning this improved data caching efficiency and reduced disk spills. ➡️ spark.executor.memoryOverhead (800MB): Provides extra memory for Spark's internal operations and non-heap memory usage, preventing out-of-memory errors during intensive processing. 🔍 Insights and Learnings: ➡️ Proper configuration can drastically enhance Spark's performance, reduce job execution time, and improve resource efficiency. ➡️ Balancing storage and execution memory is crucial for caching RDDs, DataFrames, and performing complex transformations.

16

When creating a search index with Databricks Vector Search, always keep in mind you own your data and have controlled access to any transformations or enhancements. By choosing to compute your embeddings using an embedding model registered in Model Serving, you can sync your embeddings to a writeback table in the same database where your indexed table and source table exist (vector search indexed tables are triggered or continuous DLTs). This writeback table contains your original primary key, embedding source column, and a new column - called db_<source_column>_vector that contains your embeddings. If (and when 😉) Unity Catalog is enabled on your workspace, you can then share these embeddings across different data teams. Pretty awesome!

19

🔍 Understanding the Difference: `DELETE TABLE` vs. `VACUUM TABLE` in Databricks In the world of data management, especially within platforms like Databricks, it's crucial to grasp the nuances between various commands to ensure efficient data handling. Today, let's explore the differences between the `DELETE TABLE` and `VACUUM TABLE` commands, and understand their respective advantages. # 🗑️ `DELETE TABLE` Command The `DELETE TABLE` command is used to remove an entire table from the database. This command is typically employed when you want to entirely discard a table and its data from your database. Advantages: - Complete Removal: It provides a straightforward way to eliminate both the table schema and its data. - Clearing Unused Data: Useful for clearing out old or unused tables to free up resources. Example: ```sql DELETE TABLE my_table; ``` This command will delete the table named `my_table` along with all its data. # 🧹 `VACUUM TABLE` Command On the other hand, the `VACUUM TABLE` command is used to clean up the storage by removing old versions of data files that are no longer needed, usually after a `DELETE` or `UPDATE` operation. This is essential for optimizing storage and performance. Advantages: - Optimizes Storage: Frees up disk space by cleaning up obsolete files. - mproves Query Performance: Reduces the overhead of reading unnecessary data files, which can enhance query performance. - Maintains Table Health: Regular vacuuming helps in maintaining the overall health and efficiency of the table. Example: ```sql VACUUM TABLE my_table; This command will clean up the old data files and optimize storage for the table named `my_table`. #🏆 When to Use Each? - Use `DELETE TABLE` when you need to completely remove a table and its data. - Use `VACUUM TABLE` to reclaim storage and maintain table performance after data modifications. Understanding and utilizing these commands appropriately can lead to more efficient data management and optimized performance in your data workflows on Databricks. 🚀 #DataEngineering #Databricks #SQL #DataManagement #BigData #DataOptimization If you find this useful, please repost! 🌟 I share daily insights on Data Engineering, Databricks, Azure, Python, and SQL. Interested in connecting on any topic? Find me here: https://lnkd.in/dr_NjkJy follow Satish Mandale for more such contents

4

#snowflake #ddl #storedprocedure #audit Recently my colleague Aman Gupta and I were discussing a challenge that we were facing in our project: How can we track DDL (Data Definition Language) changes made to tables in Snowflake?. We needed a way to monitor schema modifications, not just to capture what changed but also when those changes occurred. Our goal was to create a solution that would allow us to log changes such as adding or dropping columns, along with the exact dates these changes were made. Essentially, we wanted an audit trail that could show, for example, if Table A was altered on the 1st day, and then modified again on the 2nd day, detailing each operation (column addition or deletion) along the way. This would give us visibility into the structural evolution of our tables over time and help with both auditing and ensuring data consistency. Snowflake doesn’t offer an out-of-the-box solution for logging DDL changes in this way. This led us to brainstorm how we could develop a custom solution to monitor and log DDL modifications to ensure data integrity and support auditing needs. To address this, We both implemented a DDL auditing solution using a Snowflake stored procedure and schema snapshots. Please have a look and it can be an accelerator for ACCOUNTADMIN or DBA in snowflake. To get more details please click here: https://lnkd.in/gMX7CTrb Tagging: Vishal Kaushal,Somen Swain,Abhijit Bangal,Divyansh Saxena,Lav Pandey,Karthik Raman

394

28 Comments

🚀 Happy to share another achievement from Databricks: "#Advanced_Data_Engineering_with_Databricks Certification" 🚀 Key highlights of this certification: #Incremental_Data_Processing: Emphasizes designs favoring incremental data processing, enabling systems to continuously ingest and analyze growing datasets. #Platform_Optimizations: Leverages built-in optimizations to reduce the burden of code maintenance and on-call emergencies. #Adaptability: Quickly adapt production code to new demands with minimal refactoring or downtime. #NeverStopLearning #Databricks #DataEngineering #ApacheSpark #StructuredStreaming #DeltaLake #DataLakehouse #ContinuousLearning #DATA #AZURE

17

8 Comments

#What is ForEach Activity in Azure Data Factory? The ForEach Activity defines a repeating control flow in an Azure Data Factory or Synapse pipeline. This activity is used to iterate over a collection and executes specified activities in a loop. The loop implementation of this activity is similar to Foreach looping structure in programming languages. #dataanalyst #dataengineering #Azure #ADF

7

PySpark interview questions for a data engineering role 👇 📍 What is PySpark, and how does it differ from Apache Spark? 📍 Explain the architecture of PySpark. 📍 How do you initialize a SparkSession in PySpark? 📍 What are RDDs in PySpark, and how do they work? 📍 What are the transformations and actions available in RDDs? 📍 How do you create RDDs in PySpark? 📍 What are DataFrames in PySpark, and how do they differ from RDDs? 📍 How do you perform operations on DataFrames? 📍 Explain the concept of a Dataset in PySpark. 📍 How do you work with SparkSQL in PySpark? 📍 Explain how to perform SQL operations on DataFrames. 📍 What are the advantages of using SparkSQL? 📍 How do you read and write data in various formats (CSV, JSON, Parquet) using PySpark? 📍 Explain the process of handling schema evolution in PySpark. 📍 What is Catalyst Optimizer in PySpark? 📍 How do you optimize a PySpark application? 📍 Explain the concept of lazy evaluation in PySpark. 📍 What is the significance of caching and persistence in PySpark? 📍 How do you handle skewed data in PySpark? 📍 What are broadcast variables and accumulators in PySpark? 🚨 To learn more, Join my high quality, industry & modern tech stack based and practical project driven Data Engineering BootCAMP✌🏻👇 👉 Enroll Here - https://lnkd.in/dRgYCmWd 🚀 Dedicated Doubt Support & Placement Assistance 📲 Call/WhatsApp for any query (+91) 9893181542 Grow Data Skills Shashank Mishra 🇮🇳 Sahil Choudhary Shubhankit Sirvaiya Aman Kumar

22

1 Comment

Serverless compute in Databricks simplifies infrastructure management by abstracting away server configurations and automatically scaling resources based on workload demands. It offers cost efficiency with a pay-as-you-go model, where you only pay for the resources you use. This approach enhances productivity by reducing setup time and allowing users to focus on their data tasks. Additionally, it provides scalability, high availability, and improved performance, while seamlessly integrating with modern data architectures. Security and compliance are also managed effectively, ensuring data protection and regulatory adherence. Kaniz Fatma Sujitha Ramamoorthy Databricks

38

Deepak Mahto: Conversion Chronicles – Invisible Indexes in Oracle to PostgreSQL Migration. 🔍 Deepak Mahto: Conversion Chronicles – Invisible Indexes in Oracle to PostgreSQL Migration. Are you migrating from Oracle to PostgreSQL and looking for a way to make indexes invisible? Look no further! Deepak Mahto has shared an insightful article on how to achieve this functionality in PostgreSQL. In Oracle, making indexes invisible has a global effect, but in PostgreSQL, it's not officially supported. However, with the HypoPG extension, we can create or mark existing indexes as invisible for the current session. It's like having stealth mode for your indexes! By leveraging the hypopg_hide_index function, you can make an index invisible and observe the changes in the execution plan using EXPLAIN. Just remember, the invisibility effect is limited to the current session and application, so you won't be disappearing from other connections. Deepak Mahto's article provides a step-by-step guide and sample code to help PostgreSQL geeks navigate this migration challenge. Give it a read and level up your PostgreSQL game! #PostgreSQL #MigrationTips #InvisibleIndexes #HypoPG [Add relevant hashtags and make sure to tag Deepak Mahto if available] You can find the full article here: https://postgr.es/p/6uT

1