Rituraj Achuthan

Interesting read on Databricks Fundamentals. Delta Lake and Photon are two pivotal technologies within the Databricks ecosystem, revolutionizing the way organizations manage and analyze their data. 1. Delta Lake - Open-source storage layer that brings ACID Atomicity, Consistency, Isolation, Durability) transactions to Apache Spark and big data workloads. It provides reliability, performance, and data management capabilities for big data analytics. Other Key Features in Delta Lake - Schema enforcement prevents users from accidentally polluting their tables with mistakes or garbage data. - Schema Evolution - Delta Lake supports schema evolution, allowing schemas to evolve over time while maintaining backward compatibility. This enables flexibility in data ingestion and schema management, reducing development overhead and improving productivity. 2. Photon: Photon is a distributed query engine optimized for Databricks Delta Lake. It accelerates query performance and improves resource utilization for analytics workloads by leveraging advanced query optimization techniques and in-memory processing. https://lnkd.in/gpxTGEDm

7

Efficient data pipelines are crucial for unlocking insights and driving innovation. They ensure secure and efficient data ingestion, clean and standardize data, validate data quality, and curate data for analysis. This process transforms raw data into valuable insights, enabling better decision-making and fostering innovation.

9

"If it works, don't touch it" is the golden rule for Data Infrastructure Engineers. I was hosting DhanvantriAI without any issues until I had a bold idea to deploy my own Kubernetes cluster on some budget servers and move Dhanvantri over to it. I successfully set up Kubernetes, but I ran into trouble getting the SSL certificate to work for the site. Feeling overly confident, I ended up breaking the existing Dhanvantri cluster, rendering the app unavailable. Luckily, after receiving a **gentle nudge from my customer, I realized the impact of my actions and was able to reset everything back to normal. So the lesson is: "If it works, you can touch it, but always with a solid backup plan 😂 "

25

5 Comments

Sharing some insights from this thought-provoking article by Ganes Kesari on building a data-driven culture! Key takeaways include: 1) the critical importance of aligning data initiatives with business objectives, 2) mastering the art of storytelling to drive adoption, 3) and defining success metrics early on for sustained success. Kesari's emphasis on executive support, grassroots initiatives, and meticulous execution resonates deeply with leaders navigating the data landscape. Let's leverage these insights to cultivate thriving data-driven cultures within our organisations. #DataDrivenLeadership #BusinessStrategy #LearningFromData

8

With Large Data Volumes and Aggressive SLAs, building low latency Cost Optimum, Declarative and continuous Data Pipelines is need of the hour. Here's my latest article on Snowflake Dynamic Tables. #delcarativepipeline #continouspipeline #dynamicrefresh Please feel free to like, Subscribe and Share my article and Newsletter.

3

Cool #Snowpark #Python use case : Extracting data from a set of files with different file formats & content layouts using a single #Python UDTF function by dynamically injecting different Python code snippets into the function during run-time. 1. Build a mapping table that contains different Python code snippets per file format/type to read the file & extract the required data values. 2. Build a Python UDTF function that the code for file.open() and data.extract() functions are fetched dynamically at run time. 3. Combine the two and have Snowflake execute it in a distributed manner to process large amounts of files with different formats & content layouts. Read for more details & sample code: https://lnkd.in/euNkq3Fb

128

3 Comments

Congrats on the insightful article on Gen AI and GraphRAG by Maruti C and Michael Hunger ! ! Their thorough explanation offers valuable perspectives on this adanavnce technology. Great job on publsihing this , Read the article here:

4

1 Comment

❄ Discover seamless collaboration: My latest Medium blog unveils the power of integrating snowflake with Git. This is my first blog and I am inspired by Somen Swain Sachin Mittal Madhivanan Anbalagan blogs and content. Share your feedback and additional details about this feature ❄

56

4 Comments

🚀 TableRAG: A New Tool for Large-Scale Table Analysis Processing large tables—like those in RBI circulars—has always been challenging. Context limits and manual effort make extracting accurate insights time-consuming. Let’s take a query: “What were the key interest rates between 2019 and 2021?” 🔍 Manual review or partial automation often led to missed details or incomplete answers. Extracting the tables and flattening them, irrespective of structure, was a daunting and error-prone task. ⚡ TableRAG uses schema retrieval and cell retrieval to target only the necessary parts of a large table. 2 important retrievals: 💡 Schema Retrieval: It identifies the key columns (e.g., 'Year', 'Repo Rate') by understanding the table structure and picking out relevant metadata like column names and data types. 💡 Cell Retrieval: It further narrows down to the exact values that matter for the query (e.g., '5.15%', '4%') by scanning the table contents for precise matches. By combining these retrievals, TableRAG ensures only the critical data is processed, minimizing token usage while retaining context. I am very hopeful that it will change the way we deal a document during RAG set-up. #advancedanalytics #deeplearning #innovation #genai

20

1 Comment

People, if your data pipelines are as precarious as this then spend some time tackling those, they will come back and haunt you. #dataengineer #datawarehouse #datapipeline #dataleaders

17

5 Comments

🚀 Exciting Highlights from Snowflake BUILD Bangalore Dec 19th 2024! 🚀 Attending the Snowflake BUILD event on December 19th was an amazing experience, packed with insights into the latest advancements and powerful features of Snowflake. Here are some of the top new features and key learnings from the event: 1. Apache Polaris - Catalog Revolutionizing how we manage and access data with an open table format and REST Catalog for seamless integration. 2. Storage Enhancements Optionally archiving tables to optimize storage costs and manage data retention effectively. 3. Document AI - Copy INTO Simplifying data ingestion with new capabilities for handling document-based data. 4. Snowflake Horizon Catalog Enabling unified governance processes, providing full visibility across your data ecosystem. 5. Snowflake Notebooks A powerful tool for collaborative data science and analytics directly within Snowflake. 6. Observability: Generative AI & ML New capabilities for deep insights, leveraging AI and ML for enhanced observability. 7. SPCS - Snowpark Container Services Enabling more flexible and efficient data processing with containerized workloads. 8. Internal Marketplace - GET Button Simplifying access to shared data and services with a unified internal marketplace. 9. Egress Cost Optimizer Optimizing costs associated with data egress to improve operational efficiency. 10. Share Fine-Tuned LLMs Empowering users to share their fine-tuned large language models (LLMs) for advanced analytics. 11. Monetize AI Apps Enabling the creation and monetization of AI-powered applications for business growth. 12. Hybrid Tables & Unistore Solving concurrency challenges and enhancing performance with hybrid storage and compute solutions. 13. MODIN - Scalable Pandas Scalable, distributed computing for Pandas users, now leveraging RAY, DASK and SNOWPARK environments. 14. Altair Library for Visualization Bringing advanced visualization capabilities to Snowflake for data-driven insights. 15. Snowflake DevOps Capabilities Enhanced DevOps features with capabilities like CREATE OR ALTER, EXECUTE IMMEDIATE, Snowflake CLI, GIT Integration, Python APIs, and more. These innovations are taking data management, governance, and analytics to the next level. The future of data is truly exciting with Snowflake's cutting-edge solutions! 🌐 #Snowflake #SnowflakeBUILD #DataInnovation #AI #MachineLearning #DataGovernance #DAP #Ericsson #DAPPLATFORM

48

1 Comment

𝐔𝐧𝐥𝐨𝐜𝐤𝐢𝐧𝐠 𝐄𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲 𝐚𝐧𝐝 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 𝐰𝐢𝐭𝐡 𝐒𝐩𝐚𝐫𝐤 𝐉𝐨𝐛𝐬 𝐨𝐧 𝐃𝐚𝐭𝐚𝐛𝐫𝐢𝐜𝐤𝐬 🚀 As organizations delve deeper into data driven decision making, the efficiency and performance of Spark jobs hosted on platforms like Databricks become crucial factors in driving success. Optimizing Spark jobs ensures not only faster execution but also cost-effectiveness and resource utilization. 💰📊 There are several recommendations as listed below that help to improve the efficiency but they cannot be applied in every scenario. 📍 It's very important to understand when and where to apply these strategies to reap the real benefits and this document has been very helpful to me to build the cost and resource efficient data pipelines on Databricks. I hope you find it useful as well. 👉 Deletion Vectors 👉 Join Strategies 👉 DBR and Photon engine 👉 Partitions 👉 File Sizes 👉 Zorder 👉 Stats and many more..... Please let me know the strategies you use to performance tune etl jobs through comments. Document credits: Databricks #databricks #sql #dataengineer #cloud #datalake #analytics #spark #optimization #efficiency #engineer #bigdata

2

🎇 Light Up Your Data Insights this Diwali with BigQuery & Vertex AI Agent Builder! 🎇 Tired of fumbling in the dark to find insights in your BigQuery data warehouse 🕶? What if you could search your massive BigQuery datasets as easily as Googling 🤔? This Diwali, illuminate your data with the power of Vertex AI Search 🚀! Dive into my latest blog post to learn how to build a powerful, AI-driven search application for the 'tourism' use case. Link: https://lnkd.in/grt36_fU #Vertex_AI hashtag#Search hashtag#BigQuery Google Cloud

1

Snowflake Optimization Tip of the Day #5 ➡ Use incremental models in dbt There is no simpler way to reduce Snowflake cost than to process less data. If you are using dbt, try to write every model incrementally. Instead of reprocessing all data every time, incremental models only process new or changed data since the last run. This is achieved primarily by using date stamps or other markers to identify what data has been added or modified since the last run. Full-refreshes are very expensive and should be avoided if possible.

8

Move over Messi vs Ronaldo The real debate in the data world is Snowflake vs Databricks. After Both CEO's post, we know the rivalry is intense now 😆 , both platforms driving innovation and transforming the way we handle big data. Which side are you on? #snowflake #databricks #dataengineering #techrivalry #ml #ai

67

7 Comments