Rituraj Achuthan

Optimise your data across the entire value chain with Databricks and DVT DVT, a certified partner in Databricks services, offers data engineering expertise crucial for the effective implementation and ROI maximisation of your Databricks projects. For more information - https://lnkd.in/dttES4bW. Karen Heydenrych Dantes de Beer #retail #dataengineering #data #ai #innovation

4

Difference between DAG & Lineage Graph #SPARK #BIGDATA Karthik K. Lineage Graph: 👉 It represents the dependencies between RDDs (Resilient Distributed Datasets) or DataFrames in a Spark application.👈 👉 Basically, it's a logical plan which tells which transformation we need to apply to create an RDD. 👈 RDD1 --> RDD2 --> RDD3 In the above flow, it's pretty clear that RDD2 gets created by RDD1 and RDD3 gets created by RDD2. 👉 The lineage graph represents the sequence of transformations applied to the original input data to produce a new RDD or DataFrame/Dataset. 👈 👉 By maintaining the lineage information, Spark ensures that it can re-compute data without having to store intermediate results, which helps to save memory and improve performance. 👈 👉👉 Lineage doesn’t have any action defined. It’s a pure transformation tracker. Direct Acyclic Graph(DAG): - The lineage in Apache Spark is represented as an acyclic graph of stages. The term acyclic means they do not contain cycles or loops. This property ensures that the computation can be efficiently parallelized and optimized. 👉 It is a set of Vertices and Edges, where vertices represent the RDDs and the edges represent the Operation to be applied on RDD. 👈 - When multiple transformations are applied to an RDD the resulting logical plan is formed. This lineage captures the sequence of transformations applied to the original data. 👉 When an Action operation is encountered the logical plan is submitted to the Catalyst Optimizer for further optimization and execution planning.👈 Logical Plan --> Catalyst Optimizer --> DAG Schedular --> Physical Plan(DAG) - The Catalyst Optimizer in Spark applies its inherent algorithms to produce the finalized plan, which is then handed over to the DAG Scheduler. The role of the DAG Scheduler is to generate the Physical Plan in the form of a Directed Acyclic Graph (DAG) splitting the execution process into stages. - DAG is very helpful in execution point of view and also tells us stage dependencies over each other.

48

2 Comments

🚀 Exciting day at Snowflake Data Cloud World Tour Mumbai 2024! ❄️ I had the privilege of attending the Snowflake Data Cloud World Tour in Mumbai, and here are some key takeaways from insightful sessions on the future of data and AI: 📊 Open Table Formats & Iceberg Support Exploring integration possibilities with Apache Iceberg and Polaris Discussions on ACID compliance, performance optimization, and data evolution 🛡️ Snowflake Horizon Unveiling a vision for unified data governance and security across clouds Concepts for simplifying management of data and apps from a centralized interface 🤖 Generative AI & LLMs Introduction to the Snowflake Arctic family of Large Language Models (LLMs) Exploring Snowflake Cortex as a pathway to democratize AI Emphasis on the "AI to Data, not Data to AI" approach for enterprises 🔒 Data Governance & Security Highlighting the importance of unified security and compliance measures Showcasing potential applications of Dynamic Data Masking for data protection 💻 Snowpark Container Service & ML API Demonstrations of running AI and full-stack apps within the Snowflake ecosystem Exploring possibilities for direct ML model execution and predictions 🔗 Native Apps & Data Sharing Discussing streamlined approaches to building and sharing data applications Insights into enhancing operational collaborations through data sharing 📄 Document AI & Unstructured Data Previewing tools aimed at extracting insights from unstructured data The event showcased Snowflake's vision for innovation, open standards, and efficient data management. It was fascinating to see the proposed solutions for various data challenges, from governance to AI implementation. Kudos to the speakers, including Sanjay Deshmukh Vijayant Rai Rahul Jain 🇮🇳 Kamesh Sampath and Sumeet Tandure Excited to contemplate how these concepts might shape our future data strategies! #SnowflakeWorldTour #DataCloud #AI #DataGovernance #Innovation

51

2 Comments

Converting a list to string, with a delimiter. Implementation of inbuilt mkString() function in SCALA. #SCALA #SCALADEVELOPER #BIGDATADEVELOPER #BIGDATA Karthik K. def main(args: Array[String]): Unit ={ //var a = "Tanuj" //lowerCase(a) var s = "" var l = List(1,2,3,4,5,6,7,8) for( i <- 0 until l.length) { if( i == l.length -1) { s = s.concat(l(i).toString) } else { s = s.concat(l(i).toString) s = s.concat(':'.toString) } } print("The elements of list when converted to string are:-" + s) } } OUTPUT:

2

Snowflake's AI Data Cloud revolutionizes data management and AI innovation. A unified, fully managed platform empowers organizations to break silos, enhance quality, and accelerate decision-making enterprise-wide.   Next Pathway CEO Chetan Mathur delves into how migrating legacy assets to Snowflake's Data Cloud unlocks AI's full potential. Supporting structured, semi-structured, and unstructured data in native formats enables comprehensive analytics and insights driving business growth.   As global leaders in Snowflake migrations, our SHIFT Product Platform accelerates the path to AI-driven data transformation. Our automated solutions streamline complex legacy data migration processes, ensuring swift and efficient adoption of Snowflake's powerful ecosystem.   Discover how Next Pathway and Snowflake shape cloud migration and AI innovation futures. Read full article: https://bit.ly/3xDHura   #NextPathway #SHIFTCloud #CloudAutomation #CloudMigration #Snowflake #Onelake #DataFactory #Informatica #ETL #AI #Aiautomation #GenerativeAI #GenAI #DataCloudSummit #snowflakesummit #TheDataCloud

8

𝗪𝗵𝗮𝘁 𝗺𝗮𝗸𝗲𝘀 𝗮 𝗹𝗲𝗮𝗱𝗲𝗿 𝘁𝗿𝘂𝗹𝘆 𝗶𝗻𝘀𝗽𝗶𝗿𝗶𝗻𝗴? For Veena Bramhakal, it’s a combination of 𝗲𝗺𝗽𝗮𝘁𝗵𝘆, 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻, and a 𝗿𝗲𝗹𝗲𝗻𝘁𝗹𝗲𝘀𝘀 𝗳𝗼𝗰𝘂𝘀 on people. When Veena joined Fractal in 2018, she didn’t just take on a job - she took on a mission. 𝗛𝗲𝗿 𝗴𝗼𝗮𝗹? To transform the 𝘁𝗮𝗹𝗲𝗻𝘁 𝘀𝘂𝗽𝗽𝗹𝘆 𝗰𝗵𝗮𝗶𝗻 into a powerhouse that drives 𝗴𝗿𝗼𝘄𝘁𝗵 while honoring Fractal’s ‘People over Process’ philosophy. Five years later, that mission earned Fractal the 𝗚𝗹𝗼𝗯𝗮𝗹 𝗜𝗻𝗱𝘂𝘀𝘁𝗿𝘆 𝗘𝘅𝗰𝗲𝗹𝗹𝗲𝗻𝗰𝗲 𝗔𝘄𝗮𝗿𝗱 for ‘Best People-Centric Workforce Management Practices.’ 𝗛𝗲𝗿 𝘀𝗲𝗰𝗿𝗲𝘁? While data insights and automation drive 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆, Veena maintains that 𝗵𝘂𝗺𝗮𝗻 𝗰𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻 remains the essential catalyst. And, technology is not about replacing people, it’s about 𝗲𝗺𝗽𝗼𝘄𝗲𝗿𝗶𝗻𝗴 them. With 20+ years of experience, she’s done more than lead; she’s 𝗶𝗻𝘀𝗽𝗶𝗿𝗲𝗱. Whether it’s helping young professionals chase their dreams or supporting new mothers returning to work, Veena’s leadership proves one timeless truth: 𝗴𝗿𝗲𝗮𝘁 𝗼𝗿𝗴𝗮𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝘀 𝗮𝗿𝗲 𝗯𝘂𝗶𝗹𝘁 𝗼𝗻 𝗴𝗿𝗲𝗮𝘁 𝗽𝗲𝗼𝗽𝗹𝗲. Here’s to Veena - a leader who shows us what it truly means to 𝗰𝗮𝗿𝗲. #PeopleFirst #WorkforceExcellence #EmpathyInAction #TalentManagement #WomenInLeadership #FutureOfWork #Care

180

19 Comments

Next Pathway CEO Chetan Mathur explores how Snowflake is enabling data democratization in his latest article. Snowflake's cloud-based platform unifies diverse data sources, breaking down silos and enabling seamless organization-wide access and analysis. Its intuitive interface empowers users with self-service analytics, fostering a data-driven culture across departments. Snowflake efficiently handles AI and ML workloads, providing a centralized repository for AI-ready data. As a leader in Snowflake migrations, Next Pathway leverages its AI-enabled SHIFT Product Platform to streamline the transition of legacy systems to Snowflake's Data Cloud, accelerating innovation and data-driven decision-making. Read the full article here: https://bit.ly/3xDHura #NextPathway #SHIFTCloud #CloudAutomation #CloudMigration #Snowflake #Onelake #DataFactory #AzureSynapes #Informatica #ETL #AI #Aiautomation #GenerativeAI

7

Understanding Narrow and Wide Transformations in Apache Spark In Apache Spark, transformations play a critical role in defining how data is processed across distributed clusters. These transformations can be broadly classified into narrow transformations and wide transformations, each influencing the efficiency and data flow within Spark applications. 1. Narrow Transformations: A narrow transformation is one in which each output partition depends only on a single input partition. Since the data required to generate an output partition resides in the same partition as the input data, Spark can process these transformations without requiring data to move across nodes. This minimizes network traffic and improves performance. Examples of Narrow Transformations: • map(): Applies a function to each element of the RDD/DataFrame without requiring data exchange between partitions. • filter(): Selects elements based on a condition, retaining them within the same partition. • flatMap(): Similar to map(), but can generate multiple elements from a single input element, still preserving the partition boundaries. 2. Wide Transformations: Wide transformations involve operations where the output partition depends on data from multiple input partitions, often triggering a shuffle—a costly operation where data is exchanged across the cluster. These transformations require coordination between nodes, leading to network I/O, which can impact performance. Examples of Wide Transformations: • groupByKey(): Groups data by key, causing a shuffle as all values associated with the same key need to be aggregated across partitions. • reduceByKey(): Similar to groupByKey(), but applies a reducing function during the shuffle phase to optimize data movement. • join(): Combines data from two datasets based on a key, triggering a shuffle to align matching keys across partitions. Key Difference: Data Shuffling The fundamental difference between narrow and wide transformations lies in data shuffling. In narrow transformations, Spark processes data locally without any network overhead. However, in wide transformations, the data must be reshuffled across partitions, leading to increased computational and network costs. For example, a map() operation applies functions independently within partitions, while a reduceByKey() requires the data to be shuffled so that values for the same key are processed together. Follow me(Raghu Raman A V) for more articles on Big Data, Data Engineering, Azure and cloud ! #BigData #ApacheSpark #DataEngineering

15

1 Comment

Types of Indexing in Apache Hudi Index TypeDescriptionUse CasesBloom IndexUtilizes Bloom filters to quickly determine the potential presence of a record in a file group, reducing the search space during upserts and deletes.- High Throughput Upserts/Deletes: Ideal for scenarios with frequent data modifications. - Large Datasets: Efficiently handles big data volumes by minimizing file scans.Global Bloom IndexAn extension of the Bloom Index that maintains Bloom filters across all file groups globally, enhancing lookup performance for massive datasets.- Massive Scale Operations: Suitable for extremely large datasets where maintaining global state significantly improves performance. - Distributed Environments: Enhances consistency and speed across distributed file groups.HBase IndexLeverages HBase to store and manage index data externally, providing low-latency record lookups and high scalability.- Low-Latency Requirements: Perfect for applications needing rapid access to specific records. - External Index Management: When integrating with HBase ecosystems for robust indexing solutions.Simple IndexA non-index-based approach that scans all file groups to locate records. Acts as a fallback when other indexing mechanisms are not enabled or suitable.- Fallback Mechanism: Ensures data operations can proceed even without specialized indexing. - Smaller Datasets: Acceptable for datasets where full scans do not significantly impact performance.In-Memory Hash IndexMaintains an in-memory hash table mapping record keys to their respective file paths, enabling quick lookups without disk I/O.- Fast Lookups: Best for applications requiring rapid access to records with minimal latency. - Memory-Abundant Environments: Suitable when ample memory resources are available to store the hash index.SQL Query Examples Using Apache Hudi Indexes Apache Hudi integrates seamlessly with Spark SQL, allowing you to specify indexing types during table creation or data write operations. Below are examples demonstrating how to leverage different indexing types in your SQL workflows. Example - Creating a Hudi Table with Bloom Index sql Copy code CREATE TABLE hudi_bloom_index_table ( id STRING, name STRING, age INT, city STRING ) USING hudi OPTIONS ( primaryKey = 'id', preCombineField = 'timestamp', hoodie.index.type = 'BLOOM' ); Explanation: hoodie.index.type = 'BLOOM': Specifies the use of Bloom Index for efficient record lookups during upserts and deletes.

6

The Future of AI and Data Management: Key Takeaways from Databricks 💡 Chris Marshall, Lead Architect at Optima Partners, recently attended the Databricks Data+AI World Tour in London, where AI and data management took centre stage. He shared key insights on: ➡ How generative AI is revolutionising complex processes and accelerating innovation. ➡ Tools like AI/BI Genie are simplifying access to valuable insights. ➡ Why open-source AI is affordable, powerful, and levelling the playing field for businesses. ➡ The growing focus on data governance and compliance in the age of AI. Read his full insights here: https://lnkd.in/eDUNcZ84 #Databricks2024 #AI #GenerativeAI #MachineLearning #OptimaPartners #Innovation #DataGovernance

31

3 Comments

Deploying a large language model like Llama-3-70b in production is very complex and demanding. Using Kubernetes ! 1. The sheer size of Llama-3-70b means that just getting it downloaded and ready to run can test your patience and your technology. 2. Choosing the right GPU isn’t just about memory and cost. It's a fine balance of availability, parallelization capabilities, and more. 3. Strategically managing GPU resources requires creating specific settings to balance performance with expense. 4. Just when you think you're set to launch, you hit a snag like the dreaded 'Cuda Error: torch.cuda.OutOfMemoryError'. 5. Finding a bigger GPU or tweaking Torch configurations to handle memory issues proves to be a narrow path with limited options. 6. Post-launch, the model runs so slowly you might find yourself switching to a quantized version for better speed. 7. The move to a quantized model sparks debates over its performance and efficiency. 8. Just as everything seems stable, a sudden increase in user requests can send you back to the drawing board. Additional considerations like logging, metrics, cost management, and caching also complicate deployment. Cloud services and other hosting options have not been satisfactory, particularly with the slow response times of Llama-3.

28

Snowflake Optimization Tip of the Day #5 ➡ Use incremental models in dbt There is no simpler way to reduce Snowflake cost than to process less data. If you are using dbt, try to write every model incrementally. Instead of reprocessing all data every time, incremental models only process new or changed data since the last run. This is achieved primarily by using date stamps or other markers to identify what data has been added or modified since the last run. Full-refreshes are very expensive and should be avoided if possible.

8

🦆 What makes a Great Data Leader? 🦆 Data Engineers London recently had the pleasure of hearing from Nik Walker, Head of Data Engineering at Co-op, who delivered an absolutely quack-tastic talk on the power of human-centric leadership for data teams. Nik certainly brought his learned own views, but not just that; he guided us through a thoughtful exploration of how empathy, trust, and open communication are the real engines driving successful teams. And yes, there were a few quacks along the way to keep things lively! 🦆😅 The best part? Nik’s insights aren’t just for those leading teams - there’s something valuable here for anyone working in data. Trust us, it’s worth a watch! Some of the highlights! 🎥 Human-Centric Leadership 🤝  People over technology. Understanding your team’s emotions and needs is key to tackling any challenge. Trust & Vulnerability 🚀  Effective leaders build trust by being open and showing vulnerability—this creates a space where teams feel empowered to innovate and take risks. Psychological Safety 💬  Safe spaces aren’t just buzzwords. They’re essential for fostering honest conversations, admitting mistakes, and driving innovation. Inclusivity 🌍  Valuing every voice on the team leads to richer ideas and solutions—diversity truly drives creativity. Continuous Learning 🧠  Never stop learning! Encouraging ongoing development keeps the team adaptable and engaged in a fast-changing field. And perhaps the most resonant takeaway? Servant Leadership 🏅 Prioritising the team’s needs above your own cultivates a loyal, motivated, and highly effective team. Check it out now! - https://lnkd.in/eN8d4rye Thanks to our sponsors Kyndryl for your support, and Zego for hosting us! And thanks from us to you all in the community! Make sure you join our community page for the next event! Anna Aleshko, Jamie Marney & Harry Duddridge #DataEngineering #Leadership #EmpathyInTech #Data #TeamCulture #ContinuousLearning #Humancentricleadership

8

3 Comments

Why Parquet Alone Is Not Enough: Iceberg & Hudi Explained with Example 🚀 🔍 Managing data lakes is hard. What happens when your datasets; - Evolve, - Need real-time updates, - or Require historical audits? Let’s explore this with a simple example and how Apache Iceberg and Apache Hudi enhance your data lake game. #Example: E-Commerce Sales Data Imagine you’re managing daily sales transactions for an e-commerce platform. #Goal’s: • Add a new column (campaign_id) to track promotions. • Debug a pricing error by querying historical data. • Update invalid records efficiently. #With_Parquet_Alone: 1. Adding campaign_id: You must rewrite all existing files, incurring time and cost. 2. Debugging historical data: You need manual backups or external versioning tools. 3. Updating invalid records: Entire files must be rewritten due to a lack of transactional support. #With_Iceberg_or_Hudi: 1. Adding campaign_id: Schema evolution allows you to add columns dynamically, without rewrites. 2. Debugging historical data: Use time travel to query the dataset before the error: SELECT * FROM sales_table FOR TIMESTAMP AS OF '2024-11-15'; 3. Updating invalid records: ACID transactions enable fast, reliable updates with simple SQL: MERGE INTO sales_table USING updates ON sales_table.order_id = updates.order_id WHEN MATCHED THEN UPDATE SET price = updates.price; #Result: Simplified operations, faster queries, and scalable data management. #Benefits of Iceberg & Hudi Over Parquet Both Iceberg and Hudi extend Parquet with: • Schema Evolution: Add, drop, or rename columns without rewriting files. • Time Travel: Query data as it existed at any point in time. • ACID Transactions: Perform reliable updates, deletes, and concurrent writes. • Partitioning Without Hassle: • Iceberg: Hidden partitions with automatic pruning. • Hudi: Optimized for upserts with built-in indexing. • Real-Time Support: • Hudi: Designed for streaming and CDC (Change Data Capture). • Iceberg: Best for batch analytics. • Metadata-Driven Optimization: Faster queries via centralized metadata. #Limitations of Parquet Alone: • No schema evolution. • No ACID transactions for updates or deletes. • Manual partitioning is error-prone. • No time travel for historical queries. • Limited real-time support for streaming or CDC. #Takeaway -If your data lake needs dynamic schemas, real-time updates, or historical insights, table formats like #Iceberg and #Hudi are game changers. They turn your static #Parquet files into a powerful data management system. Ready to make the switch? Let’s discuss how these formats can fit your data lake! 💬

55

10 Comments

CLASSES in SCALA are pretty much like CLASSES in C++ & Other OBJECT ORIENTED PROGRAMMING LANGUAGE. #SCALA #BIGDATA #CPP #C++ Karthik K. object CLASS { class Point{ private var _x = 0 private var _y = 0 private val bound = 100 def x: Int ={ return _x} 👈 //GETTER for private var _x def y: Int ={ return _y} 👈 //GETTER for private var _y def coord( x:Int, y:Int)={ 👈 //SETTER for private vars _x & _y this._x = x 👆👇 /"This" keyword store address of the object calling the function this._y = y println(this._x,this._y) } } def main(args: Array[String]):Unit = { val a = new Point 👈 //Object creation of Point in Heap Memory a.coord(20,30) print(a) } } OUTPUT:

4

3 Comments

https://lnkd.in/dcC-kGpu - Dimensional Modelling vs. One Big Table in Databricks Lakehouse - a very interesting article on data engineering and analytics architecture

27

A Complete workflow of Data Architecture!

11

🚀 TableRAG: A New Tool for Large-Scale Table Analysis Processing large tables—like those in RBI circulars—has always been challenging. Context limits and manual effort make extracting accurate insights time-consuming. Let’s take a query: “What were the key interest rates between 2019 and 2021?” 🔍 Manual review or partial automation often led to missed details or incomplete answers. Extracting the tables and flattening them, irrespective of structure, was a daunting and error-prone task. ⚡ TableRAG uses schema retrieval and cell retrieval to target only the necessary parts of a large table. 2 important retrievals: 💡 Schema Retrieval: It identifies the key columns (e.g., 'Year', 'Repo Rate') by understanding the table structure and picking out relevant metadata like column names and data types. 💡 Cell Retrieval: It further narrows down to the exact values that matter for the query (e.g., '5.15%', '4%') by scanning the table contents for precise matches. By combining these retrievals, TableRAG ensures only the critical data is processed, minimizing token usage while retaining context. I am very hopeful that it will change the way we deal a document during RAG set-up. #advancedanalytics #deeplearning #innovation #genai

20

1 Comment