Shajee Pulukkool

An under appreciated reason why data processing companies like Confluent, Databricks, Splunk etc get really big is how data scales. Data usage in a company scales O(EQS), if we use Big-O notation, E=number of employees using the data, Q=number of queries being run , S=size of data queried. The reason customer ACVs can get absolutely massive is that if you’ve built an incredible product you will be able to process more data if the customer does even one of the 3: - store more data - add more employees who want to work with the data - have more computations being done on the data Most of the time, companies that are growing have all 3 increasing simultaneously. Ofcourse, because data tools are sticky this growth also persists and even when employee growth slows down, often more data and uses of the data have emerged If you’re building any tools, or even side projects that involve data processing, I’d love to learn more and talk to you!

50

🔄 Managing Large MySQL Tables: An Efficient Archival Method 🚨 Our organization tackled significant performance issues from vast data volumes in critical tables, impacting query efficiency. With tables holding millions of rows, conventional deletion methods led to downtime and performance degradation. 💡 We introduced a systematic Archival Method: - Identify Records for Archival: Criteria-based selection determines rows for archiving. Example: SELECT Vendor_Id FROM Vendor WHERE status = 0; - Batch Processing: Manageable batch processing minimizes live database impact; Use cursors to iterate over batches of records. *** Batch size is configurable and passed as a parameter to control and avoid long-running transactions. Example: Use cursors to iterate over batches of records with a specified batch size. - Move Records to Archive Table: Transfer records to an archive table with the same structure. Include Audit Columns in the Archive Table. - Verify Data Integrity: Ensure archive table records match deletion targets. - Delete Archived Records: Safely remove records from the primary table. - Logging and Monitoring: Log the process and monitor performance. Example: INSERT INTO audit_log (action, tablename, record_count, timestamp) VALUES ('Archival', ‘request_log’, 1000, NOW()); ⚙️ Benefits include: - Performance Improvement: Reduces the load on active tables, enhancing query performance. - Efficient Data Management: Keeps the primary table size manageable, improving overall database maintenance. - Data Integrity: Ensures archived data is safely moved and verified before deletion. - Minimized Downtime: Batch processing avoids long locks and reduces the risk of downtime. - Configurable Processing: Allows for flexible batch size, adapting to different performance needs. Implementing this method with a configurable batch size, significantly enhanced our database performance and data management efficiency. This adaptable approach offers a scalable solution for handling large data volumes in MySQL. 🔗 #MySQL #DataManagement #DatabaseOptimization #ArchivalMethod #DataIntegrity #PerformanceImprovement #TechSolutions #Automation #TableMaintenance

4

1 Comment

Do you have JSON and XML tough to parse and read up? Variant data types. I have seen less people talk about it. Let's do an overview of it and check it out. Here are my thoughts on Databricks Variant data types. https://lnkd.in/g9Qrm2rb #DataBricks #Variant #FileTypes #Simplification #DataArchitecture #Beginners #StartHere #PandaLearns

26

[New Premium Video Alert] Execution Plan Compile Cost by Amit R S Bansal Watch the full video on SQLMaestros https://bit.ly/sqlmaestros More Premium Videos: https://lnkd.in/gUfaiS-Z Share. #SQLServer #SQL #TSQL #AzureSQL

1

Oracle launches HeatWave #GenAI, featuring in-database #LLMs and a vector store. This tech lets developers build AI applications without data migration or extra costs. As per Oracle, it outperforms competitors like #Snowflake, Google #BigQuery, and #Databricks. Read details at https://lnkd.in/gqrcxGwH #Oracle #AI #GenAI #EnterpriseTech #HeatWaveGenAI

9

Exploring the Rise of Data Streaming in India: Confluent’s Rubal Sahni explores the transformational power of real-time insights in India’s business landscape. Discover how Confluent leverages Apache Kafka for enhanced operational efficiency and customer experiences. Read More: https://lnkd.in/g4wU4bbb #DataStreaming #BusinessTransformation #TechNews #Innovation #Confluent #ApacheKafka

108

2 Comments

Video from the session is live on Youtube. Check out What's new for developer in PostgreSQL 17 https://lnkd.in/d3MN8vb8 #PostgreSQL #WhatNew #PostgreSQL17

36

It gives us great pleasure to announce that Apache Spark in BigQuery is now officially available in Private Preview. This feature expands BigQuery’s processing capabilities by offering Apache Spark as an OSS processing engine under a unified control plane (admin, monitoring, billing, security, governance). It is the first step in delivering on the Unified Data Platform vision and was a key announcement at Next ‘24 #googlecloud #bigquery #spark

38

2 Comments

🔍 **Anatomy of a Data Platform: How to Choose Your Data Architecture** 📊 In today's data-driven world, choosing the right data architecture is critical for building a robust data platform. Here’s a guide to help you make informed decisions: ### Key Considerations 1. **Data Volume and Velocity**: - **High Volume**: Opt for distributed storage solutions like Hadoop HDFS or cloud-based data lakes (AWS S3, Azure Data Lake). - **High Velocity**: Use streaming platforms like Apache Kafka or cloud-native services like AWS Kinesis or Azure Event Hubs for real-time data processing. 2. **Data Variety**: - **Structured Data**: Relational databases like PostgreSQL or cloud-based RDS (Amazon RDS, Azure SQL Database). - **Unstructured/Semi-structured Data**: NoSQL databases like MongoDB or DynamoDB, and data lakes for flexibility. 3. **Processing Needs**: - **Batch Processing**: Hadoop MapReduce, Apache Spark, or cloud-based alternatives like AWS Glue. - **Real-Time Processing**: Stream processing with Apache Flink, Spark Streaming, or cloud-native solutions like AWS Lambda and Google Cloud Dataflow. 4. **Scalability and Flexibility**: - Choose architectures that can scale horizontally to handle growth. Cloud platforms provide managed services that auto-scale based on demand. 5. **Data Governance and Security**: - Implement robust data governance frameworks and security measures. Use tools like AWS Lake Formation, Azure Purview, and GCP Data Catalog to ensure compliance and data integrity. 6. **Cost Efficiency**: - Evaluate the cost implications of different architectures. Pay-as-you-go models in cloud platforms help optimize costs by only charging for resources used. ### Popular Architectural Patterns 1. **Data Lake Architecture**: - Centralized repository for all types of data. Ideal for large-scale storage and analytics. Supports a variety of analytics tools and ML frameworks. 2. **Data Warehouse Architecture**: - Optimized for structured data and complex queries. Cloud data warehouses like Amazon Redshift, Snowflake, and Google BigQuery offer high performance and scalability. 3. **Lambda Architecture**: - Combines batch and real-time processing. Best for applications needing both historical and real-time data analysis. 4. **Microservices Architecture**: - Decoupled services for flexibility and scalability. Often used with event-driven architectures for real-time data flows. ### Implementation Tips - **Start Small, Scale Gradually**: Begin with a pilot project to validate your architecture choices. - **Leverage Cloud Services**: Use managed cloud services to reduce operational overhead and focus on core business logic. - **Ensure Interoperability**: Use APIs and data integration tools to ensure seamless data flow across different systems. - **Monitor and Optimize**: Continuously monitor performance and optimize configurations to meet changing business needs.

3

Difference between DAG & Lineage Graph #SPARK #BIGDATA Karthik K. Lineage Graph: 👉 It represents the dependencies between RDDs (Resilient Distributed Datasets) or DataFrames in a Spark application.👈 👉 Basically, it's a logical plan which tells which transformation we need to apply to create an RDD. 👈 RDD1 --> RDD2 --> RDD3 In the above flow, it's pretty clear that RDD2 gets created by RDD1 and RDD3 gets created by RDD2. 👉 The lineage graph represents the sequence of transformations applied to the original input data to produce a new RDD or DataFrame/Dataset. 👈 👉 By maintaining the lineage information, Spark ensures that it can re-compute data without having to store intermediate results, which helps to save memory and improve performance. 👈 👉👉 Lineage doesn’t have any action defined. It’s a pure transformation tracker. Direct Acyclic Graph(DAG): - The lineage in Apache Spark is represented as an acyclic graph of stages. The term acyclic means they do not contain cycles or loops. This property ensures that the computation can be efficiently parallelized and optimized. 👉 It is a set of Vertices and Edges, where vertices represent the RDDs and the edges represent the Operation to be applied on RDD. 👈 - When multiple transformations are applied to an RDD the resulting logical plan is formed. This lineage captures the sequence of transformations applied to the original data. 👉 When an Action operation is encountered the logical plan is submitted to the Catalyst Optimizer for further optimization and execution planning.👈 Logical Plan --> Catalyst Optimizer --> DAG Schedular --> Physical Plan(DAG) - The Catalyst Optimizer in Spark applies its inherent algorithms to produce the finalized plan, which is then handed over to the DAG Scheduler. The role of the DAG Scheduler is to generate the Physical Plan in the form of a Directed Acyclic Graph (DAG) splitting the execution process into stages. - DAG is very helpful in execution point of view and also tells us stage dependencies over each other.

48

2 Comments

Questions on COALESCE in SPARK #BIGDATA #SPARK Karthik K. 𝐖𝐡𝐚𝐭 𝐢𝐬 𝐭𝐡𝐞 𝐮𝐬𝐞 𝐨𝐟 𝐜𝐨𝐚𝐥𝐞𝐬𝐜𝐞 𝐢𝐧 𝐒𝐩𝐚𝐫𝐤? In Apache Spark, coalesce is a transformation that is used to reduce the number of partitions in an RDD (Resilient Distributed Dataset) or DataFrame. This is particularly useful for optimizing the performance of certain operations by reducing the shuffling of data across the network. 𝐖𝐡𝐲 𝐔𝐬𝐞 𝐜𝐨𝐚𝐥𝐞𝐬𝐜𝐞? 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐢𝐧𝐠 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞: By reducing the number of partitions, coalesce can help optimize performance for certain operations, such as actions or joins, by minimizing the overhead of managing too many partitions. 𝐀𝐯𝐨𝐢𝐝𝐢𝐧𝐠 𝐅𝐮𝐥𝐥 𝐒𝐡𝐮𝐟𝐟𝐥𝐞: Unlike repartition, coalesce avoids a full shuffle of the data. This is particularly useful when reducing the number of partitions without significantly changing the data distribution. 𝐂𝐨𝐝𝐢𝐧𝐠 𝐐𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 𝐟𝐨𝐫 𝐈𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐁𝐚𝐬𝐢𝐜 𝐔𝐧𝐝𝐞𝐫𝐬𝐭𝐚𝐧𝐝𝐢𝐧𝐠: 1] What is the purpose of coalesce in Spark? 2] How does coalesce differ from repartition? 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐒𝐜𝐞𝐧𝐚𝐫𝐢𝐨: 1] Given a DataFrame with 10 partitions, how would you reduce it to 3 partitions without causing a full shuffle? 2] What are the potential benefits of reducing the number of partitions using coalesce before performing an action like count? 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 𝐂𝐨𝐧𝐬𝐢𝐝𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬: 1] Explain a scenario where using coalesce would significantly improve performance. 2 ]What are the implications of using coalesce on data skew and load balancing?

9

Understanding Narrow and Wide Transformations in Apache Spark In Apache Spark, transformations play a critical role in defining how data is processed across distributed clusters. These transformations can be broadly classified into narrow transformations and wide transformations, each influencing the efficiency and data flow within Spark applications. 1. Narrow Transformations: A narrow transformation is one in which each output partition depends only on a single input partition. Since the data required to generate an output partition resides in the same partition as the input data, Spark can process these transformations without requiring data to move across nodes. This minimizes network traffic and improves performance. Examples of Narrow Transformations: • map(): Applies a function to each element of the RDD/DataFrame without requiring data exchange between partitions. • filter(): Selects elements based on a condition, retaining them within the same partition. • flatMap(): Similar to map(), but can generate multiple elements from a single input element, still preserving the partition boundaries. 2. Wide Transformations: Wide transformations involve operations where the output partition depends on data from multiple input partitions, often triggering a shuffle—a costly operation where data is exchanged across the cluster. These transformations require coordination between nodes, leading to network I/O, which can impact performance. Examples of Wide Transformations: • groupByKey(): Groups data by key, causing a shuffle as all values associated with the same key need to be aggregated across partitions. • reduceByKey(): Similar to groupByKey(), but applies a reducing function during the shuffle phase to optimize data movement. • join(): Combines data from two datasets based on a key, triggering a shuffle to align matching keys across partitions. Key Difference: Data Shuffling The fundamental difference between narrow and wide transformations lies in data shuffling. In narrow transformations, Spark processes data locally without any network overhead. However, in wide transformations, the data must be reshuffled across partitions, leading to increased computational and network costs. For example, a map() operation applies functions independently within partitions, while a reduceByKey() requires the data to be shuffled so that values for the same key are processed together. Follow me(Raghu Raman A V) for more articles on Big Data, Data Engineering, Azure and cloud ! #BigData #ApacheSpark #DataEngineering

15

1 Comment

Enter the ultimate face-off: Apache Spark vs. Apache Hadoop. These two powerful tools serve distinct purposes.                Know more ......          👇👇👇👇👇👇👇👇👇👇👇👇👇 https://lnkd.in/gYAFCM7b #Apachespark #ApacheHadoop #DataProcessing #RealTimeAnalytics #BatchProcessing #BigDataTools #DataAnalytics #CloudComputing #DataStorageSolutions #MachineLearning #DataArchitecture

2

Maximizing Efficiency with DataProc: Ephemeral vs. Serverless Clusters Ephemeral Clusters Definition: Ephemeral clusters are temporary clusters that are created to run a specific job and then terminate once the job is completed. Advantages: Cost-Effective: You pay for the resources only while the cluster is running. Once the job completes, the cluster is terminated, reducing costs. Optimized for Short-Lived Jobs: Ideal for batch processing or ad-hoc analysis where the workload is not continuous. Customizable: You can configure the cluster according to the specific requirements of your job (e.g., machine types, number of workers). Disadvantages: Setup Time: There may be a delay in starting up the cluster, as it needs to be provisioned before the job runs. Management Overhead: Requires management of cluster lifecycle, which can add complexity. Serverless Clusters Definition: Serverless clusters allow you to run jobs without the need to manage the underlying infrastructure. You simply submit your jobs, and the service automatically provisions resources. Advantages: No Infrastructure Management: You don’t need to worry about provisioning, configuring, or managing clusters. Google handles this automatically. Instant Scaling: Automatically scales resources based on workload, making it suitable for varying loads. Cost Efficiency: You are charged only for the compute time used by your jobs, without any idle resource costs. Disadvantages: Limited Customization: You have less control over the specific configuration of the environment. Potentially Higher Costs for Long-Running Jobs: If jobs are long-lived, costs might exceed those of an ephemeral cluster depending on usage patterns. Chat Summary User 1: "I'm trying to decide between ephemeral and serverless clusters for my DataProc jobs. What do you think?" User 2: "It really depends on your workload. If you have short, one-off jobs, ephemeral clusters are great because they’re cost-effective. You only pay for what you use." User 1: "True, but I’ve noticed the setup time can be a hassle. I want to avoid waiting around to start my jobs." User 2: "That's a good point. Serverless clusters might be better for that. You don’t have to manage the infrastructure, and they scale automatically based on your job needs." User 1: "I like the idea of not managing anything, but can I customize the environment like I can with ephemeral clusters?" User 2: "Customization is more limited with serverless clusters. You’ll have to consider if that’s a dealbreaker for you. Plus, keep in mind that serverless can get pricey if you have long-running jobs." User 1: "Got it. I’ll weigh the setup time against customization needs and cost. Thanks for the insights!" User 2: "Anytime! Let me know if you need help with the setup." #data #bigdata #dataengineering #gcp

5

Let's examine a sample architecture in GCP to implement a Real-time Dashboard with an AI chatbot ✅Pub/ Sub ingests data from streams ✅DataFlow runs the ETL pipeleine ✅Big Query Analyze the data ✅Looker displays the dashboard ✅Vertex AI use the Language models to create a Chatbot 🛫Real-World use case: Gojek - A delivery app in Indonesia which started as a call center to book taxis, now utilize all of these features in GCP for their users to book Bikes and delivery services. The AI chatbot named Dira is also used with the help of Vertex AI #GCP #RealtimeDashboard #StreamingData

6

4 Comments

🚀 Excited to share my article on deploying Large Language Models (LLMs) as custom endpoints from Databricks Marketplace! 💡 Discover how these LLMs can be seamlessly integrated into Databricks notebooks and various applications like Retrieval Augmented Generation (RAG). Let's unlock the potential of AI together! #AI #Databricks #LLMs #OpenSource #Notebooks #RAG #Innovation #DataScience #MachineLearning

32

1 Comment

Being #cloudagnostic or having the ability to port across cloud providers is essential for #product companies. The interesting aspect of storage / data migration has been elaborated well. However for #enterprise customers, though they adopt multiple #clouds, their usage of multiple clouds towards strategic workload placement and it might not be required or impossible to implement portability in their applications owing to various reasons like cost to implement, lack of access to latest innovation and increased complexity to manage. https://lnkd.in/gijsS6zF

3