Srinivasan Shanmugasundaram

Some thoughts/observations from my recent visit to the Databricks Data+AI summit. If you're a small enterprise:- Looking for a data lake/mesh+BI+AI/ML platform? Just buy Databricks and be done with it. At best, do a POC with another platform e.g. Snowflake and go for the lowest bidder. These are evolving rapidly and feature parity is a constant catchup so don't go only based on what exists today. If you're a query engine/federation tool (e.g. Starburst/Dremio):- Delta (the table format) has been open source for a while but Photon (the engine) isn't (and may never be). So Databricks has been under pressure from the vendor/partner ecosystem as well as large enterprises to support non-vendor-lock-in options. It's acquisition of Tabular makes this space interesting and here's hoping the ambition to support Iceberg as a first-class citizen is genuine. Snowflake also made the Polaris announcement for Iceberg support recently. Freshen up you're price/performance stats, you're going to need it. If you're a large enterprise Governance/Unity Catalog Soo many presentations all claiming Unity as the 'single' point of governance solving all their problems. But is any large customer really all-in on Unity? Nobody I spoke with can claim that 'single' is a reality. Now they all have one more catalog. Every large enterprise that uses Databricks also uses <name-your-fav-alternative>. While Unity helps with governance when data/models are accessed from within Databricks - it doesn't push down to the underlying external platform - e.g. it isn't going to generate Lake Formation grants for you. So you'll need two enforcement points for folks who access that Glue table from LF vs from Databricks/Unity. Immuta - can you wake up quickly to this opportunity pls? ABAC (Attribute Based Access Control) Zeashan Pappa/Kristen Wilder had an interesting talk on what's coming with Unity and ABAC. They're going with tags at the Account level as part of the design. Most Databricks customers will have one account for the entire org. Some very large customers probably have one account per LOB within the org. Either way, curious to know how many such orgs have governance centralized at such a high level instead of being federated deeper. If you are more federated, are we back to CS's fav problem - naming things? Domains/namespaces etc. E.g. two different parts of an org may not interpret 'client' the same way. ML capabilities and integrations Every AutoML vendor will need to compete with code-gen, it will not be sufficient for any auto-generated model to be adopted without a need to tweak the underlying code. The Unity and git integrations around MLFlow make it very compelling. How they continue to evolve Mosaic and better integrate into the platform will be interesting to follow. Still feels like very early days on that - feels a bit square-peg-round-hole still. All in all - tremendous pace of innovation. Lots of hype but much of it is deserved.

61

2 Comments

🚀 Exciting day at Snowflake Data Cloud World Tour Mumbai 2024! ❄️ I had the privilege of attending the Snowflake Data Cloud World Tour in Mumbai, and here are some key takeaways from insightful sessions on the future of data and AI: 📊 Open Table Formats & Iceberg Support Exploring integration possibilities with Apache Iceberg and Polaris Discussions on ACID compliance, performance optimization, and data evolution 🛡️ Snowflake Horizon Unveiling a vision for unified data governance and security across clouds Concepts for simplifying management of data and apps from a centralized interface 🤖 Generative AI & LLMs Introduction to the Snowflake Arctic family of Large Language Models (LLMs) Exploring Snowflake Cortex as a pathway to democratize AI Emphasis on the "AI to Data, not Data to AI" approach for enterprises 🔒 Data Governance & Security Highlighting the importance of unified security and compliance measures Showcasing potential applications of Dynamic Data Masking for data protection 💻 Snowpark Container Service & ML API Demonstrations of running AI and full-stack apps within the Snowflake ecosystem Exploring possibilities for direct ML model execution and predictions 🔗 Native Apps & Data Sharing Discussing streamlined approaches to building and sharing data applications Insights into enhancing operational collaborations through data sharing 📄 Document AI & Unstructured Data Previewing tools aimed at extracting insights from unstructured data The event showcased Snowflake's vision for innovation, open standards, and efficient data management. It was fascinating to see the proposed solutions for various data challenges, from governance to AI implementation. Kudos to the speakers, including Sanjay Deshmukh Vijayant Rai Rahul Jain 🇮🇳 Kamesh Sampath and Sumeet Tandure Excited to contemplate how these concepts might shape our future data strategies! #SnowflakeWorldTour #DataCloud #AI #DataGovernance #Innovation

51

2 Comments

Databricks surely gonna be ruling the Data and AI space. A right step towards right direction at right time 💥 Databricks has acquired Tabular, a data management startup founded by Apache Iceberg creators, for over $1 billion. The acquisition aims to build a unified data management platform based on open lakehouse formats, reducing silos and friction for customers. This move highlights Databricks' commitment to open formats and open source data in the cloud. Read the detailed news over here: https://lnkd.in/gqVcxjcv #dataengineering #dataplatform #datastrategy #storage #nosql #cloud

6

Someone rightly said : Snowflake is for rich, low to no-tech departments like Finance. Databricks is for data engineers. In the end, both are successful, multi-billion dollar companies. Choice in the end lies in defining a clear customer segment. #snowflake #databricks #dataengineering #customersegment

41

3 Comments

Actively Seeking New Opportunities | Senior Data Engineer 🌟| C2C With 10 years of experience in data engineering, I am actively seeking new opportunities to contribute my expertise in developing cutting-edge data solutions. My experience spans across modern data stack technologies, cloud platforms, and big data architectures. 🔧 Key Skills: Cloud Platforms: AWS, GCP, Azure Data Warehousing: Snowflake, Big Query, Redshift ETL/ELT Pipelines: Apache Airflow, dbt, Talend Big Data Technologies: Spark, Hadoop, Kafka Programming Languages: Python, SQL, Scala, Java Containerization & Orchestration: Docker, Kubernetes CI/CD & DevOps: Jenkins, GitLab, Terraform Data Lakes & Streaming: Delta Lake, Kinesis I’m passionate about optimizing data workflows, improving data quality, and scaling architectures to meet the demands of today’s fast-paced data environments. If you’re looking for someone who can transform complex data into meaningful insights, let’s connect! Excited to collaborate and contribute to innovative projects. 📩 DM or reach out via email [email protected] to discuss opportunities! hashtag #repost #DataEngineering #CloudArchitecture #ETL #BigData #DataOps #DataWarehousing #DevOps #OpenToWork #Seniordataengineer #c2c #c2h #java #kafka #python #sql #aws #azure

3

You asked for memes But on a serious note - the core differentiator of lakeflow vs. the rest of the Databricks suite is twofold; there is a GUI for building pipelines (nice!) which is presumably code-based (.yml /json see docs "declarative data pipelines") but where the Tasks you can implement are also pretty cool (e.g. Oracle CDC task, very nice). In Azure Data Factory, you get: - GUI for building pipelines - .yml under the hood - cool data processing capabilities for the Azure suite This is pretty much what Lakeflow is Different things - Integration with the rest of the Databricks platform (e.g. Unity Catalog, really cool) - Integrations with third party sources (something ADF has always done TERRIBLY) We know what ADF's strengths and weaknesses are. It is great for specific data processing tasks in Azure. it is not the way to unify your data platform. Bricksters now face a strange choice - do they unify with dbx workflows, lakeflow pipelines or something else? Personally, I also would have created a standalone streaming service rather than bundle it in to something ADF-like (there is precedent for this. Streaming is hard. Estuary, RedPanda, Confluent etc. all streaming focussed ELT vendors that think it's sufficiently hard to look like their products do and not ADF..) but I can see the short-term appeal #databricks #lakeflow #adf

150

20 Comments

Snowflake Optimization - Summary of the week 1. Remove failing workloads Failing workloads are just using credits and not performing any real work. To identify them, use this: https://lnkd.in/gP8bgV-a 2. Identify unused storage Identify and remove unused storage. Here's a blog to identify: https://lnkd.in/gMTrHxmM 3. Set query timeouts Set appropriate query timeouts based on the size of the warehouse and the business use case. 4. Split or merge files before ingestion The appropriate file size for ingestion into Snowflake is 100-250MB. Split or merge files before ingesting to ensure the file size is correct. 5. Use deferred merge If the frequency of ingestion is high, use the concept of deferred merge. Here's a blog: https://lnkd.in/gk4jgz2g

20

3 Comments

One of the most common question asked in Data Engineering Interview: Difference between ETL, ELT, and Reverse ETL. 1️⃣ ETL: The Traditional Powerhouse Extract, Transform, Load (ETL) has been the go-to process for decades. It involves: 1. Extracting data from various sources 2. Transforming it to fit operational needs 3. Loading it into the target system (usually a data warehouse) 2️⃣ ELT: Flipping the Script Extract, Load, Transform (ELT) emerged with the rise of cloud computing and big data. The key difference: - Data is loaded into the target system before transformation - Leverages the power of modern data warehouses for transformation - Offers more flexibility and scalability 3️⃣ Reverse ETL: Closing the Loop A newer player in the field, Reverse ETL: - Moves processed data from warehouses back into operational systems - Enables data activation, turning insights into action - Bridges the gap between analytics and operations For career guidance in Data Analytics, job search strategies, resume review, interview preparation, etc., you can book a session with me here: https://topmate.io/itshb

53

I like Databricks' dynamic resource allocation; I think it is great. However, I would be cautious when a developer sets the maximum worker nodes to 20. You risk wasting resources and spending a lot of money because the allocation and deallocation of resources are not simple processes behind the scenes. Often, it takes a while to deallocate the newly added executors, leading to overpayment for low utilization. A quick look at the Metrics UI can provide insight into this issue. Instead, try to resize your cluster compute to a larger size, ensuring you do not exceed a maximum of 8 worker nodes in your upper limit setting. #databricks #dataengineer

1

Snowflake Optimization Tip of the Day #5 ➡ Use incremental models in dbt There is no simpler way to reduce Snowflake cost than to process less data. If you are using dbt, try to write every model incrementally. Instead of reprocessing all data every time, incremental models only process new or changed data since the last run. This is achieved primarily by using date stamps or other markers to identify what data has been added or modified since the last run. Full-refreshes are very expensive and should be avoided if possible.

8

One of the most important and commonly asked Data Engineering #interview questions is : 📍Why Spark transformations are lazy? Now, there are 2 types of operations in spark: 1. Action. e.g. display, save etc. 2. Transformation. e.g. filter, map etc. 📍How do lazy transformations help? Let's say, I am trying to figure out Top 50 salaries from the employee table and I want to order them by name. 📝If I order first and then filter,then that is a waste of time and resources. Spark being lazy,will filter first and then sort.Even if I ask it do otherwise. Spark will execute the command in an optimised manner only when an action is called and will sit idle no matter how many transformations are called before that. That's how being lazy helps Spark to be more efficient. >>Link in the comments below: #Dataengineer #dataanalytics #datascience #problemsolving #dataoptimize #datacouple #video #interview

4

1 Comment

This note is to high-light the need and relevance of Data Engineers in Hybrid Cloud environment. In my 25+ years journey at CAMS, I have created CoE on following areas and have enabled 250+ professionals to shape their career... 1. Database Administrators 2. SQL Programmers 3. ETL Programmers 4. BI Developers This note will help them and also anyone who is working on these roles now... No doubt these roles still relevant and are in demand. But Oraganizations are now expecting you to upgrade your knowledge to fit into the Cloud world. And because of change in regulator mindset and the compelling benefit of Cloud computing, Organizations have started their cloud journey which includes large financial platforms too... In next one year, professionals on these roles are expected to become DATA ENGINEERS. Interestingly, the cloud data engineering concepts are not new. In fact, we are much experienced with the problem statements or topics discussed in data engineer scope. Cloud is now helping us to solve these probems with various new algorithms and tools (Eg. Open Table Formats, Data Mesh Strategy, Handling PII data) making te job easier... Also there will be huge demand for data migration/data transformation projects for next 5 years. I strongly suggest these professionals to upgrade & take certification course on data engineering (with any cloud platform) and get certified within next 3 months. At a much higher-level, the scope includes the following. Let me assure you that you will find these topics much easier to understand.. Typically, Data Engineer Course covers following topics: 1. Data Transformation, Data pipelines 2. Data Governance, Security and Cataloging 3. Ingesting Batch and Streaming Data 4. Data Mesh, Data Lake Formation 5. Data Marts & Data Warehouses 6. Transactional Data Lakes 7. Data Analytics, Data Visualization and Machine Learning I will keep writing on this topic for next 30 days. Please call me or write to me if you need any help on this topic...

26

1 Comment

Week 6 of Elite Data Engineering Master's Program Summary Sumit Mittal and TrendyTech Enforcing the Schema – Schema should not be inferred and it should be enforced due to below issues – 1.   Spark may not infer data types correctly. 2.   Spark needs to scan more data in order to infer the schema which hampers the performance and adds more load to Apache Spark. Scanning more data takes more time. Through samplingRatio we can ask Apache Spark to scan some part/percentage of data. There are two ways to enforce the schema – 1.   Schema DDL 2.   StructType How to deal with date types – Apache spark understands yyyy-MM-dd date format. If we give date in another format and do not specify the format, spark will throw and error. There are two ways to handle date formats – 1.   Specify the date format while reading the data frame. 2.   Load the date as string and later convert the data type to relevant form. Read Modes – There are three modes to read the data frame – 1.   failfast – 2.   permissive – 3.   dropmalformed – Different way of data frame creation – 1.   Standard data frame reader API – spark.read 2.   Shortcut methods to read the dataframe – spark.read.csv, spark.read.json, spark.read.orc, spark.read.parquet 3.   The output of spark.sql is always a data frame 4.   Spark Table/Spark SQL can be converted to data frame using spark.read.table/spark.table 5.   Spark.range() will give single column dataset for testing purpose 6.   spark.createDataFrame to create data frame from local list Converting RDD to data frame – Below are the ways to convert RDD to data frame. 1.   df = spark.createDataFrame(rdd).toDF(column_names) 2.   df = rdd.toDF(orders_schema) 3.   df = spark.createDataFrame(rdd,schema) Nested Schema – Nested schema can also be handled using SchemaDDL and StructType. Also, local lists having nested schema also can be converted to data frame using createDataFrame. Data Frame Transformations – select vs selectExpr To add new column or modify existing column – withColumn transformation To rename column – withColumnRenamed transformation To drop column – drop transformation selectExpr will accept the column names and expressions as well. We can create new column and update existing column using expressions in withColumn, select and selectExpr. Removal of Duplicates from Data Frame – .distinct is good when we are considering the duplicated across all the columns. dropDuplicates is best option when we want to remove duplicates with selected columns. Spark Session in Details – Spark Session is an entry point to spark cluster. Spark Session is needed for higher level APIs and RDD/Spark Core APIs use spark context as and entry point to the spark cluster. Spark Session holds other contexts (spark context, hive context, SQL context). #bigdatadeveloper #trendytech #bigdatatechnologies

61

5 Comments

#HIRING-FEEDBACK While hiring for data scientists or machine learning engineers, I always look out for one major thing in candidates take home assignments - their curiosity and adherence to the data science lifecycle - also known as CRISP-DM. The purpose why this framework exists is to streamline and standardize data science workflows. Yet I see innumerable cases while evaluating the scripts where numerous parts of this workflow are skipped. For e.g. when you are proposing model A is better than model B, cite your reasons, ground them in quantitative metrics like RMSE, or MAPE. Argue your case! Do not recommend a model without evaluating it on the test set. Talk about limitations of data and the approach, show you have taken time to understand the problem thoroughly Most importantly, perform EDA. There is a high chance you have not even understood the data if no EDA is performed, showing a casual attitude to the problem at hand. Data scientists are by nature curious. They are intrigued by the data mining process, making correlations and unearthing hidden patterns. Hopefully this feedback will help you crack more take home assignments to demonstrate your data skills. #Hiringfeedback #datascience

174

11 Comments

Snowflake Interview Questions. 1. What is Snowgrid ? #Answer: If you share data within the same cloud provider and region, there is no need to copy the data to share it. If you share data to a different region or different cloud provider, Snowflake replicates the data. However, it does so via auto-fulfillment, so you don’t need to do any manual work. Snowgrid is the term to describe data replication for the purpose of facilitating data sharing. Snowgrid is global, seamlessly connecting Snowflake users who may be separated by region or cloud. Snowgrid achieves this through replication via auto-fulfillment. Even when sharing data across clouds and regions, the shares are transactionally consistent, meaning the source of truth is still maintained. Within Snowgrid are all the native cross-cloud governance controls that serve as the foundational building blocks for enabling federated governance. 2. What are Materialised views? Which is faster, querying materialized views or using cached results? #Answer: Materialized view is a pre-computed data set derived from a query specification (the SELECT in the view definition) and stored for later use. Because the data is pre-computed, querying a materialized view is faster than executing a query against the base table of the view. Materialized views are more flexible than, but typically slower than, cached results.Materialized views are faster than tables because of their “cache” (i.e. the query results for the view); in addition, if data has changed, they can use their “cache” for data that hasn’t changed and use the base table for any data that has changed. 3. Altering the base table will it have any impact to the materialised view? #Answer: Adding a new column to the base table will not cause a new column to be added to the materialized view. If changes are made to an existing column in the base table, such as changing the label or dropping the column, all materialized views on that base table will be suspended. Altering the base table does not change a materialized view; instead, a new materialized view will need to be created. Follow Avinash Sharma Get ultimate interview resources: https://lnkd.in/dRehzivv

72

1 Comment

🚀 Data Engineering in Action: Oracle CDC to Snowflake with Confluent Kafka & Dynamic Tables In my latest project, I had the opportunity to work on a seamless data pipeline by integrating a Change Data Capture (CDC) Oracle database with Confluent Kafka to Snowflake, where we built dynamic tables to enhance our data warehouse architecture. 🔗 Key Highlights: Utilized CDC on Oracle to capture real-time changes and stream them into Snowflake. Leveraged Confluent Kafka as the data streaming platform for efficient, scalable, and real-time data ingestion. Built dynamic tables using SQL on Snowflake to create a flexible and scalable data warehouse architecture that could adapt to changing business needs. Ensured low-latency, high-throughput data movement from Oracle to Snowflake, enhancing overall data availability and analytical capabilities. 💡 This experience reinforced the power of real-time data streaming and the flexibility that Snowflake offers for building robust data warehouses. It was an exciting challenge, and I’m looking forward to more projects that push the boundaries of data engineering and architecture. #DataEngineering #Kafka #ConfluentKafka #Snowflake #Oracle #CDC #DataWarehouse #RealTimeData #SQL #BigData

19

🚀 Have you heard of tech-first and tech-enabled companies? Say, Google,Tesla & Walmart,Pfizer respectively - Understanding the distinction is crucial in today’s rapidly evolving business landscape. ⚠️ Caution: The percentages provided are approximations based on my experience in software & data engineering and may vary significantly across different organizations and industries. Share your insights or corrections in the comments. CIO: In tech-first companies (40%), the CIO has strategic oversight, focusing on innovation. In tech-enabled companies (90%), they drive technology adoption and integration for operational efficiency. CTO: Tech-first companies (90%) see the CTO central to tech innovation, while in tech-enabled companies (40%), the focus shifts to integrating technology rather than core development. CDO: In tech-first firms (80%), the CDO innovates data strategies; in tech-enabled companies (60%), they align data strategy with business goals. Data Product Manager: Tech-first (75%) focuses on technology-driven solutions, while tech-enabled (60%) ensures data products meet user needs. Business Analyst: In tech-first companies (60%), they leverage data for new opportunities, while tech-enabled (75%) integrates data into business processes. Data Analyst: Tech-first (70%) emphasizes tech-driven analysis for initiatives, while tech-enabled (80%) focuses on extracting insights for reporting. Pricing Manager: They optimize strategies in tech-enabled (75%) firms and develop dynamic models in tech-first (60%) companies. Data Architect: In tech-first (80%) companies, they create innovative solutions; in tech-enabled (60%), they design data systems. Analytical Translator: Tech-first (75%) firms rely on them to translate insights into strategies, while tech-enabled (65%) firms bridge technical and non-technical teams. Analytics Engineer: They innovate solutions in tech-first (80%) companies, while in tech-enabled (55%), they build analytics infrastructure. Decision Scientist: Their role is more pronounced in tech-first (80%) companies, employing data for strategic decisions, while in tech-enabled (70%), they also use data effectively. Machine Learning Engineer: In tech-first companies (85%), they focus on developing advanced solutions; in tech-enabled firms (50%), they implement models. Applied Data Scientist: Tech-first (85%) firms use them for advanced methodologies; in tech-enabled (60%), they apply algorithms with domain knowledge. Data Scientist: They are deeply integrated in tech-first (90%) companies for product development, while in tech-enabled (10%), they specialize in analysis. Data Engineer: In tech-first (80%), they innovate data systems; in tech-enabled (70%), they maintain data pipelines. Data Governance Officer: They ensure data compliance in both settings (tech-first 75%, tech-enabled 65%). Data Quality Analyst: They focus on data quality in tech-first (80%) and maintain integrity in tech-enabled companies (70%).

27

6 Comments

********* Beyond the Hype********** Microservices aren't always the answer. We actually consolidated some services for better performance. · Caching is still king. Proper Redis implementation cut our database load by 40%. · Don't underestimate the power of good logs. Our custom logging saved us countless hours during debugging. · Sometimes, the boring solution is the right one. Ditching a trendy NoSQL DB for good ol' PostgreSQL was a game-changer for our data integrity. · Team communication trumps tech stack every time. Monthly architecture reviews kept everyone aligned and caught potential issues early. #TechInnovatio #AI #CloudComputing #Cybersecurity #DevOps #MachineLearning #DataScience #Blockchain #IoT #FullStackDevelopment #Hiring #JobSearch #CareerOpportunities #TalentAcquisition #RecruitmentTrends #HRTech #EmployerBranding #RemoteWork #DiversityAndInclusion #TechTalent

3

2 Comments