Jerome Montino

Seeing more and more people get confused about Databricks reference architecture. You can of course use Databricks for everything right? They're the complete data platform, but should you? Without a shadow of a doubt the pace of innovation there is ridiculous. However there are some areas where a lot of enterprise data architects I speak to are not convinced of. These mainly relate to tools that *need* integrations with other pieces of data and cloud infrastructure. You cannot use Databricks workflows to orchestrate end-to-end processes including stuff outside of Databricks without pain. You cannot give Databricks Data Quality Monitors to Data Stewards and tell them to run with it (at least, people don't seem willing to do that I speak to). Unity Catalog is a great piece of work; but people also want to use other things for discovery like Purview. And of course, what about dbt? What about if you're using that? Should you be using dbt Cloud if Databricks can *just run dbt* for you? See link here for thoughts 🖇 https://lnkd.in/eXp8MmCe #dataengineering #databricks #referencearchitecture

50

5 Komento

Handling and querying messy JSON is now easy with the use of the variant data type in Databricks. This is a common challenge when using APIs. Now we can easily handle changing JSON structures in the data pipeline.

16

Snowflake Dynamic Iceberg Summary Tables: 1. You run near real-time Auto refresh. No need to write stored procedures and Tasks. 2. Sync with Polaris's external Catalog with RBAC features. 3. Spark can read these tables. Save Cost Snowflake

2

Probably one of my more important posts for data leaders about data model governance: Like it or not, creating an aligned source of truth for business logic is currently hamstrung by the constant race to keep dbt and BI tools in sync. A complete data model (transformations and semantics) in dbt is a significant step forward. But business logic doesn’t bloom behind the scenes by data engineers; It’s developed by business analysts on the front lines, as part of their everyday tasks within their native BI environments like Looker or Tableau. These analysts operate under tight deadlines with a business-centric approach. Expecting them to become data engineers overnight is unrealistic. The transition to dbt is more than a technical upgrade — it’s a cultural shift for analysts. Their expertise is in swiftly delivering data analysis using Looker’s or Tableau’s flexible interface, and they will always find their workarounds to avoid friction. That’s when logic descends into silos, and we hope things will eventually find their way to the central data model. While we have the engineering tools and solid understanding for modeling within dbt that can be synced with Looker or Tableau, the reverse flow — capturing the inception of business logic crafted by analysts in BI tools and embedding it into dbt — remains a territory awaiting innovation. I like to refer to this reverse flow approach as **dynamic data modeling**. The reason is straightforward: business logic is dynamic, so data models must be too! This approach consists of three key elements: → Meet business analysts where they are: Let analysts leverage the tools they’ve mastered for rapid prototyping, from SQL to their preferred BI tools. → Track usage of business logic: Use automation to get full visibility into the actual utilization of your business logic within Looker or Tableau, from terms to tables. → Promote mature logic to dbt: Adopt tools that seamlessly shift mature business logic to dbt, into the gold and mart layers. The goal of dynamic data modeling is crystal clear: enabling everyone in the organization to participate in the evolution of business logic, no matter their technical skill level. And this approach offers three immediate benefits for data teams: → You free up business analysts to do what they do best and focus on answering business questions. → You power-up analytics engineers to maintain consistent data models at any scale, without compromising analyst autonomy. → You equip data and business teams with workflows that foster collaboration between different roles. *** Finding the delicate balance between governed data models and the freedom to explore fresh concepts and iterate on new data products is the best formula for scalability. When you embrace dynamic data modeling as your main governance approach, the greater the chances your data operation scales the way you envisioned it 🤞 I’d love to get your thoughts on this. Leave a comment?

36

3 Komento

When you talk with me about data content, you will hear me complain a lot about the lack of analyst content. Usually, people show how to set things up but not how to work with the data. But then there is Parker Rogers and paradime.io - Parker created the first cool NBA analysis videos. With Paradime, they launched the first dbt & analyst competition with amazing results (I loved to go through the results). And what is even better: A new competition is coming up, and I am honored to be one of the judges. It's called: ➡️ dbt™ Data Modeling Challenge - Movie Edition 🍿 And truth be told, I like Basketball, but movies, this is something totally different. In a nutshell, here is how the competition works: From now until May 26th, data professionals from around the globe will use real movie and TV series data to craft SQL queries, develop dbt™ models, and derive insights, all for a chance to win a $1,500 Amazon gift card! Second and third-place prizes are $1,000 and $500, respectively! If you’re interested in joining or want to learn more, check out the link below! https://lnkd.in/dsdnhr8a

27

2 Komento

I like the features of Databricks Unity Catalog. However, if I had one wish, it would be to have a checkbox to enable or disable Unity Catalog when creating a compute cluster. This would simplify the process. Currently, Unity Catalog enabled compute cluster is enabled based on the Access Mode settings, which doesn't directly relate to Unity Catalog, making it less intuitive. #databricks #dataengineer

3

Wondering how to make your data lake ready for near real-time analytics? Struggling with data latency in your analytics pipeline? 🤔😟 Discover how Shi Kai Ng and SHUGUANG XIANG from Grab tackled this challenge in their latest article. Using Hudi and Flink, they enabled near real-time analytics on the data lake. They detail how using Merge On Read (MOR) and Copy On Write (COW) tables for different data sources, along with optimizing connections to Kafka and RDS, led to faster access to fresh data and improved fraud detection. 🛠️🔍 The result? Quicker data insights and enhanced fraud detection capabilities. ⚡ Click on the link in the comments section to learn more! #DataLake #RealTimeAnalytics #DataLatency

9

1 Komento

The importance of understanding various industry domain data as a Data Scientist, Data Analyst, or Data Engineer; and the role it plays in database and data warehouse design and analysis. I will always remember my lecturers Ming Jiang Ken McGarry Kate MacFarlane @Ashley Williamson for stressing domain data understanding. Read more by clicking the link in the comment.

4

2 Komento

Data Storytelling is more art than data. Great, useful present (rare!) to any data scientist/AI engineer who came into the subject matter from engineering rather than stats, finance or any fundamental research. #datastorytelling #infographics #aiengineer #datascientist

5

2 Komento

When working with an unfamiliar JSON string and needing to identify all keys, using a JSON_KEYS extractor can be extremely useful. #bigquery https://lnkd.in/eY3nKTE6

6

Databricks just did a huge release you probably didn't hear about. Serverless compute for workflows. In addition to Serverless SQL Warehouses, Databricks have launched serverless compute for Tasks within jobs. This means that instead of managing your own EC2 or other box you can just leverage Databricks' serverless compute instead. Is there a catch? Not sure - I am sure some folks can chime in here Tom Youngman Ryan Chynoweth . The price per DBU of Photon is $0.15 but serverless is $0.37 - but you obviously don't also "bring your own box" with serverless. In any event - it's cool that companies like Databricks are also making it easier for folks running data workflows by removing the need to manage infrastructure. This allows them to focus on driving value from #data and #ai. This is exactly what we're doing at Orchestra - decoupling orchestration from task execution so you don't need a platform team or a massive cluster to stitch together and monitor your pipelines. Join the serverless Orchestration movement🔗 https://lnkd.in/eBeZxUYa #dataengineering #serverless

82

15 Komento

𝐃𝐚𝐭𝐚 𝐃𝐞𝐭𝐞𝐜𝐭𝐢𝐯𝐞'𝐬 𝐓𝐨𝐨𝐥𝐤𝐢𝐭: 3 𝐆𝐨-𝐓𝐨 𝐏𝐲𝐭𝐡𝐨𝐧 𝐂𝐨𝐦𝐦𝐚𝐧𝐝𝐬 𝐟𝐨𝐫 𝐃𝐚𝐭𝐚 𝐏𝐫𝐨𝐬 As data professionals, we often rely on a core set of commands to unravel the gems in our data. Here are three commands I use DAILY. ▶ 𝐔𝐧𝐯𝐞𝐢𝐥𝐢𝐧𝐠 𝐭𝐡𝐞 𝐃𝐚𝐭𝐚 𝐒𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞: First things first - I always start by peeking at my column names. df.columns This simple command gives me a bird's-eye view of the dataset structure, helping me plan my preprocessing strategy. At this stage, I usually have a few ideas of what I need to do to clean the data or process it. ▶ 𝐒𝐭𝐫𝐢𝐧𝐠 𝐒𝐩𝐥𝐢𝐭𝐭𝐢𝐧𝐠 𝐌𝐚𝐠𝐢𝐜 Ever needed to extract specific parts of a string? df['address'].str.split(' ').str[0] This line is a lifesaver when dealing with address data. It splits the string on spaces and grabs the first element. So '222 Jetson Street' becomes just '222'. Just adjust the value inside of the square brackets to get a different splitted item. Perfect for isolating house numbers or postcodes! ▶ 𝐌𝐢𝐬𝐬𝐢𝐧𝐠 𝐃𝐚𝐭𝐚 𝐃𝐞𝐭𝐞𝐜𝐭𝐢𝐯𝐞: Last but not least, let's talk about handling those pesky missing values: df.isna().sum() This combo move tells me exactly how many missing values I'm dealing with in each column. It's crucial for deciding whether to impute, drop, or investigate further. 𝑰𝒇 𝒚𝒐𝒖 𝒘𝒂𝒏𝒕 𝒕𝒐 𝒍𝒆𝒗𝒆𝒍 𝒖𝒑 𝒚𝒐𝒖𝒓 𝑫𝒂𝒕𝒂 𝑺𝒄𝒊𝒆𝒏𝒄𝒆, 𝒇𝒐𝒍𝒍𝒐𝒘 𝒎𝒆 𝒇𝒐𝒓 𝒎𝒐𝒓𝒆. 𝑰 𝒑𝒐𝒔𝒕 𝑫𝑨𝑰𝑳𝒀 𝒂𝒃𝒐𝒖𝒕 𝑫𝒂𝒕𝒂 𝑺𝒄𝒊𝒆𝒏𝒄𝒆.

5

When working with streaming data, keeping up with schema changes can feel like trying to hit a moving target 🎯. Luckily, Databricks Auto Loader offers four flexible schema evolution options that help you stay ahead of the curve. Here’s a breakdown of each: 1️⃣ Add New Columns (addNewColumns (default)) ✅ Auto Loader fails the current streaming job, but automatically adds new columns to the schema metadata. Note that you can always rerun job and it will pick up where it left off with the new schema added. This is perfect when the incoming data has new fields that weren’t in your original schema. No need to update manually—just sit back and let Auto Loader handle it! 📝 Use Case: Your IoT sensor data starts sending additional metrics mid-stream. 2️⃣ Fail on New Columns (failOnNewColumns) 🚫 If new columns appear, the stream fails. Stream will keep failing till schema is updated or bad file removed This option is great for those who want to strictly control the schema and ensure no unexpected fields sneak in. Use it when schema consistency is critical. 📝 Use Case: Regulatory reports that need to follow strict data formats. 3️⃣ No action (none) ⚠️ Does not evolve the schema, new columns are ignored, and data is not rescued unless the rescuedDataColumn option is set. Stream does not fail due to schema changes. This is perfect for exploratory analysis 📝 Use Case: Research and data analysis 4️⃣ Rescue Data (rescue) 🛟 Captures unexpected columns and stores them in a special _rescued_data column. With this option, you can maintain your stream without disruptions. It’s great for exploratory use cases—unexpected fields are stored for review, ensuring no data is lost. Schema is never evolved and stream does not fail due to schema changes. 📝 Use Case: Handling messy real-world data with unknown fields (like social media feeds or logs). 💡 Pro Tip: Choose your schema evolution strategy based on your data needs and use case. Balancing flexibility and control is the key to managing streaming data effectively! What’s your favorite schema evolution option? Let me know in the comments!👇 Reference: https://lnkd.in/gJ8DkGrW

15

Learn two useful functions to nest and un-nest datasets in PySpark — explore and collect_list — by following along Tom Reid's latest tutorial.

8

🔸 The Strategic Move to Modernization 🔸 The Promise of High Availability 🔸 The Future of Data Security 🔸 The Possibilities of Postgres and AI These were the key topics at EDB Postgres AI Day Jakarta. Modern data infrastructure will make the critical difference for enterprises looking to stay ahead of the curve in the AI generation. As businesses embrace digital transformation, integrating AI into database management will enhance performance and ensure systems remain secure and resilient in the face of accelerating demand. Thanks to our knowledgeable EDB Postgres AI Day Jakarta panelists, PT Bank Danamon Indonesia, Tbk. Head of IT Database Infrastructure Hassan Ridwan, Telkomsel Officer of Database Administration Taufik Safar Hidayat, and EDB Sales Engineer Habiburrokhman Sjarbini. Discover key insights from the EDB Postgres AI Day Jakarta panel discussion in our latest blog here ➡️ https://bit.ly/47WtdU4 #EDBPostgresAI #JustSolveItWithPostgres #DatabaseModernization #HighAvailability #ITInfrastructure #PostgreSQL #EDBPostgresAIDay #Jakarta

12

So many great nuggets in the 1st Apache Hudi Newsletter! Some of my favourite picks: ✅ Cost Optimization Strategies for scalable Data Lakehouse - Halodoc Engineering ✅ Enabling near real-time data analytics on the data lake - Grab Engineering ✅ How to use StarRocks & Apache Hudi to build a robust open data lakehouse architecture ✅ Combine Transactional Integrity and Data Lake Operations with Yugabyte and Apache Hudi Link in comments. Subscribe for all the upcoming ones! #dataengineering #softwareengineering

31

2 Komento

Databricks offers granular access control mechanisms that empower organizations to enforce data governance and privacy policies effectively. Through column-level and row-level security, companies can safeguard sensitive data while maintaining flexibility in data access. The use of dynamic views enables Access Control Lists (ACLs) to tailor visibility based on user roles and privileges. This ensures that authorized users have full access, while restricted users receive filtered or redacted data, aligned with predefined governance rules. In scenarios where personal identifiers—such as email addresses, first names, and last names—are present, restricting field-level access becomes crucial to comply with data privacy regulations like GDPR. With dynamic views, organizations can dynamically adjust access without duplicating data or creating multiple datasets, making compliance scalable and efficient. This approach not only simplifies compliance with regulations but also ensures seamless data governance across departments, providing the right people with the right level of access. By leveraging Databricks' capabilities, organizations can strengthen data security, reduce risks, and optimize data-sharing practices within an auditable framework. #DataGovernance #Databricks #DataSecurity #AccessControl #DynamicViews #PrivacyCompliance #GDPR #AI #CloudSecurity

2

Good post Hoan Nguyen! But I see some disadvantages of RAG as below: 1. Methodological Limitations: RAG faces challenges due to overly simplistic methods, resulting in inefficient document processing and suboptimal search algorithms. 2. Performance Issues: Fragmented context in real-world applications leads to decreased accuracy in complex tasks, with basic retrieval methods sometimes outperforming advanced techniques. 3. Cost and Efficiency Concerns: The current RAG approach is characterized by high latency and costly retrieval processes.

4

4 (na) Komento