Frederik Huetzen

👇🏻 Check out how Erste Group is leveraging synthetic data to democratize #Data & #AI for everyone using MOSTLY AI integrated with the Databricks #DataIntelligence Platform 👇🏻

11

Ready for the next dbt Talk? My upcoming session on Thursday 04.07 14hs CET is all about "Materialization strategies in dbt + Data Vault". I will start by discussing the different materializations available on dbt to then connect them into how we use them on the different Data Vault 2.0 layers and entities. Join me: https://scalefr.ee/46f3p0

6

Data Engineering: Working with semi-structured data types. In our previous post, we demonstrated how to parse comma-separated data into a structured file format within the Snowflake project that utilizes the medallion architectural framework. In this post, we’ll introduce another file format that you’ll frequently encounter in your data engineering projects: the JSON File Format. JSON (JavaScript Object Notation) is an open standard file format used for sharing data. It employs human-readable text to store and transmit data objects, which consist of attribute-value pairs and arrays. This file format is commonly used for transmitting data between web applications and servers, making it very popular and suitable for our discussion and demonstration. To explain in more detail, we have three common types of JSON: JSON objects, JSON nested objects, and JSON arrays. JSON objects: “User_Name” :{ “Fist_Name”: “Henry”, “Last_Name”: “Godson” } Nested JSON objects: “User_Informations”: { “Date”: “2024-05-21”, “Sports”: {“Football”: “Good”, “Swimming”: “Excellent” } “Name”: “Henry } JSON arrays: (JSON are written inside the square brackets). “Employees”: [ { “First_Name”: “Henry”, “Last_Name”: “Godson” }, { “First_Name”: “Endy”, “Last_Name”: “Junior” }, ] We also have JSON Array in the simple format as “Favourit_Sports”: [“Football”, “Baseball”]. While certain data migration tools, such as Azure Data Factory, possess built-in intelligence for transforming data from one format to another, most data engineering ETL projects necessitate data engineers to perform similar transformations using either SQL or Python. To transform a JSON array using the Medallion Architectural Framework, we have two major approaches: Copy to Bronze and Transform: In this approach, we copy the JSON file into the Bronze layer and then transform the data to a structured format before moving it into the Silver table. Direct Transformation to Bronze: The second approach involves transforming the data directly to a structured format and depositing it into the Bronze layer straight from the staging environment. This approach is recommended, and we’ll demonstrate how to achieve it in our upcoming post. Stay tuned!

3

Join Databricks on November 7th for Data + AI World Tour Munich! Discover how leading companies like Fraport AG, Deichmann SE, Bayer AG, and CARIAD SE (A Volkswagen Group Company) are taking control of their data and building custom AI on the Databricks Data Intelligence Platform. Register now!

14

🚀 Data Warehouse vs. Data Lake: Which One Should You Choose? 🚀 When it comes to modern data management, Data Warehouses and Data Lakes are two of the most powerful and popular approaches. But how do you choose the right one for your business? 🤔 Here's a quick breakdown: 🔹 Data Warehouse: Ideal for structured, processed data, tailored for Business Intelligence (BI) users and reporting. It supports enterprise-wide decision-making, but comes with a longer implementation time due to its complexity. 🔹 Data Lake: Best suited for Data Engineers and Data Scientists who work with raw, unstructured data. It offers more flexibility, enabling broader analysis at scale, and it's perfect for machine learning and advanced analytics. 📊 Key Differences: Scope: Data Warehouse = organization-wide | Data Lake = specific departments Data: Data Warehouse handles massive, processed data | Data Lake stores raw data Users: BI users vs. Data Engineers & Data Scientists Which one fits your needs? Let me know your thoughts in the comments below! 👇 #DataScience #BigData #BusinessIntelligence #DataAnalytics #CloudComputing #DataLakes #DataWarehousing #MachineLearning #Snowflake

3

1 Kommentar

𝗘𝘅𝗰𝗶𝘁𝗶𝗻𝗴 𝘁𝗶𝗺𝗲𝘀 𝗮𝘁 𝘁𝗵𝗲 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗗𝗮𝘁𝗮 + 𝗔𝗜 𝗦𝘂𝗺𝗺𝗶𝘁 𝟮𝟬𝟮𝟰! 🚀 As a proud member and partner of the Databricks community, we are thrilled to share the latest innovations that will transform the way we work with data and AI. 𝗦𝗲𝗿𝘃𝗲𝗿𝗹𝗲𝘀𝘀 𝗖𝗼𝗺𝗽𝘂𝘁𝗲: Databricks announced the general availability of its serverless computing feature, allowing users to only pay for what they use and eliminating versioning. This is a major step towards the future of the platform, with autoscaling and the Photon query engine automatically enabled. 𝗟𝗮𝗸𝗲𝗵𝗼𝘂𝘀𝗲 𝗜𝗤: The new Lakehouse IQ feature is a game-changer, providing a knowledge engine that learns the nuances of your business to enable natural language access to data. No more complex SQL queries - just ask and get the insights you need! 𝗨𝗻𝗶𝘁𝘆 𝗖𝗮𝘁𝗮𝗹𝗼𝗴 𝗢𝗽𝗲𝗻 𝗦𝗼𝘂𝗿𝗰𝗶𝗻𝗴: Databricks is open-sourcing its Unity Catalog, the unified data governance solution. This will allow developers to contribute and further enhance this powerful tool for managing data assets. 𝗡𝘃𝗶𝗱𝗶𝗮 𝗣𝗮𝗿𝘁𝗻𝗲𝗿𝘀𝗵𝗶𝗽: Databricks is expanding its collaboration with Nvidia, bringing native support for Nvidia GPUs and accelerated computing to boost the efficiency, accuracy, and performance of AI development pipelines. 𝗠𝗼𝘀𝗮𝗶𝗰 𝗔𝗜 𝗨𝗽𝗱𝗮𝘁𝗲𝘀: Enhancements to the Mosaic AI unified tooling product, including improved support for building AI systems, model quality, and governance capabilities. 𝗧𝗮𝗯𝘂𝗹𝗮𝗿 𝗔𝗰𝗾𝘂𝗶𝘀𝗶𝘁𝗶𝗼𝗻: Databricks has acquired Tabular, this will ensure seamless integration and management of Iceberg tables. 𝗢𝗽𝗲𝗻 𝗩𝗮𝗿𝗶𝗮𝗻𝘁 𝗗𝗮𝘁𝗮 𝗧𝘆𝗽𝗲: Faster processing and more flexibility when working with semi-structured data Many More.. These announcements showcase Databricks' commitment to innovation and their mission to democratize data and AI. I can't wait to see how these new features will transform the way we work. Let me know your thoughts in the comments! #Data #databricks

9

Join Databricks on November 7th for Data + AI World Tour Munich! Discover how leading companies like Fraport AG, Deichmann SE, Bayer AG, and CARIAD SE (A Volkswagen Group Company) are taking control of their data and building custom AI on the Databricks Data Intelligence Platform. Register now!

15

How do data teams keep their CFOs happy? By keeping costs low & delivering value with their Data + AI. We're making some big advancements to help customers manage their costs. Check out this 3-part series with our "Money Team" Greg Kroleski and MVP Josue A. Bogran to understand what this looks like. Part 1: 👉 Why we're investing in cost management 👉 How to get started using tags 👉 Cost Allocation/Tag Implementation Challenges Part 2: 👉 Why aren't tags used more? 👉 Changes/Improvements Databricks has made to cost reporting in 2024 👉 Why should users trust Databricks' efforts to improve performance? 👉 How can Databricks' Cost Dashboards help? Part 3: 👉 Costly mistake prevention mechanism inside of Databricks' Serverless 👉 AI/BI Genie for cost reporting inside of Databricks 👉 Balancing between user managed vs Databricks managed cost controls #FinOps

2

We woke up a bit hungry today, so... are you ready to mix some data modelling tips and a bit of food content? Query-driven, money-driven, to-Spain-driven... Here's our own Thomas in't Veld talking about how the domain model is like the Bacon 🥓 in a BLT. #DataModelling #DataWarehouse #Bacon #Casualfriday

7

Next stop: Berlin, Germany 🇩🇪 "We've had failed data catalog implementations before. We realized all the potential falls apart when adoption struggles," shared a Berlin data leader, cutting straight to a universal truth in data governance. Their solution? Clear OKRs. "That's the only thing that has helped us align the entire organization toward creating the value we want," they explained. This focus on measurable outcomes has become their north star in driving adoption. But the conversation quickly turned to AI. "It's fascinating - just a few years ago, AI was just a buzzword. Now it's table stakes," they noted. The shift has been dramatic: users expect AI to unblock teams, eliminate manual effort, and deliver hyper-personalized solutions. The message from Berlin is clear: the race isn't just about implementing tools anymore - it's about meeting rapidly evolving user expectations while ensuring sustainable adoption across the organization.

20

I submitted the following papers to #apexconn25: - Fighting Bad PL/SQL & SQL with VS Code - The Trivadis PL/SQL & SQL Coding Guidelines are Dead – What Now? Since I feel bad submitting only in German, I tried something new. I submitted the "same" abstract in both English and German.

17

❄ Super simple tip to boost your Snowflake's query performance 👇   Select fewer columns.   I told you that was simple.   Ok, but seriously.   Why it matters?   Snowflake is based on micro-partitions which stores data in columnar format with byte-range.   It means every time you run a query, engine will search for data only in particular micro-partitions, thus avoid scanning all of them.   In simple words, it checks if particular micropartition contains data in range needed by a query.   So, by selecting less columns, query runs faster.   But wait…   If your query runs on a virtual warehouse and you pay for every second your warehouse runs…   It's not only faster, but also a cheaper :)   "So shall I now leave everything and check if my queries are using only columns which they really need?"   Not really, but as requirements change over time, it's good to check them once a while.   ♻️ Repost if you found this useful.   ❤️ Follow me for tips and insights that will help you level up your data career!

106

12 Kommentare

💡 𝗘𝘅𝗽𝗹𝗼𝗿𝗲 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀 𝘄𝗶𝘁𝗵 𝗢𝘂𝗿 𝗟𝗮𝘁𝗲𝘀𝘁 𝗕𝗹𝗼𝗴 💡  𝗦𝗽𝗮𝗿𝗸 𝗗𝗮𝗻𝗴𝗲𝗿: 𝗣𝗶𝘃𝗼𝘁 𝗶𝘀 𝗮𝗻 𝗔𝗰𝘁𝗶𝗼𝗻! Curious about Spark? Explore our latest blog post to discover the real impact of the pivot operation on your Spark applications. Our experts, Michał Możdżonek and Sebastian Skiba reveal surprising performance issues in Apache Spark in their article ’Spark Insight: Pivot is an Action!’. 👨‍💻 This blog highlights often overlooked aspects of Spark that might be slowing down your applications. The pivot function, usually seen as a simple transformation can make a mess of your code if you don’t use it correctly  Packed with code examples and optimization tips, this article is a must-read for anyone looking to expand their Spark knowledge. 📖 Whether you’re tweaking your current projects or starting with Spark, our blog provides valuable insights to help you spot potential problems and improve data processing efficiency. Think you know Spark? Think again! Check out our blog to learn more ⬇  https://lnkd.in/djrxCnd4 #ApacheSpark #BigData #DataScience #DataEngineering #DataDebugging #Programming #ProblemSolving #Datumo

20

4 Kommentare

With the latest version, Nussknacker has become a powerful tool for those working with Apache Iceberg based Data Lakehouses. It can handle both 𝗯𝗮𝘁𝗰𝗵 𝗮𝗻𝗱 𝗻𝗲𝗮𝗿 𝗿𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 workloads. By integrating Nussknacker with Apache Iceberg, you can do the following: 🟢 Ingest data: Load data into your Data Lakehouse. 🟠 Transform data: Clean, filter, and restructure your data. 🔴 Aggregate data: Summarize and group data. 🔵 Enrich data: Use ML inference, joins etc to add context to your data. 🟣 Apply 𝗯𝘂𝘀𝗶𝗻𝗲𝘀𝘀 𝗹𝗼𝗴𝗶𝗰 to data In this blog post, Arkadiusz Burdach will show you how to use Nussknacker to build a data pipeline in this setup 👉 https://lnkd.in/dQg48qSi #Iceberg #DataLakehouse

22

Do you need to store raw events in the lakehouse for performing analytics and training ML models on structured and semi-structured data? 📈 Organizing and managing vast quantities of structured and semi-structured data in object storage is easy with Iceberg’s open table format. After data is loaded into the lakehouse, you can evolve or rearrange large datasets without facing costly rewrites. Data can be accessed from an OLAP engine such as ClickHouse as well as other modern processing engines like Apache Spark and Flink. Upsolver jobs enable you to load your data where needed, ensuring consumers have the latest information at their fingertips. You can load data into tables in the lakehouse, where we will continuously stream your data and manage table optimization to ensure files are compacted for maximum performance. Book a demo to see how you can create jobs with Upsolver to move your data where you need it [link is in the comments]. #ClickHouseCloud #Upsolver #DataAnalytics #RealTimeInsights #DataTransformation

10

2 Kommentare

𝗡𝗘𝗪 𝗕𝗟𝗢𝗚 𝗣𝗢𝗦𝗧: 𝗪𝗵𝘆 𝘆𝗼𝘂 𝘀𝗵𝗼𝘂𝗹𝗱 𝘀𝘁𝗮𝗿𝘁 𝘂𝘀𝗶𝗻𝗴 𝗨𝗻𝗶𝘁𝘆 𝗖𝗮𝘁𝗮𝗹𝗼𝗴 We noticed that there are still lots of organizations that use Databricks but do not leverage Unity Catalog. Hopefully, this blog can show the added value and give these organizations an extra boost. ✅Firstly, we briefly explain the basis of Unity Catalog. ✅Secondly, some examples are given to increase data governance in Databricks by using simple SQL queries. ✅Thirdly, we display how lineage makes it possible to visually retrace your data to the source. Fourthly, the need for data exploration is stressed, enhancing the searchability of your data. ✅In addition, by limiting the amount of searchable data, it is shown how Unity Catalog can help facilitate self-service. ✅Lastly, we deep dive into one of the most recent features called Delta Sharing, which makes data sharing across organizations a walk in the park. Read the full blog post via the link below 👇🏻 https://lnkd.in/eznAqbXu #blogpost #IntellusGroup #UnityCatalog

1

Have you tried running the DBRX models from Databricks? In our latest blog post we provide a tutorial to get you started with the model on Ori’s public cloud. We were very impressed with DBRX’s performance on the Hugging Face Open LLM Leaderboard before the updates last week. While DBRX Instruct did not score as highly under the new scoring system, it's still one of the best open models for Language Understanding (MMLU), Programming (HumanEval), and Math (GSM8K). We also used BeFOri, the benchmarking framework from Ori to determine DBRX Base performs inference most efficiently between 40 and 256 concurrent requests on one of Ori's 8 x H100 PCIe bare metal servers. Check out the blog to learn how you can use BeFOri to optimise self-hosted LLM performance on any infrastructure! https://lnkd.in/gi55WEks

43