Wesley Backelant

REALLY digging the Data Transporter tool by Bram Colpaert. This thing is making moving data from 4 different #dataverse environments all the way up to production a sinch! The beauty of it is it keeps the same GUIDS across all environments with options to create/update/delete.

7

1 Comment

#DataMesh and #MicrosoftFabric go together like peanut butter and chocolate.... No wait... Like peanut butter and peanuts is a MUCH better example. Why? Because if you are using data, you are ALREADY using a data mesh. You just don't know it! Join Kevin Arnold and myself in 15 min for this fun conversation and DEMO. #PowerBI https://lnkd.in/gmyXiTJM

8

👩 A new customer is starting its journey on Databricks today. They need 👉 👉👉 YOU to join DATAPAO, get onboarded and start YOUR #Databricks journey. 🚀 Databricks will be the default data platform in five years, and you will be a Databricks champion by then. Shouldn't you start today? #databricks #azure

25

2 Comments

Large Language Models (LLMs) are popular in tech. In Belgium and the Netherlands, the podcast "Nerdland" is a favorite for tech and science fans. It covers topics like bioscience, space, robotics, and AI. With over 100 episodes, "Nerdland" is a goldmine of information. So, why not create a chatbot for "Nerdland" fans? This chatbot uses podcast content to engage and inform users. It allows the "Nerdland" community to interact with the content in new ways and makes the information accessible in many languages, thanks to LLMs' multi-language capabilities. This blog post explains the project's technical details, including the LLMs used, integration process, and deployment on Azure. https://lnkd.in/ddQyZ3KG #Azure #OpenAI

2

🔶 Digital sustainability 🌱 helps organisations navigate. It serves as a North Star 🌠 to unleash innovation and provides concrete, no-regret steps for advancing the twin transition. It offers organisation practical guidance on integrating digital and environmental goals to foster economic 📈 and social 👩‍👩‍👧‍👦 progress. Looking forward to discuss this and more in Bled next week, feel welcome to join! #DataSpaces #DataSharing #DigitalSustainability Innopay, Oliver Wyman

32

3 Comments

Today we are enhancing Amazon Bedrock Knowledge Bases with structured data retrieval for RAG from Amazon SageMaker Lakehouse, including Redshift, S3 and S3 Tables. Create Knowledge Bases with support for structured data stored in the Lakehouse/Redshift. #amazon #bedrock #kb #rag #structureddata

262

1 Comment

Eraneos Netherlands 𝗽𝗿𝗲𝘀𝗲𝗻𝘁𝘀: 𝗗𝗲𝗲𝗽 𝗱𝗶𝘃𝗲 𝗶𝗻𝘁𝗼 𝗔𝘇𝘂𝗿𝗲 𝗗𝗮𝘁𝗮 𝗘𝘅𝗽𝗹𝗼𝗿𝗲𝗿 | 𝗙𝗮𝗯𝗿𝗶𝗰 𝗥𝗲𝗮𝗹-𝘁𝗶𝗺𝗲 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 𝗠𝗲𝗲𝘁𝗨𝗽 𝗼𝗻 𝗝𝘂𝗹𝘆 𝟭𝟬𝘁𝗵 Spark, Databricks and Snowflake are typical technologies that come to mind when talking about ‘Big Data’. However, Azure has a fully managed analytics solution few people know about even though it handles Petabytes every day... The technology we are talking about is Microsoft Fabric’s Real-Time Analytics which is also known as Azure Data Explorer (ADX). It is purposefully built to provide real-time insights and analytics on large streaming datasets and typically used in monitoring, IoT / OT and web analytics scenarios. It’s ease of use, blazing performance and rich set of features give rise to its growing popularity. Microsoft uses it for all Azure monitoring, ingesting an estimated 18 PB of data and executing 30 Billion queries every day! During this session we will introduce Azure Data Explorer, talk about its value in real customer cases and get our hands dirty solving some ‘data puzzles’ using this awesome technology! Sign up via 👉 https://okt.to/pwqEf6 and see you there! #MeetUp #Azure #DataExplorer #ADX #BigData

3

Winsome Wim Stoop & I offer a practical guide to building your modern data architecture. Some assimilate the word #modern with #future or #emerging, but it denotes #recency, and failing to stay modern portends getting #leftbehind. #ModernDataArchitecture #DataMesh #DataFabric #OpenDataLakehouse Cloudera CDO Magazine https://lnkd.in/gzUkfj7Q

36

2 Comments

Exciting News! 🚀 I'm happy to announce that Data Catering supports the Open Data Contract Standard (Bitol) for testing data pipelines! Why is this important? 🤔 Implementing the Open Data Contract Standard ensures that data pipelines are robust, transparent, and interoperable. This standard, in combination with Data Caterer, empowers testers, developers, data engineers, and organizations to: 🔹 Simplify Testing: Use contract metadata to generate and validate data 🔹 Boost Efficiency: Streamline testing and validation processes, reducing development cycles 🔹 Improve Reliability: Detect issues before they reach production 🔹 Be Secure: No production data or connections required, fully metadata-driven At Data Catering, we believe in the power of open standards to drive innovation and collaboration. By supporting this initiative, we're not only enhancing our own services but also contributing to a more open and reliable data ecosystem. Join us in embracing this standard and take your data pipeline testing to the next level. https://lnkd.in/gQFJG4RQ

33

3 Comments

Check out my Data Story Book summary of Collibra's DATA CITIZENS 24 Show in NYC. I am always looking for the Data Stories and I found plenty! Collibra CEO & Founder Felix Van de Maele opened the day with his vision to help customers "create the unified view of their data" We all have to be on the same page!👍 McDonald's John Tucker presented his unified view in the shape of a Quarter Pounder with cheese!* 🍔 We’ll take some AI fries with that! 🍟 * No, I did not ask what they call that system in France! 😂 DATAcated’s Kate Strachnyi🤩 showed up for a group shot with John! Thomas Brence still buzzing from the amazing New York Mets victory, deftly moderated his panel through endless baseball metaphors! ⚾ David Janson led off with healthcare giant Cigna Evernorth’s "self-healing data." Then Linh Nguyen threw an imagery curveball with an ’80s pop-inspired mnemonic for financial services firm Edward Jones, where data is "T’Pau: trusted, protected, accessible, understood." Chris Tabb, COO of LEIT DATA and Mayor of #MeanDataStreets arrived in his #DATAPUPPETS T-shirt! 🐝 Michael Rahm makes the case for AI governance: “Without constant access to trustworthy data, AI projects don’t get off the ground.” There's sure a lot of traffic on that runway! ✈️ Seth Clark vows to “capture as much metadata as possible” And I believe him! He never meta data he didn’t like!😆 OMG!! Ran into Ping F. and the DQ team from NBCUniversal, including Rahel Mehreteab and Runqian (Sarah) Huang. Ping was carrying my book TELLING YOUR DATA STORY and asked “Would you sign it?” WOULD I?!?! 😎 Then CMO Denise Vu Broady wrapped up a great day of Data Stories! Thanks, Collibra! 🙏 #datastorytelling #datacatalog #datamanagement #datagovernance #aigovernace

89

18 Comments

🚀 DE Interview 📢 Cloud Optimization ⌛ 1-Min-Read Optimizing Cloud Storage Using Spark With a Housekeeping Script 🧔🏽‍♂️Interviewer: Can you explain how you optimized cloud storage using Spark with a housekeeping script? 👨‍💻Candidate: 1️⃣ In one of our projects, we were facing challenges with rapidly increasing storage costs due to accumulating data in AWS S3. To address this, I implemented an optimization strategy using Spark along with a housekeeping script. 🧔🏽‍♂️Interviewer: That sounds interesting. Could you walk me through the steps you took? 👨‍💻Candidate: 1️⃣ First, I analyzed the storage patterns and identified that a significant portion of the data was either stale or less frequently accessed. 2️⃣ I then developed a Spark job to classify this data based on its usage and relevance, segregating it into active and inactive datasets. 3️⃣ The inactive data was compressed using efficient algorithms, and a housekeeping script was created to automate the compression process and manage the data lifecycle. 4️⃣ The script would regularly run to compress, archive, and move infrequently accessed data to a low-cost storage tier while deleting unnecessary files. Versioning in S3 was enabled to safeguard against accidental deletion. 🧔🏽‍♂️Interviewer: How did this impact your overall cloud storage management? 👨‍💻Candidate: 1️⃣ The implementation of the Spark-based solution and housekeeping script led to significant cost savings by reducing the storage footprint. 2️⃣ It ensured that only relevant data occupied primary storage layers, improving data retrieval speeds and system efficiency. 3️⃣ Additionally, the automated process minimized manual intervention, reducing errors and operational overhead. Summary: 1️⃣ Problem: Escalating cloud storage costs and inefficient data management. 2️⃣ Solution: Implemented Spark-based optimization and automated housekeeping script for data lifecycle management. 3️⃣ Impact: Reduced storage costs, enhanced data management efficiency, and streamlined cloud operations. #CloudOptimization #Spark #AWS #DataManagement #CostSavings #Automation #TechInnovation #CloudComputing #S3 #DataLifecycle

14

Super excited about the introduction of LakeFlow by Databricks! LakeFlow connect is more than welcome to facilitate data ingestion. However, the keynote left some questions unanswered. Will the pipeline component continue to support Python/PySpark like DLT does? 🤔 Fingers crossed! What are your thoughts on this new development?

4

Iceberg performance on Snowflake is almost as fast as native tables & 25x faster than #external tables on parquet files. Just because both options use #parquet files as the storage method does not mean they work or perform the same

30

Avoid these 7 Common Business-related Mistakes On Data Projects https://lnkd.in/eV_WGAbh #kamphuisgroep #caronne # #ArtificialIntelligence #MachineLearning #bigdata #businessintelligence

3

Great blog on Spark AQE!

12

Are you wondering how Microsoft Fabric and Databricks can work together? Piethein Strengholt wrote a nice article about it with several options on how to use both strengths with all pros and cons for each combination https://lnkd.in/eEN9JkFu #Microsoft #MicrosoftFabric #AzureDatabricks

127

Databricks SQL is one of the fastest-growing software launches in recent memory - going from 0 to 8,000 customers since our 2022 launch. And since then we've become 4x faster! And with less $$$, serverless option, governance through Unity Catalog, and more -- it's no wonder so many companies are migrating their data warehouses. "Simply brilliant" was how one customer phrased it to me last week whilst I was in Amsterdam. OK, so now that you are excited what have been recent product updates? Read here: https://lnkd.in/gdcgwc5U #databricks #dbsql #dw #datawarehousing #data #ai

17

2 Comments

𝗝𝗼𝗶𝗻 𝘂𝘀 𝗱𝘂𝗿𝗶𝗻𝗴 𝗼𝘂𝗿 𝗱𝗲𝗲𝗽 𝗱𝗶𝘃𝗲 𝗶𝗻𝘁𝗼 𝗔𝘇𝘂𝗿𝗲 𝗗𝗮𝘁𝗮 𝗘𝘅𝗽𝗹𝗼𝗿𝗲𝗿 𝗼𝗻 𝗝𝘂𝗹𝘆 𝟭𝟬𝘁𝗵! During this session we will introduce Azure Data Explorer, talk about its value in real customer cases and get our hands dirty solving some ‘data puzzles’ using this awesome technology 🚀 Spark, Databricks and Snowflake are typical technologies that come to mind when talking about ‘Big Data’. However, Azure has a fully managed analytics solution few people know about even though it handles Petabytes every day... The technology we are talking about is Microsoft Fabric’s Real-Time Analytics which is also known as Azure Data Explorer (ADX). It is purposefully built to provide real-time insights and analytics on large streaming datasets and typically used in monitoring, IoT / OT and web analytics scenarios. It’s ease of use, blazing performance and rich set of features give rise to its growing popularity. Microsoft uses it for all Azure monitoring, ingesting an estimated 18 PB of data and executing 30 Billion queries every day! Join our MeetUp via 👉 https://okt.to/zmOCdW #MeetUp #Azure #DataExplorer #ADX #BigData

3