Geert Van Landeghem

Why does Stijn (Stan) Christiaens the Co-Founder of Collibra have the title 'Chief Data Citizen'? He explained all on Driven By Data: The Podcast a couple of weeks ago! A great episode which you can listen to below ⬇️ #data #value

38

5 Comments

The chapter 1 | A world underwater of 📖 Data integration a piece of cake 🍰is now available! In this chapter Raymond Meester takes us into the invisible layer of data integration. Even though we don't see it: it is everywhere! Read Chapter 1 | A world underwater: https://hubs.la/Q02rYw950 #dataintegration #storytelling #book #integrationmadeeasy

4

Exciting News! 🚀 I'm happy to announce that Data Catering supports the Open Data Contract Standard (Bitol) for testing data pipelines! Why is this important? 🤔 Implementing the Open Data Contract Standard ensures that data pipelines are robust, transparent, and interoperable. This standard, in combination with Data Caterer, empowers testers, developers, data engineers, and organizations to: 🔹 Simplify Testing: Use contract metadata to generate and validate data 🔹 Boost Efficiency: Streamline testing and validation processes, reducing development cycles 🔹 Improve Reliability: Detect issues before they reach production 🔹 Be Secure: No production data or connections required, fully metadata-driven At Data Catering, we believe in the power of open standards to drive innovation and collaboration. By supporting this initiative, we're not only enhancing our own services but also contributing to a more open and reliable data ecosystem. Join us in embracing this standard and take your data pipeline testing to the next level. https://lnkd.in/gQFJG4RQ

33

3 Comments

𝗝𝗼𝗶𝗻 𝘂𝘀 𝗱𝘂𝗿𝗶𝗻𝗴 𝗼𝘂𝗿 𝗱𝗲𝗲𝗽 𝗱𝗶𝘃𝗲 𝗶𝗻𝘁𝗼 𝗔𝘇𝘂𝗿𝗲 𝗗𝗮𝘁𝗮 𝗘𝘅𝗽𝗹𝗼𝗿𝗲𝗿 𝗼𝗻 𝗝𝘂𝗹𝘆 𝟭𝟬𝘁𝗵! During this session we will introduce Azure Data Explorer, talk about its value in real customer cases and get our hands dirty solving some ‘data puzzles’ using this awesome technology 🚀 Spark, Databricks and Snowflake are typical technologies that come to mind when talking about ‘Big Data’. However, Azure has a fully managed analytics solution few people know about even though it handles Petabytes every day... The technology we are talking about is Microsoft Fabric’s Real-Time Analytics which is also known as Azure Data Explorer (ADX). It is purposefully built to provide real-time insights and analytics on large streaming datasets and typically used in monitoring, IoT / OT and web analytics scenarios. It’s ease of use, blazing performance and rich set of features give rise to its growing popularity. Microsoft uses it for all Azure monitoring, ingesting an estimated 18 PB of data and executing 30 Billion queries every day! Join our session via MeetUp 👉 https://okt.to/zTj5EP #MeetUp #Azure #DataExplorer #ADX #BigData

5

🇳🇱 European DataWarehouse's 2024 Dutch Securitisation Event is currently underway, where Nationale-Nederlanden (Woonnu)'s Martijn Breed & Koen Cop are presenting their experience, as an #ENGAGE4ESG Pilot, in extracting data to complete the #ENGAGETemplates. ▶ In case you were unable to join the event in #Amsterdam today, you can learn more about #NNBank's data extraction process by watching last month's webinar on-demand here: https://lnkd.in/eUE3vquY 📽 You can also hear more from Martijn in the short interview below. "Wie is Woonnu? Wat is de rol van Woonnu in ENGAGE?" *ENGAGE is co-funded by the European Union #EUGreenDeal #LIFEprogramme #SustainableFinanceEU #EUTaxonomy #GreenTransition #FitFor55 #securitisation #ESG #greenfinance

8

With release 2024.05, Collibra offers their newest UI/UX in DIC. Curious what these changes mean for you as an organisation? Read all about it in our latest blog! 👉 https://lnkd.in/eGN5WJiJ. #userinterface #collibra #dataintelligence

22

𝐃𝐨 𝐲𝐨𝐮 𝐰𝐚𝐧𝐭 𝐭𝐨 𝐮𝐧𝐥𝐨𝐜𝐤 𝐭𝐡𝐞 𝐟𝐮𝐥𝐥 𝐩𝐨𝐰𝐞𝐫 𝐨𝐟 Databricks 𝐟𝐨𝐫 𝐲𝐨𝐮𝐫 𝐝𝐚𝐭𝐚? Then check out our new “Databricks in a Week” workshop! In this 5 day workshop we will (optionally together with you): ✅ rapidly prototype and validate your data-driven product or solution ✅ showcase the Databricks technology hands-on with your data in your Databricks account ✅ demonstrate feasibility and value to your stakeholders Sounds good? Then head on over to our website and get in touch with our team! 👉 https://lnkd.in/dxHyFgfK What is your biggest challenge with Databricks at the moment? Let us know in the comments and let’s discuss! #databricks #data #DataIntelligence #DatabricksInAWeek

36

1 Comment

REALLY digging the Data Transporter tool by Bram Colpaert. This thing is making moving data from 4 different #dataverse environments all the way up to production a sinch! The beauty of it is it keeps the same GUIDS across all environments with options to create/update/delete.

7

1 Comment

Large Language Models (LLMs) are popular in tech. In Belgium and the Netherlands, the podcast "Nerdland" is a favorite for tech and science fans. It covers topics like bioscience, space, robotics, and AI. With over 100 episodes, "Nerdland" is a goldmine of information. So, why not create a chatbot for "Nerdland" fans? This chatbot uses podcast content to engage and inform users. It allows the "Nerdland" community to interact with the content in new ways and makes the information accessible in many languages, thanks to LLMs' multi-language capabilities. This blog post explains the project's technical details, including the LLMs used, integration process, and deployment on Azure. https://lnkd.in/ddQyZ3KG #Azure #OpenAI

2

Have you ever wondered what it means to be a Data Engineer and Cloud Architect? 🤔 Our Coder, Marc Vea, tells you about his journey to become a Data Engineer: his studies, experience and what he enjoys the most on the day to day. 👏 Do you want to know more about our roles and positions? Let us know in the comments! 💬👇 #SEATCODE #Coders #Careers #Jobs #Engineer

38

Our colleagues, Martijn Severijns and Kevin Segers, that were present at the Collibra #DataCitizens event in Orlando reconnected with old data friends, met new ones and above all, they kept their eyes and ears open. Here are Datashift's top 3 takeaways from Data Citizens 2024: 1. Empowering Data & IT teams The presentations on 'push & pull metadata', 'harnessing active metadata', and 'the role of CI/CD & data pipelines' underscore the imperative for all departments, especially Data & IT teams, to drive a cultural shift towards effective Data Governance. It's evident that proactive steps in this direction are essential for organizational transformation. 2. Exciting developments at Collibra Having worked with Collibra for years, I must say, this latest release is monumental. The strides made in UX, Data Quality, Data Catalog/Lineage, Data Products, and AI Governance are truly remarkable. I encourage everyone to stay updated via official Collibra channels to explore these and upcoming releases. 3. Leveraging AI for enhanced data intelligence It's a no-brainer that utilizing AI to accelerate and refine your data intelligence program is crucial. However, ensuring proper governance in this aspect is equally vital. Let's commit to both aspects to drive meaningful outcomes. Data Ownership/Stewardship, Change Management, and Data Literacy also remain crucial topics in the sphere of Data Governance in 2024. Want to dive deeper on any of the above topics? We are always happy to talk and ready to help!   #datagovernance #dataintelligence

41

Many data leaders are asking what are data mesh and data fabric? How does my business successfully use them? Semarchy joined DBTA in a roundtable webinar discussing data fabric and data mesh architectures and how companies can utilize them. Watch the replay to learn more: https://hubs.ly/Q02y8tMm0 #mdm #dataplatform #datamesh #datafabric #datamanagement #data #dataarchitecture

3

Hello, everyone! I have witnessed experienced Data Engineers in the community, in particular 👨🏻‍💻 Jaco van Gelder, complaining that it is pretty unfortunate that we rarely see unit tests in data pipelines -- and they are right! Although Data Engineering is not Software Engineering, best practices such as testing and documentation are applicable and should be the standard. So, if your data team is still in this phase where it knows tests are important but still don't know how to start implementing them, check this article where we use Dagster and dlt to create and manage pipelines in a way that makes unit tests easy to implement and integrate in the development workflow. https://lnkd.in/eW3Ri_4R #dataplatformengineering #dagster #dlt #testing

94

10 Comments

Avoid these 7 Common Business-related Mistakes On Data Projects https://lnkd.in/eV_WGAbh #kamphuisgroep #caronne # #ArtificialIntelligence #MachineLearning #bigdata #businessintelligence

3

How to create data for your built-in database! 👇 In this video Tristan Van Poucke explains how you can easily create data within your data formats. 😎 ** NoCode-X simplifies data management with flexible data formats and intuitive creation tools. Learn how to: ➡ Define Data Formats: Create custom data formats like 'Soccer Team' with attributes such as team name and number of players. ➡ Add Properties and Constraints: Specify data types, set character limits, and add descriptions to ensure data accuracy. ➡ Generate Data: Populate your database with new data entries directly from the back office or via API, pages, or files. Watch the full video to see how easy it is to manage your data with NoCode-X! 🚀 #nocode #technology #dataformats

5

1 Comment

New post, and new website look all in one ! In this post I go over how I figured out how to use Python code in Databricks to generate a dynamic list of tables using Delta Live Tables! One notebook to ingest thousands of tables ? Sounds pretty cool to me. Check it out and let me know your thoughts ! https://lnkd.in/d-NnMZEk #databricks #dataengineering #data #automation #deltalivetables #dlt

15

1 Comment

Data Fabric OR Data Mesh? 🤔 A question that has been on your mind finally answered! Jonas De Keuster from one of our Sponsors VaultSpeed is giving you BOTH. Unlock the potential of your data with a hybrid architecture for data management. Also Unlock the potential of “Big Food” at today’s Lunch also sponsored by Vaultspeed 😶‍🌫️ #DataFabric #DataMesh #DataManagement

11

“If data is the new gold, then you could say reliable and authentic data is the new platinum.” – Frederik Rosseel, CEO at Docbyte Maxime Vermeir and Frederik Rosseel unpack the importance of quality, integrity and authenticity in the data used to train AI models in this episode of the AI Pulse Podcast by ABBYY. Want to hear these AI experts explore the role of business data in ensuring trustworthy AI? Check out their conversation here: https://hubs.li/Q02w2NrK0 #AI #ArtificialIntelligence #IntelligentAutomation #Data #Privacy #TrustworthyAI #GenerativeAI #LLMs #Podcast

25

🚀 DE Interview 📢 Cloud Optimization ⌛ 1-Min-Read Optimizing Cloud Storage Using Spark With a Housekeeping Script 🧔🏽‍♂️Interviewer: Can you explain how you optimized cloud storage using Spark with a housekeeping script? 👨‍💻Candidate: 1️⃣ In one of our projects, we were facing challenges with rapidly increasing storage costs due to accumulating data in AWS S3. To address this, I implemented an optimization strategy using Spark along with a housekeeping script. 🧔🏽‍♂️Interviewer: That sounds interesting. Could you walk me through the steps you took? 👨‍💻Candidate: 1️⃣ First, I analyzed the storage patterns and identified that a significant portion of the data was either stale or less frequently accessed. 2️⃣ I then developed a Spark job to classify this data based on its usage and relevance, segregating it into active and inactive datasets. 3️⃣ The inactive data was compressed using efficient algorithms, and a housekeeping script was created to automate the compression process and manage the data lifecycle. 4️⃣ The script would regularly run to compress, archive, and move infrequently accessed data to a low-cost storage tier while deleting unnecessary files. Versioning in S3 was enabled to safeguard against accidental deletion. 🧔🏽‍♂️Interviewer: How did this impact your overall cloud storage management? 👨‍💻Candidate: 1️⃣ The implementation of the Spark-based solution and housekeeping script led to significant cost savings by reducing the storage footprint. 2️⃣ It ensured that only relevant data occupied primary storage layers, improving data retrieval speeds and system efficiency. 3️⃣ Additionally, the automated process minimized manual intervention, reducing errors and operational overhead. Summary: 1️⃣ Problem: Escalating cloud storage costs and inefficient data management. 2️⃣ Solution: Implemented Spark-based optimization and automated housekeeping script for data lifecycle management. 3️⃣ Impact: Reduced storage costs, enhanced data management efficiency, and streamlined cloud operations. #CloudOptimization #Spark #AWS #DataManagement #CostSavings #Automation #TechInnovation #CloudComputing #S3 #DataLifecycle

14