Joris Van den Borre

🚀 DE Interview📢Data Imputation⌛1-Min-Read 🧔‍♂️Interviewer: Can you explain how you improved data accuracy in your project, leading to significant business impact? 👶Candidate: In our project, we were facing issues with inaccurate business results due to missing values in our raw data layer. We knew which records were missing values and had a clear understanding of what those values should be. To tackle this, I developed a data imputation framework and created a new layer as a parallel pipeline. Here’s how I approached it: Data Imputation: I designed a framework that identified missing values and filled them with the appropriate data. This involved analyzing patterns and ensuring the imputed values were accurate and relevant. Parallel Pipeline: I set up a parallel pipeline to compare the imputed data with the actual data. This allowed us to validate the effectiveness of the imputation process. Comparison and Validation: By running the imputed data through the pipeline, we could see that the imputed data was much closer to the actual results compared to the raw data. Production Implementation: Due to the significant improvement in data accuracy, the team decided to move my imputation code into production. This ensured that business results were more accurate and reliable. By implementing this data imputation framework, we achieved several impacts: Increased Accuracy: The imputed data significantly reduced discrepancies, resulting in more accurate business insights. Improved Efficiency: Automating the imputation process saved time and resources, allowing the team to focus on more critical tasks. Enhanced Reliability: The production implementation of the imputation framework ensured consistent and reliable data quality. Real-time Validation: The parallel pipeline provided real-time insights, helping us quickly identify and address any data issues. Summary: 1️⃣ Problem: Inaccurate business results due to missing values in raw data. 2️⃣ Solution: Developed a data imputation framework and parallel validation pipeline. 3️⃣ Impact: Increased accuracy, improved efficiency, and enhanced data reliability. #DataEngineering #DataImputation #DataAccuracy #Automation #Python #DataValidation #TechInnovation #BusinessIntelligence #DataQuality #Efficiency

22

1 Comment

I used to think complex models were everything in AI. Then I learned: real value starts with simple, goal-aligned data. Simplicity drives results. 🧩 Want to hear more lessons from my data journey? Hit follow! #AIstrategy #DataDriven

5

1 Comment

Where should you handle data modeling and joining? The data warehouse? The visualization tool? As an analyst, it can be tempting to do this in the visualization tool (Tableau, DOMO, CJA, Excel). You can start creating right away without having to call on BI and the data engineers. But in general, it is better to handle the modeling and joining in the warehouse for the following reasons: ▶ Performance Data warehouses such as Snowflake and Big Query were built for this purpose. A SQL query joining millions of rows will run much faster than a VLOOKUP on 100k rows in Excel. ▶ Source of Truth When analysts join their own datasets in the viz tools, you end up with conflicting numbers. It is best practices to work off a single source of truth in your warehouse and have all the reporting sit on top of that foundation. ▶ Scalability Joining 100k rows in Tableau or DOMO might not be too bad. But what happens when you grow to millions, or billions of rows? Today's data warehouses were purpose-built to process these data sizes. --> Analysts, it is important to partner with your data engineers (or become a data engineer yourself) and guide them to the data sets you need to reach company goals. #data #marketinganalytics #datawarehouse #snowflake #bigquery

12

3 Comments

This morning, I came across a post by Martyn Booth about the new INFO.VIEW DAX function in Power BI, introduced in the October release. I just implemented it in one of my reports, and it’s super cool! No more boring manual documentation—this function fully automates the process. Very useful function Check out his blog here: https://lnkd.in/dxJMjXQn #PowerBI #DAX

13

3 Comments

Sometimes, when we get used to using a tool, we can overlook some functionality.  I recently learned about a superb Power BI option from Jeroen Dekker: improving all font sizes in a table and matrix in Power BI at once.  This only works if you have not yet manually changed the font size before (if you did, you can easily reset it to default). #PowerBI #DataViz #DataVisualization 

43

1 Comment

Did I quit my own 30-Day Alteryx Challenge? No. Shifted priorities? Yes. And I feel awful about it. Let me explain. I made this video two weeks ago. At the time, I was devoting most of my time to getting my new website up and running. Also I had quite a lot of business and clients to close out on the old platform. I continued to work on that for the rest of the month. Yes, I've made some halfhearted content efforts, but nothing much. To those few who were following the Alteryx Advanced Cert challenge, I will get back to it soon. The site and what follows is going to be awesome. In the meantime, there's always guilt and shame when you can't follow through on a promise or a goal. Whether the promise was to someone else, or to yourself. But when you figure out for certain that you have a higher priority, and only time to devote to the number one goal, you shift fires. I've made my peace with it. Back with better things soon. How do you shift priorities? 🪖

35

15 Comments

Data Quality is so important, even more now that we are dying to use #ai. We are all still struggling to get the right level of quality for our purposes. The Working Group Data Quality of DAMA NL is working hard on stepping up the game! They are working on extending all the good work that they have done to a Data Quality Management System. Not just a framework, but a real MS, that can be defined and even externally audited! Look at some of the ISO Management Systems: - ISO 9001 - Quality Management Systems - ISO 14001 - Environmental Management Systems - ISO 45001 - Occupational Health and Safety Management Systems - ISO/IEC 27001 - Information Security Management Systems - ISO 50001 - Energy Management Systems - ISO 22000 - Food Safety Management Systems That is what they are working on, now for data quality! Very cool and necessary! These people seem to have a lot of fun, look at the knowledge graph they made and you can find out all on: https://lnkd.in/egqu3Uib We will hear more from them, for sure at our yearly DAMA NL conference in Culemborg on 17 September 2024. We will also have Peter Aiken, President of DAMA International as a speaker. Tickets can be purchased on https://lnkd.in/ehzsSwUj DAMA International #data #datamanagement #dataquality #dama #dmbok

90

23 Comments

💥 CONTEST RESULTS: GIFTS FOR EVERYONE! 💥 📚 Two weeks ago, we launched a review contest for my new book "Data Visualization with Microsoft Power BI: How to Design Savvy Dashboards" and got so excited about showering you with gifts and bonuses that we slightly overcomplicated the mechanics. We admit it! 🤩 BUT! We're used to turning mistakes into advantages, so EVERYONE who participated gets gifts, no exceptions! 1. The review contest now runs forever! Leave a review on Amazon and get a Power BI guideline with designed charts and dashboard templates as a gift (send us your screenshot). 2. And for those who missed the contest, we're attaching the book trailer with brief information about our wonderful book, which you can still buy on Amazon (drop a comment to get the link). 3. Thanks to everyone who's already participated, check your DMs for the digital copy of the book. 🔥 Thanks to our community for teaching us about simplicity. Sometimes the best visualization is one clear chart, and the best contest is when everyone gets gifts!

39

2 Comments

Copilot in Power BI: yet another AI solution in search of a problem it can solve.😐 It's hard to read this very extensive test of #Copilot capabilities in BI scenarios and not to think that the product should not have been released yet. Because these tools in particular are not supposed to be toys you can break. The numbers need to be correct - because otherwise what's the point of business intelligence?📉 Kurt Buhler tells it like it is: "The reality of this is surreal – that you might expect a business user to base important decisions on a tool that literally says on the front of the tin 'FYI, I can’t be trusted'. What exactly is the expectation, here… that the business user should tolerate this? That they’ll risk making a bad decision; that they’ll just wing it?" As usual, the promise here is that it will get better over time. I'm sure it will, because getting any worse would be quite an achievement... Who would have thought that a Large Language Model would not be the ideal user experience for tools like Excel and Power BI that are not about language? I do believe that LLMs can become useful assistants for writing formulas. Yet if Nadella hadn't declared MS to be The Copilot Company, neither of these AI assistants would surely be out there yet - for customers to use and get disappointing results. Check out the original article from the Data Goblins blog, so you know what to expect from the current Copilot in Power BI: https://lnkd.in/dzAbNAFJ

120

26 Comments

𝐒𝐩𝐞𝐧𝐝𝐚𝐭𝐚: A True 𝐄𝐧𝐭𝐞𝐫𝐩𝐫𝐢𝐬𝐞 𝐀𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬 Solution (𝘐𝘵'𝘴 𝘯𝘰𝘵 𝘺𝘰𝘶𝘳 𝘨𝘳𝘢𝘯𝘥𝘱𝘢'𝘴 𝘴𝘱𝘦𝘯𝘥 𝘢𝘯𝘢𝘭𝘺𝘴𝘪𝘴 𝘢𝘯𝘺𝘮𝘰𝘳𝘦!) No "data warehouse", "data lake", or "data lakehouses" needed either! Nor do you need a star schema to avoid suffering in computational limbo either! (But if you still think PowerBI is a great presentation tool, it can build you a perfect star schema on millions of records in seconds ... ) Just real spend analysis, which includes the ability to: * extend the schema at any time * add new derived dimensions at any time * change those mappings at any time * build derivations, data views, and mappings that are dependent on other derivations, mappings, views, inputs, linked datasets, and so on, with real-time “recalculation” * create new views and reports WITHOUT dumping the data to Excel * preserve ALL of the above on cube data refreshes (that's right! you DON'T have to rebuild the cube from scratch on every refresh!) * do so in your OWN private copy of the cube (that inherits from the shared master cube) so you don't have to change the common cube for your own custom analysis and reporting needs * get the answer you need RIGHT NOW (not on the next refresh next month when you've forgotten why you were told to get the answer in the first place) So what else can it do besides every spend analysis report ever invented under the sun? * 𝐒𝐚𝐥𝐞𝐬 𝐃𝐢𝐬𝐜𝐨𝐮𝐧𝐭 𝐕𝐚𝐫𝐢𝐚𝐭𝐢𝐨𝐧 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬 Over Time By Salesperson … and Client Type ... that dynamically updates in real time on every parameter or "what-if" change * Real 𝐏𝐨𝐬𝐭-𝐌𝐞𝐫𝐠𝐞𝐫 𝐖𝐡𝐢𝐭𝐞-𝐒𝐩𝐚𝐜𝐞 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬 (if you think you can do this now, think again) * 𝐌𝐮𝐥𝐭𝐢-𝐯𝐚𝐫𝐢𝐚𝐭𝐞 𝐁𝐢𝐥𝐥 𝐑𝐚𝐭𝐞 𝐀𝐧𝐚𝐥𝐲𝐬𝐢𝐬 ... and more! https://lnkd.in/eM9Y4fez

9

Earlier this year I reviewed a few different companies' databases and data warehouses and was shocked at what I saw. So many basic concepts ignored and perhaps never understood. Now these aren’t one-off cases but they did make me want to review denormalization and normalization. The funny thing is that many of these data modeling issues I saw could be fixed with a quick read on basic design patterns for databases. For example, understanding concepts such as many-to-many resolutions, hierarchical data models, or other similar patterns would have helped guide some of the recent data models I have seen. But all this starts with understanding two big concepts… Normalization and denormalization. https://lnkd.in/gP7mVn2m

236

16 Comments

Hi Power BI Enthousiast, You probably know how to organize your measures in a folder structure. In the model view, you can ‘display folder’ your measure and with back slash (/) you can even create subfolders.   But did you know that if you use a semicolon (;) between folder structures, you can show that measure in multiple folders. Without duplicating the measure.   This is very useful if you have one (1) measure that is used in, for example, multiple reports. You can then create a folder structure for a measure per report. This will help you keep track of where your measure in being used. Thankyou for to my good colleague Jeroen van Laak for showing me this functionality. 👉 Interested in more of these handy power bi tidbits? Check out my website. https://lnkd.in/eTida24w

958

38 Comments

Missed our livestream about Dynamic Queries in M? Im happy to say I presented at the Persian usergroup with Vahid Doustimajd. In the session we had a thorough walkthrough about areas that can break your queries. You can now see the recorded version at: https://lnkd.in/e2GVegft If you missed it, you can still look back. Question for you: What other areas do you look at to keep your queries ?

91

9 Comments

What data is in charge of YOUR business? Master data, structured data, foundational data. Whatever you want to call it these days, it’s still in charge. No matter what is happening lately, that part of the data story isn’t going to change. Check out this video with clips from long ago (2016, 2014 and 1999!) Convince me I’m wrong 😑 #datastorytelling #datamanagement #masterdata #cdo https://lnkd.in/e5bZddfS

39

1 Comment

𝐃𝐚𝐭𝐚 𝐄𝐭𝐡𝐢𝐜𝐬: As data becomes more and more ingrained in the business models of corporations, serious consideration needs to be given to data ethics.   The ubiquity of data means that more and more people can use data to make decisions and generate insights, even though accurately interpreting data is a skill that requires training and practice. Those who are not properly trained can make decisions based on false evidence or draw misleading conclusions that skew the decisions of others.   You may have heard of the Hippocratic oath and the phrase “do no harm”. Well, there are similar oaths that have been created for data scientists such as the one from the National Academies of Sciences, Engineering, and Medicine. It includes statements like: • I will not be ashamed to say, “I know not” • I will respect the privacy of my data subjects and • I will remember that my data are not just numbers without meaning or context, but represent real people and situations, and that my work may lead to unintended societal consequences, such as inequality, poverty, and disparities due to algorithmic bias.   Other groups have developed their own codes of ethics. Datapractices has released a set of twelve data principles, ranging from reproducibility to security to fairness.   DJ Patil, Hilary Mason and Mike Loukides released a thirteen-point data ethics checklist with items like: • Have we studied and understood possible sources of bias in our data? • What kind of user consent do we need to collect to use the data? • Have we tested for fairness with respect to different user groups? • Do we have a plan to protect and secure user data? • Do we have a mechanism for redress if people are harmed by the results?   Across all these oaths and checklists, there are some key concepts to understand: (1) fairness (2) privacy and security (3) transparency and reproducibility (4) the social impact of data.   #datascience #ethics #aiethics

14

Two days ago, I said OpenAI clearly had something up its sleeve for paid users. Today, they announced interactive tables and charts from live files—and it looks awesome. Data Analyser is one of ChatGPT's lesser-talked-about features because it had a somewhat rocky start. First of all, LLMs (Large Language Models) aren't great at anything related to math. This makes working with data... a problem. OpenAI circumvented this by getting ChatGPT to write a program (in code, which it's great at) and then using the code it wrote to analyse the data. This was a clever workaround, and it significantly improved the accuracy of its solution when dealing with data. But it was still somewhat clunky and sometimes took 2-3 tries to get an analysis right. Over time, GPT-4 improved its reasoning, which in turn improved its analysis, and Data Analyser has now become a tool I use almost daily to interrogate various data sets. Today's announcement, however, takes it over the utility threshold for *most* people. Building charts for presentations, understanding data sets from various angles and presenting stories with data is something many *want* to do, but few get to. That's about to change. For everyone who keeps saying, "LLMs are no longer progressing enough; we've hit a wall," this is another example of how we can make extreme productivity jumps without any underlying LLM improvement. There are 1000 more like this - some will come from OpenAI directly, others from the industry building on top of these models. The next few years are going to be interesting :) #FutureOfWork #ArtificialIntelligence #Productivity

60

14 Comments

A quick reminder - #IncrementalRefresh in #PowerBI starting from the basics and moving to using #TabularEditor and #partioning with Peter van den Bos

8

Switch before it hurts!🤕 That is the clear message our colleague Arto De Graef wrote in this latest #PowerBI blog. Lots of companies have done great without really paying too much attention to their data and its underlying processes. They have grown over time without doing so, but there is a risk in missing the next wave. Discover how Power BI will ease the switch and the vast amount of benefits this will bring to your reporting. If you aren't new to Power BI, check out our latest Webinar on Power BI + Copilot. That video is at the bottom of this blog. #PowerBI #ExcelReporting #ExcelToPowerBI

18

Here's a great article by Ken Puls, that will help you keep data entry easy and intuitive, but still set it up for flexible reporting and analysis.

5

1 Comment