Roundtable

🚨𝗜 𝗿𝗲𝗮𝗱 𝟵𝟱 𝗮𝗰𝗮𝗱𝗲𝗺𝗶𝗰 𝗽𝗮𝗽𝗲𝗿𝘀 𝗮𝗯𝗼𝘂𝘁 𝗱𝗮𝘁𝗮 𝗾𝘂𝗮𝗹𝗶𝘁𝘆 𝗮𝗻𝗱 𝘄𝗮𝗹𝗸𝗲𝗱 𝗮𝘄𝗮𝘆 𝘄𝗶𝘁𝗵 𝗼𝗻𝗲 𝗯𝗶𝗴 𝘁𝗮𝗸𝗲𝗮𝘄𝗮𝘆. 𝗜𝘁’𝘀 𝗱𝗲𝗳𝗶𝗻𝗶𝘁𝗲𝗹𝘆 𝗻𝗼𝘁 𝘄𝗵𝗮𝘁 𝘆𝗼𝘂 𝘁𝗵𝗶𝗻𝗸. My resolution for 2024 was to dedicate 100 hours (😮) to reviewing the academic literature on data quality and determining which QC checks are most effective. Why did I take on this challenge? Because we’re making too many mistakes- flagging good respondents while letting fraudsters slip through the cracks. The literature on poor data quality covers a range of satisficing behaviors, such as careless responding, random answering, insufficient effort responding, inattentiveness, straightlining, and so on. 𝗟𝗲𝘀𝘀𝗼𝗻 #𝟭: I had no idea what I was getting myself into. Despite spending over 100 hours, I feel like I’ve only scratched the surface. 😭 𝗟𝗲𝘀𝘀𝗼𝗻 #𝟮: While there are a handful of QC checks I feel more confident in, no silver bullet emerged from my research.😵💫 𝗧𝗵𝗲 𝗕𝗶𝗴 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆: The future of data quality lies in passive in-survey behavior monitoring (like cut/paste, mouse movement, etc.). Without this information, we’re relying too heavily on assumptions about participant intent. 👀You heard it here first! That’s my prediction for 2025 -we’ll see more behavioral tools (AI or not, as long as it solves the problem I don’t care!!) that flag suspicious in-survey behavior. 🛑 BUT: Who knows how long that will remain effective 🤷♀️ . The only real solution to the data quality crisis is to start validating participant identity. The other way will be to crack down on payments. There shouldn't be 60 profiles linked to the same bank account/PayPal, and we have the technology to monitor that. What are your predictions for data quality in 2025? #mrx #dataquality #surveys

This is a really fun one. I had the chance to sit down with the amazing 🎇 Dan Wasserman 🎇 of KJT and talk all things market research, AI, and data quality. Chapters - 00:00 - Introductions 01:30 - Career transition to market research 03:37 - Gen AI in market research 05:25 - Data quality 08:35 - Large language models 10:47 - Improving data quality with gen AI 14:41 - Buying AI as a market researcher 21:59 - Benefits of automated data quality 27:45 - Humility and empathy in AI research 29:49 - Learning from failure and pivoting 33:15 - Listening and learning in AI opportunities and challenges 34:14 - Virtual vendor days, prompt engineering, learning from experience #mrx #dataquality #ai

Another effect of bad survey data: customer segmentation doesn't make sense. Here, we applied latent class analysis (LCA), a common clustering/segmentation technique, to identify customer archetypes in response to our solar-powered refrigerated hats. In the good panel, we get three clusters that relate well to each other. You have the enthusiastic customers, the moderate customers, and the cautious customers. In the bad panel, it's a mess. One cluster makes sense and corresponds to enthusiastic customers. But there are two clusters with equal usefulness/price estimates but very different number of features selected. If you were trying to analyze the customer segmentation on the low-quality dataset, you would likely be banging your head against the wall trying to understand consumer preferences. Roundtable is here to protect your survey data. #mrx #dataquality

Bad data is not just annoying, it's deceptive. It leads you to false conclusions. Here, we analyzed average usefulness and frequency-of-use ratings for a hypothetical product. Good Panel - Average Frequency: 1.78 - Average Usefulness: 3.60 - Correlation Coefficient: 0.7384 Fraudulent Panel - Average Frequency: 2.55 - Average Usefulness: 5.52 - Correlation Coefficient: 0.4727 --- One of the best internal #dataquality checks to do on your #mrx study is consistency checks - do these results make sense? For this, we computed the correlation between average frequency and average usefulness scores. The good panel has a high correlation - people who tend to wear it on more occasions would use it more. The correlation is a lot lower in the fraudulent dataset.

Having a survey participant copy-and-paste ChatGPT and then change it around to make it look like their own? Now, with a fun combo between the Roundtable content and behavioral flags, we can identify these participants trying to subvert the system. #mrx #dataquality