Gerold Csendes

All models are wrong, but some are useful. I recently stumbled upon a discussion about the well-known log-log models, which are often hailed as the industry standard for estimating elasticities—especially in retail, where products are homogenous. But in my humble (and statistically significant 😊) opinion, the best way to estimate elasticities is through causal inference followed by experiments. See: https://lnkd.in/e8eFHqWc I'd love to hear from fellow practitioners. What methods do you use —particularly for heterogeneous products like fashion items (where one size never fits all) or, in my current adventure, short-term vacation rentals?

9

💻 What are the most sought-after technical skills for Data Scientists? 💻 Aleksandra Yosifova's latest report researched 1,000 Data Science, Job Postings in the US and these were the key findings: 📌 Core skills: Data analysis, statistics, and coding remain foundational. 📌 Top programming languages: Python (57%), R (33%), SQL (30%) 📌 Popular tools: Tableau (11.5%), Power BI (9.1%), Excel (10%) 📌 Emerging Trends: Rising demand for AI-related skills, especially NLP (19%) - an increase from 5% in 2023. 📌 Advanced skills in machine learning (69.3%) and AI tools like PyTorch (10.8%) and TensorFlow (10.4%). 📌 Leading sectors: Technology & Engineering (28.2%), HR (19%), Health & Life Sciences (13%). Full report in the comments 📒

17

3 hozzászólás

I am very proud of this project https://lnkd.in/dHEfVazy. The notebook does run in under an hour in the Kaggle environment, but the functions at the end are where the fun stuff is. I gathered data about clinical trials from ClinicalTrials.gov and then created two functions. The first: find a list of similar companies to a target company. The second: To graph the proximity in some meaningful way. Given that the first function included an ungraphable number of dimensions, it is much more accurate. The second is better if you want to get some visual estimate of closeness. How specifically did I do this? 1) I connected to the ClinicalTrials.gov database using the Python requests library. 2) I carefully and in some cases with error handling plucked the data I wanted navigating the ClinicalTrial.gov's data dictionary. 3) I removed data that was related to government and other single organizations, in short I only wanted Industry studies. And I only wanted studies that were sponsored. 4) I encoded data that had few possibilities for data entry, such as the population the study was conducted on (e.g. healthy, diseased). 5) I used tf-idf vectorization on the rest of the data, such as title and intervention description. (Note I combined some of these where the context was similar) 6) I combined all of that together and performed dimensionality reduction. Essentially, I used a python library to algorithmically combine some of the features. 7) I ran a loop to calculate when I had enough dimensions to explain 95% of the variance of the data. 8) I used these dimensions, which are all numerical due to the above methods, to allow me to calculate a 'distance' from one company to the next. 9) Using my domain knowledge of clinical research I adjusted the order in which I performed the above activities into it's current form and the results are much closer to expected in this order. (Before number of studies had an oversized impact) 10) I wrote a function that you can use towards the end of the notebook that will take a partial string of a company, e.g. Novartis, and return the n closest companies. (Note: You still have to spell everything in the partial string correctly to get the results for that company) 11) I wrote a function and found a way to graph the above, though compressing 400 dimensions into 2, will cause some errors. (Think a paper map vs. a globe.) But it's a fun tool, (Note I left the legend because the order companies appear in the legend is the order of closeness to the target company from the original function in 10).

6

Hi everyone! I'm thrilled to share my latest GitHub project with you – "Streamlining Your Reports: Automating Dashboards with Python, SQL, and Power BI." We're taking automation to the next level! The Challenge is that we are all drowning in a sea of reports and struggling to keep them updated in real-time is a universal challenge. But fear not! With Python's data prowess, SQL's seamless integration, and Power BI's visual wizardry, we're revolutionizing how data is managed. I've applied these powerhouse tools to a cryptocurrency project, tapping into web APIs to access real-time data on prices, trading volumes, and market trends. By harnessing this information, we gain the edge needed to make informed decisions in the fast-paced crypto market. In this project, I'm creating dynamic dashboard reports that provide real-time snapshots of key metrics like revenue growth and daily earnings. Plus, I'm automating the process to ensure these insights are always up-to-date. Imagine having a dashboard that refreshes itself with the latest data every time you open it – that's the power of dynamic reporting! With insights at your fingertips, you can track revenue trends, monitor active projects, and stay ahead of the curve in the crypto world. Feel free to ask me anything about data, automation, or anything 😊 https://lnkd.in/dhyK_-C2

24

2 hozzászólás

My first prediction model... I used simple linear regression with plate appearances as the independent variable to predict Mike Trout's total hits by season compared to his actual hits. I'm looking forward to improving and building on what I've learned so far. Full project can be found here: https://lnkd.in/gfFQNkZJ #sportsanalytics #baseballanalytics #predictivemodeling

20

This post 👇 got me thinking about something I’ve been pondering for a while: Do you calibrate your classification models? How important is it for you to have model scores that can be directly interpreted as confidence levels? To recap: In a well-calibrated model, if it assigns a score of 0.9 to a set of samples, you'd expect 90% of those samples to belong to the positive class. If your model's predictions are primarily used for ranking, calibration might not be as crucial - you’re more focused on the relative ranking of scores rather than their absolute values. But what if your business decisions depend on individual model scores, with different actions triggered by different score ranges? In that case, calibrated scores become essential. So, I’m curious: Are your model predictions calibrated? What methods do you use for calibration?

5

A simple way to understand 𝗚𝗿𝗮𝗱𝗶𝗲𝗻𝘁 𝗕𝗼𝗼𝘀𝘁𝗶𝗻𝗴 👇 𝗢𝗻𝗲 𝗟𝗶𝗻𝗲𝗿: It's an ensemble model that iterates over sequential decision trees where each DT is trained on errors of the previous DT. Consider the image below where there are multiple small pandas (Weak Learners) the initially the first panda is trained on the data and second panda is trained to predict the residual of the second panda and the prediction is multiplied by a learning parameter and added to the first prediction and so on as simple as that. 𝗠𝗮𝘁𝗵𝗲𝗺𝗮𝘁𝗶𝗰𝗮𝗹 𝗙𝗼𝗿𝗺𝘂𝗹𝗮 : M0 + Sum( Alpha * M(i)) 𝗤𝘂𝗶𝗰𝗸 𝗣𝗿𝗼𝗽𝗲𝗿𝘁𝗶𝗲𝘀 :  1. Ensemble Model 2. Can be applied for both Regression and Classification 3. Non Parametric Model 4. Sensitive to Outliers 5. Can Deal with Missing Values 6. Can OverFit 𝗪𝗵𝗲𝗻 𝘁𝗼 𝗨𝘀𝗲 𝘁𝗵𝗶𝘀 𝗺𝗼𝗱𝗲𝗹 :  1. Whenever there is subtle patterns in the Data 2. Priority is to achieve high accuracy 3. When there is imbalanched data 4. Applicable to Small to medium datasets For More Explanation i posted a video on my 𝗬𝗼𝘂𝗧𝘂𝗯𝗲 please check it out: https://lnkd.in/eS4TQh6C Note: There is a small error in the explanation lets see who can catch it if you find it Post in the Comments 🤪

7

Explain any machine learning model in your browser! Great tool for ml interpretation by Georgia Institute of Technology Polo Club of DS. https://lnkd.in/dS3tBV5N #ml #ai #ds #artificialintelligence #kaggle #datascience #machinelearning

40

2 hozzászólás

The 3rd Edition of Think Python is available now at https://lnkd.in/ejzC5_-v The print edition is available for preorder, expected to ship in June. What's new? * The entire book is in Jupyter notebooks that run on Colab, so you can read the book, run the code, and work on exercises -- without installing anything. * Each chapter includes suggestions for using virtual assistants like ChatGPT to develop, test, and debug programs, and explore additional topics. * The examples that use turtle graphics now work in Jupyter notebooks! * More testing with doctest and unittest. And a new, full color, parrot on the cover!

552

9 hozzászólás

Is Chain-of-Thought (CoT) Always Worth It? Just read an insightful paper analyzing CoT prompting in large language models (LLMs). The takeaway? CoT mainly boosts performance on math and logic tasks, with little benefit for other types of problems. The study found that on benchmarks like MMLU, directly generating answers without CoT produces nearly the same accuracy—unless the task involves symbolic reasoning (like equations). CoT’s gains come from handling intermediate steps, but symbolic solvers often perform better. Is this paper proves what we are seeing with o1 models ? The extra cost and time during inference is not worth it in most situations but will help when it comes to hard reasoning prompts https://lnkd.in/dm7PY45n #AI #MachineLearning #LLM #ChainOfThought #NLP #AIResearch

14

How do you evaluate a data science pilot? Using Probabilistic Model Evaluation in which you estimate the probability distribution of the selected performance metric using a Bayesian approach. This is something we have been thinking about for a while at nannyML. For many use cases scaling a model is expensive and risky. Predictive maintenance, demand forecasting, supply chain optimisations, basically any use case where you need to change the real world extensively. The most effective way to know whether there is a positive ROI in deploying your model across your organization, is by running a real world pilot. Given most pilots run on an extremely small amount of data it is not always easy to determine if your model is performing as expected. Or conversely running pilots longer than necessary is also expensive, so as soon as you do have enough data, you should stop the pilot. Check out this video where Wojtek Kuberski goes into some of the details behind our pilot evaluation framework and how it helps you evaluate your models during a pilot. Do you run real world pilots? Let me know in the comments!

26

9 hozzászólás

I have given Googles BERT and T5 models the same PDFs as I gave to Chat GPT to compare them, and I got significantly different results. Not sure if I have missed something on the documentation on how to implement them but the performance of T5 and BERT was very bad for this task. Have a look at the 2 summaries they generated. I have also tested questioning the text and the performance was the similar. Google's Gemini is a more fair comparison which I will be running next! Full video on how I created the 2 apps here: https://lnkd.in/duRpF_s2

18

1 hozzászólás