HEMANT SINGH SARDANA

Question : Why does an AI product manager need to monitor #AI model❓ Once AI model is developed and deployed, the job doesnt end there. Model encodes pattern of the data it has seen during training phase. There may be several patterns or edge cases which are not part of training data. If model encounters any such patterns in testing or production phase, it will produce results which wont make sense without giving any warning about it. 👉 Data drift is a common challenge faced by AI-powered products. Over time, the distribution of input data can shift, leading to a mismatch between the model's training data and the real-world data it encounters. This can result in degraded model performance, unexpected outputs, and even complete model failure. 👉 The consequences of data drift can be severe, ranging from suboptimal user experiences to regulatory compliance issues. Product managers must be vigilant in detecting and addressing these challenges to ensure the continued success of their AI-powered offerings. 👉 Sudden changes in metrics can indicate input data drift. Additionally, monitoring the distribution of model outputs can reveal output drift, which may signal a need to retrain or fine-tune the model. Detecting data drift early is crucial. Product managers can leverage techniques such as A/B testing, anomaly detection, and continuous monitoring to identify and address these issues. Once detected, the product team can take appropriate actions, such as retraining the model, adjusting the data pipeline, or even considering a model architecture change. 💡 Quick Tip: Integrate model monitoring into your product roadmap and allocate dedicated resources to ensure the long-term success of your AI-powered offerings. Establish a comprehensive model monitoring dashboard to gain real-time visibility into the health and performance of your AI models. #artificialintelligence #productmanagement #datascience #productmanager

5

Question : Why does quoting just the accuracy numbers do not make sense for an AI model❓ Its a common practice among data science practitioners to quote accuracy of an #AI model. While it is easier to understand accuracy numbers, this practice can be misleading and fail to capture the nuances of real-world scenarios, leading to misaligned expectations among business stakeholders. 👉 Accuracy alone does not provide a comprehensive picture of a model's capabilities. It fails to account for crucial factors such as the size and distribution of the training, testing, and validation datasets, the presence of class imbalances, and potential biases. Relying solely on accuracy can mask underlying issues and lead to false confidence in the model's performance. 👉 To truly understand a model's effectiveness, it is essential to consider a broader range of metrics. Precision, recall and F1 score offer valuable insights into a model's ability to correctly identify positive and negative instances, handle imbalanced data, and discriminate between classes. 👉 However, even these technical metrics may not resonate with business stakeholders who are primarily concerned with the practical implications of the model's performance. It is crucial to translate these metrics into business-relevant terms that directly align with the product's objectives and success criteria. For example, in a fraud detection system, precision could be framed as the percentage of flagged transactions that are indeed fraudulent, while recall could represent the proportion of actual fraudulent transactions correctly identified. By presenting these metrics in a business context, stakeholders can better understand the trade-offs and make informed decisions about the model's deployment and ongoing monitoring. 💡 Quick Tip: Dont quote accuracy numbers without mentioning the size of training and testing dataset. Consider a diverse set of metrics tailored to your specific use case and business objectives. Communicate these metrics in a language that resonates with stakeholders, fostering a shared understanding and aligning expectations. #artificialintelligence #productmanagement #datascience #productmanager

6

Another great talk from the PyData London 2024 - Backtesting and error metrics for modern time series forecasting by Kishan Manani Backtesting is the time series forecasting equivalent to the machine learning cross-validation approach for training and testing models. https://lnkd.in/g-DWgqmB #timeseries #forecasting #machinelearning #python

46

5 Comments

Is AI / Data Science Field Facing "Attention Span" Issue? Recently, I read about COMMITMENT / EXPLORATION aspects in the context of decreasing attention span in youngsters. In the data science field, I feel we are experiencing the same issue. When the data science field started (growing), it was HIGH COMMITMENT / LESS EXPLORATION scenario. People were willing to spend more time understanding the business problem, associated data, overall value proposition {indicating HIGH COMMITMENT}. There were less options to explore and implement those ideas in terms of tools / technologies / number of techniques {indicating LESS EXPLORATION}. Decision making was easy. Attention to details was a prime requirement. Fast forward to 2024, the field is now experiencing LESS COMMITMENT / HIGH EXPLORATION scenario. Hardly anyone wants to spend time understanding the business problem, nuances around data, less to no awareness of value proposition {indicating LESS COMMITMENT}. There is always some sense of urgency to jump on to the models, some tools, technologies or frameworks which are easily available in huge numbers {indicating HIGH EXPLORATION}. The success or failure of the work is generally attributed to the selected tools / technologies. Decision making is becoming increasingly difficult due to available choices. Arriving at a conclusion too easily and / or too early is a widespread thing. Overall, resulting in less attention to the details. There could be other two possible scenarios:  1. LESS COMMITMENT / LESS EXPLORATION : Certainly not a desirable state to be in.  2. HIGH COMMITMENT / HIGH EXPLORATOIN : I would imagine this to be the most matured state for any AI / Data Science set up.    Just to summarize, all the work in AI / Data Science space should start with HIGH COMMITMENT / LESS EXPLORATION scenario to lay strong foundations and then HIGH COMMITMENT / HIGH EXPLORATION can be a north star. Comments? #datascience #principalpune #pgs #ai #machinelearning #linkedin #linkedinfamily #pfg

33

5 Comments

 Dear LinkedIn Family, We are looking for Senior Data Scientist for Pune Location. #DataScientist LLM Architecture: Good understanding of the architecture underlying large language models, such as Transformer-based models and their variants. Design and implement deep learning model architectures using PyTorch. #LLM #PyTorch #pytorch Language Model Training and Fine-Tuning: Experience in training large-scale language models from scratch, as well as fine-tuning pre-trained models on domain data. Data Preprocessing for NLP: Skilled in preprocessing textual data, including tokenization, stemming, lemmatization, and handling of different text encoding. Transfer Learning and Adaptation: Proficiency in applying transfer learning techniques to adapt existing LLMs to new languages, domains, or specific business needs. Data Annotation and Evaluation: Skills in designing and implementing data annotation strategies for training LLMs and evaluating their performance using appropriate metrics. Scalability and Deployment: Experience in scaling LLMs for production environments, ensuring efficiency and robustness in deployment. Model Training, Optimization, and Evaluation: Evaluate the performance of PyTorch models using appropriate metrics and techniques like cross-validation, holdout sets, or online evaluation. This encompasses the complete cycle of training, fine-tuning, and validating language models. You will be designing and adapting LLMs for use in virtual assistants, Information retrieval and extraction etc. Experimentation with Emerging Technologies and Methods: Actively exploring new technologies and methodologies in language model development, including experimental frameworks and software tools. LLM Alignment: Understanding of algorithms like DPO, PPO, KPO, RLHF and using it for guardrails. AI Data Retrieval: Data retrieval from unstructured data, extract key value pairs using techniques like donut, layoutLM, table transformers. Analyze data and build EDAs to identify data patterns Hands-on and strong understanding of concepts in Deep Learning and NLP Proficient in TensorFlow and similar libraries.   Required Qualifications 5+ years of hands-on experience in developing and deploying Large Language Models, and Machine learning and working with Pytorch. A thorough understanding of machine learning, particularly deep learning techniques, including knowledge of neural network architectures, training methods, and optimization algorithms. Proficiency in AI technology, Python, including experience with NLP libraries (e.g., Hugging Face Transformers, NLTK, spaCy), text classification. Experience with frameworks: PyTorch, or Tensorflow. Experience with cloud services (AWS, Azure) and ML deployment tool Docker Familiarity with model fine-tuning and optimization techniques for LLMs. Proven track record of innovative solutions in the field of LLMs. Strong communication skills, with the ability to explain complex AI concepts to non-expert audiences.

4

1 Comment

There is a gloom simmering in the data science industry. Confetti is flowing in the wind of generative AI. But at the same time an entire batch of freshers are anxious about their placements. Hiring has been extremely low across the board. Irrespective of tier-1, 2 or 3 campuses. These days whenever any student enquires about our courses, they mainly try to understand whether we provide "placement assistance" or not. Meaning if we will be able to find them a job or not. This is a great indicator to understand the overall mood of the batch. They're sadly, desperate for an opportunity. Looking at the industry from this vantage point breaks you a little bit. So from last two weeks I've been doing research on the topic. Talking to several industry veterans, hundreds of students, reading. And the conclusion is simple and worrying. AI adoption in everyday life is the future. We're moving towards an age where "internet of things" will be a reality pretty soon. Most of the work that can be automated will definitely be automated. As we'll be increasingly a more capitalistic society, most of the necessities will be controlled by companies. And as we all very well know, automation is an effective cost cutting measure. So many people will loose jobs, a lot of people will have to learn new skills to adjust, payouts will follow a leptokurtic distribution. Overall it'll be a gradually more competitive and skill-based economy. Harder to thrive in. Even though it might sound like doom. Life always finds a way. A different way for sure. But definitely a way. Which frankly I don't know yet. I'm planning to read more about this topic. it's a ocean of knowledge to consume. I'm in the water but still near the bay. What you'll see soon is something beautiful, that has come of these research. Till then please Stay Tuned. #lastbench #resumereview #coverletter #interviewpreparation #interviewtips #placement #placementassistqnce #datascience #analytics #machinelearning #careercoach

134

2 Comments

The widespread use of deceptive patterns or dark patterns as they are more commonly known in top Indian apps is concerning! Check out our latest report “Conscious Patterns” which maps and analyses deceptive patterns used in top Indian apps. The report will reveal: -Prevalence of deceptive patterns across industries -Most commonly used deceptive patterns -Ethical design alternatives Don't miss out on this pivotal resource. Read the full web report here: https://lnkd.in/dHiEcapu #EthicalDesign #DarkPatterns #ASCI #ParallelHQ #Nasscom #UXUIDesign #ConsumerTrust #ConsciousPatterns #ASCIAcademy

16

1 Comment

I call this time series model the ARIMA Killer!! Dramatic hooks gets us more clicks right? Anyways, for people who are into time series, do read this. I am doing a series (has no relation with pandas Series, nor time series, HAHAHA, {lame}) in which I would be covering somewhat latest developments in time series forecasting/analysis, and some of the common and/or advanced time series tasks that the Internet Gurus won't teach you. I am excited to announce that I have successfully written another blog🚀💫 (sarcasm) #timeseries

8

2 Comments

F1-Score vs Accuracy: Why F1-Score Should Be Your Focus. In machine learning, it's tempting to focus on ACCURACY—the percentage of correctly predicted instances. But what if I told you that accuracy isn't always the best indicator of your model's performance? In cases where the data is imbalanced (e.g., fraud detection, disease diagnosis), accuracy can be misleading. Imagine a scenario where only 5% of transactions are fraudulent—if your model predicts "non-fraudulent" 100% of the time, you’ll get 95% accuracy but miss all the critical fraud cases! When it comes to the F1-score: - Precision tells you how many of your positive predictions were correct. - Recall tells you how many of the actual positives you correctly identified. The F1-score balances both precision and recall, giving you a better sense of how well your model performs—especially when false positives and false negatives matter. If you're working on models where missing a positive instance could have serious consequences (like healthcare, security, or finance), the F1-score should be your go-to metric. Focus on the right metrics to make smarter models and better decisions. #MachineLearning #DataScience #AI #F1Score #ModelEvaluation #TechInsights #DataDriven #BrainK

10

Sharing update from SSRN publishing... "Dear Ashutosh Bhatnagar: Your paper, "NEIGHBORHOODS OF PRIMES", was recently listed on SSRN's Top Ten download list for: Discrete Mathematics & Combinatorics eJournal and Number Theory eJournal. As of 02 October 2024, your paper has been downloaded 16 times. You may view the abstract and download statistics at: https://lnkd.in/eMmhurr4. Top Ten Lists are updated on a daily basis. Click the following link(s) to view the Top Ten list for: Discrete Mathematics & Combinatorics eJournal Top Ten and Number Theory eJournal Top Ten. Click the following link(s) to view all the papers in: Discrete Mathematics & Combinatorics eJournal All Papers and Number Theory eJournal All Papers. To view SSRN's Top Ten lists for any network, subnetwork, eJournal or topic on the Browse list (reachable through the following link: https://lnkd.in/eEmS8twm), click the "i" button to the right of the name, and then select the "Top Downloaded" link in the popup window. Your paper may be included in future Top Ten lists for other networks or eJournals. If so, you will receive additional notices at that time. Questions? Visit the SSRN Support Center. This email helps you gauge the impact of your paper. To unsubscribe from ALL notifications, including SSRN eJournals, click here. Thank you, The SSRN Team"

10

3 Comments

Anyone looking for LLM inference optimisation must look into this amazing series. Why inference optimisation matters? https://lnkd.in/duu56cAR Understanding transformer inference - https://lnkd.in/dNyktzjB Follow Mastering LLM (Large Language Model) for upcoming topics.

7

1 Comment

🚀 What Happens to a Contextual Embeddings Model if a Word is Not Present in the Vocabulary? 🌟 In contextual embeddings models like BERT, GPT-3, etc., handling out-of-vocabulary (OOV) words is crucial for maintaining performance and accuracy. These models typically address OOV words by breaking them down into subword units using techniques like WordPiece or Byte-Pair Encoding (BPE). This means that even if the entire word isn't present in the vocabulary, its parts likely are. Here's how it works: 1. Subword Tokenization: When a word is not in the vocabulary, it is split into smaller subword units. For example, the word "newword" might be broken down into "new" and "word", or even into smaller units like "ne", "##w", "##word" in the case of BERT. This approach ensures that the model can still generate embeddings for parts of the word, capturing some of its meaning and allowing it to be processed effectively. 2. Retaining Semantic Information: By breaking down OOV words into known subwords, the model can retain and leverage semantic information from the subwords. This allows the model to handle a wide range of words, even those it hasn't seen during training. For instance, the subwords "new" and "word" in "newword" carry meaningful information that the model can use to infer the context and meaning of the OOV word. 3. Handling Complex Words and Morphology: Subword tokenization is particularly useful for handling complex words and morphological variations. It reduces the number of OOV words and improves the model's ability to generalize across different forms of words. For example, the word "unhappiness" might be broken down into "un", "##happi", and "##ness", allowing the model to understand its structure and meaning more effectively. 4. Contextual Understanding: Contextual embedding models like BERT and GPT-3 are pre-trained on large corpora, which helps them understand and generate embeddings for subwords based on their context. This allows them to handle OOV words dynamically and accurately. Even if a word is rare or newly coined, the model can use its context to generate meaningful embeddings and produce relevant responses. By leveraging subword tokenization techniques, contextual embeddings models can effectively manage OOV words, ensuring robust performance and accurate language understanding. Stay tuned for more insights on NLP and AI advancements! 💡 #NLP #AI #MachineLearning #DataScience #LanguageProcessing #TechInnovation #BERT #GPT3 Please follow my posts for more updates: Tejas Nanaware on LinkedIn #machinelearning #deeplearning #artificialintelligence #datascience #statistics Feel free to connect and share your experiences or ask any questions!

5

Evaluating LLMs are very crucial for generative models. Do you know how to evaluate such models? Here are few popular metrics. 📍 Context-independent Metrics BLEU: Contxt-independent precision based metric. Number of n-grams in generated output that also appears in reference label. ROUGE: Context-independent recall based metric. Number of n-grams in reference that also occur in generated output. METEOR: Context-independent precision-recall based metric. No exact n-gram matching. Unigram matching on multiple levels (exact word matching, stemmed matching, paraphrase matching etc.) BertScore: Context-independent embedding based metric. It is same as METEOR but with embeddings. Precision-Recall is calculated by nearest cosine similarity match between generated output and reference. It is one-to-one matching (hard-alignment) between tokens of output and reference. MoverScore: Context-independent contextualized embedding based metric. Computes distance between tokens in output and reference. Mapping of semantically related words in one sequence to their counterparts in another sequence. It is many-to-one matching (soft-alignment) between tokens of output and reference. 📍 Context-dependent Metrics ROUGE-C: Context-dependent metric. Modifying ROUGE to compare output with source document instead of reference output. G-Eval: Context-dependent Chain-of-Thought (CoT) based metric. Provide task introduction and evaluation criteria to LLM and ask it to generate evaluation steps. This is very popular nowadays since GPT-4 and later models have very high Spearman correlation with human judgements. 📍 Consistency Evalulation Metrics (for NLI models) SummaC: Splitting reference document and generated output into sentences, and classify each output sentence into (neutral, aligned or inconsistent with reference texts) consistency labels. TrueTeacher: Distill LLMs (T-5 large, medium and small) by feeding them prompts and then provide these synthetic labels to NLI model in order to detect (classify) consistency. QuestEval: Precision-Recall based metric to generate questions from provided document and the summary. The generated question is compared with reference questions to compute precision-recall. There is also an eval metric for question answering where answer is generated by prompting question + document. 📍 Preference-based metrics: In between reference-based metrics and context-based metrics. Reward model is trained on human rank preferences of generated text. The model gets aligned to the outputs based on "preference-score" generated by reward model. 📍 Sampling-based metrics: When reference is not available, generated outputs of different LLMs can be compared to see the "agreement-score" to evaluate. For detailed information, please refer to the amazing article by Eugene Yan (link in comment).

9

2 Comments

💰 LLM Pricing vs MMLU 📈 MMLU (Massive Multitask Language Understanding) is a benchmark that evaluates an LLM's performance across a wide range of tasks, providing a comprehensive measure of its language understanding capabilities. I've analysed the relationship between pricing and MMLU score for various cutting-edge models and its been insightful. 💡 Key Takeaways: 1. The general trend of higher prices correlating with better MMLU scores holds, but there are notable exceptions. GPT-4 and Claude 3 Opus lead the pack in performance, but GPT-4 comes at a lower price point. 2. Llama 3 variants dominate the low-cost market, with Llama 3 70B (Groq) offering an impressive 82.0 MMLU score at just $0.79 per 1M tokens. This further cements their position as high-performing, accessible open-source options. 3. The mid-range price tier offers a diverse selection of models, catering to various needs and budgets. From Claude 3 Sonnet and Haiku to Mistral Medium and Mixtral-8x22B, this segment provides a balance between performance and cost, with MMLU scores ranging from 75.2 to 79.0. Groq's emergence as a cloud provider is a game-changer, enabling the deployment of powerful open-source models like Llama and Mixtral at extremely competitive prices. This development levels the playing field and empowers startups and businesses of all sizes to harness the potential of advanced language models. As open-source models continue to improve and offer more competitive pricing, closed-source models will need to differentiate themselves through unique features, exceptional performance, and seamless integration to justify their premium pricing. Moreover, while price and MMLU scores are important metrics, it's essential to consider other aspects such as model size, training data, API features, long-term growth plans and ease of use when selecting a language model that best suits your specific needs. #LLMs #GenAI #OpenSource

10

My Top 10 R Packages for EDA to cut your exploratory data analysis time in half. Here they are: 1. skimr: skim() 2. psych: describe() 3. corrplot: corrplot() 4. PerformanceAnalytics: chart.Correlation() 5. GGally: ggpairs() 6. DataExplorer: create_report() 7. summarytools: dfSummary() 8. SmartEDA: ExpReport() 9. janitor: tabyl() 10. inspectdf: inspect_na(), inspect_cat(), inspect_cor() Want to learn these? I made a full R Tutorial Here: https://lnkd.in/etvJf9aS If you'd like to hear more about the blueprint to prepare yourself for a 6-figure data science for business career... ... then I'd like to help. I've created a complete system called the 5-Course R-Track that teaches you everything you need to become a Business Scientist. 1,848 Lessons, 152.9 Hours of Video, and 24 Challenges to Test Your Skills Learn more in my free R masterclass: https://lnkd.in/e4R-mG3H

58

7 Comments

For hyperparameter tuning hyperopt was the go to package. I recently learnt that the open-source package is no longer maintained. Microsoft and Databricks encourage the use of optuna for hyper parameter optimisation. MLflow have created this awesome tutorial to get you started using Databricks and Optuna.

11

For all CAT aspirants, here’s a streamlined approach you can apply to tackle the Quant section: Start Sequentially (If Strong in Quant): Begin by solving questions in order. If a problem looks familiar and solvable within 3 minutes, attempt it. Mark Time-Consuming Questions: For questions that seem familiar but might take longer, mark them for review and move forward. Skip Unfamiliar Problems: If a question seems completely new or difficult, leave it for later rounds. Round 2 (If Strong in Quant): Go back to the marked questions and attempt them with more focus. Round 3 (If Time Permits): Only if time remains, revisit any skipped or unmarked questions. Target Attempts (If Strong in Quant): Typically aim for 20 attempts in the first round, followed by 8-10 more in the second. Adjust based on paper difficulty— 21 attempts in CAT 2018 resulted in a 99.5+ percentile. Skim for Easy Wins (If Quant is Weak): Quickly scan the paper to find the easiest, most familiar questions to attempt in Round 1. Mark Comfortable Topics (If Quant is Weak): Mark any questions from familiar topics, even if the problem itself isn't clear, to review later. Time-Efficient First Round: In Round 1, solve questions you're sure of within 90 seconds. Comfort Zone in Round 2: Focus on marked questions from familiar topics in Round 2. Attempt 3-5 questions in Round 1 and at least 6 more in Round 2, adjusting to the paper's difficulty. Give this strategy a shot in your mocks and see what fits best. After attempting 3-4 mocks, you'll know what works for you. And remember, as they say in The Dark Knight, “The night is darkest just before the dawn. And I promise you, the dawn is coming.” Stay focused, keep practicing, and you’ll find your rhythm. Best of luck for CAT 2024! 😊 😊 #CAT2024 Unacademy #IIM

22

NLP techniques every Data Scientist should know: 1. Tokenization 2. Stop words removal 3. Stemming and Lemmatization 4. Named Entity Recognition 5. Term Frequency-Inverse Document Frequency 6. Bag of Words Learn to implement all these techniques using Python here: https://lnkd.in/dTjeQiqm #datascience #dataanalysis #nlp #naturallanguageprocessing #python #artificialintelligence

54

8 Comments

@For researchers and PhD/DBA students 💡: 🔍 Measurement Model Validation in PLS-SEM: Introducing CTA-PLS I recently explored an insightful article: “Confirmatory tetrad analysis in PLS path modeling” by Gudergan et al. (2008). This study presents CTA-PLS, a robust method for validating measurement models in PLS-SEM. 📊 📌 Research Problem: Incorrectly specifying constructs as either formative or reflective can lead to biased parameter estimates and flawed conclusions in PLS path modeling. ✨ Key Insights: 🔍 Formative vs Reflective Models: CTA-PLS helps determine whether a construct should be modelled as formative or reflective. 📊 Bootstrapping Approach: This approach uses bootstrapping to assess the significance of vanishing tetrads, ensuring that empirical data aligns with theoretical assumptions. ✅ Avoiding Misspecification: Critical for accurate path estimations and valid model conclusions. 📘 What it Means for Researchers: 🔄 Apply CTA-PLS before measurement model assessment to confirm indicator specification. 🚀 Use it in pilot studies to refine models early and ensure robust main study results. 🔍 Incorporate CTA-PLS for post-hoc revalidation to align empirical findings with theory. 📝 Conclusion: Leverage CTA-PLS to prevent misspecification and enhance the reliability of your research for accurate measurement model evaluation in PLS-SEM. Summarised by Dr Hatim Dawood Al-lawati (PMP®, DBA-FinTech) ✍️ 📚 Reference: Gudergan, S.P., Ringle, C.M., Wende, S., & Will, A. (2008). Confirmatory tetrad analysis in PLS path modeling. Journal of Business Research, 61(12), 1238–1249. #PLSSEM #StructuralModel #CTA #SmartPLS #ResearchMethods #Formative #Reflective #MeasurementModel ResearchGate

2