Shivendra Singh

Question : Why does an AI product manager need to monitor #AI model❓ Once AI model is developed and deployed, the job doesnt end there. Model encodes pattern of the data it has seen during training phase. There may be several patterns or edge cases which are not part of training data. If model encounters any such patterns in testing or production phase, it will produce results which wont make sense without giving any warning about it. 👉 Data drift is a common challenge faced by AI-powered products. Over time, the distribution of input data can shift, leading to a mismatch between the model's training data and the real-world data it encounters. This can result in degraded model performance, unexpected outputs, and even complete model failure. 👉 The consequences of data drift can be severe, ranging from suboptimal user experiences to regulatory compliance issues. Product managers must be vigilant in detecting and addressing these challenges to ensure the continued success of their AI-powered offerings. 👉 Sudden changes in metrics can indicate input data drift. Additionally, monitoring the distribution of model outputs can reveal output drift, which may signal a need to retrain or fine-tune the model. Detecting data drift early is crucial. Product managers can leverage techniques such as A/B testing, anomaly detection, and continuous monitoring to identify and address these issues. Once detected, the product team can take appropriate actions, such as retraining the model, adjusting the data pipeline, or even considering a model architecture change. 💡 Quick Tip: Integrate model monitoring into your product roadmap and allocate dedicated resources to ensure the long-term success of your AI-powered offerings. Establish a comprehensive model monitoring dashboard to gain real-time visibility into the health and performance of your AI models. #artificialintelligence #productmanagement #datascience #productmanager

5

Question : Why does quoting just the accuracy numbers do not make sense for an AI model❓ Its a common practice among data science practitioners to quote accuracy of an #AI model. While it is easier to understand accuracy numbers, this practice can be misleading and fail to capture the nuances of real-world scenarios, leading to misaligned expectations among business stakeholders. 👉 Accuracy alone does not provide a comprehensive picture of a model's capabilities. It fails to account for crucial factors such as the size and distribution of the training, testing, and validation datasets, the presence of class imbalances, and potential biases. Relying solely on accuracy can mask underlying issues and lead to false confidence in the model's performance. 👉 To truly understand a model's effectiveness, it is essential to consider a broader range of metrics. Precision, recall and F1 score offer valuable insights into a model's ability to correctly identify positive and negative instances, handle imbalanced data, and discriminate between classes. 👉 However, even these technical metrics may not resonate with business stakeholders who are primarily concerned with the practical implications of the model's performance. It is crucial to translate these metrics into business-relevant terms that directly align with the product's objectives and success criteria. For example, in a fraud detection system, precision could be framed as the percentage of flagged transactions that are indeed fraudulent, while recall could represent the proportion of actual fraudulent transactions correctly identified. By presenting these metrics in a business context, stakeholders can better understand the trade-offs and make informed decisions about the model's deployment and ongoing monitoring. 💡 Quick Tip: Dont quote accuracy numbers without mentioning the size of training and testing dataset. Consider a diverse set of metrics tailored to your specific use case and business objectives. Communicate these metrics in a language that resonates with stakeholders, fostering a shared understanding and aligning expectations. #artificialintelligence #productmanagement #datascience #productmanager

6

For hyperparameter tuning hyperopt was the go to package. I recently learnt that the open-source package is no longer maintained. Microsoft and Databricks encourage the use of optuna for hyper parameter optimisation. MLflow have created this awesome tutorial to get you started using Databricks and Optuna.

11

I call this time series model the ARIMA Killer!! Dramatic hooks gets us more clicks right? Anyways, for people who are into time series, do read this. I am doing a series (has no relation with pandas Series, nor time series, HAHAHA, {lame}) in which I would be covering somewhat latest developments in time series forecasting/analysis, and some of the common and/or advanced time series tasks that the Internet Gurus won't teach you. I am excited to announce that I have successfully written another blog🚀💫 (sarcasm) #timeseries

8

2 Comments

 Dear LinkedIn Family, We are looking for Senior Data Scientist for Pune Location. #DataScientist LLM Architecture: Good understanding of the architecture underlying large language models, such as Transformer-based models and their variants. Design and implement deep learning model architectures using PyTorch. #LLM #PyTorch #pytorch Language Model Training and Fine-Tuning: Experience in training large-scale language models from scratch, as well as fine-tuning pre-trained models on domain data. Data Preprocessing for NLP: Skilled in preprocessing textual data, including tokenization, stemming, lemmatization, and handling of different text encoding. Transfer Learning and Adaptation: Proficiency in applying transfer learning techniques to adapt existing LLMs to new languages, domains, or specific business needs. Data Annotation and Evaluation: Skills in designing and implementing data annotation strategies for training LLMs and evaluating their performance using appropriate metrics. Scalability and Deployment: Experience in scaling LLMs for production environments, ensuring efficiency and robustness in deployment. Model Training, Optimization, and Evaluation: Evaluate the performance of PyTorch models using appropriate metrics and techniques like cross-validation, holdout sets, or online evaluation. This encompasses the complete cycle of training, fine-tuning, and validating language models. You will be designing and adapting LLMs for use in virtual assistants, Information retrieval and extraction etc. Experimentation with Emerging Technologies and Methods: Actively exploring new technologies and methodologies in language model development, including experimental frameworks and software tools. LLM Alignment: Understanding of algorithms like DPO, PPO, KPO, RLHF and using it for guardrails. AI Data Retrieval: Data retrieval from unstructured data, extract key value pairs using techniques like donut, layoutLM, table transformers. Analyze data and build EDAs to identify data patterns Hands-on and strong understanding of concepts in Deep Learning and NLP Proficient in TensorFlow and similar libraries.   Required Qualifications 5+ years of hands-on experience in developing and deploying Large Language Models, and Machine learning and working with Pytorch. A thorough understanding of machine learning, particularly deep learning techniques, including knowledge of neural network architectures, training methods, and optimization algorithms. Proficiency in AI technology, Python, including experience with NLP libraries (e.g., Hugging Face Transformers, NLTK, spaCy), text classification. Experience with frameworks: PyTorch, or Tensorflow. Experience with cloud services (AWS, Azure) and ML deployment tool Docker Familiarity with model fine-tuning and optimization techniques for LLMs. Proven track record of innovative solutions in the field of LLMs. Strong communication skills, with the ability to explain complex AI concepts to non-expert audiences.

4

1 Comment

Can LLM preference tuning degrade Multi-Agent system performance? In a recent tweet, Andrej Karpathy questioned why all LLMs sound similar, prompting Sebastian Rascha to speculate about the uniformity of alignment datasets. Aligning user preferences is a last-mile step in training large language models. Regardless of the pre-training datasets used, could the preference datasets used during alignment compromise the effectiveness of LLMs? This issue could adversely affect multi-agent systems incorporating multiple LLMs in their design.

7

1 Comment

Quentin Gallea, PhD distinguishes between predictive and causal modelling behind the machine learning behind AI technologies. In Coaching, facilitating change through causal origins underpins realisation of potential, goals and effects that clients seek. It is also central to how the coach is being, why they ask questions they do and what their attention creates intentions for. Before you can join the dots you need to place them. Alternatively, once a line or path of thought or perspective is drawn, place dots at key turning points of its path, to help yourself and others account for progress. Good to know there are people who 'X-ray' reality the way Quentin does. It's what allows us to get finer 'MRI' views of the real dynamics of how things are and how they unfold. How often is the issue a client comes to a coach for the true causal origin of their visit?

4

4-Stage Recommender Systems : Retrieval -> Filtering -> Scoring/Ranking -> Ordering 1. Retrieval: Gather a broad set of potential items using techniques like collaborative filtering or content-based filtering. 2. Filtering: Narrow down the set based on user preferences, browsing history, or contextual factors like time and location. 3. Scoring: Use machine learning models, such as neural networks or gradient boosting machines etc., to predict and assign relevance scores to each filtered item. 4. Ordering: Arrange the ranked items in a sequence that maximizes user engagement or satisfaction, considering factors like diversity, novelty, and business goals. This ordered list is then presented to the user as personalized recommendations.

5

My latest Medium post comparing SetFit, FastFit, and Semantic Router for Few-shot NLP Intent Classification has been published by Towards AI! This post covers: 1. Details of NLP Intent classification and why it is extremely important in the age of LLM chatbots 2. Algorithm details of SetFit, FastFit and Semantic Router 3. Detailed code experiments on a chatbot intent detection dataset 4. Conclusion on the best method as of today. As with most of my posts, it has lots of code you can directly use in your work! This post was jointly written by Shilpa and me, both of us humans 😅 🔗 Medium Friend Link (without paywall): https://lnkd.in/dVfAvjzr Follow me for more tips on building successful ML and LLM products! Medium: https://lnkd.in/g2jAJn5 X: https://lnkd.in/g_JbKEkM #generativeai #llm #nlp #artificialintelligence #mlops #llmops

78

9 Comments

Another great talk from the PyData London 2024 - Backtesting and error metrics for modern time series forecasting by Kishan Manani Backtesting is the time series forecasting equivalent to the machine learning cross-validation approach for training and testing models. https://lnkd.in/g-DWgqmB #timeseries #forecasting #machinelearning #python

46

5 Comments

💰 LLM Pricing vs MMLU 📈 MMLU (Massive Multitask Language Understanding) is a benchmark that evaluates an LLM's performance across a wide range of tasks, providing a comprehensive measure of its language understanding capabilities. I've analysed the relationship between pricing and MMLU score for various cutting-edge models and its been insightful. 💡 Key Takeaways: 1. The general trend of higher prices correlating with better MMLU scores holds, but there are notable exceptions. GPT-4 and Claude 3 Opus lead the pack in performance, but GPT-4 comes at a lower price point. 2. Llama 3 variants dominate the low-cost market, with Llama 3 70B (Groq) offering an impressive 82.0 MMLU score at just $0.79 per 1M tokens. This further cements their position as high-performing, accessible open-source options. 3. The mid-range price tier offers a diverse selection of models, catering to various needs and budgets. From Claude 3 Sonnet and Haiku to Mistral Medium and Mixtral-8x22B, this segment provides a balance between performance and cost, with MMLU scores ranging from 75.2 to 79.0. Groq's emergence as a cloud provider is a game-changer, enabling the deployment of powerful open-source models like Llama and Mixtral at extremely competitive prices. This development levels the playing field and empowers startups and businesses of all sizes to harness the potential of advanced language models. As open-source models continue to improve and offer more competitive pricing, closed-source models will need to differentiate themselves through unique features, exceptional performance, and seamless integration to justify their premium pricing. Moreover, while price and MMLU scores are important metrics, it's essential to consider other aspects such as model size, training data, API features, long-term growth plans and ease of use when selecting a language model that best suits your specific needs. #LLMs #GenAI #OpenSource

10

If you are thinking about Gen AI or Conversational AI, think in terms of decrement instead of increment. I said to someone, "Gen AI and conversational AI Agent Architecture can complete tasks for you". They asked "So, it's like RPA? Like workflows?". They wanted to compare it to incumbent technology to learn how to think about it and how to envision it. The intention is good, but I asked them to not compare it to a past technology. I asked them to compare it to Humans instead. I asked, "What can humans do in your business?...", "Now how can Gen AI or conversational AI assist those humans or take over some of their work?". The mental switch is to think big. Solve big problems, big issues. In corporations, in government, in society. When it comes to Gen AI or Conversational AI, think about decrement to human effort, instead of an incremental technological advance.

36

2 Comments

Evaluating LLMs are very crucial for generative models. Do you know how to evaluate such models? Here are few popular metrics. 📍 Context-independent Metrics BLEU: Contxt-independent precision based metric. Number of n-grams in generated output that also appears in reference label. ROUGE: Context-independent recall based metric. Number of n-grams in reference that also occur in generated output. METEOR: Context-independent precision-recall based metric. No exact n-gram matching. Unigram matching on multiple levels (exact word matching, stemmed matching, paraphrase matching etc.) BertScore: Context-independent embedding based metric. It is same as METEOR but with embeddings. Precision-Recall is calculated by nearest cosine similarity match between generated output and reference. It is one-to-one matching (hard-alignment) between tokens of output and reference. MoverScore: Context-independent contextualized embedding based metric. Computes distance between tokens in output and reference. Mapping of semantically related words in one sequence to their counterparts in another sequence. It is many-to-one matching (soft-alignment) between tokens of output and reference. 📍 Context-dependent Metrics ROUGE-C: Context-dependent metric. Modifying ROUGE to compare output with source document instead of reference output. G-Eval: Context-dependent Chain-of-Thought (CoT) based metric. Provide task introduction and evaluation criteria to LLM and ask it to generate evaluation steps. This is very popular nowadays since GPT-4 and later models have very high Spearman correlation with human judgements. 📍 Consistency Evalulation Metrics (for NLI models) SummaC: Splitting reference document and generated output into sentences, and classify each output sentence into (neutral, aligned or inconsistent with reference texts) consistency labels. TrueTeacher: Distill LLMs (T-5 large, medium and small) by feeding them prompts and then provide these synthetic labels to NLI model in order to detect (classify) consistency. QuestEval: Precision-Recall based metric to generate questions from provided document and the summary. The generated question is compared with reference questions to compute precision-recall. There is also an eval metric for question answering where answer is generated by prompting question + document. 📍 Preference-based metrics: In between reference-based metrics and context-based metrics. Reward model is trained on human rank preferences of generated text. The model gets aligned to the outputs based on "preference-score" generated by reward model. 📍 Sampling-based metrics: When reference is not available, generated outputs of different LLMs can be compared to see the "agreement-score" to evaluate. For detailed information, please refer to the amazing article by Eugene Yan (link in comment).

9

2 Comments

🚀 Unlocking AI/ML with Snowflake Cortex Functions 🚀 Snowflake Cortex is transforming how AI and ML are integrated into data workflows. With LLM (Large Language Models) and ML-based functions, Cortex offers seamless access to powerful AI tools directly within the Snowflake platform. Key Features: LLM Functions: Use models like "gemma-7b" to generate, translate, or summarize text through SQL queries. ML Functions: Predictive analytics for structured data. Tokenization: Understand costs by managing token usage during text processing. Role-Based Access: Secure access with specific roles like CORTEX_USER. 💡 Stay ahead with this AI-ML integration revolution! https://lnkd.in/eRMk5cfi https://lnkd.in/edeidvbz #AI #MachineLearning #DataScience #Snowflake #CloudComputing #LLM #Automation #DataEngineering

2

- SwiftKV is a novel model transformation and distillation procedure (by folks at Snowflake) designed to optimize large language model (LLM) inference. Very cool findings in the paper (as presented in the vLLM meetup on 11/13). - SingleInputKV: Prefills later layers' KV cache using earlier layer outputs, reducing computation on prompt tokens. The key point is that the first few layers of attention in LLMs are similar (nice find), allowing SwiftKV's SingleInputKV mechanism to skip computations in later layers during inference. By using a single earlier layer's output to compute the KV cache for subsequent layers, it significantly reduces both computation (by 50%) and memory usage (by 62.5%) without sacrificing model performance. - AcrossKV: Merges KV caches of neighboring layers to minimize memory usage and increase batch size for higher throughput. - For models like Llama-3.1-8B and 70B, SwiftKV reduces prefill compute by 50%, memory usage by 62.5%, and achieves up to 2× higher throughput and 60% lower time per token. https://lnkd.in/guhEBW5G

57

Spark - RDD, Dataframe and Dataset!! Interviewer: Let's start with RDDs (Resilient Distributed Datasets). Can you explain what an RDD is and its role in distributed computing? Candidate: An RDD is a fundamental data structure in Apache Spark, designed to handle large-scale data processing across clusters. It represents an immutable, partitioned collection of records that can be operated on in parallel. RDDs provide fault tolerance through lineage information, enabling recomputation of lost data partitions. Interviewer: How does Spark's RDD differ from traditional data structures like arrays or lists? Candidate: Unlike arrays or lists, RDDs are distributed across multiple nodes in a cluster, allowing for parallel processing and fault tolerance. RDDs are immutable, meaning their contents cannot be changed once created. Operations on RDDs are lazily evaluated, allowing Spark to optimize execution plans and perform transformations efficiently. Interviewer: Moving on to dataframes in Spark. What is a dataframe, and how does it differ from RDDs? Candidate: A dataframe is a distributed collection of data organized into named columns, similar to a table in a relational database or a dataframe in Pandas. Unlike RDDs, dataframes provide a higher-level abstraction, allowing for structured data processing with support for SQL queries, optimizations, and integration with other data sources. Dataframes offer better performance and ease of use compared to RDDs for structured data processing tasks. Interviewer: Can you demonstrate how you would create a dataframe in Spark and perform some basic operations on it? Candidate: Sure. We can create a dataframe by loading data from various sources like CSV, JSON, or Parquet files using SparkSession. Once created, we can perform operations like selecting columns, filtering rows, aggregating data, and joining with other dataframes using DataFrame APIs or SQL queries. Interviewer: Lastly, let's discuss datasets in Spark. How do datasets differ from dataframes, and when would you choose one over the other? Candidate: Datasets are a newer API introduced in Spark that combine the benefits of RDDs and dataframes. Like dataframes, datasets support structured data processing with optimizations and type safety. However, datasets also provide the flexibility and performance of RDDs through user-defined functions (UDFs) and custom transformations. I would choose datasets over dataframes when dealing with complex data types or when fine-grained control over serialization and performance is required. Interviewer: Thank you for your explanations. This concludes our discussion on RDDs, dataframes, and datasets in Spark. Candidate: Thank you for the opportunity to discuss these concepts. #pyspark #bigdata

63

2 Comments