Prabakaran Chandran

For hyperparameter tuning hyperopt was the go to package. I recently learnt that the open-source package is no longer maintained. Microsoft and Databricks encourage the use of optuna for hyper parameter optimisation. MLflow have created this awesome tutorial to get you started using Databricks and Optuna.

11

Interesting research paper alert : https://lnkd.in/giwG6bnZ Been thinking about evaluators and can they be self taught ! Stumbled upon this research paper and here is a Quick read about this : This paper on "Self-Taught Evaluators" introduces a novel approach to improving large language model (LLM) evaluators without relying on human-annotated data. The authors present an iterative self-improvement scheme that generates synthetic training data, allowing the LLM-as-a-Judge to self-train by generating contrasting model outputs and reasoning chains. This iterative process enables the evaluator to produce more accurate judgments, resulting in continuous performance improvement without the need for expensive and potentially outdated human annotations. Key contributions and insights from the paper include: Self-Taught Iterative Process: The core innovation is the use of synthetic data to generate contrasting model responses, which the LLM-as-a-Judge then evaluates. Over iterations, the model self-trains on its own outputs, improving its judgment accuracy. Performance Gains: The self-taught method shows impressive results, improving the Llama-3-70B-Instruct model's accuracy on RewardBench from 75.4 to 88.7, outperforming other commonly used LLM evaluators like GPT-4. Synthetic Data Creation: Synthetic preference pairs are generated without human input by prompting the LLM to produce one high-quality response and one lower-quality response to a modified version of the original instruction. This method allows the model to self-annotate its own training data. Iterative Fine-Tuning: In each iteration, the model uses the judgments it generates to improve its ability to evaluate responses. As the model's accuracy increases, the quantity of correct training examples increases as well, driving further improvement. LLM-as-a-Judge: The paper emphasizes the growing trend of using LLMs not just as response generators but also as judges, especially for open-ended or complex instructions where human evaluation is costly and subjective. The LLM-as-a-Judge provides explainable reasoning chains alongside its judgments, which enhances interpretability. Generalization: The method shows strong generalization across different domains, including reasoning, coding, and other complex tasks, highlighting its flexibility and scalability. Overall, this work demonstrates a significant step forward in LLM-based evaluation by leveraging self-training techniques and synthetic data, which reduce dependency on human annotations and accelerate model development. Saylee Muley You might want to use this, I remember you trying to use an LLM to evaluate the results from LLMs. #generativeAI #LLMs #AI

7

5 Comments

Question : Why does quoting just the accuracy numbers do not make sense for an AI model❓ Its a common practice among data science practitioners to quote accuracy of an #AI model. While it is easier to understand accuracy numbers, this practice can be misleading and fail to capture the nuances of real-world scenarios, leading to misaligned expectations among business stakeholders. 👉 Accuracy alone does not provide a comprehensive picture of a model's capabilities. It fails to account for crucial factors such as the size and distribution of the training, testing, and validation datasets, the presence of class imbalances, and potential biases. Relying solely on accuracy can mask underlying issues and lead to false confidence in the model's performance. 👉 To truly understand a model's effectiveness, it is essential to consider a broader range of metrics. Precision, recall and F1 score offer valuable insights into a model's ability to correctly identify positive and negative instances, handle imbalanced data, and discriminate between classes. 👉 However, even these technical metrics may not resonate with business stakeholders who are primarily concerned with the practical implications of the model's performance. It is crucial to translate these metrics into business-relevant terms that directly align with the product's objectives and success criteria. For example, in a fraud detection system, precision could be framed as the percentage of flagged transactions that are indeed fraudulent, while recall could represent the proportion of actual fraudulent transactions correctly identified. By presenting these metrics in a business context, stakeholders can better understand the trade-offs and make informed decisions about the model's deployment and ongoing monitoring. 💡 Quick Tip: Dont quote accuracy numbers without mentioning the size of training and testing dataset. Consider a diverse set of metrics tailored to your specific use case and business objectives. Communicate these metrics in a language that resonates with stakeholders, fostering a shared understanding and aligning expectations. #artificialintelligence #productmanagement #datascience #productmanager

6

Question : Why does an AI product manager need to monitor #AI model❓ Once AI model is developed and deployed, the job doesnt end there. Model encodes pattern of the data it has seen during training phase. There may be several patterns or edge cases which are not part of training data. If model encounters any such patterns in testing or production phase, it will produce results which wont make sense without giving any warning about it. 👉 Data drift is a common challenge faced by AI-powered products. Over time, the distribution of input data can shift, leading to a mismatch between the model's training data and the real-world data it encounters. This can result in degraded model performance, unexpected outputs, and even complete model failure. 👉 The consequences of data drift can be severe, ranging from suboptimal user experiences to regulatory compliance issues. Product managers must be vigilant in detecting and addressing these challenges to ensure the continued success of their AI-powered offerings. 👉 Sudden changes in metrics can indicate input data drift. Additionally, monitoring the distribution of model outputs can reveal output drift, which may signal a need to retrain or fine-tune the model. Detecting data drift early is crucial. Product managers can leverage techniques such as A/B testing, anomaly detection, and continuous monitoring to identify and address these issues. Once detected, the product team can take appropriate actions, such as retraining the model, adjusting the data pipeline, or even considering a model architecture change. 💡 Quick Tip: Integrate model monitoring into your product roadmap and allocate dedicated resources to ensure the long-term success of your AI-powered offerings. Establish a comprehensive model monitoring dashboard to gain real-time visibility into the health and performance of your AI models. #artificialintelligence #productmanagement #datascience #productmanager

5

Something I've written on how a T20 batting innings can be assessed. This framework can be extended to assessment of individual players as well, which I will do in the susequent posts. For now, please feel free to comment and if you like the content, please subscribe. #cricket #cricketanalytics #data #dashboards #datavisualization #sportsanalytics https://lnkd.in/gb7T-UzJ

28

6 Comments

MLOps networking question: is there anyone based in India whose known for being great at MLOps? Like maybe they’ve - given some really great talks, or - done ML end to end (a few different ways/times), or - seen the full picture done at both small and LARGE data science teams, or - thought deeply about the various tools and where they fall short, and - are friendly and active in some of the communities - gone past the basics: anyone can train a model and get a batch inference pipeline or endpoint out the door—but few have thought about the hard problems like backfilling, monitoring, collaboration/exp tracking, governance, reproducibility, feature management, GPUs, working with lakehouses, data versioning, medium and large datasets, or lineage/data contracts I’m guessing they’d be a staff or principal engineer or maybe a lead / director. (Or undertitled) I can name a few people I know like this *outside* of India, but my network isn’t as strong there. (I’m thinking of Maria Vechtomova, Başak Tuğçe Eskili, 👨🏻‍💻 Jaco van Gelder) I’m hiring! It’s either a staff, principle, and/or tech lead. There's a team in Pune they'd be guiding/mentoring. I’m new to the team and the time difference in India makes communication harder so I want to find a friend I can trust and rely on to learn with me and make our ML Platform one of the best in the world. And also have fun with. We’re on Snowflake and AWS. Looking to hire a lot and select tools this year for the largest 3rd party seller on Amazon. We get to rebuild the platform from scratch for a really willing team in a profitable, private company of ~1600 where DS/AI/ML/BI is the star of the show right now. Growing the DS team from 20 to 30 this year. Powering several UI’s and major purchasing decisions. Tons of executive enthusiasm and support will be pointed our way. Currently 95% batch serving. Possibility of real-time on the horizon. Data: tabular, text, and images. The salary budget is aimed at India.

68

24 Comments

There is a gloom simmering in the data science industry. Confetti is flowing in the wind of generative AI. But at the same time an entire batch of freshers are anxious about their placements. Hiring has been extremely low across the board. Irrespective of tier-1, 2 or 3 campuses. These days whenever any student enquires about our courses, they mainly try to understand whether we provide "placement assistance" or not. Meaning if we will be able to find them a job or not. This is a great indicator to understand the overall mood of the batch. They're sadly, desperate for an opportunity. Looking at the industry from this vantage point breaks you a little bit. So from last two weeks I've been doing research on the topic. Talking to several industry veterans, hundreds of students, reading. And the conclusion is simple and worrying. AI adoption in everyday life is the future. We're moving towards an age where "internet of things" will be a reality pretty soon. Most of the work that can be automated will definitely be automated. As we'll be increasingly a more capitalistic society, most of the necessities will be controlled by companies. And as we all very well know, automation is an effective cost cutting measure. So many people will loose jobs, a lot of people will have to learn new skills to adjust, payouts will follow a leptokurtic distribution. Overall it'll be a gradually more competitive and skill-based economy. Harder to thrive in. Even though it might sound like doom. Life always finds a way. A different way for sure. But definitely a way. Which frankly I don't know yet. I'm planning to read more about this topic. it's a ocean of knowledge to consume. I'm in the water but still near the bay. What you'll see soon is something beautiful, that has come of these research. Till then please Stay Tuned. #lastbench #resumereview #coverletter #interviewpreparation #interviewtips #placement #placementassistqnce #datascience #analytics #machinelearning #careercoach

134

2 Comments

[New series on my GitHub blog - DS / ML Back to the Basics Series] - Statistical Implication of Parameter Estimation and Standard Error. Link -🔗https://lnkd.in/gv7p96SF I have experienced some DS / ML related modeling projects, but I’ve realized there’s so much I thought I knew yet never fully understood, things I used frequently without a deep understanding. So, I’ve decided to start a new series to organize and explain what I’ve learned in a way that anyone can understand. The first post in this series addresses the question, "What does it really mean to do estimation and modeling?" While this might sound obvious, we often use the term "modeling" without considering the important statistical implications behind it. To introduce this concept, I used a simple example of mean estimation to explore the idea of Standard Error. P.S. I am pretty sure that these might feel much more familiar to my fellow UCLA Anderson School of Management MSBA classmates, thanks to our comprehensive coursework!

21

Summary of the recently concluded test match. https://lnkd.in/g6hZNruW #cricketanalytics #cricketdata #cricket #sportsanalytics

13

Unlike software engineering or software development the entry barrier for a machine learning engineer is still very high and it hasn’t changed much over the last 5 years Being an ML engineer at FAANG still requires one to have a masters or PhD. So how can you enter this field? I interview for Amazon’s applied scientist post and cleared 3 rounds. Even though I was just a bachelors degree holder I got shortlisted and performed well for the 3 rounds. So how did I do it? Why was I shortlisted? Is it because I’m from tier 1? Not necessarily, cause I did have a talk with Amazon employees as to what it takes to be an applied ML scientist at Amazon. Having a masters or phd surely increases your chances but here’s how you can get shortlisted and probably selected as well for an applied ML scientist position at Amazon or other FAANG or bit MNCs. The first important thing to have is, prior work experience to prove that you’re proficient with ML. It can either come through internships, projects or publications. Fortunately I had them all from my undergraduate studies, so it increased my chances. Having a good kaggle rating is really really important if you don’t have internships or projects to write. Very few people do kaggle regularly and it’s really a game changer. Even I didn’t do kaggle much in college. Practice maths behind ML is I cannot emphasize how important maths is. Most of the students I talked to, when I ask them why they used a certain model or certain type of architecture or regularizer for this problem stumble to give me any strong answer cause their solution was not mathematically motivated but rather something they found on tutorials or internet. It’s fine to use tutorials but you should know why you’re doing this and not something else. Maths hone your intuitive thinking as well. Given a new problem how to approach it from ground up Last but not the least Be good at Python. Don’t write crappy unreadable and non reproducible code ML is all about reproducibility of experiments.

304

10 Comments

I call this time series model the ARIMA Killer!! Dramatic hooks gets us more clicks right? Anyways, for people who are into time series, do read this. I am doing a series (has no relation with pandas Series, nor time series, HAHAHA, {lame}) in which I would be covering somewhat latest developments in time series forecasting/analysis, and some of the common and/or advanced time series tasks that the Internet Gurus won't teach you. I am excited to announce that I have successfully written another blog🚀💫 (sarcasm) #timeseries

8

2 Comments

Want to perform hyperparameter tuning on your ML and DL algorithms? Learn about how you can do it with Optuna and neptune.ai

15

While exploring various transformer-based architectures, I stumbled upon a great blog titled: "Understanding Large Language Models" by Sebastian Raschka, PhD The blog provides a concise overview of the key papers that have shaped the field of large language models (LLMs) in recent years. Sebastian mentioned the papers to read in chronological order to understand the evolution of LLM architectures and training techniques. The post starts with the original transformer paper from 2017 and covers influential works on attention mechanisms, layer normalization, and the bifurcation of LLMs into encoder-style models for predictive tasks and decoder-style models for generative tasks. It also discusses the ULMFiT approach for finetuning language models on specific tasks. The blog concludes by covering the topic of alignment, which aims to steer LLMs towards intended goals and interests. Key papers discussed include the InstructGPT approach that led to ChatGPT and a survey on parameter-efficient finetuning methods. Overall, the blog provides a well-curated reading list for researchers and practitioners looking to understand the rapid progress in LLMs over the past few years. Here's the link: https://lnkd.in/dEuq4e8p Happy Learning! #LLMs #LargeLanguageModels #GenAI #NLP #NaturalLanguageProcessing #TransformerArchitecture #ScalingLaws #ParameterEfficientFinetuning #LanguageModelAlignment #AIResearch #MachineLearning #ComputerScience

16

 Dear LinkedIn Family, We are looking for Senior Data Scientist for Pune Location. #DataScientist LLM Architecture: Good understanding of the architecture underlying large language models, such as Transformer-based models and their variants. Design and implement deep learning model architectures using PyTorch. #LLM #PyTorch #pytorch Language Model Training and Fine-Tuning: Experience in training large-scale language models from scratch, as well as fine-tuning pre-trained models on domain data. Data Preprocessing for NLP: Skilled in preprocessing textual data, including tokenization, stemming, lemmatization, and handling of different text encoding. Transfer Learning and Adaptation: Proficiency in applying transfer learning techniques to adapt existing LLMs to new languages, domains, or specific business needs. Data Annotation and Evaluation: Skills in designing and implementing data annotation strategies for training LLMs and evaluating their performance using appropriate metrics. Scalability and Deployment: Experience in scaling LLMs for production environments, ensuring efficiency and robustness in deployment. Model Training, Optimization, and Evaluation: Evaluate the performance of PyTorch models using appropriate metrics and techniques like cross-validation, holdout sets, or online evaluation. This encompasses the complete cycle of training, fine-tuning, and validating language models. You will be designing and adapting LLMs for use in virtual assistants, Information retrieval and extraction etc. Experimentation with Emerging Technologies and Methods: Actively exploring new technologies and methodologies in language model development, including experimental frameworks and software tools. LLM Alignment: Understanding of algorithms like DPO, PPO, KPO, RLHF and using it for guardrails. AI Data Retrieval: Data retrieval from unstructured data, extract key value pairs using techniques like donut, layoutLM, table transformers. Analyze data and build EDAs to identify data patterns Hands-on and strong understanding of concepts in Deep Learning and NLP Proficient in TensorFlow and similar libraries.   Required Qualifications 5+ years of hands-on experience in developing and deploying Large Language Models, and Machine learning and working with Pytorch. A thorough understanding of machine learning, particularly deep learning techniques, including knowledge of neural network architectures, training methods, and optimization algorithms. Proficiency in AI technology, Python, including experience with NLP libraries (e.g., Hugging Face Transformers, NLTK, spaCy), text classification. Experience with frameworks: PyTorch, or Tensorflow. Experience with cloud services (AWS, Azure) and ML deployment tool Docker Familiarity with model fine-tuning and optimization techniques for LLMs. Proven track record of innovative solutions in the field of LLMs. Strong communication skills, with the ability to explain complex AI concepts to non-expert audiences.

4

1 Comment

Training LLMs using reinforcement learning using human feedback. I was slacking on this topic for 6 months and finally was able to go through it, and wow I am amazed at what technologies are being worked on in production-ready systems these days. Generally in RL of LLMs, we have a reward model (let's say a good/bad text classifier) that gives a reward (prediction 1 for good, -1 for bad) and an RL algorithm (based on multiple loss functions) updates the weights of the LLM model to achieve better results. This is an iterative process between the reward model, the LLM, and the RL algorithm. Overall, my take is the most important aspect of training LLM using RLHM is obtaining that human feedback and training a reward model.

28