Kaniz Fatma

Time series forecasting using generative models continues to be a challenging area owing to multiple factors like complex temporal dynamics (trends, seasonality), effect of dimensionality curse (very high dimensional data), non-stationarity problem, sparse datasets etc to name a few. IBM research team has introduced Tiny Time Mixers (TTM), a compact and efficient pre-trained model designed for enhanced zero/few-shot multivariate time series forecasting. TTM addresses the limitations of large pre-trained models, which often require significant computational resources and overlook cross-channel and exogenous correlations. Leveraging a lightweight architecture based on TSMixer, TTM incorporates innovative techniques like adaptive patching, diverse resolution sampling, and multi-level modeling to handle diverse dataset characteristics with minimal computational requirements. TTM significantly outperforms existing benchmarks in accuracy while being more resource-efficient, making it highly accessible for broader applications, including those constrained by computational resources. The research paper highlights TTM’s versatility in various forecasting tasks and its potential for widespread adoption due to its effectiveness on both CPU and GPU environments. Paper can be accessed from the following link: https://lnkd.in/ekqw5Uwn

16

Another great talk from the PyData London 2024 - Backtesting and error metrics for modern time series forecasting by Kishan Manani Backtesting is the time series forecasting equivalent to the machine learning cross-validation approach for training and testing models. https://lnkd.in/g-DWgqmB #timeseries #forecasting #machinelearning #python

46

5 Comments

How to create Table, Listing, and Figure (TLF) Shells: The Statistical Analysis Plan (SAP) and associated TLF shells are 1A and 1B when it comes to the deliverables the biostatistician produces during the study. The SAP is the written language version of the statistician telling any reviewers of the clinical trial how the data was analyzed (what populations were used, what statistical techniques were implemented, how missing data was handled, etc.). The Shells then, are the physical representation of what the written language says. The Shells give the programming team something to program towards AND give the CSR development team (Sponsor, Medical Writer, Medical, etc.) an idea of what the final outputs will look like. Here is how to ensure you create shells that are clear and meaningful to everyone: 1️⃣ Start with the SAP - as you write the SAP and any time you specify a table, figure, or listing will be generated, make a mark in the SAP for yourself. That mark is to tell you that you are calling for an output to be created. You can then create a table of contents for all the outputs the SAP is calling for. Remember, the SAP and Shells should be a 1:1 mapping exercise - if the SAP calls for an output, there had better be a shell, and if there is a shell, there had better be text in the SAP about that shell. 2️⃣ Stand on the shoulders of giants - Very rarely have I created shells truly from scratch. I am more times than not creating shells from a template my company or the sponsor has, OR I am finding results/looking up other studies that have presented results similar to the type of results I plan on presenting - if you don't have a shell for what MMRM output should look like you can always go to clin trials dot gov and search for studies that have their statistical analysis plans or results available and see if there are any examples included (if not directly available from clintrials you can see if the drug was approved, and if it was, check out it's SBA on the FDA's website for the results and some inspiration). 3️⃣ Take your time - Often (for one reason or another) you may find yourself with a tight timeline to create TLF shells. If you have any doubts that you do not have enough time to create the shells - speak with your manager as early as possible. The shells are CRITICAL for the success of the study analysis and if you rush through them you run the risk of the programming team getting stuck because the shells are not clear, or worse, you may end up creating a shell that cannot be used by the CSR development team. 4️⃣ If ever you are not sure what to include in a shell - include as much as you can - even if it is overkill. It is much simpler to remove something than it is to add something later in the game. Shells are a huge topic and one that deserves more time to be covered - if you have any questions about creating them please feel free to send me a direct message. Happy Monday

108

3 Comments

Finding the Right Number of Clusters for K-Means In data analysis, K-means clustering stands as a pivotal algorithm for partitioning data into groups based on similarity. However, the success of this algorithm hinges on one critical decision: selecting the appropriate number of clusters, 'K'. This choice is not arbitrary; it can significantly impact the algorithm's ability to reveal insightful patterns within the data. The Elbow Method is a popular technique that aids in this decision. By plotting the sum of squared distances of samples to their nearest cluster center, analysts can observe where the rate of decrease in these distances 'bends' or forms an 'elbow'. This inflection point is often indicative of the optimal number of clusters, beyond which adding more clusters does not provide significant improvement in the variance explained. Another valuable metric is the Silhouette Score, which offers a measure of how well an object fits within its cluster compared to other clusters. A high silhouette score suggests that the object is well-matched to its own cluster and poorly matched to neighboring clusters, thus providing a clear demarcation of cluster boundaries. The Davies-Bouldin Index also serves as a useful tool, focusing on the ratio of within-cluster distances to between-cluster distances. The goal is to minimize this index, as lower values indicate that clusters are well-separated and compact, which is desirable in a well-structured dataset. Each of these methods possesses unique advantages and caters to different dataset characteristics. For instance, the Elbow Method is intuitive and easy to visualize, making it an excellent starting point for analysts. The Silhouette Score, on the other hand, provides a more nuanced view of cluster fit, which can be crucial for datasets with overlapping clusters. The Davies-Bouldin Index excels in identifying sets of clusters that are well separated, which is particularly beneficial for datasets where cluster separation is paramount. In conclusion, determining the right 'K' is a nuanced process that requires consideration of the dataset's specific attributes and the intended use of the clustering results. By employing methods like the Elbow Method, Silhouette Score, and Davies-Bouldin Index, analysts can approach this task with a toolkit designed for precision and clarity. #KMeansClustering #DataScience #MachineLearning #ElbowMethod #SilhouetteScore #DaviesBouldinIndex #ClusterAnalysis #BigData #Analytics

3

3 Comments

MLOps networking question: is there anyone based in India whose known for being great at MLOps? Like maybe they’ve - given some really great talks, or - done ML end to end (a few different ways/times), or - seen the full picture done at both small and LARGE data science teams, or - thought deeply about the various tools and where they fall short, and - are friendly and active in some of the communities - gone past the basics: anyone can train a model and get a batch inference pipeline or endpoint out the door—but few have thought about the hard problems like backfilling, monitoring, collaboration/exp tracking, governance, reproducibility, feature management, GPUs, working with lakehouses, data versioning, medium and large datasets, or lineage/data contracts I’m guessing they’d be a staff or principal engineer or maybe a lead / director. (Or undertitled) I can name a few people I know like this *outside* of India, but my network isn’t as strong there. (I’m thinking of Maria Vechtomova, Başak Tuğçe Eskili, 👨🏻‍💻 Jaco van Gelder) I’m hiring! It’s either a staff, principle, and/or tech lead. There's a team in Pune they'd be guiding/mentoring. I’m new to the team and the time difference in India makes communication harder so I want to find a friend I can trust and rely on to learn with me and make our ML Platform one of the best in the world. And also have fun with. We’re on Snowflake and AWS. Looking to hire a lot and select tools this year for the largest 3rd party seller on Amazon. We get to rebuild the platform from scratch for a really willing team in a profitable, private company of ~1600 where DS/AI/ML/BI is the star of the show right now. Growing the DS team from 20 to 30 this year. Powering several UI’s and major purchasing decisions. Tons of executive enthusiasm and support will be pointed our way. Currently 95% batch serving. Possibility of real-time on the horizon. Data: tabular, text, and images. The salary budget is aimed at India.

68

24 Comments

Question : Why does an AI product manager need to monitor #AI model❓ Once AI model is developed and deployed, the job doesnt end there. Model encodes pattern of the data it has seen during training phase. There may be several patterns or edge cases which are not part of training data. If model encounters any such patterns in testing or production phase, it will produce results which wont make sense without giving any warning about it. 👉 Data drift is a common challenge faced by AI-powered products. Over time, the distribution of input data can shift, leading to a mismatch between the model's training data and the real-world data it encounters. This can result in degraded model performance, unexpected outputs, and even complete model failure. 👉 The consequences of data drift can be severe, ranging from suboptimal user experiences to regulatory compliance issues. Product managers must be vigilant in detecting and addressing these challenges to ensure the continued success of their AI-powered offerings. 👉 Sudden changes in metrics can indicate input data drift. Additionally, monitoring the distribution of model outputs can reveal output drift, which may signal a need to retrain or fine-tune the model. Detecting data drift early is crucial. Product managers can leverage techniques such as A/B testing, anomaly detection, and continuous monitoring to identify and address these issues. Once detected, the product team can take appropriate actions, such as retraining the model, adjusting the data pipeline, or even considering a model architecture change. 💡 Quick Tip: Integrate model monitoring into your product roadmap and allocate dedicated resources to ensure the long-term success of your AI-powered offerings. Establish a comprehensive model monitoring dashboard to gain real-time visibility into the health and performance of your AI models. #artificialintelligence #productmanagement #datascience #productmanager

5

Here's a basic implementation of a Variational Autoencoder (VAE) from scratch using PyTorch: *VAE Model* ``` import torch import torch.nn as nn import torch.optim as optim from torch.distributions import Normal class VAE(nn.Module): def __init__(self, latent_dim, hidden_dim, input_dim): super(VAE, self).__init__() self.encoder = nn.Sequential( nn.Linear(input_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, latent_dim*2) ) self.decoder = nn.Sequential( nn.Linear(latent_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, hidden_dim), nn.ReLU(), nn.Linear(hidden_dim, input_dim) ) def encode(self, x): z_mean, z_log_var = self.encoder(x).chunk(2, dim=1) return z_mean, z_log_var def reparameterize(self, z_mean, z_log_var): std = torch.exp(0.5 * z_log_var) eps = torch.randn_like(std) z = z_mean + eps * std return z def decode(self, z): return self.decoder(z) def forward(self, x): z_mean, z_log_var = self.encode(x) z = self.reparameterize(z_mean, z_log_var) x_reconstructed = self.decode(z) return x_reconstructed, z_mean, z_log_var def loss_function(self, x, x_reconstructed, z_mean, z_log_var): reconstruction_loss = ((x - x_reconstructed) ** 2).sum(dim=1).mean() kl_loss = -0.5 * (1 + z_log_var - z_mean**2 - torch.exp(z_log_var)).sum(dim=1).mean() return reconstruction_loss + kl_loss ``` *Training Loop* ``` vae = VAE(latent_dim=2, hidden_dim=256, input_dim=784) optimizer = optim.Adam(vae.parameters(), lr=0.001) for epoch in range(100): for x in train_loader: x = x.view(-1, 784) x_reconstructed, z_mean, z_log_var = vae(x) loss = vae.loss_function(x, x_reconstructed, z_mean, z_log_var) optimizer.zero_grad() loss.backward() optimizer.step() print(f'Epoch {epoch+1}, Loss: {loss.item()}') ``` *Example Use Case* To use this VAE, you can train it on a dataset like MNIST, and then use it to generate new samples or perform anomaly detection. For example, to generate new samples, you can sample from the latent space and pass the samples through the decoder: ``` z = torch.randn(1, 2) x_reconstructed = vae.decode(z) ``` This will generate a new sample that is similar to the training data. Note that this is a basic implementation, and you may want to add additional features such as batch normalization, convolutional layers, or more complex architectures. Additionally, you can experiment with different loss functions, optimizers, and hyperparameters to improve the performance of the VAE.

7

#DataPipeline with #S3, #AWSGlue , #AmazonAthena, #AmazonQuickSight High-Level Overview: 1. Data Ingestion: Data is ingested and stored in Amazon S3. 2. Data Transformation: AWS Glue is used to catalog and transform the data. 3. Data Querying: Amazon Athena queries the data cataloged by AWS Glue. 4. Data Visualization: Amazon QuickSight is used to visualize the data.  Step-by-Step Guide: 1. Data Ingestion - Step 1.1: Upload your raw data to an S3 bucket.  - Create an S3 bucket (if not already created).  - Upload your raw data files (e.g., CSV, JSON) to this S3 bucket. 2. Data Transformation with AWS Glue - Step 2.1: Create an AWS Glue Crawler.  - Open the AWS Glue console.  - Go to Crawlers and click "Add crawler".  - Define the crawler name and specify the data store as the S3 bucket where your raw data resides.  - Configure the IAM role for the crawler to have appropriate permissions to read from S3 and write to the Glue Data Catalog.  - Choose a database in Glue Data Catalog where the metadata will be stored.  - Run the crawler to catalog the data. - Step 2.2: Create an AWS Glue ETL Job.  - Open the AWS Glue console.  - Go to Jobs and click "Add job".  - Define the job name and choose the same IAM role used by the crawler.  - For the ETL source, select the table created by the crawler.  - Define the transformations needed (e.g., data cleaning, normalization).  - Define the target data store, which could be another S3 bucket or folder for the transformed data.  - Configure the job properties (e.g., script, worker type, number of workers).  - Save and run the Glue job to transform the data. - Step 2.3: Update the Glue Data Catalog with transformed data.  - Create another Glue Crawler for the transformed data if necessary, following the same steps as above.  - Run the crawler to catalog the transformed data.  3. Data Querying with Amazon Athena - Step 3.1: Query the cataloged data.  - Open the Amazon Athena console.  - Make sure you are connected to the Glue Data Catalog.  - Run SQL queries on the tables created by the Glue crawler. Example Query:  SELECT * FROM my_database.transformed_data WHERE column1 = 'some_value';  4. Data Visualization with Amazon QuickSight - Step 4.1: Set up Amazon QuickSight.  - Open the Amazon QuickSight console.  - Choose to "Manage data" and then click "New data set". - Step 4.2: Connect QuickSight to Athena.  - Select "Athena" as your data source.  - Provide a name for the data source and click "Create data source".  - In the "Choose your table" step, select the database and table created in Athena. - Step 4.3: Prepare the data in QuickSight.  - Use QuickSight's data preparation features to modify and enhance the dataset as needed.  - Click "Save & visualize". - Step 4.4: Create visualizations in QuickSight.  - Use the data to create various charts, graphs, and dashboards.  - Customize the visualizations according to your needs.  - Share the dashboards with stakeholders. #DataEngineering

3

Spark - RDD, Dataframe and Dataset!! Interviewer: Let's start with RDDs (Resilient Distributed Datasets). Can you explain what an RDD is and its role in distributed computing? Candidate: An RDD is a fundamental data structure in Apache Spark, designed to handle large-scale data processing across clusters. It represents an immutable, partitioned collection of records that can be operated on in parallel. RDDs provide fault tolerance through lineage information, enabling recomputation of lost data partitions. Interviewer: How does Spark's RDD differ from traditional data structures like arrays or lists? Candidate: Unlike arrays or lists, RDDs are distributed across multiple nodes in a cluster, allowing for parallel processing and fault tolerance. RDDs are immutable, meaning their contents cannot be changed once created. Operations on RDDs are lazily evaluated, allowing Spark to optimize execution plans and perform transformations efficiently. Interviewer: Moving on to dataframes in Spark. What is a dataframe, and how does it differ from RDDs? Candidate: A dataframe is a distributed collection of data organized into named columns, similar to a table in a relational database or a dataframe in Pandas. Unlike RDDs, dataframes provide a higher-level abstraction, allowing for structured data processing with support for SQL queries, optimizations, and integration with other data sources. Dataframes offer better performance and ease of use compared to RDDs for structured data processing tasks. Interviewer: Can you demonstrate how you would create a dataframe in Spark and perform some basic operations on it? Candidate: Sure. We can create a dataframe by loading data from various sources like CSV, JSON, or Parquet files using SparkSession. Once created, we can perform operations like selecting columns, filtering rows, aggregating data, and joining with other dataframes using DataFrame APIs or SQL queries. Interviewer: Lastly, let's discuss datasets in Spark. How do datasets differ from dataframes, and when would you choose one over the other? Candidate: Datasets are a newer API introduced in Spark that combine the benefits of RDDs and dataframes. Like dataframes, datasets support structured data processing with optimizations and type safety. However, datasets also provide the flexibility and performance of RDDs through user-defined functions (UDFs) and custom transformations. I would choose datasets over dataframes when dealing with complex data types or when fine-grained control over serialization and performance is required. Interviewer: Thank you for your explanations. This concludes our discussion on RDDs, dataframes, and datasets in Spark. Candidate: Thank you for the opportunity to discuss these concepts. #pyspark #bigdata

63

2 Comments

Prometheus 2: open-source Language Model (LM) evaluation! Researchers have introduced Prometheus 2, a powerful open-source LM specifically designed for evaluating other LMs. This addresses growing concerns about the transparency, controllability, and affordability of relying solely on proprietary LMs like GPT-4 for evaluation purposes. Challenges with Existing Open-Source Evaluator LMs: *  Discrepancies with Human Judgement: Scores generated by these models often deviate significantly from human evaluations. *  Limited Evaluation Formats: They lack the flexibility to handle both direct assessments and pairwise rankings, which are the two most common evaluation methods. *  Restricted Evaluation Criteria: Existing models focus on general attributes like helpfulness and harmlessness, without the ability to incorporate custom criteria. Prometheus 2: A Game Changer *  Alignment with Human Judgements: Prometheus 2 demonstrates a remarkable ability to closely mirror human evaluations, as well as those of GPT-4. *  Versatile Evaluation Formats: It effectively handles both direct assessments and pairwise rankings. *  Customizable Evaluation Criteria: Users can define specific criteria to guide the evaluation process, allowing for greater control and precision. Results: Across various benchmarks, Prometheus 2 achieves the highest correlation and agreement with human judgements and proprietary LMs among all tested open-source evaluator LMs. Open Access: The models, code, and data for Prometheus 2 are all publicly available, fostering transparency and collaboration within the research community. Link to access the resources: https://lnkd.in/gudkqVyv Read more about the "Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models" in this research paper: https://lnkd.in/giW-rmRW #LanguageModels #LLM #AI #NLP #opensource #MachineLearning #LargeModels

26

Something I've written on how a T20 batting innings can be assessed. This framework can be extended to assessment of individual players as well, which I will do in the susequent posts. For now, please feel free to comment and if you like the content, please subscribe. #cricket #cricketanalytics #data #dashboards #datavisualization #sportsanalytics https://lnkd.in/gb7T-UzJ

28

6 Comments

His teaching method is exceptional, presenting Data Science as more than just a series of code lines. Ayush Singh extensive experience enables us to gain a deeper understanding and view Data Science from a unique perspective. Honestly Data science is not just about pd.read_csv() or doing EDA just for the sake of doing it without understanding why we need to do it and what important insights we will be getting from this EDA. It’s far more than this…. #datascience #machinelearning

17

2 Comments

Another way of handling missing values in your modelling projects!! In data science, handling missing values is a common and crucial task. Common methods include imputation, deletion, or using algorithms that can handle missing values natively. However, another effective way is using missing value indicators. Scenario: Imagine we are modeling loan approval of loan applicants based on marital status, as one of the features. In our sample, 30% of individuals are unmarried, thus their marital status is missing. This missingness isn't random; it indicates that these individuals are actually unmarried. Instead of just imputing these missing values with the mode, we can use an indicator to signify missingness (1 for unmarried and 0 for married). This provides the model with additional information. More info and practical example here: https://lnkd.in/drMhfbHz #machinelearning #datascience #missingvalues #missingvalueimputation #handlingmissingvalues #statistics

17

The Power of Discipline in Data Science: Consistency Is Key In the ever-changing world of data science, it is easy to get caught up in the excitement of new technologies and cutting-edge algorithms. But today, I want to talk about something less flashy yet equally crucial: the power of discipline, consistency, and routine in data science work. Here's why these habits are game-changers: 1. Continuous Learning: The field evolves rapidly. A disciplined approach to daily learning keeps you ahead of the curve. 2. Data Quality: Consistently following data cleaning and validation routines ensures your analyses start on solid ground. 3. Reproducibility: A structured workflow reproduces your work, benefiting you and your team. 4. Efficiency: Routine automates decision-making for repetitive tasks, freeing up mental energy for complex problems. 5. Project Management: Consistent progress, even in small increments, leads to timely project completion. 6. Skill Mastery: Regular practice of core skills (coding, statistics, visualization) leads to mastery over time. 7. Problem-Solving: A disciplined approach helps in methodically tackling complex data challenges. 8. Work-Life Balance: Established routines can help manage workload and prevent burnout. Remember, in data science, brilliance matters—but consistency often wins the day. What daily habits have you found most impactful in your data science journey? #DataScience #Discipline #ContinuousLearning #ProductivityTips

7

Our Sentiment Analysis independent research paper is finally live! Please check it out and comment your thoughts 😁 https://lnkd.in/gCfjgx2J Here’s the abbreviated summary I gave a friend: There's a commonly used emotion wheel that was created in 1980 by Robert Plutchik. This emotion wheel continues to be very impactful in fields such as psychology and psychiatry. That wheel organized emotions in various rungs of intensity. Our research showed that, most of the emotions were portrayed differently by the four LLMs we examined — aka the "more intense" emotions were actually less intense and vice versa. In one particularly intriguing instance, the words of varying intensity (Anticipation & Vigilance) weren't consistent on positive or negative sentiment. The four LLMs themselves portrayed sentiments wildly differently. The emotional consistency from most to least consistent was ChatGPT > Claude > Llama > Gemini. Consistency isn’t inherently good or bad —professional uses usually call for more consistency while creative uses usually call for more variability. I’ll include the emotion wheel in the comments too! Thanks to Dylan Ward, Dana J., Ph.D, A.R. Lantrip, M.A., and everyone else for being such incredible research mates 🥰🎉

35

3 Comments