Karthick's Sunday Learning (17/11)

As I strive to learn every day, here are my this week's learning and deep reading as I took some time to recap/reflect on those topics for my network and newsletter. Are you motivated to learn more every day? If not, ask yourself what is missing and go after it.

1. Bayesian Inference

2. Bias and Variance

Bayesian Inference

Bayesian inference is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as more evidence or information becomes available. Named after Thomas Bayes, an 18th-century statistician and philosopher, this approach is fundamental in the field of statistics and has applications across a wide range of disciplines, including machine learning, data science, and artificial intelligence.

At the heart of Bayesian inference lies Bayes' theorem, which describes the probability of an event, based on prior knowledge of conditions that might be related to the event. Bayes' theorem can be stated mathematically as:

P(H|E) = [P(E|H) * P(H)] / P(E)

Where:

·      P(H|E) is the posterior probability: the probability of hypothesis H given the evidence E.

·      P(E|H) is the likelihood: the probability of evidence E given that hypothesis H is true.

·      P(H) is the prior probability: the initial probability of hypothesis H.

·      P(E) is the marginal likelihood: the total probability of evidence E under all possible hypotheses.

Bayesian inference is a specific way to learn from data that is heavily used in statistics for data analysis. Bayesian inference is used less often in the field of machine learning, but it offers an elegant framework to understand what “learning” actually is. It is generally useful to know about Bayesian inference.

What is Bayesian Inference?

Bayesian inference is like a special way of thinking that helps you make smart guesses based on what you already know and what you see happening.

Imagine you have a big jar full of jellybeans. There are red ones, green ones, blue ones, and yellow ones. Now, let’s say you want to figure out how many of each color are in the jar without looking inside. This might sound tricky, but there’s a super cool way called "Bayesian inference" that can help us make really good guesses!

How Does It Work?

Let’s break it down step by step with our jellybean jar:

Step 1: Start with What You Know

First, you think about what you already know or what you believe about the jellybeans. Maybe you've seen similar jars before, and you think there are usually about the same number of each color. This is called your "prior belief."

Step 2: Get New Information

Next, you get some new information. You reach into the jar, grab a handful of jellybeans, and count them. Let’s say you grab 10 jellybeans and find out that 3 are red, 4 are green, 2 are blue, and 1 is yellow.

Step 3: Update What You Know

Now, you mix what you already knew with the new information you got from the handful of jellybeans. You use this to make a better guess about the whole jar. This is called "updating your belief."

Bayesian inference is super useful because it helps you make better guesses even when you don’t have all the information. It’s like having a magic tool that makes your guesses smarter every time you learn something new.

Key Concepts in Bayesian Inference

Prior Distribution

The prior distribution represents the initial beliefs about the parameters before any evidence is observed. It encapsulates all the information known about the parameter prior to the current data. The choice of prior can significantly influence the results of Bayesian analysis, and it can be based on previous studies, expert knowledge, or non-informative assumptions.

Likelihood

The likelihood function measures the plausibility of the observed data under different parameter values. It is a critical component in Bayesian analysis as it connects the prior distribution to the observed data, allowing for the updating of beliefs about the parameter values.

Posterior Distribution

The posterior distribution combines the prior distribution and the likelihood of the observed data to form a new, updated belief about the parameter. It is the result of the Bayesian updating process and represents the updated state of knowledge after observing the evidence.

Marginal Likelihood

Also known as the evidence, the marginal likelihood is the probability of observing the data under all possible parameter values. It acts as a normalising constant in Bayes' theorem to ensure that the posterior distribution is a valid probability distribution.

Applications of Bayesian Inference

Machine Learning

Bayesian inference plays a crucial role in machine learning, particularly in the development of probabilistic models. It allows for the integration of prior knowledge into the learning process, leading to more robust and interpretable models. Techniques such as Bayesian neural networks, Gaussian processes, and latent Dirichlet allocation are examples of Bayesian methods in machine learning.

Data Science

In data science, Bayesian inference is used for parameter estimation, hypothesis testing, and model comparison. Its ability to provide a probabilistic framework for uncertainty quantification makes it valuable for making informed decisions based on data. Bayesian methods are employed in various fields, including healthcare, finance, and marketing, to analyze complex data and derive actionable insights.

Artificial Intelligence

Bayesian inference is fundamental in artificial intelligence for reasoning under uncertainty. It is used in Bayesian networks, which are graphical models representing the probabilistic relationships among variables. These models are applied in areas such as natural language processing, computer vision, and robotics to perform tasks like prediction, classification, and decision-making.

Economics

Economists use Bayesian inference for estimating economic models, making policy decisions, and forecasting future economic trends. By incorporating prior knowledge and updating beliefs with new data, Bayesian methods provide a flexible and rigorous approach to economic analysis.

Advantages of Bayesian Inference

Incorporation of Prior Knowledge

One of the most significant advantages of Bayesian inference is its ability to incorporate prior knowledge into the analysis. This feature allows for more informed and contextually relevant inferences, especially when dealing with limited or noisy data.

Probabilistic Interpretation

Bayesian inference provides a probabilistic interpretation of parameter estimates and predictions. This approach allows for the quantification of uncertainty and the ability to make probability-based statements about the data and hypotheses.

Flexibility

The Bayesian framework is highly flexible and can be applied to a wide range of models and data structures. It accommodates complex models with hierarchical structures, latent variables, and missing data, making it suitable for diverse applications.

Model Comparison

Bayesian methods facilitate model comparison through the computation of the marginal likelihood. By comparing the evidence for different models, Bayesian inference can identify the model that best explains the observed data.

Challenges and Criticisms

Choice of Prior

The selection of an appropriate prior distribution can be challenging and subjective. Inappropriate priors can lead to biased or misleading results, and the influence of the prior may dominate the posterior distribution, particularly with limited data.

Computational Complexity

Bayesian inference often involves complex integrals that are analytically intractable. This complexity necessitates the use of computational techniques such as Markov Chain Monte Carlo (MCMC) methods, which can be computationally intensive and time-consuming.

Interpretation of Results

The interpretation of Bayesian results requires a thorough understanding of the probabilistic framework and the influence of prior assumptions. Communicating Bayesian results to non-experts can be challenging and may require additional explanation and visualisation.

Using Bayesian Inference in Real Life

Bayesian inference isn’t just for jellybeans. People use it in real life for all sorts of things!

Here are a few examples:

• Weather Forecasting: Meteorologists (the people who predict the weather) use Bayesian inference to guess if it will rain or be sunny based on past weather patterns and current information.

• Medical Diagnosis: Doctors use it to figure out what might be wrong with a patient based on symptoms and medical history.

• Games and Sports: Coaches use it to make decisions about which players to put in the game based on how they’ve been playing recently.

Bayesian inference is like having a superpower for making smart guesses. It helps you use what you know and what you learn to figure out things more accurately. So, next time you’re trying to guess how many jellybeans are in a jar, remember you can use Bayesian inference to make the best guess ever!

Resources

• Bayesian Inference intro - https://www.wallstreetmojo.com/bayesian-inference/

Bayesian inference is a powerful and versatile approach to statistical analysis that incorporates prior knowledge and updates beliefs based on new evidence.

• Its applications span various fields, including machine learning, data science, artificial intelligence, and economics.

• *Despite its challenges and criticisms, Bayesian inference offers a probabilistic framework for making informed decisions and understanding uncertainty.

• As computational techniques continue to advance, the adoption and impact of Bayesian methods are likely to grow, further enhancing their relevance and utility in scientific research and practical applications.

Bias and Variance

This topic will focus on Bias and Variance in Machine Learning.

In the realm of statistics and machine learning, two critical concepts often discussed are bias and variance. These fundamental elements play a pivotal role in determining the performance and accuracy of predictive models. Understanding the balance between bias and variance is essential for developing robust and efficient machine learning algorithms.

What is Bias?

Bias refers to the error introduced by approximating a real-world problem, which may be complex, by a simpler model. In other words, bias is the difference between the average prediction of our model and the correct value we are trying to predict. High bias can cause the model to miss relevant relations between features and target outputs, leading to systematic errors.

• Machine learning algorithms use mathematical or statistical models with inherent errors in two categories: reducible and irreducible error. Irreducible error, or inherent uncertainty, is due to natural variability within a system. In comparison, reducible error is more controllable and should be minimized to ensure higher accuracy.

• Bias and variance are components of reducible error. Reducing errors requires selecting models that have appropriate complexity and flexibility, as well as suitable training data. Data scientists must thoroughly understand the difference between bias and variance to reduce error and build accurate models.

Bias is like if you kept throwing the darts and they always landed far from the bullseye, maybe all of them landing to the left. This means your aim is off in a certain direction. In other words, you have a consistent mistake in your throws.

High Bias

When a model has high bias, it implies that the model is too simple. It makes strong assumptions about the data and fails to capture the underlying patterns. This often results in underfitting, where the model performs poorly on both the training data and new, unseen data.

Common causes of high bias include:

• Using a linear model for non-linear data

• Insufficient features

• Overly simplistic algorithms

Low Bias

A model with low bias generally has a more complex structure and is better at capturing the underlying patterns in the data. However, while low bias is desirable, it needs to be balanced with variance to avoid overfitting.

What is Variance?

Variance refers to the model's sensitivity to fluctuations in the training data. It measures how much the predictions of the model change when different data sets are used. High variance indicates that the model learns the noise in the training data, which can lead to overfitting.

Variance is like if you kept throwing the darts and they landed all over the place – some close to the bullseye, some far, some to the left, some to the right. You are not consistent in your throws. Sometimes you hit close, sometimes far, but you don't have a predictable pattern.

High Variance

A model with high variance pays too much attention to the training data, including noise and outliers. This results in excellent performance on the training data but poor generalisation to new, unseen data. High variance models are highly flexible but lack robustness.

Common causes of high variance include:

• Using overly complex models

• Excessive feature engineering

• Insufficient training data

Low Variance

Low variance implies that the model is stable and consistent across different data sets. However, it must be balanced with bias to ensure the model does not become too simplistic, leading to underfitting.

Bias-Variance Trade-off

Bias- Variance trade-off is about balancing and about finding a sweet spot between error due to bias and errors due to variance.

The bias-variance trade-off is a critical concept in machine learning model development. It represents the balance between the error introduced by bias and variance. The ultimate goal is to find the sweet spot where both bias and variance are minimised, leading to a model that generalises well to new data.

• If you have high bias, your darts are consistently missing the bullseye in a specific direction.

• If you have high variance, your darts are all over the place without a consistent pattern.

• The best is to have low bias and low variance, where your darts are not only close to the bullseye but also consistently hitting near the center.

• That's how bias and variance work – they help us understand the kinds of mistakes we might be making, whether we are off in a particular direction or just not consistent.

The bias/variance graph shows a plot of Error against Model Complexity. It shows:

• Relationship of variance and Model Complexity: As we increase the variance, the variance increases

• Relationship of bias and Model Complexity: As the bias increase, the model complexity reduces

• Relationship of variance and Error: As the variance increases, the error increases.

• Relationship of bias and Error: As the bias increases, the error increases.

Strategies to mitigate Bias and Variance

Reducing Bias

• Add more features: Including relevant features can help the model capture more complex patterns.

• Increase model complexity: Using more sophisticated algorithms can reduce bias.

• Feature engineering: Creating new features that capture the underlying patterns in the data can improve model performance.

Reducing Variance

• Increase training data: Providing more data helps the model learn more general patterns rather than noise.

• Use regularisation techniques: Applying regularisation can prevent the model from fitting to noise.

• Ensemble methods: Techniques like bagging and boosting combine multiple models to reduce variance.

Resources

• Bias-Variance Trade-off in Machine Learning: Concept & Tutorial - https://www.bmc.com/blogs/bias-variance-machine-learning/

• Guide for Beginners - https://www.analyticsvidhya.com/blog/2020/08/bias-and-variance-tradeoff-machine-learning/

Understanding and managing the bias-variance trade-off is crucial for developing efficient and accurate machine learning models. By striving for the right balance, we can create models that perform well on both training and unseen data, ultimately leading to more reliable and robust predictions.

As machine learning continues to evolve, the principles of bias and variance remain foundational, guiding practitioners in their quest to build better models and harness the full potential of data-driven insights.

What are you learning? May be we can learn together!

Happy learning!