Special Issue

Introduction to Machine Learning, Neural Networks,

and Deep Learning
Rene Y. Choi1,* , Aaron S. Coyner2,* , Jayashree Kalpathy-Cramer3 , Michael F. Chiang1,2 ,
and J. Peter Campbell1
1
Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University (OHSU), Portland, Oregon, United States
2
Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon, United States
3
Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown,
Massachusetts, United States

Correspondence: J. Peter Campbell, Purpose: To present an overview of current machine learning methods and their use in
Casey Eye Institute, Oregon Health & medical research, focusing on select machine learning techniques, best practices, and
Science University, 3375 SW deep learning.
Terwilliger Blvd, Portland OR 97239.
e-mail: [email protected]
Methods: A systematic literature search in PubMed was performed for articles perti-
nent to the topic of artificial intelligence methods used in medicine with an emphasis
Received: October 31, 2019 on ophthalmology.
Accepted: November 1, 2019
Published: February 27, 2020 Results: A review of machine learning and deep learning methodology for the audience
without an extensive technical computer programming background.
Keywords: deep learning; machine
learning; artificial intelligence Conclusions: Artificial intelligence has a promising future in medicine; however, many
Citation: Choi RY, Coyner AS,
challenges remain.
Kalpathy-Cramer J, Chiang MF, Translational Relevance: The aim of this review article is to provide the nontechnical
Campbell JP. Introduction to readers a layman’s explanation of the machine learning methods being used in medicine
machine learning, neural networks, today. The goal is to provide the reader a better understanding of the potential and
and deep learning. Trans Vis Sci Tech. challenges of artificial intelligence within the field of medicine.
2020;9(2):14,
https://doi.org/10.1167/tvst.9.2.14

simulate it.”7 They described this principle as “artifi-

Introduction cial intelligence.”7 Simply put, AI is a field focused
on automating intellectual tasks normally performed
Over the past decade, artificial intelligence (AI) has by humans, and ML and DL are specific methods of
become a popular subject both within and outside of achieving this goal. That is, they are within the realm
the scientific community; an abundance of articles in of AI (Fig. 1). However, AI includes approaches that
technology and non-technology-based journals have do not involve any form of “learning.” For instance,
covered the topics of machine learning (ML), deep the subfield known as symbolic AI focuses on hardcod-
learning (DL), and AI.1–6 Yet there still remains confu- ing (i.e., explicitly writing) rules for every possible
sion around AI, ML, and DL. The terms are highly scenario in a particular domain of interest. These rules,
associated, but are not interchangeable. In this review, written by humans, come from a priori knowledge
we (attempt to) forgo technical jargon to better explain of the particular subject and task to be completed.
these concepts to a clinical audience. For example, if one were to program an algorithm to
In 1956, a group of computer scientists proposed modulate room temperature of an office, he or she
that computers could be programmed to think and likely already know what temperatures are comfortable
reason, “that every aspect of learning or any other for humans to work in and would program the room
feature of intelligence [could], in principle, be so to cool if temperatures rise above a specific thresh-
precisely described that a machine [could] be made to old and heat if they drop below a lower threshold.

Copyright 2020 The Authors

tvst.arvojournals.org | ISSN: 2164-2591 1

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Downloaded from tvst.arvojournals.org on 06/22/2023

 Introduction to ML, Neural Networks, and DL TVST | Special Issue | Vol. 9 | No. 2 | Article 14 | 2

data. In contrast to classical programming (Fig. 2A),

in which an algorithm can be explicitly coded using
known features, ML uses subsets of data to generate an
algorithm that may use novel or different combinations
of features and weights than can be derived from first
principles (Fig. 2B).8,9 In ML, there are four commonly
used learning methods, each useful for solving differ-
ent tasks: supervised, unsupervised, semisupervised,
and reinforcement learning.8–10 To better understand
these methods, they will be defined via an example of
a hypothetical real estate company that specializes in
predicting housing prices and features associated with
those houses.

Supervised Learning
Figure 1. Umbrella of select data science techniques. Artificial Suppose the real estate company would like to
intelligence (AI) falls within the realm of data science, and includes predict the price of a house based on specific features
classical programming and machine learning (ML). ML contains of the house. To begin, the company would first gather
many models and methods, including deep learning (DL) and artifi- a dataset that contains many instances.8,9,11 Each
cial neural networks (ANN).
instance represents a singular observation of a house
Although symbolic AI is proficient at solving clearly and associated features. Features are the recorded
defined logical problems, it often fails for tasks properties of a house that might be useful for predicting
that require higher-level pattern recognition, such as prices (e.g., total square-footage, number of floors, the
speech recognition or image classification. These more presence of a yard).8,9,11 The target is the feature to be
complicated tasks are where ML and DL methods predicted, in this case the housing price.8,9,11 Datasets
perform well. This review summarizes machine learn- are generally split into training, validation, and testing
ing and deep learning methodology for the audience datasets (models will always perform optimally on the
without an extensive technical computer programming data they are trained on).8,9 Supervised learning uses
background. patterns in the training dataset to map features to
the target so that an algorithm can make housing
price predictions on future datasets. This approach
Methods is supervised because the model infers an algorithm
from feature-target pairs and is informed, by the
We conducted a literature search in PubMed for target, whether it has predicted correctly.8–10 That is,
articles that were pertinent to leading artificial intel- features, x, are mapped to the target, Y, by learning
ligence methods being utilized in medical research. the mapping function, f, so that future housing prices
Selection of articles was at the sole discretion of the may be approximated using the algorithm Y = f(x).
authors. The goal of our literature search was to The performance of the algorithm is evaluated on the
provide the nontechnical readers a layman’s explana- test dataset, data that the algorithm has never seen
tion of the machine learning methods being used in before.8,9 The basic steps of supervised machine learn-
medicine today. ing are (1) acquire a dataset and split it into separate
training, validation, and test datasets; (2) use the train-
Results ing and validation datasets to inform a model of the
relationship between features and target; and (3) evalu-
ate the model via the test dataset to determine how
We found 33 articles that were pertinent to the main well it predicts housing prices for unseen instances. In
AI methods being used in medicine today. each iteration, the performance of the algorithm on
the training data is compared with the performance
Discussion on the validation dataset. In this way, the algorithm is
tuned by the validation set. Insofar as the validation
Introduction to Machine Learning set may differ from the test set, the performance of
the algorithm may or may not generalize. This concept
ML is a field that focuses on the learning aspect of will be discussed further in the section on performance
AI by developing algorithms that best represent a set of evaluation.

Downloaded from tvst.arvojournals.org on 06/22/2023

 Introduction to ML, Neural Networks, and DL TVST | Special Issue | Vol. 9 | No. 2 | Article 14 | 3

Figure 2. Classical programming versus machine learning paradigm. (A) In classical programming, a computer is supplied with a dataset
and an algorithm. The algorithm informs the computer how to operate upon the dataset to create outputs. (B) In machine learning, a
computer is supplied with a dataset and associated outputs. The computer learns and generates an algorithm that describes the relationship
between the two. This algorithm can be used for inference on future datasets.

The most common supervised learning tasks are uses a clustering algorithm on its dataset and it finds
regression and classification.8–10 Regression involves three distinct clusters. Upon further investigation, it
predicting numeric data, such as test scores, laboratory might find that the clusters represent the three separate
values, or prices of an item, much like the housing price architects responsible for designing the homes in their
example.8–10 Classification, on the other hand, entails dataset, which is a feature that was not present in the
predicting to which category an example belongs.8–10 training dataset.
Sticking with the previous example, imagine that rather
than predicting exact housing prices in a fluctuat- Semisupervised Learning
ing market, the real estate company would now like Semisupervised learning can be thought of as the
to predict a range of prices for which a house will “happy medium” between supervised and unsuper-
likely sell, such as (0, 125K), (125K, 250K), (250K, vised learning and is particularly useful for datasets
375K), and (375K, ∞). To accomplish this, data that contain both labeled and unlabeled data (i.e., all
scientists would transform the numeric target variable features are present, but not all features have associated
into a categorical variable by binning housing prices targets).10 This situation typically arises when label-
into separate classes. These classes would be ordinal, ing images become time-intensive or cost-prohibitive.
meaning that there is a natural order associated with Semisupervised learning is often used for medical
the categories.9 However, if their task was to determine images, where a physician might label a small subset of
whether houses had wood, plastic, or metal siding, images and use them to train a model. This model is
classes would be nominal; they are independent of one then used to classify the rest of the unlabeled images in
another and have no natural order.9 the dataset. The resultant labeled dataset is then used
to train a working model that should, in theory, outper-
Unsupervised Learning form unsupervised models.10
In contrast to supervised learning, unsupervised
learning aims to detect patterns in a dataset and Reinforcement Learning
categorize individual instances in the dataset to Finally, reinforcement learning is the technique of
said categories.8–10 These algorithms are unsupervised training an algorithm for a specific task where no single
because the patterns that may or may not exist in a answer is correct, but an overall outcome is desired.9,10
dataset are not informed by a target and are left to It is arguably the closest attempt at modeling the
be determined by the algorithm. Some of the most human learning experience because it also learns from
common unsupervised learning tasks are clustering, trial and error rather than data alone.9,10 Although
association, and anomaly detection.8–10 Clustering, as reinforcement learning is a powerful technique, its
the name suggests, groups instances in a dataset into applications in medicine are currently limited and thus
separate clusters based upon specific combinations of will be presented with a new example. Imagine one
their features.8–10 Say the real estate company now would like to train an algorithm to play the video game

Downloaded from tvst.arvojournals.org on 06/22/2023

 Introduction to ML, Neural Networks, and DL TVST | Special Issue | Vol. 9 | No. 2 | Article 14 | 4

Super Mario Bros, where the purpose of the game is be expected to work in another population. Finally, if
to move the character Mario from the left side of the training performance increases far more than valida-
screen to the right side in order to reach the flag pole tion set performance (e.g., the model has an accuracy
at the end of each level while avoiding hazards such of 99% on the data it was trained on, but only 80%
as enemies and pits. There is no correct sequence of on the validation data), the model is overfit. That is,
controller inputs; there are sequences that lead to a it has learned features specific to the training dataset
win and those that do not. In reinforcement learning, population at the expense of generalizability to another
an algorithm would be allowed to “play” on its own. population. Although the validation dataset is not
It would attempt many different controller inputs and specifically used to train the algorithm, it is used to
when it finally moves Mario forward (without receiving iteratively tune the algorithm. Therefore, the validation
damage), the algorithm is “rewarded” (i.e., the behav- dataset is not necessarily a reliable indicator of model
ior is reinforced). Through this process, the algorithm performance on unseen data.8,9
begins to learn what behavior is desired (e.g., moving Upon completion of the training phase, a data
forward is better than moving backward, jumping over scientist has, ideally, trained a highly generalizable
enemies is better than running into them). Eventually, model; however, this must be confirmed via a separate
the algorithm learns how to move from start to finish. test dataset. In the case of supervised learning, which
Although reinforcement has its place in the field of will be the focus of this review from here on, the perfor-
computer science and machine learning, it has yet to mance of a learned model can be evaluated in a number
make a substantial impact in clinical medicine. of ways, but is most commonly evaluated based on
prediction accuracy (classification) or error and resid-
Performance Evaluation uals (regression).8,9 As previously mentioned, the test
dataset contains instances of the original dataset that
To maximize the chance of generalizability to the have not been seen by the algorithm during the train-
performance of the algorithm on unseen data, the ing phase. If the predictive power of a model is strong
training dataset is usually split into a slightly smaller on the training dataset, but poor on the test dataset,
training dataset and a separate validation dataset.8,9 then the model is too specific to the patterns from the
Metrics used for evaluation of a model depend upon training data and is considered to be overfit to the train-
the model itself and whether it is in the training ing dataset.8,9 That is, it has memorized patterns rather
or testing phase. The validation dataset is meant to than learned a generalizable model. An underfit model,
mimic the test dataset and helps data scientists tune on the other hand, is one that performs poorly on both
an algorithm by identifying when a model may gener- training and test datasets and has neither learned nor
alize well and work in a new population. Because the memorized the training dataset and still is not gener-
validation dataset is a small sample of the true (larger) alizable.8,9 An ideally fitted model is one that performs
population, it may not accurately represent the popula- strongly on both datasets, suggesting it is generalizable
tion itself due to an unknown sampling bias. There- (i.e., it will perform well on other similar datasets).8,9
fore, model performance and generalizability should With regression models, the average mean squared
not be assessed via validation set performance. It is error (MSE) can be an indicator of model perfor-
conceivable that a data scientist could create a valida- mance.8,9 MSE measures how close a predicted
tion dataset with an unknown bias and use it to tune value is to the intended target value. MSE is calcu-
a model. Although the model might perform well on lated by summing the differences between predicted
the validation dataset, it would likely not perform well values and target values, squaring the results,
on the much larger test dataset (i.e., it would not be a and dividing  by the total number of instances
generalizable model) (MSE = 1n ni=1 (yi − ŷi )2 ).8,9 There are many other
Typically, model performance is monitored via some measures of performance for regression models that
form of accuracy on the training and validation are out of the scope of this review.
datasets during this phase. So long as the accuracy For binary classification, the output of the model
of the model on the training set (X%) and valida- is a class. However, before the class designation, the
tion set (Y%) are increasing and converging after probability of an instance belonging to class A or class
each training iteration, the model is considered to B is determined.8,9 Normally, this probability thresh-
be learning. If both converge, but do not increase old is set at 0.5. A receiver operating characteristic
(e.g., X converges on Y at 50%), the model is not curve evaluates a model’s true positive rate (TPR; i.e.,
learning and may be underfit to the data, that is, sensitivity, recall), the number of samples correctly
it may not have learned enough of the relationship identified as positive divided by the total number of
between features and targets in a way that it would positive samples, versus its false-positive rate (FPR;

Downloaded from tvst.arvojournals.org on 06/22/2023

 Introduction to ML, Neural Networks, and DL TVST | Special Issue | Vol. 9 | No. 2 | Article 14 | 5

Figure 3. Sensitivity, specificity, positive predictive value, and negative predictive value. A population (dataset) is represented as circles
colored blue if positive or orange if negative. The dataset is input to an algorithm that predicts each instance’s class association. If an instance
is correctly predicted as positive or negative, it is a true positive (TP) or true negative (TN), respectively. If an instance  is incorrectly labeled
positive or negative, it is a false positive (FP) or false negative (FN), respectively. (A) A model with perfect sensitivity ( T P T+P FN ) and specificity

( T NT+N FP ). (B) A model with perfect sensitivity (ability to correctly classify all positive cases), but poor specificity (ability to correctly classify all
negative cases) and (C) a model with perfect specificity, but poor sensitivity. Although a model might have perfect sensitivity (B), it can have
many false positives. Similarly, a model with perfect specificity (C) might have many false negatives. Therefore, it is also useful to evaluate
TP TN
the positive predictive value (PPV; T P + FP
) and the negative predictive value (NPV; T N + FN
). PPV and NPV are also thus dependent on
the prevalence of disease in a population.

Figure 4. Example receiver operating characteristics and precision-recall curves. Red line: a model that performs no better than chance has
an area under the curve (AUC) of the receiver operating characteristics curve (AUROC) of 0.50 or area under the precision-recall curve (AUPR)
at the class ratio (red shaded area). Blue line: a model that performs better than chance, but not perfectly, will have an AUC between 0.50 and
1.00 (blue + red shaded areas). Green line: a model that performs perfectly has an AUC of 1.00 (red + blue + green shaded areas).

i.e., 1 - specificity), the number of samples incor- its recall (Fig. 3, Fig. 4B).8,9 Each curve is evaluated
rectly identified as positive divided by the total number across the range of model probability thresholds from
of negative samples (Fig. 3, Fig. 4A).8,9 Similarly, 1 to 0, left to right. A receiver operating characteristic
the precision-recall curve evaluates a model’s positive curve starts at the point (FPR = 0, TPR = 0), which
predictive value (PPV; i.e., precision), the number of corresponds to a decision threshold of 1 (every sample
samples correctly identified as positive divided by the is classified as negative, and thus there are no false or
total number of samples identified as positive, versus true positives). It ends at the point (FPR = 1, TPR = 1),

Downloaded from tvst.arvojournals.org on 06/22/2023

 Introduction to ML, Neural Networks, and DL TVST | Special Issue | Vol. 9 | No. 2 | Article 14 | 6

which corresponds to a decision threshold of 0 (where each describing to what degree each feature influences
every sample is classified as positive, and thus all points the target.8,9
are either truly or falsely labeled positive). The points In practice, there is rarely a single function that fits
in between, which create the curve, are obtained by a dataset perfectly. To measure the error associated
calculating the TPR and FPR for different decision with a fit, the residuals are measured. Conceptually,
thresholds between 1 and 0, trading off sensitivity residuals are the vertical distances between predicted
(minimizing false negatives) with specificity (minimiz- values, ŷ, and actual values, y. In machine learn-
ing false positives). The area under the curve (AUC) of ing, the cost function is a calculus derived term that
the receiver operating characteristics curve (AUROC) aims to minimize errors associated with a model.8,9
can be calculated and used as a metric for evaluat- The process of minimizing the cost function involves
ing the overall performance of a classifier, assuming an iterative optimization algorithm known as gradi-
the classes of the dataset are balanced. If classes are ent descent, of which the mathematical calculations
not balanced, the area under the precision-recall curve involved are outside the scope of this article.8,9,12 In
(AUPR) may be a better metric of model performance linear regression, the cost function is the previously
because the threshold (set at 0.5 in Fig. 4B) may be described MSE. Minimizing this function often obtains
adjusted. For example, if a dataset comprised 75% of estimates of a and b that best model a dataset. All
class A and 25% of class B, the ratio between the model-based learning algorithms have a cost function,
two would be computed as the threshold (0.75). In and the goal is to minimize this function to find the
practice, an AUROC value of 0.50 indicates a model best-fit model.8,9
that performs no better than chance, and an AUC of
1.00 indicates that the model performs perfectly; the
Logistic Regression
higher the value of the AUC, the stronger the perfor-
Logistic regression is a classification algorithm
mance of the ML model.8,9 Similarly, an AUPR value
where the goal is to find a relationship between features
at the preset threshold indicates a model that performs
and the probability of a particular outcome. Rather
no better than chance, and an AUPR value of 1.00
than using the straight line produced by linear regres-
indicates a perfect model.8,9
sion to estimate class probability, logistic regression
uses a sigmoidal curve to estimate class probability
Classic Machine Learning Methods (Fig. 5). This curve is determined by the sigmoid
function, y = 1 +1e−x , which produces an S-shaped
There are many machine learning algorithms used in curve that converts discrete or continuous numeric
medicine. Described next are some of the most popular features (x) into a single numerical value (y) between
to date. 0 and 1.8,9 The major advantage of this method is that
probabilities are bounded between 0 and 1 (i.e., proba-
bilities cannot be negative or greater than 1). It can
Linear Regression
be either binomial, where there are only two possible
Linear regression is arguably the simplest ML
outcomes, or multinomial, where there can be three or
algorithm. The main idea behind regression analy-
more possible outcomes.8,9
sis is to specify a relationship between one or more
numeric features and a single numeric target.8,9 Linear
regression is an analysis technique used to solve a Decision Trees and Random Forests
regression problem by using a straight line to describe A decision tree is a supervised learning technique,
a dataset. Univariate linear regression, a regression primarily used for classification tasks, but can also be
problem where only a single feature is used for predict- used for regression.8,9 A decision tree begins with a root
ing a target value, can be represented in a slope- node, the first decision point for splitting the dataset,
intercept form: y = ax + b.8,9 Here, a is a weight and contains a single feature that best splits the data
describing the slope, which describes how much a line into their respective classes (Fig. 6).8,9 Each split has
increases on the y-axis for each increase in x. The inter- an edge that connects either to a new decision node
cept, b, describes the point where the line intercepts the that contains another feature to further split the data
y-axis. Linear regression models a dataset using this into homogenous groups or to a terminal node that
slope-intercept form, where the machine’s task is to predicts the class. This process of separating data into
identify values of a and b such that the determined line two binary partitions is known as recursive partition-
is best able to relate the supplied values of x values to ing.8,9 A random forest is an extension of this method,
the values of y. Multivariate linear regression is similar; known as an ensemble method, that produces multi-
however, there are multiple weights in the algorithm, ple decision trees.8,9 Rather than using every feature

Downloaded from tvst.arvojournals.org on 06/22/2023

 Introduction to ML, Neural Networks, and DL TVST | Special Issue | Vol. 9 | No. 2 | Article 14 | 7

Neural Networks and Deep Learning

An artificial neural network (ANN) is a machine
learning algorithm inspired by biological neural
networks.8,9,21 Each ANN contains nodes (analogous
to cell bodies) that communicate with other nodes via
connections (analogous to axons and dendrites). Much
in the way synapses between neurons are strengthened
when their neurons have correlated outputs in a biolog-
ical neural network (the Hebbian theory postulates that
“nerves that fire together, wire together”), connections
between nodes in an ANN are weighted based upon
their ability to provide a desired outcome.8,9,21

Feedforward Neural Networks

A perceptron is a machine learning algorithm that
Figure 5. Example class probability prediction using linear and takes in a series of features and their targets as input
logistic regression. Presented are linear (blue line) and logistic (red and attempts to find a line, plane, or hyperplane
line) regression models for predicting the probability of various that separates the classes in a two-, three-, or hyper-
samples (gray circles) as belonging to a particular class using a dimensional space, respectively.9,22,23 These features
single variable, variable X, which ranges from -10 to 10. With logis-
are transformed using the sigmoid function (Fig. 7A).
tic regression, variable X is transformed into class probabilities that
are bounded between 0 and 1 using the sigmoid function. Simple Thus, this method is similar to logistic regression;
linear regression attempts to estimate class probabilities, but is not however, it only provides class associations, and not the
bounded between 0 and 1; thus, it breaks a fundamental law of probability of an instance belonging to a class.
probability that does not allow for negative probabilities or those When multiple perceptrons are connected, the
greater than 1. model is referred to as a multilayer perceptron
algorithm or an ANN. Commonly, ANNs contain a
layer of input nodes, a layer of output nodes, and
a number of “hidden layers” between the two.9 In
to create every decision tree in a random forest, a
simple ANNs, there exists an input layer between zero
subsample of features are used to create each decision
and three hidden layers and an output layer, whereas
tree. Trees then predict a class outcome, and the major-
deep neural networks contain tens or even hundreds of
ity vote among trees is used as the model’s final class
hidden layers.9,24 For most tasks, ANNs feed informa-
prediction.8,9
tion forward. This is known as a feedforward neural
network, meaning information from each node in the
Classic Machine Learning in Ophthalmology previous layer is passed to each node in the next layer,
Although DL has become a highly popular transformed, and passed forward to each node in the
technique in ophthalmology, there are a multitude next layer (Fig. 7B).9 In recurrent neural networks,
of examples of classic ML algorithms being used which are out of the scope of this paper, information
in the field. Simple linear models have been used to can be passed between nodes within a layer or to previ-
predict patients who would develop advanced age- ous layers, where their output is operated on and fed
related macular degeneration and to discern which forward once again.22
factors separate patients into who will respond to anti- Each layer in an ANN can contain any number of
vascular endothelial growth factor treatment versus nodes; however, the number of nodes in the output
those who will not.13–16 Random forest algorithms layer typically corresponds to the number of classes
have been used to discover features that are most being predicted if the goal is multiclass classifica-
predictive of progression to geographic atrophy in tion, a single node with a sigmoidal activation for
age-related macular degeneration and find prognostic binary classification, or a linear activation function if
features for visual acuity outcomes of intravitreal the goal is regression.9,24 These activation functions
anti-vascular endothelial growth factor treatment.17,18 simply transform a node’s input into a desired output
Random forest classifiers have also been applied to (Fig. 7C). Each node in an ANN contains an activa-
diagnose and grade cataracts from ultrasound images, tion function (not just the output layer; Fig. 7B). These
as well as identify patients with glaucoma based on activation functions, although not always linear, do not
retinal nerve fiber layer and visual field data.19,20 have to be complex. For instance, the rectified linear

Downloaded from tvst.arvojournals.org on 06/22/2023

 Introduction to ML, Neural Networks, and DL TVST | Special Issue | Vol. 9 | No. 2 | Article 14 | 8

Figure 6. Structure of a decision tree. Splitting of the dataset begins at the root node. Each split connects to either another decision node,
which results in further splitting of the data, or a terminal node that predicts the class of the data.

Figure 7. Components of a neural network. (A) The basis of an artificial neural network, the perceptron. This algorithm uses the sigmoid
function to scale and transform multiple inputs into a single output ranging from 0 to 1. (B) An artificial neural network connects multiple
perceptron units, so that the output of one unit is used as input to another. Additionally, these units are not limited to using the sigmoid
activation function. (C) Examples of four different activation functions: sigmoid, hyperbolic tangent, identity, and rectified linear unit. The
sigmoid scales inputs between 0 and 1 using an S-shaped curved. Similarly, the hyperbolic tangent function uses an S-shaped curve, but
scales inputs between -1 and 1. The identity function can multiply its input by any number to produce a linear output. The rectified linear
unit is similar to the identity function, however all inputs < 0 are given an output value of 0. There are other activation functions outside of
these, but these are arguably.

Downloaded from tvst.arvojournals.org on 06/22/2023

 Introduction to ML, Neural Networks, and DL TVST | Special Issue | Vol. 9 | No. 2 | Article 14 | 9

Figure 8. Example of a digital image convolved with a filter. The image (left) is transformed into the feature map (right) via a convolutional
filter (center). The convolutional filter is designed to locate diagonal lines running from top left to bottom right of the image. The filter passes
over the image in a specified manner and each element in the image (red) is multiplied by the corresponding element in the convolutional
filter (blue). The summation of these elements (orange) is output into a new matrix that reports the presence of a diagonal line. The feature
map indicates 2 when the specified diagonal line is found, 1 if a portion of it is found, and 0 if none of it is found.

unit applies a linear transformation to inputs ≥ 0, and image is color (red, green, and blue color channels).28
sets inputs < 0 to 0.25 It follows that as inputs proceed These matrices contain values, typically between 0 and
through an ANN, they are progressively modified at 255, that represent pixels in the image and the inten-
each layer so that at the final layer they no longer resem- sity of each color channel at each pixel.28 A convolu-
ble their original state. However, this final representa- tional filter is a much smaller matrix that is typically
tion of the input is, in theory, more predictive of the square and range in size from 2 × 2 to 9 × 9.28
specified outcome. This filter is passed over the original image and, at
each position, element-wise matrix multiplication is
Convolutional Neural Networks performed (Fig. 8).28 The output of this convolution is
For image recognition tasks, each input into a mapped to a new matrix (a feature map) that contains
feedforward ANN corresponds to a pixel in the image. values corresponding to whether or not the convolu-
However, this is not ideal because there are no connec- tional filter detected a feature of interest.24,26–29
tions between nodes in a layer. In practice, this means In CNNs, filters are trained to extract specific
that the spatial context of features in the image are features from images (e.g., vertical lines, U-shaped
lost.24,26,27 In other words, pixels that are close to one objects,) and mark their location on the feature
another in an image are likely more correlated than map.26,27 A deep CNN then uses the feature map as
pixels on opposite sides of the image, but a feedforward input for the next layer, which uses new filters to create
ANN does not take this into account. another new feature map.24,26,27 This can continue
A convolutional neural network (CNN) is a special for many layers and, as it continues, the extracted
case of the ANN that overcomes this issue by preserv- features become abstract, but highly useful for predic-
ing the spatial relationship between pixels in an tion. The final features maps are then compressed from
image.24,26,27 Rather than using single pixels as input, their square representations and input to a feedforward
a CNN feeds patches of an image to specific nodes in ANN, where classification of the image based on the
the next layer of nodes (rather than all nodes), thereby extracted features and textures can occur.24,26,27 This
preserving the spatial context from which a feature process is referred to as DL.24
was extracted.9,24,26,27 These patches of nodes learn Aside from image classification tasks, DL has shown
to extract specific features and are known as convolu- promise for image segmentation tasks.1,30,31 Rather
tional filters. than classifying images as a whole, this method aims
Convolutions are widely used in the realm of image to identify objects within an image. To accomplish this
processing, and are often used to blur or sharpen task, DL classifies individual pixels given surrounding
images, or for other tasks such as edge detection.28 A pixel information. For example, in diabetic retinopa-
visible-light digital image is simply a single matrix if thy, a segmentation algorithm might segment (outline)
the image is grayscale or three stacked matrices if the the retinal vasculature by assigning probabilities to

Downloaded from tvst.arvojournals.org on 06/22/2023

 Introduction to ML, Neural Networks, and DL TVST | Special Issue | Vol. 9 | No. 2 | Article 14 | 10

individual pixels as belonging to a retinal blood vessel izable. Thus, developing models for identification of
or not belonging to a retinal blood vessel. A similar rare diseases, where large datasets may not be readily
method for breast cancer detection could mark pixels available, is especially challenging. On the other hand,
as belonging to a mass or not belonging to a mass, and although one might assume that more data will always
the output image could be provided to a radiologist for lead to better models, if the quality of the training
further review. data is imprecise, mislabeled, or somehow systemati-
cally different than the test population, training on very
Deep Learning in Ophthalmology large datasets may result in models that do not perform
The popularity for DL has especially risen in the well in real-world scenarios. Furthermore, there is an
field of ophthalmology for image-based diagnostic implicit assumption that datasets are accurately labeled
systems. On the simpler end of visual interpreta- by human graders. Unfortunately, this is often not the
tion tasks, Coyner et al. devised a DL system for case, and noisy and/or missing labels are often a bane
automated assessment of retinal fundus image quality for data scientists.
with an output of “acceptable” or “not acceptable” DL methods also suffer from the “black box”
based on multiple graded expert labels.3 Presum- problem: input is supplied to the algorithm and an
ably, the network learned that the retinal vascula- output emerges, but it is not exactly clear what features
ture must be easily distinguishable for an image to be were identified or how they informed the model
deemed acceptable. In a more complex task, Gulshan output.29,32,33 In contrast, simple linear algorithms,
et al. demonstrated that DL could classify diabetic although not always as powerful as DL, are easily
retinopathy, in agreement with the Early Treatment for interpretable. The computed weights for each feature
Diabetic Retinopathy Study scale, using only retinal are supplied upon completion of the training process,
fundus images as input and the consensus diagnoses of which allow for one to interrogate exactly how the
multiple clinicians as the “class labels.”2 The presence model works and possibly discover important predic-
of features such as microaneurysms, intraretinal tors that may be useful for prevention of a disease. With
hemorrhages, or neovascularization were not supplied deep learning, a complex series of matrix multiplica-
to the DL method as signs of diabetic retinopathy. tion and abstract filters makes interpretability signif-
Rather, the DL model either learned these features icantly more challenging.29,32,33 Activation maps, or
or learned novel features that aid in the diagnosis of heatmaps, are methods that attempt to address the
diabetic retinopathy. Further, Brown et al. trained a “black box” issue by highlighting areas of images that
similar DL network for the diagnosis of plus disease highlight regions of an image that “fire together” with
in retinopathy of prematurity. First, an algorithm the output classification label.29,32,33 Unfortunately,
was trained to segment retinal vasculature into binary these methods still require human interpretation, as
vessel maps. Then another DL algorithm was trained they are often not examined critically (examples are
to examine the vessel maps and conclude whether cherry picked for publication, highly subject to confir-
the vasculature appeared normal or abnormal.1 This mation bias, etc.), and thus this remains an active area
network, too, performs on par or better than most of research. For instance, if a DL model classifies a
experts in the field. One of the most impressive fundus image as having proliferative diabetic retinopa-
examples of DL in ophthalmology was conducted by thy, a heatmap will highlight feature areas on that
De Fauw et al. Using three-dimensional optical coher- fundus image that contributed to the decision of being
ence tomography images, a DL framework was trained classified as having proliferative diabetic retinopathy.
to not only detect a single disease, but more than 50 It is up to the physician to interpret whether these
common retinal diseases.6 DL model identified features are the same features the
physician would use to diagnose the disease, and the
Challenges with DL Models implications of such findings.
In recent years, DL has become a hot topic within
the field of medicine given the digital availability of
information; however, many challenges still exist. DL
is limited by the quantity and quality of data used to Conclusion
train the model. It is difficult to estimate how much
data are necessary to sufficiently and reliably train DL AI methods have shown to be a promising tool in
systems because it depends both on the quality of the the field of medicine. Recent work has demonstrated
input training data as well as the complexity of the task. that these methods can develop effective diagnostic
Typically, thousands of training examples are required and predictive tools to identify various diseases. In
to create a model that is both accurate and general- the future, AI-based programs may become an integral

Downloaded from tvst.arvojournals.org on 06/22/2023

 Introduction to ML, Neural Networks, and DL TVST | Special Issue | Vol. 9 | No. 2 | Article 14 | 11

part of patients’ clinic visits with their ability to assist in for diagnosis and referral in retinal disease.
diagnosis and management of various diseases. Physi- Nat Med. 2018;24:1342–1350. doi:10.1038/
cians should take an active approach to understand the s41591-018-0107-6.
theories behind AI and its utility in medicine with the 7. Moor J. The Dartmouth College Artificial Intel-
goal of providing optimal patient care. ligence Conference: the next fifty years. AI Mag.
2006;27:87–87. doi:10.1609/aimag.v27i4.1911.
8. James G, Witten D, Hastie T, Tibshirani R, eds.
Acknowledgments An Introduction to Statistical Learning: With Appli-
cations in R. New York: Springer; 2013.
This project was supported by grants R01EY19474, 9. Hastie T, Tibshirani R, Friedman JH. The Ele-
K12 EY027720, and P30EY10572 from the National ments of Statistical Learning: Data Mining, Infer-
Institutes of Health; SCH-1622679, SCH-1622542, ence, and Prediction. 2nd ed. New York, NY:
and SCH-1622536 from the National Science Founda- Springer; 2009.
tion; and by unrestricted departmental funding and a 10. NVIDIA Blog: Supervised Vs. Unsupervised
Career Development Award (JPC) from Research to Learning. The Official NVIDIA Blog. https:
Prevent Blindness. //blogs.nvidia.com/blog/2018/08/02/supervised-
unsupervised-learning/. Published August 2, 2018.
Disclosure: R.Y. Choi, None; A.S. Coyner, None; Accessed October 24, 2019.
J. Kalpathy-Cramer, None; M.F. Chiang, None; J.P. 11. Molnar C. Interpretable Machine Learning: A
Campbell, None Guide for Making Black Box Models Explainable.
Morrisville, North Carolina: Lulu Press, Inc.;
* RYC and ASC contributed equally to the submit- 2019. https://christophm.github.io/interpretable-
ted work. ml-book/.
12. Mei S, Montanari A, Nguyen P-M. A mean field
view of the landscape of two-layer neural net-
References works. Proc Natl Acad Sci U S A. 2018;115:E7665–
E7671. doi:10.1073/pnas.1806579115.
1. Brown JM, Campbell JP, Beers A, et al. Auto- 13. Bogunovic H, Waldstein SM, Schlegl T, et al.
mated diagnosis of plus disease in retinopathy Prediction of Anti-VEGF Treatment Require-
of prematurity using deep convolutional neural ments in Neovascular AMD Using a Machine
networks. JAMA Ophthalmol. 2018;136:803–810. Learning Approach. Invest Ophthalmol Vis Sci.
doi:10.1001/jamaophthalmol.2018.1934. 2017;58:3240–3248. doi:10.1167/iovs.16-21053.
2. Gulshan V, Peng L, Coram M, et al. Development 14. de Sisternes L, Simon N, Tibshirani R, Leng
and validation of a deep learning algorithm for T, Rubin DL. Quantitative SD-OCT imaging
detection of diabetic retinopathy in retinal fundus biomarkers as indicators of age-related macular
photographs. JAMA. 2016;316:2402–2410. doi:10. degeneration progression. Invest Ophthalmol
1001/jama.2016.17216. Vis Sci. 2014;55:7093–7103. doi:10.1167/iovs.
3. Coyner AS, Swan R, Campbell JP, et al. Auto- 14-14918.
mated fundus image quality assessment in 15. Schmidt-Erfurth U, Waldstein SM, Klimscha S,
retinopathy of prematurity using deep convo- et al. Prediction of individual disease conversion in
lutional neural networks. Ophthalmol Retina. early AMD using artificial intelligence. Invest Oph-
2019;3:444–450. doi:10.1016/j.oret.2019.01.015. thalmol Vis Sci. 2018;59:3199–3208. doi:10.1167/
4. Rajpurkar P, Irvin J, Zhu K, et al. CheXNet: iovs.18-24106.
radiologist-level pneumonia detection on chest X- 16. Vogl W-D, Waldstein SM, Gerendas BS, Schlegl
rays with deep learning. ArXiv171105225 Cs Stat. T, Langs G, Schmidt-Erfurth U. Analyzing and
November 2017. http://arxiv.org/abs/1711.05225. predicting visual acuity outcomes of anti-VEGF
Accessed October 23, 2019. therapy by a longitudinal mixed effects model
5. Jones LD, Golan D, Hanna SA, Ramachandran of imaging and clinical data. Invest Ophthal-
M. Artificial intelligence, machine learning and the mol Vis Sci. 2017;58:4173–4181. doi:10.1167/iovs.
evolution of healthcare: a bright future or cause 17-21878.
for concern? Bone Jt Res. 2018;7:223–225. doi:10. 17. Bogunovic H, Montuoro A, Baratsits M, et al.
1302/2046-3758.73.BJR-2017-0147.R1. Machine learning of the progression of inter-
6. De Fauw J, Ledsam JR, Romera-Paredes mediate age-related macular degeneration based
B, et al. Clinically applicable deep learning on OCT imaging. Invest Ophthalmol Vis Sci.

Downloaded from tvst.arvojournals.org on 06/22/2023

 Introduction to ML, Neural Networks, and DL TVST | Special Issue | Vol. 9 | No. 2 | Article 14 | 12

2017;58:BIO141–BIO150. doi:10.1167/iovs.17- volutional Neural Networks. In: Pereira F,

21789. Burges CJC, Bottou L, Weinberger KQ, eds.
18. Niu S, de Sisternes L, Chen Q, Rubin DL, Advances in Neural Information Processing
Leng T. Fully automated prediction of geo- Systems 25. Red Hook, NY: Curran Asso-
graphic atrophy growth using quantitative spectral- ciates, Inc.; 2012:1097–1105. http://papers.nips.
domain optical coherence tomography biomarkers. cc/paper/4824-imagenet-classification-with-
Ophthalmology. 2016;123:1737–1750. doi:10.1016/ deep-convolutional-neural-networks.pdf.
j.ophtha.2016.04.042. 27. Hinton GE, Salakhutdinov RR. Reducing the
19. Caixinha M, Amaro J, Santos M, Perdigao F, dimensionality of data with neural networks.
Gomes M, Santos J. In-vivo automatic nuclear Science. 2006;313:504–507. doi:10.1126/science.
cataract detection and classification in an animal 1127647.
model by ultrasounds. IEEE Trans Biomed Eng. 28. Kaehler A, Bradski GR. Learning OpenCV 3:
2016;63:2326–2335. doi:10.1109/TBME.2016.252 Computer Vision in C++ with the OpenCV Library.
7787. Sebastopol, CA: O’Reilly Media; 2017.
20. Kim SJ, Cho KJ, Oh S. Development of machine 29. Zhou B, Khosla A, Lapedriza A, Oliva A, Tor-
learning models for diagnosis of glaucoma. PloS ralba A. Learning deep features for discriminative
One. 2017;12:e0177726. doi:10.1371/journal.pone. localization. ArXiv151204150 Cs. December 2015.
0177726. http://arxiv.org/abs/1512.04150. Accessed October
21. Hebb DO. Animal and physiological psychology. 23, 2019.
Annu Rev Psychol. 1950;1:173–188. doi:10.1146/ 30. Ronneberger O, Fischer P, Brox T. U-Net: con-
annurev.ps.01.020150.001133. volutional networks for biomedical image seg-
22. Rumelhart DE, Hinton GE, Williams RJ. Learn- mentation. ArXiv150504597 Cs. May 2015. http:
ing representations by back-propagating errors. //arxiv.org/abs/1505.04597. Accessed October 28,
Nature. 1986;323:533–536. doi:10.1038/323533a0. 2019.
23. Orbach J. Principles of neurodynamics. Per- 31. Ghosh S, Das N, Das I, Maulik U. Under-
ceptrons and the theory of brain mechanisms. standing deep learning techniques for image seg-
Arch Gen Psychiatry. 1962;7:218–219. doi:10.1001/ mentation. ArXiv190706119 Cs. July 2019. http:
archpsyc.1962.01720030064010. //arxiv.org/abs/1907.06119. Accessed October 28,
24. LeCun Y, Bengio Y, Hinton G. Deep learn- 2019.
ing. Nature. 2015;521:436–444. doi:10.1038/nature 32. Zhang Q, Zhu S-C. Visual interpretability for deep
14539. learning: a survey. ArXiv180200614 Cs. February
25. Hahnloser RHR, Sarpeshkar R, Mahowald MA, 2018. http://arxiv.org/abs/1802.00614. Accessed
Douglas RJ, Seung HS. Digital selection and ana- October 23, 2019.
logue amplification coexist in a cortex-inspired 33. Zeiler MD, Fergus R. Visualizing and understand-
silicon circuit. Nature. 2000;405:947–951. doi:10. ing convolutional networks. ArXiv13112901 Cs.
1038/35016072. November 2013. http://arxiv.org/abs/1311.2901.
26. Krizhevsky A, Sutskever I, Hinton GE. Accessed October 23, 2019.
ImageNet Classification with Deep Con-

Downloaded from tvst.arvojournals.org on 06/22/2023