Features Cover story

The algorithms
that run your life
They shape our viewing
habits, spot disease and
keep our emails safe.
Meet the algorithms
that you rely on more
than you might think

I
T IS almost impossible to go
a day without interacting with
an algorithm. They help direct
the whole of our online experience,
recommending what we should buy,
read, watch and listen to. Some 74 per
cent of adults in the US use Facebook
at least once a day – and what they see
is decided entirely by an algorithm.
Offline, they are increasingly used to
help us make tricky decisions, screening
job applications, moderating exam
results and even directing which crimes
police investigators focus on.
As they have become ubiquitous,
algorithms have generated a mixture
of hype and concern. On the one hand,
we are regularly told that they can be
opaque and biased. On the other, we
hear that they can be incredibly handy,
pulling off tasks that humans can
struggle with, from optimising complex
trade logistics to spotting the earliest
signs of disease in medical scans.
So what’s the truth about algorithms?
It helps to understand that the word can
mean quite different things (see “What
is an algorithm?”, page 36). It also helps
to get to know some of the algorithms
that shape our lives – so that’s what
we’ll do over the next few pages.

34 | New Scientist | 19 June 2021

 1
SOCIAL MEDIA
Facebook’s news feed
Few algorithms wield as much power politicians, campaigners and academics, one
as those under the bonnet of Facebook. of whom is technology law specialist Nicolas
The social media giant’s algorithms control Suzor at Queensland University of Technology
which updates its 2.8 billion monthly users in Brisbane, Australia. “There are big problems
see from which friends and what headlines to address, such as whether the way that news
they read on their news feed. is amplified on social media aligns with our
When we speak of the “Facebook algorithm”, social aims,” says Suzor. “These problems
we’re actually referring to dozens of pieces cannot be addressed until Facebook becomes
of software that are based on a range of more transparent about how the algorithms
technologies and are constantly being tweaked. work.” Matthew Sparkes
This software analyses what the firm calls the
“inventory”: the collection of posts from those
people, pages or groups that a user follows. 2
It then uses neural networks, a form of artificial WEATHER FORECASTING
intelligence, to score those posts on various
factors. As far back as 2014, Facebook employees The Unified Model
reported that the news feed was taking 100,000 The British obsession with the weather may
factors into account. Eventually, it combines baffle the rest of the world, but one good thing
these scores into a single ranking for each post. to come of it is a first-class weather forecasting
This is used to curate what a user sees. service. At the heart of the Met Office’s
Facebook rarely talks publicly about forecasts is an algorithm called the Unified
exactly how its algorithms work. In truth, it Model. It works by sucking in data on the state
doesn’t know itself, at least not at the level of of the atmosphere from weather stations and
individual users. Nick Clegg, Facebook’s vice satellites, then extrapolating to calculate how
president of global affairs, recently wrote it will change. The results determine whether
that the results the algorithms produce are we see an icon for sunshine or snow.
as unique to each user as their fingerprints. The algorithm is built around the
This complexity could spell a reckoning Navier-Stokes equations, our best description
for Facebook in the coming years. There has of how liquids and gases flow. These are how
long been tension between the company’s we predict the flow of dry and moist air in the
business model and what is perceived as atmosphere, a major factor in determining the
good for society at large. Facebook wants to weather. It isn’t possible to solve the equations
keep users on its site for longer so it can sell exactly, but you can do so approximately if
more ads. This has caused concern that its you take a series of incremental changes in
algorithms show people more posts about the state of the atmosphere and add them
controversial topics, such as Brexit, because together. This is what the Unified Model does.
they tend to prompt higher engagement. For decades, computer processors have been
The company has long used human getting quicker, allowing algorithms like the
moderators to help decide whether Unified Model to process measurements made
BRATISLAV MILENKOVIC

controversial content should be removed at ever shorter increments and so produce

or not. Recently, it set up the independent better, faster predictions. It can seem like
Oversight Board to review especially difficult weather forecasts get it wrong all too often,
cases. The board is composed of 20 journalists, but the Met Office’s next-day temperature >

19 June 2021 | New Scientist | 35

What is an
algorithm?
Ask a computer scientist this question School exams
and they will tell you it is a sequence in England were
of instructions that takes an input, moderated by
performs some repeatable computation algorithm in 2020,
on it and provides an output. Think of an causing controversy
algorithm like a super-precise recipe,
usually written in the cold logic of a JPEG is named after the Joint Photographic
programming language. Experts Group, which devised this standard
A simple example is the bubble back in 1992. The algorithm itself is quite
sort, which arranges a list of numbers sophisticated, but it is based on simple insights
in ascending order. It begins by comparing about human vision. “A lot of research has
the first two numbers. If the first is greater been put into what humans can perceive
than the second, it swaps them. Otherwise, in images and sound, and it is a lot less than
it moves on to the next pair. It cycles the files are capable of transmitting,” says
through the list again and again until it Jeff Waldock at Sheffield Hallam University
passes through without any swaps needed, in the UK. “So, for efficiency’s sake,
at which point it outputs an ordered list. If compression algorithms focus on what’s
you are shopping online and filter products important for an eye or ear to perceive.”
by price, the bubble sort algorithm is Cone cells in the human eye, which register
kicking into gear behind the scenes. colour, are 500 to 1000 times less sensitive to
These days, popular use of the word light than the eye’s rod cells, which are most
algorithm is morphing: it is increasingly sensitive to brightness. Among other things,
used to describe almost anything that JPEG exploits this by dropping some of the
a computer accomplishes. That includes information about colour, while retaining
the realms of artificial intelligence (AI) the brightness more precisely.
and machine learning, where the steps There are situations where you wouldn’t
AMER GHAZZAL/ALAMY

in the recipe aren’t always quite so want to use a loss-inducing format like JPEG,
clearly laid out. such as in medical imaging or digital art. With
Take neural networks, a type of AI compression algorithms, it is always a trade-off
system that mimics the human brain in that between quality and speed of transmission.
it can be trained to perform a task based on Data speeds have increased drastically since
looking at examples of correct and incorrect 1992, but demand has skyrocketed too, so
results. Such “algorithms” can be incredibly forecasts are accurate to within 2°C on 92 per compression is still important. Other kinds
powerful, but it is usually hard to look cent of occasions. So, while it can’t forecast of compression algorithms can squeeze down
inside and determine how they really work. every shower with total precision, it is a other types of file – MP3 is often used for music
There are those who find the loosening pretty reliable algorithm. files, for instance.
of the term algorithm to include AI unhelpful. To get even better, the Unified Model is There are plenty of would-be successors
“Now people use ‘algorithm’ to mean going to have to change. “We are no longer to JPEG designed for even more efficient
almost anything,” says Martin Dyer at getting faster processors, we’re just getting data transmission. One of these is JPEG XL,
the University of Leeds, UK. “I’ve become more processors,” says Ben Shipway at the which Jyrki Alakuijala at Google is working
so annoyed at people misusing it.” Met Office. He and his colleagues are now on. “Images are the heart of the internet,” he
Dyer warns that, in future, we may redesigning the model so it can be run on says. “We are trying to introduce a new way
increasingly lean on machine learning parallel processors. Anna Demming to model colours.” The idea is that this will
as an “easy way out” – a route to solve produce images that are even better tuned
problems without fully understanding to human vision. Chelsea Whyte
them ourselves.
3
He says we ought to apply the right
DIGITAL SHARING
kind of algorithm in the right context.
JPEG
There are times when a rigid set of
predictable steps is desirable and times
when highly capable but ambiguous AI
If you recognise the acronym JPEG, you might
think of it as merely a type of file. In reality, it
92%
can be beneficial. “It’s fine if it gets wrong is an algorithm used to compress the amount of Met Office next-day
temperature forecasts
whether you like this book or not, but it’s of data in an image. The blistering speed with
are accurate to within 2°C
not fine if it crashes your car,” says Dyer. which we can share pictures online is partly
Matthew Sparkes thanks to compression algorithms like this one. SOURCE: UK MET OFFICE
“We will surely be grappling with the
legacy of PageRank for years to come”

4 (see “Facebook’s news feed”, page 35).

WORLD WIDE WEB It is difficult to overstate how much

PageRank 74% this algorithmic indexing has changed the

world. Google’s supremacy allows it to sell
Back in 1998, two entrepreneurs filed a patent of adults in the US targeted advertising that nets it vast profits:
for an algorithm called PageRank. They were use Facebook at $46.2 billion of revenue in the third quarter
Larry Page and Sergey Brin, the founders of a least once a day of 2020 alone. But it has also sucked support
medium-sized internet search engine called from other industries, not least the media,
SOURCE: PEW RESEARCH, 2019
Google. From there, Google quickly grew to which once enjoyed reliable streams of
be the clear leader in online search – and it revenue from adverts. We will surely be
was PageRank that made this happen. better, more accurate results. grappling with the legacy of PageRank
Before Page and Brin filed that patent, The PageRank algorithm no longer has for years to come. Alexander McNamara
search algorithms tended to rely on analysing the influence it once did. Instead, Google
the words on a web page. This determined now uses a series of different algorithms to
which websites a user would be served when analyse hundreds of different factors that 5
they entered a search query. PageRank worked decide where websites rank, drawing clues FINANCE
differently: it assigned each link a rank based from all of Google’s many products. You may
on the number and authority of any websites not realise it, but your location, the time of day Trading algorithms
linking to it. If site A links to site B, site B will and the device you are searching from – plus Financial trading has always been an arena
inherit some of A’s PageRank, boosting its many other subtle factors, not all of which ripe for automation. It isn’t just that humans
position on Google’s results page. This turned Google reveals – affect your search results. frequently lose their nerve and make mistakes.
the search market on its head, negating the As with so many other parts of the online There is also decades of price information
tricks and workarounds that website owners experience, algorithms personalise what ready to be analysed and used to develop
were using to game the rankings. It provided you see in a way that’s hard to fully understand rules on when to buy, when to hold and when
to sell. Today, algorithms control a huge
portion of the global economy.
Stock market “Good traders follow good rules, but as
traders can make a human, you let emotions come in and
mistakes, so they you maybe panic or get too greedy,” says
built algorithms Nir Vulkan, a business economist at the
to make rational University of Oxford. “Algorithms are the
buy-and-sell ultimate way to be systematic.”
decisions for them One breed of finance algorithm carries out
what is called high-frequency trading, where
computers make vast numbers of trades at
incredible speed. Each transaction is likely to
generate only a tiny profit, but this adds up.
These algorithms have been accused of causing
financial crashes in the past. But they are only
a small part of the market.
More common are algorithms that lay
out the circumstances under which particular
financial assets, such as shares and bonds, are
bought and sold. It isn’t so much about being
quick, but about minimising human bias.
SIMON BELCHER/ALAMY

Such digital rule books are mainly developed

and used by funds handling huge pots of
money and their inner workings aren’t public.
An increasing number of global financial >

19 June 2021 | New Scientist | 37

 “Many of the medicines we rely on wouldn’t
exist without Monte Carlo algorithms”

Some algorithms does this with a quirk of prime numbers.

can read medical Multiplying primes together is simple, but
images more factorising the number produced to get back
accurately than the original primes takes a lot more time.
human doctors In real life, the RSA algorithm deals in
truly gigantic primes and the system is
practically impossible to break – for now, at
least. Factorising numbers into primes is only
hard for classical computers. For a big enough
quantum computer, it will be a breeze. But fear
not – work is under way on cryptography that
relies on other, more quantum-resistant
mathematics. Timothy Revell
MARTIN BARRAUD/GETTY IMAGES

7
HEALTH
Triage algorithms
Imagine you start feeling a crushing chest
pain. You quickly phone the emergency
decisions are made according to these only the intended recipient can unscramble services for help. In many countries, it will be
algorithms. “It’s growing all the time,” it. A simple encryption method would be to an algorithm on the end of the line – well, sort
says Vulkan. “Hedge funds have secrets, move each letter in a message one along in the of. The call handler will take you through a
so nobody can tell for sure, but I would alphabet. You send “Ij!” and then the recipient series of questions governed by an algorithm
guess it’s about half of the market now.” knows to decrypt it by reversing the process to to work out if you are having a heart attack or
We do know that trading algorithms are get the original “Hi!”. something else. The outcome determines if an
becoming more complex. Some funds are Trouble is, the first time you send a message, ambulance needs to be sent and how quickly.
going beyond simple rules and using artificial you must overtly tell the recipient how to “If you put everything as a priority, then
intelligence and machine learning. Systems decrypt it, which weakens the system. But here nothing is a priority,” says Richard Webber
based on these can use not only stock market is a cunning trick: if you use an encryption key at the UK’s College of Paramedics.
data as an input variable, but also things like that comes in two parts, one public and one Elsewhere in medicine, algorithms are
the number of positive and negative words in private, you can encrypt your message with beginning to be used without much, if any,
a CEO’s media interview. In time, this might the recipient’s public key but they alone can human intervention. In the online version of
give firms with the smartest algorithms an decrypt it with their private key. NHS 111, the UK’s non-emergency healthcare
advantage. Leah Crane The way to pull this off is to have a process triage service, users are taken through a series
that is easy to do in one direction (encrypting of completely automated questions to help
6 the message), but hard to do in the other direct them. There are also several “symptom
ENCRYPTION (decrypting the message). The RSA algorithm checker” apps available. These ask about your
medical problems and then, based on an
The RSA algorithm algorithm, suggest a probable diagnosis.
Ever used the internet? Then you have However, the UK’s Royal College of General
used RSA cryptography, a combination Practitioners has said that such apps can’t yet
of algorithms and protocols that makes
it possible to send information privately
2.1 million replace doctors’ abilities to make decisions
based on their breadth of experience, their
between computers. Those secrets could be images are training and, sometimes, their gut instincts.
anything from an email to your bank details. shared online More sophisticated machine-learning
every minute
Methods for sending secrets rely on algorithms are being used too. There are
scrambling the message in such a way that SOURCE: STATISTICA, 2019 several systems that can read medical images,

38 | New Scientist | 19 June 2021

spotting signs of a broken bone in an X-ray, 9
for instance, or the onset of diabetes-related SCIENCE
blindness in an eye scan. Algorithms like
this are developed by training the software 40,000 Monte Carlo algorithms
on thousands of images that have been searches are
The scientists developing the atomic
categorised by doctors. In some cases, processed by bomb during the second world war needed
these systems have outperformed trained Google every second to understand nuclear chain reactions – and
professionals and some hospitals are already there was no room for error. The problem was,
SOURCE: INTERNET LIVE STATS
using them, usually double-checked by a the physics was far too complex to analyse
doctor, to help make diagnoses far more with conventional methods. So mathematician
quickly. Clare Wilson only 4.3 billion IP addresses were available. Stanisław Ulam devised a set of algorithms
Thankfully, the internet got an upgrade. that could solve the problem in a new way,
The new address system, IPv6, has more harnessing the nascent computers of the time.
8 possible addresses than there are atoms on The idea was to fake the physics many
INTERNET the surface of Earth. So, even with trillions times over and see what tended to happen on
of future devices to be connected to the average, producing not a direct answer but the
The internet protocol suite internet, we won’t run out of addresses most probable outcomes. Ulam named the
When you fire off an email or type into any time soon. Donna Lu algorithm after the card game Monte Carlo –
a browser, you are asking your computer sometimes called solitaire – because he first
to exchange messages with another one tested it by modelling outcomes of the game.
somewhere in the vast digital expanse of These days, Monte Carlo algorithms are
the internet. This miraculous connection is employed incredibly widely, from forecasting
governed by a sprawling set of algorithms stock market trends to the output of wind
and rules known as the internet protocol suite. farms. “They are important because they can
The purpose of these rules is to make sure be used to understand complex situations
traffic gets where it needs to go. To that end, the where there is no analytical formula that
algorithms don’t specify exact routes through would allow one to directly compute the
the cables that make up the internet in case result,” says computer scientist John Guttag
parts of it get disrupted. Instead, the message is at the Massachusetts Institute of Technology.
chopped into packets and the algorithms help They are especially useful for scientists
them find a way by any viable route, with each wrestling with complex phenomena. For
packet basing its trajectory on feedback from instance, they are routinely used to figure
the one in front. Once at the destination, the out what shapes of molecule can best glom
packets are put back together. on to important proteins, providing a lead
This algorithmic rule book was written on compounds that may be useful drugs.
by computer scientists Vint Cerf and Robert Many of the medicines we rely on wouldn’t
Kahn and implemented in 1974. It has formed exist without Monte Carlo algorithms.
the backbone of the internet ever since – but We are still finding new uses for
it has needed a few tweaks along the way. them. One devilish problem in physics is
PETER SCHOLEY/ALAMY

One reason for this is that it works hand-in- understanding the behaviour of materials
glove with the internal protocol (IP) address in which lots of electrons whizz around
system, which gives each connected device, and mutually affect each other. The maths
from servers to smartphones, a unique, is nigh-on impossible, which means we can’t
machine-readable identifier. As the world predict how materials we haven’t yet made
became ever more connected, the number will behave. Roger Melko at the Perimeter
of available IP addresses dwindled, and in One of the most Institute in Waterloo, Canada, has found a
2017 we all but ran out. This was because important algorithms shortcut. He uses Monte Carlo algorithms
IP version 4, the first publicly used version in science was first to do the work instead, applying them to hunt
of the internet, was structured such that tested on a card game exotic new states of matter. Daniel Cossins ❚

19 June 2021 | New Scientist | 39