CS4231

Parallel and Distributed Algorithms

Lecture 8
Instructor: Haifeng YU
 Review of Last Lecture
 Chapter 13 “Leader Election”

 Leader election on a ring

 Chang-Roberts Algorithm
 Interesting avg performance analysis

 Leader election on a general graph

 Spanning tree construction

CS4231 Parallel and Distributed Algorithms 2

 Today’s Roadmap
 Chapter 15 “Agreement”
 Also called consensus

 Motivation

 5 Different Versions of Distributed Consensus

CS4231 Parallel and Distributed Algorithms 3

 Motivation
 Distributed consensus
 A set of nodes want to agree on something
 Each node has an input
 E.g., whether the flight tickets for segments of your journey
should be purchased

 Version 0: No failures
 Everyone send its input to everyone else
 Do some deterministic computation on the n values, Decide,
and Done

CS4231 Parallel and Distributed Algorithms 4

 Motivation
 But things fail
 Nodes can crash
 Internet can be down
 Obviously we still want to book our flight tickets despite all
those
 Goal: Distributed consensus despite failures
 Think carefully what will go wrong in the previous protocol
when there are failures…

 There are many different version of distributed

consensus depending on the precise goal, failure
model, and timing model
CS4231 Parallel and Distributed Algorithms 5
 5 Versions of Distributed Consensus

Failure Model and Timing Model

Ver0: No node or link failures Trivial
Ver1: Node crash failures; Channels are This lecture
reliable; Synchronous;
Ver2: No node failures; Channels may drop This lecture
messages (the coordinated attack problem)
Ver3: Node crash failures; Channels are Next lecture
reliable; Asynchronous;
Ver4: Node Byzantine failures; Channels Next lecture
are reliable; Synchronous; (the Byzantine
Generals problem)

CS4231 Parallel and Distributed Algorithms 6

 Distributed Consensus Version 1: Consensus
with Node Crash Failures/Synchronous
 Failure model:
 Nodes may experience crash failure
 Communication channels are reliable
 Timing model
 Synchronous: Message delay has a known upper bound x, and node
processing delay has a known upper bound y
 Synchronous timing model implies the possibility of accurate failure
detection
 Goal:
 Termination: All nodes (that have not failed) eventually decide
 Agreement: All nodes that decide should decide on the same value
 Validity: If all nodes have the same initial input, that value should be
the only possible decision value. Otherwise nodes are allowed to
decide on anything (except that they still need to satisfy the
Agreement requirement)

CS4231 Parallel and Distributed Algorithms 7

 Synchronous System and Rounds
 Under a synchronous timing model, we can always make
all processes proceed in inter-locked rounds, where in
each round:
 Every process sends one message to every other process
 Every process receives one message from every other process
 Every process does some local computation
 Why is this general enough?

 Why is round useful? (In fact, VERY useful)

 Conceptually very clear – All processes make progress round by
round, no one is left behind
 Messages sent during a round is received during that round
 Accurate failure detection

CS4231 Parallel and Distributed Algorithms 8

 Implementing Rounds via Accurate Clocks

 Each process starts a timer at the beginning of a round.

 Consider the first round

 Each process first performs a send (taking time no more that t1).
 Message propagation and receiving the message no more than t2.
 Local processing no more than t3.

 Hence set the length of each round to be t1+t2+t3. A new round

starts once t1+t2+t3 time has passed.

CS4231 Parallel and Distributed Algorithms 9

Implementing Rounds via Clocks with Bounded Error
 Bounded error: Consider an example where the clock on a certain
process is twice as fast as an accurate clock.
 To compensate, everyone sets the timer duration to be 2(t1+t2+t3)
 But different process will still start the next round at different time.
Need to do two things to fix:
 If a process receives a round 2 message while in round 1, it
immediately start round 2
 Everyone sets the timer duration to be 2((t1+t2+t3) +t1+t2), since the
round starting time at different processes may differ by t1+t2.
time 0 round 1 round 2
By this design, a process
must have finished
processing round 1 (i.e., P1
the green parts) when
receiving a message for P2
round 2.

P3

CS4231 Parallel and Distributed Algorithms 10

 Synchronous System and Rounds

 Summary: With the previous protocol

 We can now use the notion of round
 Equivalent to all processes make progress in lock steps

 Only a synchronous system enables us to define a

round as above
 The notion of rounds is very convenient and powerful!
 But remember that we do need the previous protocol in
practice

CS4231 Parallel and Distributed Algorithms 11

Distributed Consensus Version 1: Protocol Intuition

 Intuition: How do we do consensus if there are no

failures
 Each process sends its input to all others
 Pick the min (or max)

 What’s the problem if there are failures?

 A process may send its input to only a subset of the
processes

 Solution: Keep forwarding the values

CS4231 Parallel and Distributed Algorithms 12

Distributed Consensus Version 1: Protocol

round 1

round 2

round 3

CS4231 Parallel and Distributed Algorithms 13

 Distributed Consensus Version 1: Protocol
input = 2 input = 1 input = 3

round 1
{1, 2, 3} {2, 3}

round 2
{1, 2, 3} {1, 2, 3}

Need one extra round for each failure !  f+1 rounds to tolerate f failures

f needs to be an input to the protocol --- namely, the

protocol needs the user to indicate the maximum
number of failures to be tolerated

CS4231 Parallel and Distributed Algorithms 14

 Distributed Consensus Version 1: Protocol

 Each node does the following:

Set S initialized to {my input}
for (int i = 1; i <= f+1; i++) {
// do this for f+1 rounds
send S to all other nodes
receive n-1 sets;
for each set T received: set S = S  T
}

Decide on min(S);
CS4231 Parallel and Distributed Algorithms 15
 Distributed Consensus Version 1:
Correctness Proof for the Protocol
 Termination: Obvious since there is no wait
 Validity: Obvious since S will have only a single element

 Agreement: We will prove that S is the same on all nodes when

the protocol terminates (the proof below is different from the
proof in the textbook and is slightly easier to understand)
 A node is nonfaulty during round r if it has not crashed by the
end of round r
 A round is good if there is no node failure during that round
 With f+1 round and f failures, there must be at least one good
round

CS4231 Parallel and Distributed Algorithms 16

 Distributed Consensus Version 1:
Correctness Proof for the Protocol
 Claim: At the end of any good round r, all non-faulty nodes during
round r have the same S

 Claim: Suppose r is a good round, and consider any round t after

round r. During round t, the value of S on any non-faulty nodes
does not change.

 Claim: All nonfaulty processes at round f+1 will have the same S

CS4231 Parallel and Distributed Algorithms 17

 Distributed Consensus Version 1:
Lower Bound on the Number of Rounds
 Theorem: With f crash failures, any consensus
protocol will take (f) rounds.
 In fact, any consensus protocol will take at least f+1 rounds.
 The proof of this theorem is beyond the scope of this module.

CS4231 Parallel and Distributed Algorithms 18

 Distributed Consensus Version 2:
Consensus with Link Failures
 (Also called the coordinated attack problem)
 Failure model:
 Nodes do not fail
 Communication channels may fail – may drop arbitrary
(unbounded) number of messages
 Timing model
 Synchronous: Message delay has a known upper bound x,
and node processing delay has a known upper bound y

CS4231 Parallel and Distributed Algorithms 19

 Distributed Consensus Version 2:
Consensus with Link Failures
 Goal:
 Termination: All nodes eventually decide
 Agreement: All nodes decide on the same value
 Validity: If all nodes have the same initial input, they should all
decide on that. Otherwise nodes are allowed to decide on
anything
 It is impossible to reach the above goal using a
deterministic algorithm – since the communication
channel can drop all messages
 Trivial to prove, but the technique is important – see next slide
 Execution  is indistinguishable from execution β for some node
if the node sees the same things (messages and inputs) in both
executions
CS4231 Parallel and Distributed Algorithms 20
 Impossibility Proof via Contradiction
A B A B due to
due to
input=0 input=0 input=1 input=1 indistingu
validity decision=0 decision=? ishability
decision=0 decision=0

A B A B due to
due to
input=1 input=0 indistingu input=1 input=1 agreement
decision=? decision=0 ishability decision=0 decision=0

This last execution

A B contradict with validity
due to
input=1 input=0
decision=0 decision=0 agreement

CS4231 Parallel and Distributed Algorithms 21

 Distributed Consensus Version 2:
Consensus with Link Failures
 But what if we weaken our goal?
 Weakened Validity: Assuming input either 0 or 1
 If all nodes start with 0, decision should be 0
 If all nodes start with 1 and no message is lost throughout the
execution, decision should be 1
 Otherwise nodes are allowed to decide on anything

 Theorem: It is still impossible to reach the above

weakened goal using a deterministic algorithm

CS4231 Parallel and Distributed Algorithms 22

 Impossibility Proof via Contradiction
input =1 input =1
decision = 1 decision = 1
input =1 input =1 because of because
decision = 1 decision = 1 agreement indistinguishable

how to
rigorously
define last
lost message?

CS4231 Parallel and Distributed Algorithms 23

 Distributed Consensus Version 2:
Allowing Limited Disagreement
 Goal:
 Termination: All nodes eventually decide
 Agreement: All nodes decide on the same value with
probability of (1 – error_prob)
 Validity:
If all nodes start with 0, decision should be 0
If all nodes start with 1 and no message is lost throughout the
execution, decision should be 1
Otherwise nodes are allowed to decide on anything

CS4231 Parallel and Distributed Algorithms 24

 Distributed Consensus Version 2:
More Precise Notion of Randomization
 Processes can flip coins on the fly
 Adversary tries to maximize error_prob by:
 Setting the inputs of the processes
 Cause message losses
 In reality, there is no human adversary – we are trying to
best the worst case scenario (i.e., where the adversary is not
only intelligent but in fact the toughest)
 The outcomes of random coin flips are not known by
adversary beforehand
 The key to beat the adversary

CS4231 Parallel and Distributed Algorithms 25

 A Randomized Algorithm
 For simplicity, consider two processes (can
generalize to multiple): P1 and P2
 Algorithm has a predetermined number (r) of rounds
 Adversary determines which messages get lost,
before seeing the random choices

 P1 picks a random integer bar  [1...r] at the

beginning
 The protocol allows P1 and P2 to each maintain a
level variable (L1 and L2), such that
 level is influenced by the adversary, but
 L1 and L2 can differ by at most 1
CS4231 Parallel and Distributed Algorithms 26
 Simple Algorithm to Maintain Level
 P1 sends msg to P2 each round
P1 P2
 P1 sends msg to P2 each round
0 0
 bar, input and current level
attached on all messages round 1
0 1

 Upon P2 receiving a msg with

round 2
L1 attached: P2 sets L2 = L1+1
2 1

 L1 maintained similarly
round 3
2 3

CS4231 Parallel and Distributed Algorithms 27

 Simple Algorithm to Maintain Level
 Lemma: L1 and L2 never
P1 P2
decreases in any round,
and at the end of any 0 0
round, L1 and L2 differ by
at most 1. round 1
0 1

round 2
2 1

round 3
2 3

CS4231 Parallel and Distributed Algorithms 28

 Inductive Proof for the Lemma

round k round k
L1 L1-1 L1 L1-1

round k+1 round k+1

L1 L1+1 L1 L1-1

round k round k
L1 L1-1 L1 L1-1

round k+1 round k+1

L1 L1+1 L1 L1-1

CS4231 Parallel and Distributed Algorithms 29

 Decision Rule
 At the end of the r rounds, P1 decides on 1 iff
 P1 knows that P1’s input and P2’s input are both 1, and
 L1  bar
 (This implies that P1 will decide on 0 if it does not see P2’s input.)

 At the end of the r rounds, P2 decides on 1 iff

 P2 knows that P1’s input and P2’s input are both 1, and
 P2 knows bar, and
 L2  bar
 (This implies that P2 will decide on 0 if it does not see P1’s input or
if it does not see bar.)

CS4231 Parallel and Distributed Algorithms 30

 When does error occur?
 For P1 and P2 to decide on different values: One must decide on 1
while the other decide on 0
 For someone to decide on 1, P1’s input and P2’s input must be both 1
 Case 1:
 P1 sees P2’s input, but P2 does not see P1’s input or does not see bar
 Then L1 = 1 and L2 = 0. Error occurs only when bar = 1.
 Case 2:
 P2 sees P1’s input and bar, but P1 does not see P2’s input
 Then L1 = 0 and L2 = 1. Error occurs only when bar = 1.
 Case 3:
 P1 sees P2’s input, and P2 sees P1’s input and bar
 Define Lmax = max(L1, L2)
 Error occurs only when bar = Lmax.

CS4231 Parallel and Distributed Algorithms 31

 Correctness Proof
 Termination: Obvious (r rounds)
 Validity:
 If all nodes start with 0, decision should be 0 – obvious
 If all nodes start with 1 and no message is lost throughout the
execution, decision should be 1 – If no messages are lost, then
L1=L2=r  P1 and P2 will decide on 1
 Otherwise nodes are allowed to decide on anything
 Agreement: With (1-1/r) probability – by arguments on previous
slide

CS4231 Parallel and Distributed Algorithms 32

 Summary
Failure Model and Timing Model Consensus Protocol
Ver 0: No node or link failures Trivial – all-to-all broadcast
Ver 1: Node crash failures; Channels (f+1)-round protocol can
are reliable; Synchronous; tolerate f crash failures
Ver 2: No node failures; Channels may Impossible without error
drop messages (the coordinated attack Randomized algorithm with
problem) 1/r error prob
Ver 3: Node crash failures; Channels Next lecture
are reliable; Asynchronous;
Ver 4: Node Byzantine failures; Next lecture
Channels are reliable; Synchronous;
(the Byzantine Generals problem)

CS4231 Parallel and Distributed Algorithms 33

 Homework Assignment
 In the randomized algorithm for the coordinated attack problem, suppose
that r = 3 and the adversary knows that the bar chosen by P1 is 2.
Construct a scenario where the adversary ensures disagreement between
the two processes by carefully selecting certain messages to drop

 Page 249, Problem 15.6

Show by an example that if the consensus algorithm (for Version 1)
decided the final value after f rounds instead of f+1 rounds, then it might
violate the agreement property.

 Homework due a week from today

CS4231 Parallel and Distributed Algorithms 34