Aci 3545

BRKACI-3545
Mastering ACI Forwarding

Behavior
– A day in the life of a packet –
Takuya Kishida, Technical Leader, Service

Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKACI-3545
BRKACI-3545 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda
• Introduction
• ACI Overlay VxLAN and TEP
• ACI Forwarding components

• Endpoints, EPG, EP Learning, COOP and How it all works
• BD, VRF forwarding scope and detailed options
• Spine-Proxy and ARP Glean
• Forwarding Software Architecture and ASIC Generation
• ACI Packet Walk

• Walk through the life of a packet going through ACI
Reference Slide
Basic Acronyms/Definitions
VxLAN packet acronyms
Acronyms Definitions
Acronyms Definitions
ACI Application Centric Infrastructure
dXXXo Outer Destination XXX
APIC Application Policy Infrastructure Controller (dIPo = Outer Destination IP)
sXXXo Outer Source XXX
EP Endpoint
(sIPo = Outer Source IP)
EPG Endpoint Group dXXXi Inner Destination XXX
(dIPi = Inner Destination IP)
BD Bridge Domain
sXXXi Inner Source XXX
VRF Virtual Routing and Forwarding (sIPi = Inner Source IP)
COOP Council of Oracle Protocol GIPo Outer Multicast Group IP
VxLAN Virtual eXtensible LAN VNID Virtual Network Identifier
Agenda
• Introduction

• ACI Packet Walk

ACI Overlay VxLAN and TEP ※ TEP : Tunnel EndPoint
Anycast
VRF overlay-1 TEP
TEP1 TEP2 TEP3
VRF1 VRF1 VRF1
BD1 BD1 BD2 BD2 L3OU

T
EPG1 EPG2 EPG EPG EPG

2 3 4
EP1-1 EP2-1 EP2-2 EP3-1 EP3-2 EP4-1 WAN
ACI Overlay VxLAN and TEP ※ TEP : Tunnel EndPoint
Anycast
VRF overlay-1 TEP
dMACo sMACo sIPo dIPo VxLAN dMACi sMACi sIPi dIPi
TEP1 TEP2 TEP3
VRF1 VRF1 VRF1

dMAC sMAC sIP dIP dMAC sMAC sIP dIP

T

2 3 4
ACI Overlay VxLAN and TEP
Scenario 1 : source LEAF knows the destination

VRF overlay-1
Anycast ( on the same LEAF )
TEP
Scenario 2 : source LEAF knows the destination ( on another LEAF X )

TEP1 TEP2 TEP3
Scenario 3 : source
VRF1
LEAF does NOT know
VRF1
the destinationVRF1
(Spine-Proxy)
T
Scenario 4 : source
EPG1LEAF does NOT know
EPG2 EPG theEPG
destinationEPG
(Flood)
2 3 4
Source LEAF knows the destination ( on the same LEAF )
Anycast
VRF overlay-1 TEP
TEP1 TEP2 TEP3
VRF1 VRF1 VRF1

T
Local EndPoint EPG1 EPG2 EPG EPG EPG

2 3 4
Quite Similar to
normal L2/L3
Forwarding

VRF overlay-1
TEP

TEP1 TEP2 TEP3
Scenario 3 : source
VRF1
LEAF does NOT know
VRF1
the destinationVRF1
(Spine-Proxy)
T
Scenario 4 : source
EPG2 EPG theEPG
destinationEPG
(Flood)
2 3 4
Source LEAF knows the destination ( on the remote LEAF )
Anycast
VRF overlay-1 TEP
TEP1 TEP2 TEP3
1 Send to LEAF2 (TEP2)

VRF1 VRF1 VRF1
dMAC sMAC sIP dIP

T
Remote EndPoint
2 3 4
Packet goes
through infra
network with EP1-1 EP2-1 EP2-2 EP3-1 EP3-2 EP4-1 WAN
additional encaps
(iVxLAN)
※ dMAC, sMAC may change if it’s routing traffic
VxLAN has VRF or BD VNID
Anycast
VRF overlay-1 TEP
sIPo dIPo
dMACo sMACo VxLAN dMACi sMACi sIPi dIPi
(TEP1) (TEP2)
2 Add VxLAN header

TEP1 TEP2 TEP3

VRF1 VRF1 VRF1
dMAC sMAC sIP dIP

T
Remote EndPoint
2 3 4
Packet goes
through infra
additional encaps
(iVxLAN)
VxLAN has VRF or BD VNID Anycast TEP is used for proxy
not used in this scenario
Anycast
VRF overlay-1 TEP
3 Forward based on outer IP (dIPo)
sIPo dIPo
(TEP1) (TEP2)
2 Add VxLAN header

TEP1 TEP2 TEP3

VRF1 VRF1 VRF1
dMAC sMAC sIP dIP

T
Remote EndPoint
2 3 4
Packet goes
through infra
additional encaps
(iVxLAN)
Anycast
VRF overlay-1 TEP
sIPo dIPo
(TEP1) (TEP2)
2 Add VxLAN header

TEP1 TEP2 TEP3
4 Decapsulate VxLAN
VRF1 VRF1 VRF1

T
Remote EndPoint
2 3 4
Packet goes
through infra
additional encaps
(iVxLAN)
Anycast
VRF overlay-1 TEP
sIPo dIPo
(TEP1) (TEP2)
2 Add VxLAN header

TEP1 TEP2 TEP3
4 Decapsulate VxLAN
VRF1 VRF1 VRF1
BD1 BD1 BD2 5 BD2

Send to EP L3OU
T
Remote EndPoint
2 3 4
Packet goes
through infra
additional encaps
(iVxLAN)

VRF overlay-1
TEP

TEP1 TEP2 TEP3
Scenario 3 : source
VRF1
LEAF does NOT know
VRF1
the destinationVRF1
(Spine-Proxy)
T
Scenario 4 : source
EPG2 EPG theEPG
destinationEPG
(Flood)
2 3 4
Source LEAF does NOT know the destination (Spine-Proxy)
Anycast
VRF overlay-1 TEP
TEP1 TEP2 TEP3
1 Send to SPINE (Anycast TEP)

VRF1 VRF1 VRF1
dMAC sMAC sIP dIP

T
Remote EndPoint
Packet goes 2 3 4
through infra
additional
encaps (iVxLAN)
Anycast
3 VRFIPoverlay-1
Forward based on outer (dIPo) TEP
sIPo dIPo
(TEP1) (acast TEP)
2 Add VxLAN header TEP1 TEP2 TEP3

VRF1 VRF1 VRF1
dMAC sMAC sIP dIP

T
Remote EndPoint
Packet goes 2 3 4
through infra
additional
encaps (iVxLAN)
4 Lookup dMACi or dIPi with BD or VRF VNID in VxLAN
Anycast
3 VRFIPoverlay-1
sIPo dIPo
(TEP1) (acast TEP)

VRF1 VRF1 VRF1
dMAC sMAC sIP dIP

T
Remote EndPoint
Packet goes 2 3 4
through infra
additional
encaps (iVxLAN)
Anycast
3 VRFIPoverlay-1
sIPo 5 Send to LEAF2 (TEP2)

dIPo
(TEP1) (acast TEP)
sIPo dIPo
(TEP1) (TEP2)

VRF1 VRF1 VRF1
dMAC sMAC sIP dIP

T
Remote EndPoint
Packet goes 2 3 4
through infra
additional
encaps (iVxLAN)
Anycast
3 VRFIPoverlay-1

dIPo
(TEP1) (acast TEP)
sIPo dIPo
(TEP1) (TEP2)
6 Decapsulate VxLAN
VRF1 VRF1 VRF1
dMAC sMAC sIP dIP
dMAC sMAC sIP dIP

T
Remote EndPoint
Packet goes 2 3 4
through infra
additional
encaps (iVxLAN)
Anycast
3 VRFIPoverlay-1

dIPo
(TEP1) (acast TEP)
sIPo dIPo
(TEP1) (TEP2)
6 Decapsulate VxLAN
VRF1 VRF1 VRF1
dMAC sMAC sIP dIP
dMAC sMAC sIP dIP

7 Send to EP
T
Remote EndPoint
Packet goes 2 3 4
through infra
additional
encaps (iVxLAN)

VRF overlay-1
TEP

TEP1 TEP2 TEP3
Scenario 3 : source
VRF1
LEAF does NOT know
VRF1
the destinationVRF1
(Spine-Proxy)
T
Scenario 4 : source
EPG2 EPG theEPG
destinationEPG
(Flood)
2 3 4
Source LEAF does NOT know the destination (Flood)
ftag tree
ROOT
VRF overlay-1
TEP1 TEP2 TEP3
1 Flood with BD mcast TEP and tree

VRF1 VRF1 VRF1
dMAC sMAC sIP dIP

T
Remote EndPoint EPG1 EPG2 EPG EPG EPG

2 3 4
Packet goes
through infra EP1-1 EP2-1 EP2-2 EP3-1 EP3-2 EP4-1 WAN
network with
additional encaps
(iVxLAN)
ftag tree
ROOT
VRF overlay-1
TEP1 TEP2 TEP3

VRF1 VRF1 VRF1
dMAC sMAC sIP dIP

T

2 3 4
Packet goes
network with
additional encaps
(iVxLAN)
ftag tree
ROOT
VRF overlay-1
3 Flood based on ftag tree
sIPo GIPo
(TEP1) (mcast TEP)

VRF1 VRF1 VRF1
dMAC sMAC sIP dIP

T

2 3 4
Packet goes
network with
additional encaps
(iVxLAN)
ftag tree
ROOT
VRF overlay-1
sIPo GIPo
(TEP1) (mcast TEP)

VRF1 VRF1 VRF1
dMAC sMAC sIP dIP

T

2 3 4
Packet goes
network with
additional encaps
(iVxLAN)
ftag tree
ROOT
VRF overlay-1
sIPo GIPo
(TEP1) (mcast TEP) 5 Flood based on ftag tree
1 Flood with BD mcast TEP and tree 5 Decapsulate VxLAN

VRF1 VRF1 VRF1
T

2 3 4
Packet goes
network with
additional encaps
(iVxLAN)
ftag tree
ROOT
VRF overlay-1
sIPo GIPo
(TEP1) (mcast TEP) 5 Flood based on ftag tree
1 Flood with BD mcast TEP and tree 5 Decapsulate VxLAN

VRF1 VRF1 VRF1
6 Send to EP T

2 3 4
Packet goes
network with
additional encaps
(iVxLAN)

VRF overlay-1
TEP

TEP1 TEP2 TEP3
Scenario 3 : source
VRF1
LEAF does NOT know
VRF1
the destinationVRF1
(Spine-Proxy)
T
Scenario 4 : source
EPG2 EPG theEPG
destinationEPG
(Flood)
2 3 4
How does LEAF pick one of these scenario?

EP1-1 EP2-1 EP2-2 WAN
EP3-1 EP3-2 EP4-1
 based on EP information
Agenda
• Introduction

• ACI Packet Walk

ACI Forwarding Component 1
• Endpoint
• EPG (EndPoint Group)
• VLAN Type in ACI
• Endpoint Type
• Endpoint Learning
• COOP (Council of Oracle Protocol)
End Point (EP)
What is an EP?
• It stands for hosts, in other words MAC address with IP(s)
 sometimes MAC only
 IP in EP is always /32
What Forwarding Table is used?

• End Point Table
 host information (MAC and /32 IP address)
MAC A MAC B MAC C
• LPM(Longest Prefix Match) Table IP A IP B IP C
 non /32 IP route information (exception: /32 for SVI or L3OUT route)
These
Theseare Endpoints
are End Points
Legacy ACI
RIB ( non-/32 & /32 ) RIB ( non /32 )
Forwarding table lookup order
MAC EndPoint ( mac & /32 ip ) 1. EndPoint Table (show endpoint)
2. RIB (show ip route)
ARP ARP (only for L3OUT)
RIB : Routing Information Base
End Point Group (EPG)
What is an EPG?
• Logical grouping of hosts (EPs)
• Each EPG belongs to a Bridge Domain (BD).
What is the EPG used for?

• To implement traffic filtering
 Traffic within the same EPG doesn’t get blocked
 Traffic across EPGs always blocked without a VLAN 1 VLAN 2 VLAN 3
contract
• A contract is applied between EPGs to allow traffic
MAC A1 MAC B1
MAC A2 EPG-B
EPG-A IP B1
What is a VLAN in ACI? IP A1 IP A2
• VLAN is just an identifier to classify EPs to each EPG

• BD is the L2 domain instead of VLAN
No Contract
What happens with forwarding?

• It will be done by BD or VRF to which EPG belongs
How to check End Points Fabric Wide
From APIC GUI ( Fabric perspective ) Visibility
shows where EPs are
learned
From LEAF CLI ( LEAF perspective )

leaf1# show endpoint vrf TK:VRF1
Legend:
s - arp O - peer-attached a - local-aged S - static
V - vpc-attached p - peer-aged M - span L - local 0000.0000.5151 0000.1111.5151
192.168.1.1 192.168.1.11
B - bounce H - vtep
+-----------------------------------+---------------+-----------------+--------------+-------------+
Good for forwarding

VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
verification
+-----------------------------------+---------------+-----------------+--------------+-------------+
TK:VRF1 192.168.1.11 tunnel8
17/TK:VRF1 vxlan-15826915 0000.1111.5151 tunnel8
19 vlan-5 0000.0000.5151 L eth1/1 shows how EPs look from
TK:VRF1 vlan-5 192.168.0.51 L eth1/1 each LEAF
BRKACI-3545 36
VLAN types in ACI
※ PI-VLAN : Platform Independent VLAN
VLAN ID for external devices Internal ID on LEAF For forwarding

(user configured value) (not shared across LEAFs) (global value for entire fabric)
Access Encap VLAN PI-VLAN VxLAN ID PI-VLAN Access Encap VLAN

(VNID)
LEAF 1 LEAF 2
VRF1 2293760
VRF1
BD1 For BD SVI

BD1
17 15826915 31
EPG1 vxlan- vxlan-

EPG1
20 8388608 33
8388608 8388608
vlan-5 19 9492 30 vlan-5
EP EP EP EP
PI-VLAN for EPG and BD CLI
• Endpoint Table
leaf1# show endpoint ip 192.168.0.51
19 vlan-5 0000.5555.1111 L eth1/1
TK:VRF1 vlan-5 192.168.0.51 L eth1/1
PI-VLAN Access Encap VLAN
• VLAN Table
NOT Access Encap VLAN.
PI-VLAN 17, 19 “extended” option to display Access Encap VLAN
leaf1# show vlan id 17,19 extended

VLAN Name Status Ports
---- -------------------------------- --------- ----------------------
17 TK:BD1 active Eth1/1, Eth1/2, Po6
19 TK:AP1:EPG1 active Eth1/1
VLAN Type Vlan-mode Encap

---- ----- ---------- -------------------------------
17 enet CE vxlan-15826915
19 enet CE vlan-5
PI-VLAN Access Encap VLAN

PI-VLAN for EPG and BD CLI
leaf1# show vlan id 17,19 extended

---- -------------------------------- --------- -------------------------------
17 TK:BD1 active Eth1/1, Eth1/2, Po6
19 TK:AP1:EPG1 active Eth1/1

---- ----- ---------- -------------------------------
19 enet CE vlan-5
BD
PI-VLAN
EPG
PI-VLAN leaf1# show system internal epm vlan 19
+----------+---------+-----------------+----------+------+----------+-----------
VLAN ID Type Access Encap Fabric H/W id BD VLAN Endpoint
(Type Value) Encap Count
+----------+---------+-----------------+----------+------+----------+-----------
19 FD vlan 802.1Q 5 8294 14 17 2
How to check details of EndPoints
With MAC keyword :
show system internal epm endpoint mac 0000.5555.1111
leaf1# show system internal epm endpoint ip 192.168.0.51
MAC : 0000.5555.1111 ::: Num IPs : 1

PI-VLAN IP# 0 : 192.168.0.51 ::: IP# 0 flags :
for EPG
Vlan id : 19 ::: Vlan vnid : 9492 ::: VRF name : TK:VRF1
VNID for
EPG(Vlan), BD and VRF
BD vnid : 15826915 ::: VRF vnid : 2293760
Phy If : 0x1a000000 ::: Tunnel If : 0
Interface this EP is
Interface : Ethernet1/1
learned on Flags : 0x80004c04 ::: sclass : 49154 ::: Ref count : 5
EP Create Timestamp : 05/03/2017 09:33:56.654606
EP Update Timestamp : 05/04/2017 16:11:37.584734
EP Flags : local|IP|MAC|sclass|timer|
::::
End Point Types
Legend:
all commands on O - peer-attached
L - local
Physical Local Endpoint (PL)

• An endpoint attached to this LEAF
fab1-leaf1# show endpoint ip 192.168.0.51
19 vlan-5 0000.5555.1111 L eth1/1
TK:VRF1 vlan-5 192.168.0.51 L eth1/1 EP
Virtual Local Endpoint (VL)

• An endpoint on AVS/AVE attached to this LEAF
14 vxlan-8388608 0050.5680.34eb L tunnel10 AVS
TK:VRF1 vxlan-8388608 192.168.66.2 L tunnel10
EP AVS/AVE is treated as virtual LEAF with vTEP
Access Encap VLAN (VxLAN)

Remote Endpoint (Xr)
• An endpoint on a remote LEAF
BD VNID (not Access Encap VLAN)
fab1-leaf1# show endpoint mac 0000.5555.2222
fab1-leaf1# show endpoint ip 192.168.0.52 EP

TK:VRF1 192.168.0.52 tunnel8
On-Peer Endpoint
• An endpoint connected to an orphan port on vPC peer
19 vlan-5 0000.5555.2222 O tunnel8
TK:VRF1 vlan-5 192.168.0.52 O tunnel8 EP
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
End Point Learning (Local EP)
EP
PI-VLAN ID(19) of EPG

to which MAC belongs leaf1# show endpoint ip 192.168.0.51
19 vlan-5 0000.5555.1111 L eth1/1
TK:VRF1 vlan-5 192.168.0.51 L eth1/1
VRF to which
MAC & IP belong Access Encap VLAN
of EP(MAC+IP)
Local Endpoint (MAC)

A leaf learns MAC A as local if a packet with src MAC A comes in from its front panel port.
Local Endpoint (/32 host IP)

A leaf learns IP A /32 as local
• if a packet with src IP A comes in from its front panel port AND IP lookup is done on ACI.
(which means IP addr is learned only when a leaf handles L3 traffic)
or
• if ARP request with sender IP A comes in from its front panel port. (regardless of ARP Flooding setup)
EPG/BD/VRF is based on Access Encap VLAN ID
What APIC GUI shows are these local Endpoints

End Point Learning (Remote EP = cache)
PI-VLAN(17) of BD BD VNID
EP
and VRF to which
MAC belongs fab1-leaf1# show endpoint mac 0000.5555.2222
VRF to which fab1-leaf1# show endpoint ip 192.168.0.52

MAC & IP belong TK:VRF1 192.168.0.52 tunnel8
tunnel represents
Remote Endpoint (MAC) destination leaf TEP
A leaf learns MAC A as remote when L2 traffic with src MAC A comes in from SPINE.
Remote Endpoint (/32 host IP)

A leaf learns IP A as remote when L3 traffic with src IP A comes in from SPINE.
• Remote MAC and remote IP is learned separately

• BD(for MAC) / VRF(for IP) is based on VNID in VxLAN header
VNID is
BD when L2 traffic
VRF when L3 traffic (not both)
dMACo sMACo sIPo dIPo VxLAN dMACi sMACi sIPi dIPi
How to check Tunnel Interface (TEP)
leaf1# show int tunnel 8 | grep Tun
Tunnel8 is up
Tunnel protocol/transport is ivxlan
Tunnel source 11.0.200.92/32 (lo0)
Tunnel destination 11.0.48.95 TEP
TEP IP address
address
leaf1# acidiag fnvread

ID Pod ID Name Serial Number IP Address Role State LastUpdMsgId
--------------------------------------------------------------------------------------------------------------
101 1 leaf1 FDO20160AAA 11.0.200.92/32 leaf active 0
102 1 leaf2 FDO20160BBB 11.0.48.95/32 leaf active 0
103 1 leaf3 FDO20240CCC 11.0.200.91/32 leaf active 0
201 1 spine1 FGE12345678 11.0.200.94/32 spine active 0
202 1 spine2 FGE87654321 11.0.200.93/32 spine active 0
admin@apic1:~> moquery -c vpcDom | egrep 'virtualIp|dn|#'

# vpc.Dom
dn : topology/pod-1/node-101/sys/vpc/inst/dom-1
virtualIp : 11.0.64.65/32
# vpc.Dom Tunnel
Tunnelmay
maypoint
pointto
tovPC
vPCvTEP
TEP
dn : topology/pod-1/node-102/sys/vpc/inst/dom-1 This is vTEP for vPC LEAF 101-102
virtualIp : 11.0.64.65/32
# vpc.Dom
virtualIp : 11.0.192.64/32
# vpc.Dom
virtualIp : 11.0.192.64/32
COOP (End Point Learning on Spine)
COOP Table
SPINEs do NOT learn EP from data plane like LEAF
MAC/IP A -> LEAF 1
MAC/IP B -> LEAF 2
SPINEs receive all EP data from Leafs MAC/IP C -> LEAF 3
1. LEAF learns EP (either MAC or/and IP) as local
2. LEAF reports local EP to Spine via COOP process
3. SPINE stores these in COOP DB and synchronize with other I know EP B
SPINEs
I know EP C
What is the purpose of COOP? I know EP A

When Leaf doesn’t know dst EP, LEAF can forward
packet to Spine in order to let Spine decide where to send.
This behavior is called Spine-Proxy.
LEAF 1 LEAF 2 LEAF 3
Note :
• Normally SPINE doesn’t push COOP DB entries to MAC C
MAC A MAC B
each LEAF. It just receives and stores. The IP A IP B IP C
exception is for bounce entries.
• Remote Endpoints are stored on each Leaf nodes

as cache. This is not reported to Spine COOP.
How to check COOP DB on Spine
fab5-spine2# show coop internal info repo ep key 15826915 0000.5555.1111 | egrep 'vnid|mac|id|Real'
EP bd vnid : 15826915 BD VNID

EP mac : 00:00:55:55:11:11 EP data
Vrf vnid : 2293760
Epg vnid : 0
Ep vpc-id : 0
Ep vpc virtual switch-id : 0.0.0.0
publisher id : 11.0.200.92 TEP address of LEAF
Real IPv4 EP : 192.168.5.111
EP data
fab5-spine2# show coop internal info ip-db key 2293760 192.168.5.111
IP address : 192.168.5.111 VRF VNID

Vrf : 2293760
Flags : 0 EP data
EP bd vnid : 15826915
EP mac : 00:00:55:55:11:11
Publisher Id : 11.0.200.92 TEP address of LEAF
---- snip ----
Bounce Entry
What is Bounce Entry? COOP
COOP Table
Table
MAC/IP A1 -> Leaf1
• a
• Remote EPs created by COOP when • aMAC/IP B -> LEAF 2
• aMAC/IP A2 -> Leaf2
an EP moved • aMAC/IP B1 -> Leaf3
 The exception where SPINE

pushes COOP DB to a LEAF Local Endpoint
• MAC/IP
A B -> e1/1
What is Bounce Entry for? Remote Endpoint
Remote Endpoint
• An old LEAF (LEAF2) can bounce • aIP B -> LEAF 2
• a
• aa I know EP B
packets to the latest EP location
in case other LEAFs (LEAF1) with
old Remote EPs still send
packets to the old LEAF (LEAF2) MAC B
IP B
For more details : ACI Fabric Endpoint Learning Whitepaper

Bounce Entry
COOP Table
Table EP B moved
MAC/IP A1 -> Leaf1
• a to LEAF3
• Remote EPs created by COOP when • aMAC/IP B -> LEAF 2
LEAF 3

• MAC/IP
A B -> e1/1 I know EP B
Remote Endpoint
• An old LEAF (LEAF2) can bounce • aIP B -> LEAF 2
•
•
a
aa
packets to the old LEAF (LEAF2) MAC B MAC B
IP B IP B
EP Move

Bounce Entry
COOP Table
Table EP B moved
MAC/IP A1 -> Leaf1
• a to LEAF3
• Remote EPs created by COOP when Your EP B moved • aMAC/IP B -> LEAF 2
to LEAF 3 LEAF 3

No Change
• MAC/IP
Remote Endpoint
• An old LEAF (LEAF2) can bounce • MAC
a -> LEAF 3
• aIP B -> LEAF 2 • IP
aa -> LEAF 3
packets to the old LEAF (LEAF2) MAC B MAC B
IP B IP B
EP Move

Bounce Entry
COOP Table
Table EP B moved
MAC/IP A1 -> Leaf1
• a to LEAF3
• Remote EPs created by COOP when Your EP B moved • aMAC/IP B -> LEAF 2
to LEAF 3 LEAF 3

No Change
• MAC/IP
Remote Endpoint
• An old LEAF (LEAF2) can bounce • MAC
a -> LEAF 3
• aIP B -> LEAF 2 • IP
aa -> LEAF 3
packets to the old LEAF (LEAF2) Bounce Entries MAC B MAC B
IP B IP B
EP Move
leaf2# show end vrf TK:VRF1
TK:VRF1 192.168.1.11 B tunnel4

60/TK:VRF1 vxlan-15826915 0000.1111.5151 B tunnel4

Agenda
• Introduction

• ACI Packet Walk

• Pervasive Gateway (BD SVI)
• Forwarding Scope (VRF or BD)
• Forwarding mode in BD
Pervasive Gateway(BD SVI)
What is pervasive GW for?
• To be a default GW for EPs in the Fabric
 All EPs can have consistent gateway IP
address one hop away
• To represent subnets(IP ranges) for a BD

 ACI knows which BD may have potential
hidden/silent EPs
How is pervasive GW deployed?
• Installed as an SVI on LEAFs
 PI-VLAN for BD is used to represent a
leaf1# show ip route vrf TK:VRF1 pervasive GW SVI
Pervasive route
 A pervasive SVI has secondary IP when
192.168.0.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.184.64%overlay-1, [1/0], 04:32:16, static multiple pervasive GWs are configured on
the same BD
Pervasive SVI
192.168.0.254/32, ubest/mbest: 1/0, attached
*via 192.168.0.254, vlan10, [1/0], 04:32:16, local, local
 User can choose a primary address
BD SVI with PI-VLAN

Pervasive Gateway(BD SVI) example
Tenant
VRF1 VRF2
BD1 BD2 BD3
BD-SVI BD-SVI (2ndary) BD-SVI BD-SVI
192.168.0.254/24 192.168.1.254/24 5.0.0.254/24 10.0.0.254/24
EPG1 EPG2 EPG3 EPG4 EPG5

EP-A EP-B EP-C EP-D EP-E EP-F
192.168.0.1 192.168.0.2 192.168.0.3 192.168.1.1 5.0.0.1 10.0.0.1
VRF1
VRF1 VRF1 VRF1 5.0.0.254/24
•192.168.0.254/24
a •192.168.0.254/24
a •192.168.0.254/24
a VRF2
•192.168.1.254/24
a •192.168.1.254/24
a •192.168.1.254/24
a 10.0.0.254/24
none
EP-A EP-D EP-B EP-C EP-E EP-F

192.168.0.1 192.168.1.1 192.168.0.2 192.168.0.3 5.0.0.1 10.0.0.1
If Leaf has IfEPLEAF has

for BD, EPG for
subnets will BD,
be programed
Both pervasive route and SVI Pervasive GW
When ? ( A. will be
depends programed
on Immediacy)
Pervasive Gateway(BD SVI) example
Tenant
VRF1 VRF2
BD1 BD2 BD3
BD-SVI BD-SVI (2ndary) BD-SVI BD-SVI
192.168.0.254/24 192.168.1.254/24 5.0.0.254/24 10.0.0.254/24

192.168.0.1 192.168.0.2 192.168.0.3 192.168.1.1 5.0.0.1 Only routes. SVI is not pushed.
10.0.0.1
192.168.0.0/24
5.0.0.0/24 192.168.1.0/24
VRF1
VRF1 VRF1 VRF1 5.0.0.254/24
•192.168.0.254/24
a •192.168.0.254/24
a •192.168.0.254/24
a VRF2
•192.168.1.254/24
a •192.168.1.254/24
a •192.168.1.254/24
a 10.0.0.254/24
none
EP-A EP-D EP-B EP-C EP-E EP-F

192.168.0.1 192.168.1.1 192.168.0.2 192.168.0.3 5.0.0.1 10.0.0.1
If a contract is tied to EPGs, pervasive routes are

exchanged across LEAFs within the same VRF
Pervasive Gateway(BD SVI) cont.
Why does ACI push pervasive routes to other LEAFs after a contract?
 Pervasive routes are required for Spine-Proxy
what if no pervasive route, no remote EP? with pervasive route and no remote EP?
5.0.0.0/24
Spine-Proxy Spine-Proxy
192.168.0.0/24 192.168.0.0/24
LEAF 1 itself 192.168.0.254/32 LEAF 1 itself 192.168.0.254/32
no remote EP yet no remote EP yet

EP-C EP-E EP-C EP-E
192.168.0.3 5.0.0.1 192.168.0.3 5.0.0.1
No Spine-Proxy for 5.0.0.1 Spine-Proxy for 5.0.0.1

It may be either dropped or forwarded to L3OUT if a With the contract, ACI knows the LEAF needs to
default route exists reach out to 5.0.0.0/24
Pervasive Gateway EP-C EP-E
192.168.0.3 5.0.0.1
Without contract With contract

L101# show ip route vrf TK:VRF1 L101# show ip route vrf TK:VRF1
192.168.0.0/24, ubest/mbest: 1/0, attached, direct, pervasive 192.168.0.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.184.64%overlay-1, [1/0], 04:32:16, static *via 10.0.184.64%overlay-1, [1/0], 04:32:27, static
192.168.0.254/32, ubest/mbest: 1/0, attached 192.168.0.254/32, ubest/mbest: 1/0, attached
*via 192.168.0.254, vlan10, [1/0], 04:32:16, local, local *via 192.168.0.254, vlan10, [1/0], 04:32:27, local, local
*via 10.0.184.64%overlay-1, [1/0], 00:00:02, static
L103# show ip route vrf TK:VRF1 Exchange pervasive route L103# show ip route vrf TK:VRF1

5.0.0.0/24, ubest/mbest: 1/0, attached, direct, pervasive 5.0.0.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.184.64%overlay-1, [1/0], 00:00:06, static *via 10.0.184.64%overlay-1, [1/0], 00:00:32, static
5.0.0.254/32, ubest/mbest: 1/0, attached 5.0.0.254/32, ubest/mbest: 1/0, attached
*via 192.168.2.254, vlan13, [1/0], 00:00:06, local, local *via 192.168.2.254, vlan13, [1/0], 00:00:32, local, local
 Pervasive routes are pushed to other LEAFs with contracts
ACI Forwarding Scope Concepts
Need VRF-Leaking
Tenant (Shared-Service)
VRF1 VRF2
L3
L3
BD1 BD-SVI BD-SVI (2ndary) BD2 BD-SVI BD3

192.168.0.254/24 192.168.1.254/24 5.0.0.254/24
BD-SVI
0022.bdf8.19ff 0022.bdf8.19ff 0022.bdf8.19ff
10.0.0.254/24
L2 Contract Contract Contract 0022.bdf8.19ff
L2
192.168.0.1 192.168.0.2 192.168.0.3 192.168.1.1 5.0.0.1 10.0.0.1
0000.0000.1111 0000.0000.2222 0000.0000.3333 0000.1111.1111 0000.5555.5555 0000.1010.1010
Forwarding Scope
Basic L2/L3 lookup is same as legacy switch
IP Lookup will be done with VRF scope even though
Tenant subnets are configured under BD
scope : VRF-VNID
VRF1 192.168.0.1 -> EP-A 5.0.0.1 -> EP-E
192.168.0.2 -> EP-B 192.168.0.254 -> BD1 SVI
192.168.0.3 -> EP-C 192.168.1.254 -> BD1 SVI
192.168.1.1 -> EP-D 5.0.0.254 -> BD2 SVI
scope : BD-VNID scope : BD-VNID

BD1 BD2
0000.0000.1111 -> EP-A 0000.1111.1111 -> EP-D 0000.5555.5555 -> EP-E
0000.0000.2222 -> EP-B 0022.bdf8.19ff -> BD SVI 0022.bdf8.19ff -> BD SVI
0000.0000.3333 -> EP-C
EPG1 EPG2 EPG3 EPG4

EP-A EP-B EP-C EP-D EP-E
192.168.0.1 192.168.0.2 192.168.0.3 192.168.1.1 5.0.0.1
0000.0000.1111 0000.0000.2222 0000.0000.3333 0000.1111.1111 0000.5555.5555
Forwarding Scope L2 traffic(=same subnet) use only MAC
hence BD lookup only
Tenant
scope : VRF-VNID
VRF1 192.168.0.1 -> EP-A 5.0.0.1 -> EP-E
192.168.0.2 -> EP-B 192.168.0.254 -> BD1 SVI
192.168.0.3 -> EP-C 192.168.1.254 -> BD1 SVI
192.168.1.1 -> EP-D 5.0.0.254 -> BD2 SVI

BD1 BD2
0000.0000.1111 -> EP-A 0000.1111.1111 -> EP-D 0000.5555.5555 -> EP-E
0000.0000.3333 -> EP-C
EPG1 EPG2 EPG3 EPG4

192.168.0.1 192.168.0.2 192.168.0.3 192.168.1.1 5.0.0.1
0000.0000.1111 0000.0000.2222 0000.0000.3333 0000.1111.1111 0000.5555.5555
EP-A (0000.0000.1111) -> EP-B (0000.0000.2222) © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Forwarding Scope It’s same even if EPG is different
Tenant
scope : VRF-VNID
VRF1 192.168.0.1 -> EP-A 5.0.0.1 -> EP-E
192.168.0.2 -> EP-B 192.168.0.254 -> BD1 SVI
192.168.0.3 -> EP-C 192.168.1.254 -> BD1 SVI
192.168.1.1 -> EP-D 5.0.0.254 -> BD2 SVI

BD1 BD2
0000.0000.1111 -> EP-A 0000.1111.1111 -> EP-D 0000.5555.5555 -> EP-E
0000.0000.3333 -> EP-C
EPG1 EPG2 EPG3 EPG4

192.168.0.1 192.168.0.2 192.168.0.3 192.168.1.1 5.0.0.1
0000.0000.1111 0000.0000.2222 0000.0000.3333 0000.1111.1111 0000.5555.5555
EP-A (0000.0000.1111) -> EP-C (0000.0000.3333) © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Forwarding Scope L3 traffic(=different subnet) use IP Lookup
1. Dst MAC hits default gw svi mac
2. IP Lookup in VRF
Tenant even though EPs are in the same BD
scope : VRF-VNID
VRF1 192.168.0.1 -> EP-A 5.0.0.1 -> EP-E
192.168.0.2 -> EP-B 192.168.0.254 -> BD1 SVI
192.168.0.3 -> EP-C 192.168.1.254 -> BD1 SVI
192.168.1.1 -> EP-D 5.0.0.254 -> BD2 SVI

BD1 BD2
0000.0000.1111 -> EP-A 0000.1111.1111 -> EP-D 0000.5555.5555 -> EP-E
0000.0000.3333 -> EP-C
EPG1 EPG2 EPG3 EPG4

192.168.0.1 192.168.0.2 192.168.0.3 192.168.1.1 5.0.0.1
0000.0000.1111 0000.0000.2222 0000.0000.3333 0000.1111.1111 0000.5555.5555
EP-A (192.168.0.1) -> EP-D (192.168.1.1)

Forwarding Scope
It’s same even if BD is different
Tenant
scope : VRF-VNID
VRF1 192.168.0.1 -> EP-A 5.0.0.1 -> EP-E
192.168.0.2 -> EP-B 192.168.0.254 -> BD1 SVI
192.168.0.3 -> EP-C 192.168.1.254 -> BD1 SVI
192.168.1.1 -> EP-D 5.0.0.254 -> BD2 SVI

BD1 BD2
0000.0000.1111 -> EP-A 0000.1111.1111 -> EP-D 0000.5555.5555 -> EP-E
0000.0000.3333 -> EP-C
EPG1 EPG2 EPG3 EPG4

192.168.0.1 192.168.0.2 192.168.0.3 192.168.1.1 5.0.0.1
0000.0000.1111 0000.0000.2222 0000.0000.3333 0000.1111.1111 0000.5555.5555
EP-A (192.168.0.1) -> EP-E (5.0.0.1) © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
ACI BD Forwarding Option
• Unicast Routing
• L2 Unkown Unicast
• L3 Unknown Multicast Flooding
• Multi Destination Flooding
• ARP Flooding
※ Please check a whitepaper “ACI Fabric EP Learning” for EP learning options

https://www.cisco.com/c/en/us/solutions/collateral/data-center-
virtualization/application-centric-infrastructure/white-paper-c11-739989.html
On
Unicast Routing
Tenant scope : VRF-VNID
VRF1 192.168.0.1 -> EP-A 5.0.0.1 -> EP-E
192.168.0.2 -> EP-B 192.168.0.254 -> BD1 SVI
192.168.0.3 -> EP-C 192.168.1.254 -> BD1 SVI
Unicast Routing : OFF Unicast Routing : ON
192.168.1.1 -> EP-D 5.0.0.254 -> BD2 SVI
BD1 0000.0000.1111 -> EP-A 0000.1111.1111 -> EP-D BD2
0000.0000.2222 -> EP-B 0022.bdf8.19ff -> BD SVI 0000.5555.5555 -> EP-E
0000.0000.3333 -> EP-C 0022.bdf8.19ff -> BD SVI
EPG1
EP-A EP-B
EPG2
EP-C
EPG3
EP-D
EPG4 EP-E
192.168.0.1 192.168.0.2 192.168.0.3 192.168.1.1 5.0.0.1
0000.0000.1111 0000.0000.2222 0000.0000.3333 0000.1111.1111 0000.5555.5555
If Unicast Routing is disabled, (BD1 in above) In above example :

• IP Learning is disabled on BD • EP-A <->
• EP-A <->
EP-B
EP-C
:
:
GOOD
GOOD
(L2
(L2
forwarding)
forwarding)
• BD SVI is disabled • EP-A <-> EP-D : FAIL (L3 forwarding)
=> Only L2 Forwarding is available • EP-A <-> EP-E : FAIL (L3 forwarding)
On
ARP Flooding
• ARP Flood On • ARP Flood Off ( = Spine-Proxy)
Tenant Unicast
Tenant to SPINE
target-IP Redirect to
VRF1 L2 Flood ---------- IP Data ---------- VRF1 LOCAL
---------- IP Data ---------- or SPINE
BD1 ARP
--------- MAC Data --------- Req BD2
-- MAC -- BD1 --------- MAC Data --------- BD2
-- MAC --
ARP Req
EPG1 EPG2 EPG3 EPG4 EPG1 EPG2 EPG3 EPG4
EP-A EP-B EP-C EP-D EP-E EP-A EP-B EP-C EP-D EP-E
Always flood ARP Request within the same BD ARP Request is handled as L3 Unicast with Target-IP
• Flood as broadcast if DST-MAC is FFFF.FFFF.FFFF • If IP is learned on ingress Leaf,

• Flood to other Leaf switches through Spine  Ingress Leaf forwards ARP Req directly to dest
• EP IP Data is not used for forwarding but still • If IP is not learned on ingress Leaf, Spine-Proxy
 Ingress Leaf forwards ARP Req to Spine
Sender-IP is learned if Unicast Routing is enabled.
Spine will forward it to Leaf on which DstIP resides
• Good option when BD is supposed to be pure L2
without Unicast Routing like legacy VLAN • If IP is not learned even on Spine,
 Drop and ARP Glean (only within BD)
※ ARP is not filtered by a contract by default
※ if dst mac is not bcast, ARP request is bridged©based
BRKACI-3545 onand/or
2019 Cisco dst itsmac regardless
affiliates. of ARP
All rights reserved. CiscoFlooding
Public mode
68
L2 Unknown Unicast
• Flood • Hardware Proxy ( = Spine-Proxy)
Tenant Tenant SPINE Unicast to Dest
Knows dst MAC? Knows dst MAC? or Drop on Spine
VRF1 No -> L2 ----------
Flood IP Data ---------- VRF1
No -> Spine
---------- IP Data ----------
BD1 --------- MAC Data --------- BD2

-- MAC -- BD1 --------- MAC Data --------- BD2
-- MAC --
EPG1 EPG2 EPG3 EPG4 EPG1 EPG2 EPG3 EPG4
EP-A EP-B EP-C EP-D EP-E EP-A EP-B EP-C EP-D EP-E
Always flood L2 Unknown Unicast within the same L2 Unknown Unicast is sent to Spine-Proxy
BD
• Flood as well as legacy VLAN. • If DST-MAC is learned on Spine,
• Flood happens locally and on other Leaf switches.  Spine forwards it directly to dest Leaf
• Good option when BD is supposed to be pure L2
without Unicast Routing as in legacy VLAN • If DST-MAC is not learned even on Spine
• Good option when there are silent L2 hosts  Drop
※EP IP Data is not used even if Leaf knows DST-IP. L2 Unknown is still L2 traffic.
L3 Unknown Multicast Flooding
• Flood Flood in ingress LEAF and • OMF (Optimized Multicast Flood)
1st Generation LEAF LEAF with a router-port 1st Generation LEAF only to router-ports
BD1 BD1
Leaf1 Leaf2 Leaf3 Leaf1 Leaf2 Leaf3
EP EP EP EP EP EP EP EP EP EP
querier querier
2nd Generation LEAF Flood 2nd Generation LEAF only to router-ports
BD1 BD1
Leaf1 Leaf2 Leaf3 Leaf1 Leaf2 Leaf3
EP EP EP EP EP EP EP EP EP EP
querier querier
L3 Unknown Multicast = IP multicast group unknown to LEAF IGMP snooping

 Controls flooding unknown IGMP snooping groups
Multi Destination Flooding This mode does not apply to
OSPF/OSPFv6, BGP, EIGRP, CDP, LACP, LLDP, ISIS,
Flooding mode for L2 multicast, Broadcast and link-local IGMP, PIM, ST-BPDU, ARP/GARP, RARP, ND
BD1 BD2 BD1 BD2

• Flood in BD EPG1 EPG2 EPG3 EPG4 EPG1 EPG2 EPG10 EPG3 EPG4
Flood within the same BD Encap1 Encap2 Encap3 Encap4 Encap5 Encap6 Encap1 Encap3 Encap4
regardless of EPG or VLAN.
Leaf1 Leaf2
Behavior change from 3.1

BD1 BD2 BD1 BD2
• Flood in Encapsulation EPG1 EPG2 EPG3 EPG4 EPG1 EPG2 EPG10 EPG3 EPG4
Flood within the same
Encap1 Encap2 Encap3 Encap4 Encap5 Encap6 Encap1 Encap3 Encap4
access encap VLAN and BD
regardless of EPG. Leaf1 Leaf2
BD1 BD2 BD1 BD2

• Drop EPG1 EPG2 EPG3 EPG4 EPG1 EPG2 EPG10 EPG3 EPG4
No Flood. Just drop. Encap1 Encap2 Encap3 Encap4 Encap5 Encap6 Encap1 Encap3 Encap4
Leaf1 Leaf2
Flood in Encapsulation
BD
EPG
L2 Unknown Unicast ARP Request Unknown L3 Mcast other multi-dest

Traffic Type
Flood Flood Flood traffic
BD1 BD1 BD1 BD1
EPG1 EPG2 EPG3 EPG1 EPG2 EPG3 EPG1 EPG2 EPG3 EPG1 EPG2 EPG3
Prior to 3.1 Encap1 Encap2 Encap1 Encap1 Encap2 Encap1 Encap1 Encap2 Encap1 Encap1 Encap2 Encap1
BD1 BD1 BD1 BD1

From 3.1 EPG1 EPG2 EPG3 EPG1 EPG2 EPG3 EPG1 EPG2 EPG3 EPG1 EPG2 EPG3
Encap1 Encap2 Encap1 Encap1 Encap2 Encap1 Encap1 Encap2 Encap1 Encap1 Encap2 Encap1
From 3.1 and 2nd generation LEAF, Including OSPF, BGP etc.
all packets are flooded within encapsulation without exceptions
※ Flood in Encap behavior stays same on 1st generation LEAF

※ If traffic is not to be flooded in the first place due to other options, it will not be flooded.
ACI BD Forwarding Option (cont.)
-- TIPS --
When Unicast Routing is OFF,
ARP Flooding is enabled internally
even though config shows off
leaf1# show vlan | grep TK:BD1
22 TK:BD1 active Eth1/1, Eth1/2
leaf1# vsh_lc -c 'show system internal eltmc info vlan 22 detail' | grep _mode
fwd_mode: bridge
arp_mode: unicast Unicast Routing - Off
hw_arp_mode: flood ARP Flooding - Off
unk_uc_mode: proxy
ARP Flooding in H/W - On
L2 Unknown Unicast - Hardware Proxy

Agenda
• Introduction

• ACI Packet Walk

Spine Proxy Summary
Forward to Forward to Flood Spine Forward to Forward to Spine Forward to
within BD Proxy Drop
local port remote leaf local port remote leaf Proxy Border Leaf
Hardware
Flood
Proxy
Dst IP is
L3OUT Routes?
Is Dst MAC on Is Dst IP on LEAF has

Local Leaf? What is BD config? Local Leaf? BD Subnets
Yes No Yes for Dst IP? No
Yes No
LEAF knows LEAF knows Dst IP

Dst MAC? Yes as EndPoint? No
Yes No
L2 L3
L2 or L3 ?
If ARP Flooding is OFF, ARP

target-IP is used for this L3 flow
Packet coming in
to Leaf © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
How to check Spine-Proxy TEP
BD Subnet (Pervasive Route)
leaf1# show ip route vrf TK:VRF1
next-hop should be
192.168.0.0/24, ubest/mbest: 1/0, attached, direct, pervasive SPINE-PROXY
leaf1# show isis dteps vrf overlay-1 | grep PROXY next-hop of Pervasive Route
10.0.16.65 SPINE N/A PHYSICAL,PROXY-ACAST-MAC is IPv4 Spine Proxy TEP
10.0.16.64 SPINE N/A PHYSICAL,PROXY-ACAST-V4
10.0.16.67 SPINE N/A PHYSICAL,PROXY-ACAST-V6
Three types of Spine Proxy TEP

• Proxy-Acast-MAC
 Spine-Proxy for L2 traffic (L2 Unknown Unicast mode “Hardware Proxy”)
• Proxy-Acast-V4
 Spine-Proxy for IPv4 traffic (includes ARP Request with ARP Flooding mode “OFF”)
• Proxy-Acast-V6
 Spine-Proxy for IPv6 traffic
ARP Glean (Silent Host Tracking)
What if even SPINE COOP doesn’t know the destination when proxy’ed?
 L2 Traffic : Drop
 L3 Traffic : ARP Glean
5 Drop
4 No COOP entry Anycast 6 Generates new packets called glean for the unknown IP
VRF overlay-1 TEP
3 Spine Proxy
TEP1 7 LEAFsTEP2
check its BD subnets TEP3
2 Hit Pervasive Route
8 LEAF generates ARP Request 8 LEAF ignores Request from Spine
1 Unicast IP
If BD subnet for the
unknown IP doesn’t present
If BD subnet for the unknown
IP presents on LEAF
Agenda
• Introduction

• ACI Packet Walk

ACI Forwarding Table & Software Architecture
on the Supervisor Engine: APIC
APIC
EPG BD L3OUT Contract
EPM (EndPoint Manager): manages host MAC & IP learning
uRIB (Unicast RIB): contains the unicast routing information

SUP
LEAF
Pervasive Static Routing (ibash, vsh)
Policy Mgr : manages contracts between EPGs or L3OUT.
Route Route Protocol
mac, ip contract
ip
on the Linecards: EPM uRIB Policy Mgr
EPMc (EndPoint Manager Client): learns host MAC & IP
LEAF
addresses from hardware(dataplane) via HAL EPMc uFIB ACL QoS
uFIB (Unicast FIB): programs the hardware unicast routing

table via HAL ASIC USD / HAL
ACL QoS : programs contracts via HAL Cloud Scale ASIC

Line Card (vsh_lc)
HAL (Hardware Abstraction Layer): pass the messages
※ ASIC USD (User Space Driver) is only for 1st generation ASIC
between hardware(ASIC) and software
ACI Forwarding Table & Software Architecture
on the Supervisor Engine: ibash (default) APIC
APIC
EPG BD L3OUT Contract
EPM show endpoint
show system internal epm ….
SUP
LEAF
uRIB show ip route vrf xxx
Pervasive Static Routing (ibash, vsh)
Policy Mgr show system internal policymgr …. Route Route Protocol
mac, ip contract
ip
on the Linecards: vsh_lc EPM uRIB Policy Mgr
EPMc show system internal epmc …
LEAF
EPMc uFIB ACL QoS
uFIB show forwarding …

ASIC USD / HAL
ACL QoS show system internal aclqos … Cloud Scale ASIC

Line Card (vsh_lc)
HAL show platform internal hal …
※ ASIC USD (User Space Driver) is only for 1st generation ASIC
※ LST: Local Station Table, GST: Global Station Table
LEAF ASIC Generations ※ FP Tile: Forwarding and Policy Tile

※ HAL: Hardware Abstraction Layer
1st generation 2nd generation (or later) Tile X: IP

Dest EP Remote Tile Y: MAC
Lookup EP Learn etc.
To SPINE To SPINE
GST GST
Cisco ASIC
ingress egress
Cloud Scale FP
LST LST CPU Tiles
ASIC
ingress egress
CPU Broadcom
Local EP Dest EP
Learn Lookup
N9K-C9332PQ N9K-C9396PX • Complete separation of N9K-C93180YC-EX N9K-C93180YC-FX • More flexible/scalable

N9K-C9372PX N9K-C9396TX + Ingress and Egress N9K-C93108TC-EX N9K-C93108TC-FX with configurable tiles
N9K-C9372PX-E N9K-C93120TX + Source Learn and N9K-C93180LC-EX N9K-C9348GC-FXP • Abstracted with HAL
N9K-C9372TX N9K-C93128TX Destination Lookup N9K-C9336C-FX2 • Tile X for both source
N9K-C9372TX-E • Separate GST/LST for IP N9K-C93240YC-FX2 learn and destination
and MAC lookup
※ number of ASIC per card depends on model
SPINE ASIC Generations
1st generation 2nd generation (or later)
SUP Fabric card SUP Fabric card

COOP Database COOP Database
CPU Broadcom CPU Cisco ASIC
Line card Line card
Cloud Cloud
Cisco Cisco TEP Information TEP Information
Scale Scale
ASIC ASIC
ASIC ASIC
Line card Fabric card Line card Fabric card

N9K-X9736PQ N9K-C9504-FM N9K-X9732C-EX N9K-C9504FM-E
N9K-C9508-FM N9K-X9736C-FX N9K-C9508FM-E
Box spine N9K-C9516-FM Box spine N9K-C9508FM-E2
N9K-C9336PQ N9K-C9364C N9K-C9516FM-E2
N9K-C9332C
Agenda
• Introduction

• ACI Packet Walk

• Logical Topology • Physical Topology
COOP Table
--- empty ---
VRF 1
VRF1 VRF1
BD1 192.168.0.254/24 BD1 192.168.0.254/24
SVI SVI (secondary)
192.168.0.254/24
BD 1 192.168.1.254/24 BD1 192.168.1.254/24 BD1 192.168.1.254/24
EPG 1 EPG 2 EP Table EP Table

--- empty --- --- empty ---
EP-A EP-B
192.168.0.1 192.168.1.1 EP-A EP-B
0000.0000.1111 0000.0000.2222 192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
ARP Flooding : OFF ARP Table ARP Table

Unicast Routing : ON --- empty --- --- empty ---
PING : EP-A (192.168.0.1) -> EP-B (192.168.1.1)

1. ARP Request to default GW
EP-A Data 1. ARP Req is sent out to GW (192.168.0.254)
2. LEAF1 learns src IP/MAC from ARP.
notified by Leaf1 COOP Table
 Leaf1 notify that to Spine COOP
--- empty ---
3. LEAF1 sends ARP reply to EP-A.
VRF1
BD1 192.168.0.254/24
BD1 192.168.1.254/24
Learn EP-A
EP Table EP Table
--- empty --- --- empty ---
ARP ARP
Request Reply
(192.168.0.254)
EP-A EP-B
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
ARP Table ARP Table

--- empty --- --- empty ---
Learn Default GW MAC

CLI notes (VLAN/EPG/BD programming)
LEAF1# show vlan id 69 extended
---- -------------------------------- --------- -------------------------------
69 TK:APP1:EPG1 active Eth1/11
VLAN and
VLAN Type Vlan-mode Encap I/F mapping
---- ----- ---------- -------------------------------
69 enet CE vlan-753
LEAF1# show system internal epm vlan 69

+----------+---------+-----------------+----------+------+----------+-----------
(Type Value) Encap Count BD PI-VLAN
+----------+---------+-----------------+----------+------+----------+-----------
69 FD vlan 802.1Q 753 8994 68 68 1
LEAF1# show ip interface vlan 68 vrf TK:VRF1

vlan68, Interface status: protocol-up/link-up/admin-up, iod: 86, mode: pervasive BD
IP address: 192.168.0.254, IP subnet: 192.168.0.0/24 Pervasive GW
IP address: 192.168.1.254, IP subnet: 192.168.1.0/24 secondary
CLI notes (Source learning)
LEAF1# show endpoint ip 192.168.0.1 detail
Legend:
EndPoint Table
O - peer-attached H - vtep a - locally-aged S - static
V - vpc-attached p - peer-aged L - local M - span (= host table)
s - static-arp B - bounce
+--------------------+---------------+-----------------+--------------+-------------+-------------------+
VLAN/ Encap MAC Address MAC Info/ Interface Endpoint Group
Domain VLAN IP Address IP Info Info
+--------------------+---------------+-----------------+--------------+-------------+-------------------+
69 vlan-753 0000.0000.1111 L eth1/11 TK:APP1:EPG1
TK:VRF1 vlan-753 192.168.0.1 L eth1/11
LEAF1# show ip route vrf TK:VRF1

*via 10.0.184.65%overlay-1, [1/0], 01w08d, static
192.168.0.254/32, ubest/mbest: 1/0, attached, pervasive
*via 192.168.0.254, vlan68, [1/0], 01w08d, local, local RIB
192.168.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive (= LPM table)
*via 192.168.1.254, vlan68, [1/0], 01w08d, local, local
CLI notes (COOP sync)
LEAF1# show vrf TK:VRF1 detail extended | grep vxlan VRF VNID
Encap: vxlan-2228224

---- -------------------------------- --------- -------------------------------
68 TK:BD1 active Eth1/11
BD VNID
---- ----- ---------- -------------------------------
fab2-spine1# show coop internal info ip-db key 2228224 192.168.0.1

COOP on SPINE
IP address : 192.168.0.1 (with IP as a key)
Vrf : 2228224
Flags : 0 fab2-spine1# show coop internal info repo ep key
EP bd vnid : 16711542 16711542 0000.0000.1111
EP mac : 00:00:00:00:11:11
Publisher Id : 10.0.8.95 EP bd vnid : 16711542
URIB Tunnel Info EP mac : 00:00:00:00:11:11 COOP on SPINE
Num tunnels : 1 Vrf vnid : 2228224 (with MAC as a key)
Tunnel address : 10.0.8.95 publisher id : 10.0.8.95
Tunnel ref count : 1 Real IPv4 EP : 192.168.0.1
※ command outputs are sniped
COOP Table
1. ARP Req is sent out to GW (192.168.0.254)
192.168.0.1
2. LEAF1 learns src IP/MAC from ARP.
0000.0000.1111
 Leaf1 notify that to Spine COOP
-> LEAF 1
Spine does not know
VRF1
192.168.1.1 either 2. ICMP from EP-A to EP-B (192.168.1.1)
BD1 192.168.0.254/24
-> drop 1. Dst MAC is ACI MAC (BD SVI router-mac)
BD1 192.168.1.254/24  L3 Lookup within VRF
EP Table 2. LEAF1 doesn’t know 192.168.1.1 but knows
192.168.0.1 it’s subnet (192.168.1.0/254)
0000.0000.1111 EP Table  Spine-Proxy
-> eth 1/11 --- empty --- 3. Spine COOP lookup
Proxy 1. COOP doesn’t know 192.168.1.1 either
 drop
ICMP
(192.168.1.1)
EP-A EP-B
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
ARP Table ARP Table

192.168.0.254 -> ACI MAC --- empty ---
CLI notes (Destination lookup)
Legend: EndPoint Table
--- snip --- (= host table)
+---------------------+-------------+-----------------+--------------+-------------+-----------------+
+---------------------+-------------+-----------------+--------------+-------------+-----------------+
<----- no output ----->

192.168.0.254/32, ubest/mbest: 1/0, attached, pervasive RIB
*via 192.168.0.254, vlan68, [1/0], 01w08d, local, local (= LPM table)
*via 192.168.1.254, vlan68, [1/0], 00:00:06, local, localn
LEAF1# show isis dteps vrf overlay-1 | grep 10.0.184.65 NextHop is

10.0.184.65 SPINE N/A PHYSICAL,PROXY-ACAST-V4 IPv4 Proxy
LEAF1# show ip route 10.0.184.65 vrf overlay-1

10.0.184.65/32, ubest/mbest: 2/0 NextHop I/F
*via 10.0.48.97, eth1/50.2, [115/2], 02w14d, isis-isis_infra, L1 (to SPINE)
*via 10.0.48.94, eth1/49.1, [115/2], 02w14d, isis-isis_infra, L1 90
BRKACI-3545 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
CLI notes (Coop Check)
LEAF1# show vrf TK:VRF1 detail extended | grep vxlan VRF VNID
Encap: vxlan-2228224
COOP on
LEAF1# show coop internal info ip-db key 2228224 192.168.1.1 SPINE
Key not found in ip db
COOP Table
1. ARP Req is sent out to GW (192.168.0.254)
192.168.0.1 EP-B Data is 2. LEAF1 learns src IP/MAC from ARP.
0000.0000.1111 notified by LEAF2  Leaf1 notify that to Spine COOP
-> LEAF 1
VRF1
2. ICMP from EP-A to EP-B (192.168.1.1)
BD1 192.168.0.254/24
1. Dst MAC is ACI MAC (BD SVI router-mac)
BD1 192.168.1.254/24  L3 Lookup within VRF
EP Table 2. LEAF1 doesn’t know 192.168.1.1 but knows
192.168.0.1 Learn EP-B it’s subnet (192.168.1.0/254)
0000.0000.1111 EP Table  Spine-Proxy
-> eth 1/11 ARP Glean --- empty --- 3. Spine COOP lookup
1. COOP doesn’t know 192.168.1.1 either
 drop
ARP Request ARP Reply 4. ARP Glean for 192.168.1.1 to each LEAFs
(192.168.1.1) 1. LEAF1 and LEAF2 has a BD with
EP-B 192.168.1.0/24 subnet
EP-A
192.168.1.1  Both LEAFs generates an ARP Request for
192.168.0.1
0000.0000.2222 192.168.1.1 out of ports on the BD
0000.0000.1111
2. EP-B sends ARP Reply to LEAF2
ARP Table ARP Table 3. LEAF2 learns EP-B IP/MAC
192.168.0.254 -> ACI MAC --- empty ---  LEAF2 notifies that to Spine COOP
CLI notes (LEAF2 VLAN/EPG/BD programming)
---- -------------------------------- --------- -------------------------------
10 TK:APP1:EPG2 active Eth1/11
VLAN and
VLAN Type Vlan-mode Encap I/F mapping
---- ----- ---------- -------------------------------
10 enet CE vlan-754
LEAF2# show system internal epm vlan 10

+----------+---------+-----------------+----------+------+----------+-----------
(Type Value) Encap Count BD PI-VLAN
+----------+---------+-----------------+----------+------+----------+-----------
10 FD vlan 802.1Q 754 8987 7 9 1
LEAF2# show ip interface vlan 9 vrf TK:VRF1

vlan9, Interface status: protocol-up/link-up/admin-up, iod: 80, mode: pervasive BD
IP address: 192.168.0.254, IP subnet: 192.168.0.0/24 Pervasive GW
IP address: 192.168.1.254, IP subnet: 192.168.1.0/24 secondary
CLI notes (LEAF2 Source learning)
Legend:
EndPoint Table
O - peer-attached H - vtep a - locally-aged S - static
V - vpc-attached p - peer-aged L - local M - span (= host table)
s - static-arp B - bounce
+--------------------+---------------+-----------------+--------------+-------------+-------------------+
+--------------------+---------------+-----------------+--------------+-------------+-------------------+
TK:VRF1 vlan-754 192.168.1.1 L eth1/11

192.168.1.254/32, ubest/mbest: 1/0, attached, pervasive RIB
*via 192.168.1.254, vlan9, [1/0], 01w08d, local, local (= LPM table)
*via 192.168.1.254, vlan9, [1/0], 01w08d, local, local
4. EP-A sends 2nd ICMP to EP-B (192.168.1.1)
COOP Table 1. Dst MAC is ACI MAC (BD SVI router-mac)
192.168.0.1 192.168.1.1
 L3 Lookup within VRF
0000.0000.1111 0000.0000.2222
2. LEAF1 still doesn’t know 192.168.1.1 but knows
-> LEAF 1 -> LEAF 2
it’s subnet (192.168.1.0/254)
 Spine-Proxy
Spine knows VRF1 5. Spine COOP lookup for 2nd ICMP
192.168.1.1 BD1 192.168.0.254/24 1. Now COOP knows 192.168.1.1
BD1 192.168.1.254/24 2. Spine sends it to Leaf2
EP Table EP Table
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
-> eth 1/11 -> eth 1/11
Proxy
2nd ICMP
(192.168.1.1)
EP-A EP-B
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
ARP Table ARP Table

192.168.0.254 -> ACI MAC --- empty ---
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
-> LEAF 1 -> LEAF 2
it’s subnet (192.168.1.0/254)
 Spine-Proxy
VRF1 5. Spine COOP lookup for 2nd ICMP
BD1 192.168.0.254/24 1. Now COOP knows 192.168.1.1
EP Table EP Table
192.168.0.1 192.168.1.1 6. LEAF2 learns EP-A as a remote EP
0000.0000.1111 0000.0000.2222  The packet is routed = sent out with VRF
-> eth 1/11 Learn -> eth 1/11 VNID.
EP-A IP Only IP is learned
7. LEAF2 sends it out to EP-B
EP-A EP-B
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
ARP Table ARP Table

192.168.0.254 -> ACI MAC --- empty ---
CLI notes (remote EP learning)
LEAF2# show endpoint vrf TK:VRF1 detail
Legend: Learn Remote EP
--- snip ---
+-------------------------+---------------+-----------------+--------------+-------------+-------------------+
+-------------------------+---------------+-----------------+--------------+-------------+-------------------+
TK:VRF1 192.168.0.1 tunnel11
TK:VRF1 vlan-754 192.168.1.1 L eth1/11
LEAF2# show int tunnel 11 | grep dest Tunnel points to

Tunnel destination 10.0.8.95 LEAF1
LEAF2# acidiag fnvread | grep 8.95
101 1 LEAF1 ABC1234DEFG 10.0.8.95/32 leaf active 0
LEAF2# show sys int epm endpoint ip 192.168.0.1

MAC : 0000.0000.0000 ::: Num IPs : 1
IP# 0 : 192.168.0.1 ::: IP# 0 flags :
Vlan id : 0 ::: Vlan vnid : 0 ::: VRF name : TK:VRF1
BD vnid : 0 ::: VRF vnid : 2228224 Packet was L3 traffic
Phy If : 0 ::: Tunnel If : 0x1801000b -> MAC won’t be learned
Interface : Tunnel11
Flags : 0x80004400 ::: sclass : 5479 ::: Ref count : 3
--- snip ---
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
-> LEAF 1 -> LEAF 2
it’s subnet (192.168.1.0/254)
 Spine-Proxy
BD1 192.168.0.254/24
EP Table 1. Now COOP knows 192.168.1.1
192.168.1.1
EP Table 0000.0000.2222
192.168.0.1 -> eth 1/11
6. LEAF2 learns EP-A as a remote EP
0000.0000.1111 192.168.0.1
 The packet is routed = sent out with VRF
-> eth 1/11 -> LEAF 1
VNID.
Only IP is learned
ARP ARP 7. LEAF2 sends it out to EP-B
Request Reply
(192.168.1.254) 8. EP-B resolves ARP for its gateway
EP-A EP-B (192.168.1.254)
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
ARP Table ARP Table

192.168.0.254 -> ACI MAC --- empty ---
Learn Default GW MAC

192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
-> LEAF 1 -> LEAF 2
it’s subnet (192.168.1.0/254)
 Spine-Proxy
BD1 192.168.0.254/24 1. Now COOP knows 192.168.1.1
Learn EP-B IP EP Table
192.168.1.1
EP Table 0000.0000.2222
192.168.0.1 -> eth 1/11
0000.0000.1111 192.168.0.1
-> eth 1/11 -> LEAF 1
VNID.
To Leaf1 Only IP is learned
ICMP reply
(to 192.168.0.1) 8. EP-B resolves ARP for its gateway
EP-A EP-B (192.168.1.254)
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222 9. EP-B sends ICMP reply
1. LEAF2 already knows where EP-A IP is
ARP Table ARP Table  Directly sends it to LEAF1
192.168.0.254 -> ACI MAC 192.168.1.254 -> ACI MAC 10. LEAF1 learns EP-B IP as a remote EP
 Only IP is learned as well
CLI notes (remote EP learning)
LEAF1# show endpoint vrf TK:VRF1
Legend:
Learn Remote EP
--- snip ---
+-----------------------------------+---------------+-----------------+--------------+-------------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
TK:VRF1 192.168.1.1 tunnel6
69 vlan-753 0000.0000.1111 L eth1/11
TK:VRF1 vlan-753 192.168.0.1 L eth1/11
LEAF1# show int tunnel 6 | grep dest

Tunnel points to
Tunnel destination 10.0.8.90
LEAF2
LEAF1# acidiag fnvread | grep 8.95
102 1 LEAF2 ABC5678DEFG 10.0.8.90/32 leaf active 0
LEAF1# show sys int epm endpoint ip 192.168.1.1

MAC : 0000.0000.0000 ::: Num IPs : 1
IP# 0 : 192.168.1.1 ::: IP# 0 flags :
Vlan id : 0 ::: Vlan vnid : 0 ::: VRF name : TK:VRF1 Packet was L3 traffic
BD vnid : 0 ::: VRF vnid : 2228224 -> MAC won’t be learned
Phy If : 0 ::: Tunnel If : 0x18010006
Interface : Tunnel6
Flags : 0x80004400 ::: sclass : 49160 ::: Ref count : 3
--- snip ---
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
-> LEAF 1 -> LEAF 2
it’s subnet (192.168.1.0/254)
 Spine-Proxy
EP Table BD1 192.168.0.254/24
EP Table 1. Now COOP knows 192.168.1.1
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222
-> eth 1/11 -> eth 1/11
192.168.1.1 192.168.0.1
-> LEAF 2 -> LEAF 1
VNID.
Only IP is learned
ICMP reply
(to 192.168.0.1)
8. EP-B resolves ARP for its gateway
EP-A EP-B (192.168.1.254)
192.168.0.1 192.168.1.1
0000.0000.1111 0000.0000.2222 9. EP-B sends ICMP reply
1. LEAF2 already knows where EP-A IP is
ARP Table ARP Table  Directly sends it to LEAF1
192.168.0.254 -> ACI MAC 192.168.1.254 -> ACI MAC 10. LEAF1 learns EP-B IP as a remote EP
 Only IP is learned as well
Appendix
ELAM Assistant in ACI AppCenter
Interested in more detail packet forwarding verification ?
 ELAM Assistant!!
https://aciappcenter.com
ELAM (Embedded Logic Analyzer Module)
• Perform an ASIC level packet capture
ELAM Assistant
• You can perform ELAM like a TAC
engineer!
• With a nicely formatted result report
Detail Explanations:
• https://aciappcenter.cisco.com/elam-assistant-beta-2-
2-1n.html
 How to use video, pictures
 A download link for ELAM Assistant
• https://learningnetwork.cisco.com/docs/DOC-34985
 ACI webinar for ELAM Assistant
ELAM Assistant in ACI AppCenter (example)
1. Perform ELAM
Set Parameters
Triggered!!
ELAM Assistant in ACI AppCenter (example)
2. Read a report
Zoom
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKACI-3545
Complete your online
session survey
• Please complete your Online Session
Survey after each session
• Complete 4 Session Surveys & the Overall
Conference Survey (available from
Thursday) to receive your Cisco Live T-
shirt
• All surveys can be completed via the Cisco
Events Mobile App or the Communication
Stations
Don’t forget: Cisco Live sessions will be available for viewing

on demand after the event at ciscolive.cisco.com
Continue Your Education
Demos in Meet the Related

Walk-in
the Cisco engineer sessions
self-paced
Showcase labs 1:1
meetings
Thank you

Aci 3545

Uploaded by

Copyright:

Available Formats

Aci 3545

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Aci 3545

Uploaded by

Copyright:

Available Formats

BRKACI-3545

Mastering ACI Forwarding

Takuya Kishida, Technical Leader, Service

• ACI Forwarding components

• ACI Packet Walk

VxLAN Virtual eXtensible LAN VNID Virtual Network Identifier

• ACI Forwarding components

• ACI Packet Walk

TEP1 TEP2 TEP3

VRF1 VRF1 VRF1

BD1 BD1 BD2 BD2 L3OU

EPG1 EPG2 EPG EPG EPG

EP1-1 EP2-1 EP2-2 EP3-1 EP3-2 EP4-1 WAN

dMACo sMACo sIPo dIPo VxLAN dMACi sMACi sIPi dIPi

TEP1 TEP2 TEP3

VRF1 VRF1 VRF1

BD1 BD1 BD2 BD2 L3OU

EPG1 EPG2 EPG EPG EPG

EP1-1 EP2-1 EP2-2 EP3-1 EP3-2 EP4-1 WAN

Scenario 1 : source LEAF knows the destination

Scenario 2 : source LEAF knows the destination ( on another LEAF X )

EP1-1 EP2-1 EP2-2 EP3-1 EP3-2 EP4-1 WAN

TEP1 TEP2 TEP3

VRF1 VRF1 VRF1

BD1 BD1 BD2 BD2 L3OU

Local EndPoint EPG1 EPG2 EPG EPG EPG

Scenario 1 : source LEAF knows the destination

Scenario 2 : source LEAF knows the destination ( on another LEAF X )

EP1-1 EP2-1 EP2-2 EP3-1 EP3-2 EP4-1 WAN

TEP1 TEP2 TEP3

1 Send to LEAF2 (TEP2)

BD1 BD1 BD2 BD2 L3OU

2 Add VxLAN header

1 Send to LEAF2 (TEP2)

BD1 BD1 BD2 BD2 L3OU

2 Add VxLAN header

1 Send to LEAF2 (TEP2)

BD1 BD1 BD2 BD2 L3OU

2 Add VxLAN header

BD1 BD1 BD2 BD2 L3OU

2 Add VxLAN header

BD1 BD1 BD2 5 BD2

Scenario 1 : source LEAF knows the destination

Scenario 2 : source LEAF knows the destination ( on another LEAF X )

EP1-1 EP2-1 EP2-2 EP3-1 EP3-2 EP4-1 WAN

TEP1 TEP2 TEP3

1 Send to SPINE (Anycast TEP)

BD1 BD1 BD2 BD2 L3OU

2 Add VxLAN header TEP1 TEP2 TEP3

1 Send to SPINE (Anycast TEP)

BD1 BD1 BD2 BD2 L3OU

2 Add VxLAN header TEP1 TEP2 TEP3

1 Send to SPINE (Anycast TEP)

BD1 BD1 BD2 BD2 L3OU

sIPo 5 Send to LEAF2 (TEP2)

1 Send to SPINE (Anycast TEP)

BD1 BD1 BD2 BD2 L3OU

sIPo 5 Send to LEAF2 (TEP2)

BD1 BD1 BD2 BD2 L3OU