BRKDCN 3900

A Network Engineer’s
Blueprint for ACI Forwarding
Joe Young, ACI Technical Leader, Customer Experience
BRKDCN-3900
Cisco Webex App
Questions?
Use Cisco Webex App to chat
with the speaker after the session
How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install the Webex App or go directly to the Webex space Enter your personal notes here
4 Enter messages/questions in the Webex space
Webex spaces will be moderated

until February 24, 2023.
BRKDCN-3900 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
• What’s Different About ACI Forwarding?
• (iVXLAN, contracts, endpoint learning)
• Proxy Forwarding
ACI Forwarding Tables
Agenda •
• Endpoint tables, routing tables, hardware
lookups
• Understanding the Configuration Options
• The Anatomy of an ACI Switch
• Understanding the Tools
• UI Tools
• Elam
• Ftriage
Agenda •
•
Span / ERSPAN
Flow Telemetry / netflow
• Debugging and Walking Through ACI
Flows
• (Routed, Bridged, BUM, Proxied)
Glossary of Acronymns
VxLAN packet acronyms
Acronyms Definitions
Acronyms Definitions
ACI Application Centric Infrastructure
dXXXo Outer Destination XXX
APIC Application Policy Infrastructure Controller (dIPo = Outer Destination IP)
sXXXo Outer Source XXX
EP Endpoint
(sIPo = Outer Source IP)
EPG Endpoint Group dXXXi Inner Destination XXX
(dIPi = Inner Destination IP)
BD Bridge Domain
sXXXi Inner Source XXX
VRF Virtual Routing and Forwarding (sIPi = Inner Source IP)
COOP Council of Oracle Protocol GIPo Outer Multicast Group IP
VxLAN Virtual eXtensible LAN VNID Virtual Network Identifier
What’s Different
About ACI
Forwarding?
What is “Application Centric”?
• Traditional networks use ACL’s to
classify traffic
• Usually based on L3 or L2 addresses
• Makes security decisions (permit,
Host1
deny, log, etc) EPG1
• Makes forwarding decisions (policy

App
based routing) Host2
EPG4
EPG2
• ACI can classify traffic based on its

EPG Host3
EPG3
• Traffic inherits the forwarding and

security policy of the EPG
How is “Application Centric” Achieved?
Sources and Destinations Must be Classified into EPG’s
Endpoints Policy-Prefixes PcTags Contracts
• Used by App EPG’s • Used by External • The security ID of • Defines security

EPG’s an EPG and sometimes
• Represents the forwarding (pbr)
network identity of an • Classifies destination • Used in contracts. policy between
end device by longest prefix Ex: Permit PcTag epgs
match 1000 to PcTag
• Learned dynamically 2000 • Essentially an ACL
or configured • Also used for between PcTags
statically shared-services • Sclass/dclass imply
PcTag direction • Consumer/Provider
• Configured rather than src/dest
Vlan Types
※ PI-VLAN : Platform Independent VLAN
VLAN ID for external devices Internal ID on LEAF For forwarding

(user configured value) (not shared across LEAFs) (global value for entire fabric)
Access Encap VLAN PI-VLAN VxLAN ID PI-VLAN Access Encap VLAN

(VNID)
LEAF 1 LEAF 2
VRF1 2523136
VRF1
BD1 For BD SVI

BD1
17 16613259 31
EPG1 vxlan- vxlan-

EPG1
20 8388608 33
8388608 8388608
vlan-5 19 12661 30 vlan-5
EP EP EP EP
What is an Endpoint?
At the APIC level an Endpoint is a Mac address with zero or more
IP/IPv6 Addresses
fvCEp
<epg-dn>/cep-00:00:00:00:0a
fvIp
<epg-dn>/cep-00:00:00:00:0a/ip-[10.0.0.10]
At the Switch level an Endpoint is a Mac address OR an IP/IPv6

Address
Hardware Entry #1
Endpoint
Mac – aaa.bbb.ccc
IP – 10.0.0.1 IP
Hardware Entry #2
What is an Endpoint?
An Endpoint joins both forwarding and security policy
Local Learn VNID Remote Learn
leaf103# show system internal epm end ip 192.168.200.11 leaf103# show system internal epm endpoint ip 192.168.100.10
MAC : 0000.1111.2222 ::: Num IPs : 1 MAC : 0000.0000.0000 ::: Num IPs : 1
IP# 0 : 192.168.200.11 ::: IP# 0 flags : ::: l3-sw-hit: No IP# 0 : 192.168.100.10 ::: IP# 0 flags : ::: l3-sw-hit: No
Vlan id : 2 ::: Vlan vnid : 12661 ::: VRF name : CL2022:vrf1 Vlan id : 0 ::: Vlan vnid : 0 ::: VRF name : CL2022:vrf1
BD vnid : 16613259 ::: VRF vnid : 2523136 BD vnid : 0 ::: VRF vnid : 2523136
Phy If : 0x40018000 ::: Tunnel If : 0 Phy If : 0 ::: Tunnel If : 0x18010001
Interface : Ethernet1/25/1 Interface : Tunnel1
Flags : 0x80005c04 ::: sclass : 32771 ::: Ref count : 5 Flags : 0x80004400 ::: sclass : 49154 ::: Ref count : 3
EP Create Timestamp : 11/01/2021 14:06:25.769904 EP Create Timestamp : 11/04/2021 16:38:13.570615
EP Update Timestamp : 11/04/2021 18:51:54.387104 EP Update Timestamp : 11/04/2021 18:51:54.386595
EP Flags : local|IP|MAC|host-tracked|sclass|timer| EP Flags : IP|sclass|timer|
Interface/TEP
PcTag
What is a TEP? (Tunnel Endpoint)
• IP addresses allocated for overlay communication
• VXLAN Traffic is sent to the TEP + VNID of destination
Most Common TEP Types
TEP Type What is it? What is it for?

Unique Overlay IP Address for each Non-vpc dataplane, l3out communication, apic-leaf
Physical TEP (PTEP) individual Leaf/Spine comm, etc
Unique Overlay IP Address for each Traffic destined to endpoints that are connected
VPC TEP (VTEP) VPC Pair behind VPC
Spine Anycast IP’s used for proxy Leafs send to these TEPs when doing proxy
Proxy TEP traffic forwarding
a-leaf101# show ip interface loopback0

IP Interface Status for VRF "overlay-1"
lo0, Interface status: protocol-up/link-up/admin-up, iod: 4, mode: ptep
What are Tunnels?
• Leafs/Spines Install Tunnel Interface to each known TEP.
• Used for VXLAN Dataplane How are Tunnels Learned?
leaf# moquery -c tunnelIf -f 'tunnel.If.id=="tunnel1"'
id : tunnel1
dest : 10.0.72.67
Dataplane Learns idRequestorDn : sys/*/db-dtep/dtep-[10.0.72.67]
id : tunnel1
Through BGP dest : 10.0.72.64
(l3out routes) idRequestorDn : sys/bgp/*/db-dtep/dtep-[10.0.72.64]
# tunnel.If
id : tunnel1
Local POD ISIS dest : 10.0.152.64
Database idRequestorDn : sys/isis/*/lvl-l1/db-dtep/dtep-[10.0.152.64]
How is an Endpoint Learned? How does the Egress leaf classify
traffic into the correct EPG?
Spine Spine
4 Leaf Installs Remote 3
Endpoint learn from Leaf Updates COOP
dataplane Database on spines
2
Ingress leaf classifies smac and sIP
(if IP learning enabled) into EPG
based on some info such as vlan.
Leaf Leaf Leaf Endpoint entry installed
EP2 EP1 1
10.1.1.2/24 10.1.1.1/24 Source sends some
4444.5555.6666 0000.1111.2222 type of traffic
Overlay iVXLAN Bit pos 4 – Source Policy Applied
Bit pos 5 – Destination Policy Applied
ACI uses VXLAN with some additional bits Bit pos 7 – Don’t learn
Spine Spine
VNID (3 bytes) PcTag/Sclass (2 bytes) Flags (1 byte)
L4/Payload Proto DIP SIP 802.1Q SMAC DMAC VXLAN DSCP DIP SIP 802.1Q SMAC DMAC
Leaf Leaf Leaf

Dataplane VXLAN
contains all information
EP2 EP1 needed for endpoint
10.1.1.2/24 10.1.1.1/24
4444.5555.6666 classification
0000.1111.2222
How is an Endpoint Learned?
Spine
leaf103# show system internal epm vlan 2 detail
VLAN 2
VLAN type : FD vlan
hw id : 34 ::: sclass : 32771
access enc : (802.1Q, 100)
fabric enc : (VXLAN, 12661)
Object store EP db version : 4
Leaf BD vlan id : 1 ::: BD vnid : 16613259 ::: VRF vnid : 2523136
Valid : Yes ::: Incomplete : No ::: Learn Enable : Yes
leaf103# show vlan encap-id 100

EP Sends Encap Vlan 100
some traffic
VLAN Name Status Ports
EP1 ---- ---------------------- ---------
10.1.1.1/24 PI-VLAN 2 CL2022:ap1:epg2 active Eth1/25/3
0000.1111.2222
Checking Endpoints
Reference commands can be run from leafs or apics
#Check object model for Mac Address Endpoint
moquery -c epmMacEp -f 'epm.MacEp.addr=="00:00:AA:AA:BB:BB"'
#Check object model for IP Address Endpoint

moquery -c epmIpEp -f 'epm.IpEp.addr=="192.168.200.11"'
Reference commands can be run from leafs only

#Check endpoint manager process directly
show system internal epm endpoint mac 0000.aaaa.bbbb
show system internal epm endpoint ip 192.168.200.11
#Check hardware level endpoint process directly

vsh_lc -c "show system internal epmc endpoint mac 0000.aaaa.bbbb"
vsh_lc -c "show system internal epmc endpoint ip 192.168.200.11"
How is an Endpoint Learned?
The Leaf Updates COOP on Spines
Spine spine1005# show coop internal info ip-db | grep -B 1 -A 15
192.168.200.11
------------------------------
IP address : 192.168.200.11
Vrf : 2523136 VNID info should match
Update Flags : 0 the info on leaf
COOP EP bd vnid : 16613259
EP mac : 00:00:AA:AA:BB:BB
Publisher Id : 10.0.64.70
Leaf Record timestamp : 11 05 2021 17:02:56 217794556
Publish timestamp : 11 05 2021 17:02:56 220584642
Seq No: 0
Remote publish timestamp: 01 01 1970 00:00:00 0
URIB Tunnel Info
Num tunnels : 1
Tunnel address : 10.0.64.70
EP1 Tunnel ref count : 1
10.1.1.1/24
0000.1111.2222 Leaf TEP that owns this EP:
#From APIC
moquery -c ipv4Addr -f 'ipv4.Addr.addr=="10.0.64.70"'
Checking COOP
Reference commands can be run from spines or apics
Query COOP for l2 entry:
moquery -c coopEpRec -f 'coop.EpRec.mac=="00:00:AA:AA:BB:BB"'
Query COOP for l3 entry and get parent l2 entry:

moquery -c coopEpRec -x rsp-subtree=children 'rsp-subtree-filter=eq(coopIpv4Rec.addr,"1.1.1.1")' rsp-subtree-
include=required
Query COOP for l3 only entry (such as an SVI IP):

moquery -c coopIpOnlyRec -f 'coop.IpOnlyRec.addr=="192.168.100.10"'
Query COOP for l3 ep:

moquery -c coopIpv4Rec -f 'coop.Ipv4Rec.addr=="192.168.100.10"'
How is Traffic Classified with no EP Learn?
In most of these cases, the pcTag is based on a policy-prefix lookup
• There will be no endpoint learn in several cases
• Source/dest is behind an l3out
• Source/dest is in another vrf
• Endpoint learning is disabled by some option
• If ingress leaf doesn’t apply policy, egress leaf should (indicated via
policy-applied bits in ivxlan header)
Destination Behind L3out
leaf101# vsh_lc -c "show forwarding route 10.99.99.100 platform vrf CL2022:vrf1"
!
Policy Prefix 10.99.99.0/24
!
vrf: 16(0x10), routed_if: 0x0 epc_class: 32772(0x8004)
Classification based on
longest l3out policy prefix
Destination is unknown and is proxied
leaf101# show ip route 192.168.200.20 vrf CL2022:vrf1
192.168.200.0/24, ubest/mbest: 1/0, attached, direct, pervasive “Pervasive” indicates this is a

*via 10.0.176.66%overlay-1, [1/0], 4d05h, static, tag 4294967294 BD or EPG subnet (fvSubnet).
recursive next hop: 10.0.176.66/32%overlay-1 Send to spine proxy-addr
leaf101# vsh_lc -c "show forwarding route 192.168.200.20 platform vrf CL2022:vrf1"

!
!
Vrf: 16(0x10), routed_if: 0x0 epc_class: 1(0x1)
Don’t apply policy, Forward to proxy Anycast!

-pcTag of 1 indicates the fabric owns leaf101# show isis dtep vrf overlay-1 | egrep "Type|PROXY"
the subnet, don’t apply policy DTEP-Address Role Encapsulation Type
-policy applied flags not set in ivxlan 10.0.176.66 SPINE N/A PHYSICAL,PROXY-ACAST-V4
header 10.0.176.65 SPINE N/A PHYSICAL,PROXY-ACAST-MAC
10.0.176.64 SPINE N/A PHYSICAL,PROXY-ACAST-V6
Destination is in shared services Shared Services
provider EPG (different vrf) Classification
leaf# show ip route 192.168.255.10 vrf CL2022:vrf1
192.168.255.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.176.66%overlay-1, [1/0], static, tag !!!, rwVnid: vxlan-2457601 Destination is in shared services
recursive next hop: 10.0.176.66/32%overlay-1
consumer EPG (different vrf)
leaf# vsh_lc -c "show forwarding route 192.168.255.10 plat vrf CL2022:vrf1"
Prefix:192.168.255.0/24, Update_time:Fri Nov 5 20:57:00 2021 leaf# show ip route 192.168.100.10 vrf CL2022:vrf2
! 192.168.100.0/24, ubest/mbest: 1/0, attached, direct, pervasive
Policy Prefix 0.0.0.0/0 *via 10.0.176.66%overlay-1, [1/0], static, rwVnid: vxlan-2523136
! recursive next hop: 10.0.176.66/32%overlay-1
Flags: IN-HW, SHRD-SVC,
vrf: 16(0x10), routed_if: 0x0 epc_class: 36(0x24) leaf# vsh_lc -c "show forwarding route 192.168.100.10 plat vrf CL2022:vrf2"
Prefix:192.168.100.0/24, Update_time:Tue Nov 9 14:34:05 2021
! Reserved tag for shared
Policy Prefix 0.0.0.0/0 services consumer. Policy
PcTag of provider epg ! applied in consumer vrf
Flags: IN-HW, SHRD-SVC,
vrf: 10(0xa), routed_if: 0x0 epc_class: 14(0xe)
Check hidden slide for impact of “Policy
Control Enforcement Direction” setting
Contracts and Forwarding Set policy-applied bits in

ivxlan. Permit, deny, redir, log
Ingress Yes
Source Leaf
EPG2 Contract Found?
PcTag 200
No If LPM is BD/EPG subnet,
forward and don’t set
policy-applied bits in
ivxlan. Otherwise, drop!
leaf# show zoning-rule scope 2523136 src-epg 200
+---------+--------+--------+----------+--------+
| Rule ID | SrcEPG | DstEPG | FilterID | Action |
+---------+--------+--------+----------+--------+ Don’t do contract
| 4159 | 200 | 100 | 532 | permit | lookup. Forward.
+---------+--------+--------+----------+--------+ Egress Yes
Policy-Applied
Bits set?
Dest
EPG1
Leaf No
Do contract
lookup. Permit,
PcTag 100
deny, redir, log
Policy enforcement table
Where is policy enforced?
VRF Enforcement
Setting
Flow Direction INGRESS EGRESS
EPG to unknown EPG Applied Egress Unchanged
EPG to known EPG Applied Ingress Unchanged
EPG to L3out Applied Ingress/non-BL Applied Egress/BL
L3out to unknown EPG Applied Egress/non-BL Applied Egress
L3out to known EPG Applied Egress/non-BL Applied Ingress/BL
L3out to L3out Applied Ingress Applied Egress
Policy enforcement affects only traffic to or from the L3Out.

There are no behavior changes in EPG-to-EPG.
What About Flooded Traffic?
The following traffic may be flooded: How does ACI flood?
• Broadcast • Flooded traffic is sent to the BD
• Multicast
GiPo (l2 flood) or VRF GiPo (l3
flood)
• Unknown Unicast
• The GiPo is an overlay multicast
• Control Plane maintenance (EP address allocated to a BD or VRF
announce, fabric ARP, etc)
• Flooding is done on a loop-free
tree called an FTAG
Security policy NOT applied
GiPo
What are FTAGs?
3
Forward out all Outgoing
Interfaces Root for
Ftag 0
• FTAGs are loop-free trees Spine Spine

within the overlay used by
flooded traffic
2 Select ftag 0, forward
• FTAGs are picked per flow out root port*
from values 0 – 0xc

• One spine is root for each tree Leaf Leaf Leaf
• Outgoing interfaces calculated

by ISIS ARP
EP1
10.1.1.1/24
*Note, the ingress leaf communicates the selected ftag to 0000.1111.2222
the rest of the fabric by adding it to the destination gipo.
1 Who has 10.1.1.100?
If the gipo is 225.0.0.0 and the ftag is 0x9, the
Please tell 10.1.1.1
destination address would be 225.0.0.9
Checking FTAGs
Find the outgoing interfaces for a tree
Check FTAG tree Check FTAG tree

on ingress leaf on root spine
leaf101# show isis internal mcast routes ftag spine1005# show isis internal mcast routes ftag
FTAG Routes FTAG Routes This spine is the

==================================== root for ftag 0
====================================
FTAG ID: 0 [Enabled] Cost:( 1/ 7/ 0) FTAG ID: 0 [Root] [Enabled] Cost:( 0/ 0/ 0)

---------------------------------- ----------------------------------
Root port: Ethernet1/54.6 Leaf forwards to Root port: -
OIF List: root port and any OIF List:
Ethernet1/53.5 additional OIFs Ethernet1/1.20 Forward out all of
! Ethernet1/2.21 these interfaces
!ommitted rest of ftags Ethernet1/3.19
!ommitted rest of ftags
Proxy
Forwarding
What is Proxy Forwarding?
Why? Scaling out Endpoint Learning
Send traffic
destined to EP Send traffic destined
1 to Leaf 1
Spine to an Unknown EP to Spines own
any Spine separate anycast
TEP’s for mac, ipv4,
and ipv6 proxy
I am connected lookups
to Endpoint 1 Leaf 1 Leaf 2 Leaf 3 Leaf 4
Endpoint 1 Only Leaf 1 and Spines have to

program Endpoint 1 in hardware
How to check the Spine-Proxy TEP
BD Subnet (Pervasive Route)
leaf1# show ip route vrf CL2022:vrf1
next-hop should be
192.168.0.0/24, ubest/mbest: 1/0, attached, direct, pervasive SPINE-PROXY
*via 10.0.16.64%overlay-1, [1/0], 00:21:39, static
leaf1# show isis dteps vrf overlay-1 | grep PROXY next-hop of Pervasive Route
10.0.16.65 SPINE N/A PHYSICAL,PROXY-ACAST-MAC is IPv4 Spine Proxy TEP
Three types of Spine Proxy TEP

• Proxy-Acast-MAC
✓ Spine-Proxy for L2 traffic (L2 Unknown Unicast mode “Hardware Proxy”)
• Proxy-Acast-V4
✓ Spine-Proxy for IPv4 traffic (includes ARP Request with ARP Flooding mode “OFF”)
• Proxy-Acast-V6
✓ Spine-Proxy for IPv6 traffic
What is COOP?
COOP is the proxy-database of ACI
• Council of Oracles Protocol – A TCP protocol for citizens (Leafs) to
publish records to oracles (Spines).
• Used for announcing endpoints, fabric owned IP’s, multicast
information, and more
• Synced across Pods/Sites with BGP EVPN
• Each Endpoint Record contains all information to forward (VNID, leaf
TEP, mac, etc)
• COOP records pushed into hardware on spines
• For modular spines, scale is achieved by pushing each EP onto only
two Fabric Modules
What if the Endpoint isn’t in COOP? (ARP Glean)
What if Spine’s COOP DB doesn’t know the destination when proxy’ed?
X L2 Traffic : Drop
✓ L3 Traffic : ARP Glean
5 Encap original packet with special ethertype
4 No COOP entry Anycast 6 Flood this “Glean” to reserved multicast group

VRF overlay-1 TEP
3 Spine Proxy
TEP1 7 LEAFsTEP2
check its BD subnets TEP3
2 Hit Pervasive Route
8 LEAF generates ARP Request 8 LEAF ignores Request from Spine
1 Unicast IP
If BD subnet for the
unknown IP doesn’t present
If BD subnet for the unknown
IP presents on LEAF
Spine Proxy Summary
Forward to Forward to Flood Spine Forward to Forward to Spine Forward to
Proxy Drop
local port remote leaf within BD local port remote leaf Proxy Border Leaf
Hardware
Flood
Proxy
Dst IP is
L3OUT Routes?
Is Dst MAC on Is Dst IP on LEAF has

Local Leaf? What is BD config? Local Leaf? BD Subnets
Yes No Yes for Dst IP? No
Yes No
LEAF knows LEAF knows Dst IP

Dst MAC? Yes as EndPoint? No
Yes No
L2 L3
L2 or L3 ?
If ARP Flooding is OFF, ARP

Packet coming in target-IP is used for this L3 flow
to Leaf
Capturing a Glean with Tcpdump
ACI Leafs and Spines contain pseudo interfaces for traffic to and from the CPU
• Traffic on the on the knet or tahoe pseudo interface will have a special ieth header. It must be decoded.
• Starting in 3.2 the knet_parser.py script is available on the switch cli to decode
1st Gen Leaf • For traffic going to the cpu

check knet0 and kpm_inb
Phys knet0 CPU
ASIC • For traffic coming from the
Port knet1 kpm_inb
cpu check knet1 and
kpm_inb
EX (or Later) Leaf

• For traffic to and from the
Phys CPU cpu check Tahoe0 and
ASIC Tahoe0
Port kpm_inb kpm_inb
*Note, not all traffic will show up on the kpm_inb interface. However, all
traffic shows on the pseudo interface
*Gen1 and 2 Modular spines use psdev0, psdev1, and psdev2 interfaces.
Gen 2 fixed spines use tahoe0. Gen 1 fixed spines use knet0-3
Egress Leaf
Capturing a Glean with Tcpdump Verification
Gen2 or Later Leaf
tcpdump -xxxvei tahoe0 -w /bootflash/tahoe0.pcap
Decode type should
knet_parser.py --file /bootflash/tahoe0.pcap --pcap --decoder tahoe
be tahoe for tahoe
interface
Frame 111
RX sup traffic
Time: 2019-05-16T16:56:33.059831+00:00
rather than TX
Header: ieth_extn CPU Receive
sup_qnum:0x14, sup_code:0x21, istack:ISTACK_SUP_CODE_SPINE_GLEAN(0x21)
Header: ieth
sup_tx:0, ttl_bypass:0, opcode:0x6, bd:0x120e, outer_bd:0x27, dl:0, span:0, traceroute:0, tclass:0
src_idx:0x3a, src_chip:0x0, src_port:0x19, src_is_tunnel:1, src_is_peer:1
dst_idx:0x0, dst_chip:0x0, dst_port:0x0, dst_is_tunnel:0
Len: 148
Eth: 000d.0d0d.0d0d > 0100.5e7f.fff1, len/ethertype:0x8100(802.1q)
802.1q: vlan:2, cos:5, len/ethertype:0x800(ipv4)
ipv4: 10.0.116.64 > 239.255.255.241, len:130, ttl:249, id:0x0, df:0, mf:0, offset:0x0, dscp:32, prot:17(udp)
udp: (ivxlan) 0 > 48879, len:110
Switch recognizes
ivxlan: n:1, l:1, i:1,
this as a Glean
vnid: 0x2b0000 Traffic that
lb:0, dl:1, exception:0, src_policy:0, dst_policy:0, src_class:0x5c0 triggered Glean
mcast(routed:0, ingress_encap:0/802.1q), ac_bank:0, src_port:0x0
Eth: 000c.0c0c.0c0c > ffff.ffff.ffff, len/ethertype:0xfff2(aci-glean)
ipv4: 172.16.1.1 > 172.16.2.2, len:84, ttl:63, id:0x71f9, df:1, mf:0, offset:0x0, dscp:0, prot:1(icmp)
icmp: echo request id:0x9092, seq:0x1980
Egress Leaf
Capturing a Glean with Tcpdump Verification
Gen1 Leaf Example
knet0 would show Rx traffic (similar output as Tahoe0)
tcpdump -xxxvei knet0 -w /bootflash/knet0.pcap
knet_parser.py --file /bootflash/knet0.pcap --pcap --decoder knet
knet1 would show Tx traffic

tcpdump -xxxvei knet1 -w /bootflash/knet1.pcap
knet_parser.py --file /bootflash/knet1.pcap --pcap --decoder knet
No decode necessary for kpm_inb (cpu) interface…Gleans aren’t easily readable

tcpdump -xxxvei kpm_inb ether proto 0xfff2
a-leaf102# tcpdump -xxxvei kpm_inb ether proto 0xfff2
tcpdump: listening on kpm_inb, link-type EN10MB (Ethernet), capture size 65535 bytes
15:27:37.663580 00:0c:0c:0c:0c:0c (oui Unknown) > Broadcast, ethertype Unknown (0xfff2), length 94:
0x0000: ffff ffff ffff 000c 0c0c 0c0c fff2 4500
0x0010: 0054 aa4b 4000 3f01 825d 0404 0464 0303
0x0020: 0396 0800 0dc6 2384 38db 5275 dd5c 0000
0x0030: 0000 9e35 0100 0000 0000 1011 1213 1415
0x0040: 1617 1819 1a1b 1c1d 1e1f 2021 2223 2425
0x0050: 2627 2829 2a2b 2c2d 2e2f 3031 3233
Egress Leaf
Verification
Layer 3 Unicast – Glean Scenario
Verify ARP on Remote Leaf
Endpoint Learn
a-leaf205#show ip arp internal event-history event | grep -F -B 1 172.16.2.2 Installed
73) Event:E_DEBUG_DSF, length:127, at 316928 usecs after Wed May 1 08:31:53 2019
Updating epm ifidx: 1a01e000 vlan: 105 ip: 172.16.2.2, ifMode: 128 mac: 0000.1111.2222 Response
75) Event:E_DEBUG_DSF, length:152, at 316420 usecs after Wed May 1 08:31:53 2019 Received
log_collect_arp_pkt; sip = 172.16.2.2; dip = 172.16.2.254; interface = Vlan104;info = Garp Check adj:(nil) ARP Request is
77) Event:E_DEBUG_DSF, length:142, at 131918 usecs after Wed May 1 08:28:36 2019 generated by leaf
log_collect_arp_pkt; dip = 172.16.2.2; interface = Vlan104;iod = 138; Info = Internal Request Done
Glean
Glean Received, Dst IP
Group Range
log_collect_arp_glean;dip = 172.16.2.2;interface = Vlan104;info = Received pkt Fabric-Glean: 1
included
is in BD
as Bidir
Subneton IPN
log_collect_arp_glean; dip = 172.16.2.2; interface = Vlan104; vrf = CiscoLive2020:vrf1; info = Address in PSVI subnet or special VIP
How ACI
Builds
Forwarding
Tables
Building Adjacency Tables
ACI combines ARP and MAC Tables into the Endpoint Table
Legacy Behavior ACI Behavior
• ARP/ND tables map Layer 3 to Layer 2 • Endpoint table contains endpoints, which
are Layer 2 addresses OR Layer 3
• ARP/ND tables are updated by control- addresses OR a combination of Layer 2
plane messages and Layer 3 addresses
• MAC Address Table used for switching • By default, both Layer 2 and Layer 3
decisions information is updated by dataplane
• Mac Address Table updated by • Used for security and forwarding policy
dataplane
Endpoints can be
programmed via software
Building Endpoint Tables process or by hardware
dataplane learns (HAL)
Resource Table Info Commands to Verify
show system internal epm endpoint mac <addr>
EPM – Endpoint Manager
Supervisor show system internal epm endpoint ip <addr>
Sup process for managing
endpoints.
EPMC – Endpoint Manager Client vsh_lc –c “show system internal epmc endpoint mac <addr>”
Line Card Line card process that sits vsh_lc –c “show system internal epmc endpoint ip <addr>”
between hardware layer (HAL)
and EPM
vsh_lc -c "show plat internal hal ep l2 mac <addr>"

HAL – Hardware Abstraction Layer
vsh_lc -c "show plat internal hal ep l3 ip <ip/pfx len>“
Asic View of what is programmed into
!
the ASIC.
!L3 Endpoints are put into HW Routing Table
vsh_lc -c "show plat internal hal l3 routes | grep EP"
What about ARP?
ARP Tables are still
used in ACI for… Resource Table Info Commands to Verify
show ip arp vrf <name>
Adjacency Manager.
• L3outs Supervisor Programmed by ARP
process.
• Overlay adjacencies
• VXLAN Endpoints (AVE, K8s,
vsh_lc –c “show forwarding
Openstack, etc) adjacency <ip>”
Line Card UFIB
• APIC / Fabric node
adjacencies
vsh_lc -c "show plat internal

HAL – Hardware
Asic hal l3 routes”
Abstraction Layer
View of what is
programmed into the
ASIC.
Building Routing Tables
show ip route x.x.x.x/y vrf <name>
URIB / MRIB – the unicast and
Supervisor show ip mroute x.x.x.x/y vrf <name>
multicast routing tables.
Programmed by route protocol
vsh_lc -c "show forwarding route <ip/pfx len> vrf <name>“

UFIB / MFIB – the unicast and
Line Card vsh_lc -c "show forwarding multicast route vrf <name>"
multicast forwarding tables on
the Line Card
vsh_lc -c "show platform internal hal l3 routes vrf <name>”

vsh_lc -c "show platform internal hal l3 mcast routes vrf <name>”
vsh_lc -c "show plat internal hal l3 routes vrf <name>" | grep MC
the ASIC.
Check Endpoint Table
Troubleshooting TIP before Routing Table
When Troubleshooting Layer 3 Flows Always…

1) Check if there is an Endpoint Learn show endpoint ip <addr>
show system internal epm endpoint ip <addr>
If not then…
2) Check if there is a BD (pervasive) static route
If not then… show ip route x.x.x.x/y vrf <name>
3) Check if there is an External Route
Programming Contracts
show zoning-rules
Policy Manager.
Supervisor Programmed by leaf policy-
element process
vsh_lc -c “show system internal aclqos zoning-rules”

Line Card ACLQOS
vsh_lc -c “show plat internal hal objects policy zoningrule”

the ASIC.
Applicable to EX and
HAL – Hardware Abstraction Layer Later Hardware
Wouldn’t it be great if there was a single point to

validate forwarding and security classification?
vsh_lc -c "show platform internal hal l3 routes”
HAL
Overflow
TRIE DLEFT TCAM Policy TCAM
ASIC
TCAM
Hardware Forwarding Tables Hardware Policy Tables
Applicable to EX and
HAL – Hardware Abstraction Layer Later Hardware
L3 Lookup of Hardware Tables

module-1# show plat internal hal l3 routes vrf CL2022:vrf1
----------------------------------------!!-------------------------
| | | | LID |!! |
| VRF | Prefix/Len | RT| Type|!!|CLSS| Flags
|-----|----------------------|---|-----|!!|----|------------------|
|
Much more info
|-----|----------------------|---|-----|!!|----|------------------| available in full
| 4626| 192.168.100.10/ 32| EP| TRIE|!!|c002| le,bne,sne, dl | output!
| 4626| 10.99.99.0/ 24| UC| TCAM|!!|8004| sc,spi,dpi |
| 4626| 192.168.255.0/ 24| UC| TCAM|!!| 24| sc,spi,dpi, dr |
| 4626| 192.168.200.11/ 32| EP| TRIE|!!|8003| sc, le,sne |
---------------------------------------!!------------------------|
Consolidated view of routes

PcTag from destination
for Endpoints, Shared
EPG…used for contract lookup
Services, and External routes
L2 Lookup of Hardware Tables Applicable to EX and
Later Hardware
module-1# show platform internal hal ep l2 all
=============================================================
BD EP L2 L2 S Much more info

BdId Name T Mac IfId Ifname Class
=============================================================
available in full
b BD-11 Pl 00:00:11:11:22:22 1a010000 Eth1/17 c003 output!
1a BD-26 Xr 00:00:22:22:33:33 18010004 Tunnel4 400f
21 BD-33 Pl 00:00:22:22:33:33 16000002 Po3 4002
Consolidated view of all PcTag from destination

learned Mac Addresses EPG…used for contract lookup
Understanding
the
Configuration
Options
VRF Level Forwarding Options
Feature What Does it Do?
Policy Control Enforcement If disabled, policy is never applied between EPGs. If enabled,
Preference contracts are enforced.
If Disabled, ACI uses legacy behavior for learning endpoints.
IP Dataplane Learning Layer 3 endpoints are learned by ARP/GARP/ND and Layer 2
endpoints are learned by dataplane.
If set to Ingress, contract enforcement for l3out flows is done
Policy Control Enforcement
on service leaf. Egress enables enforcement on Border Leaf
Direction
(requires remote learning to be enabled)
Ingress Enforcement Egress Enforcement

Ingress leaf sets policy applied bits Ingress leaf does not set policy applied bits
SLeaf BLeaf SLeaf BLeaf

L3out L3out
Egress leaf does not set policy applied bits Egress leaf sets policy applied bits
Bridge-Domain Level Forwarding Options
For non-link-local L3 multicast traffic in a PIM-disabled
L3 Unknown Multicast
BD, should a leaf with no snooping entries flood in BD
Flooding
(flood) or wait for joins (OMF)?
For L2 mcast and broadcast, flood, drop, or flood within
Multidestination Flooding epg encap? If flooding with EPG encap, proxy-arp is
required for cross-epg L2 communication
If destination mac is unicast and unknown, flood or proxy
L2 Unknown Unicast
to spines?
Proxied, L2 Unknown Unicast

is dropped if the Destination
MAC isn’t known in COOP
Bridge-Domain Level Forwarding Options
Limit IP Learning to Only learn IP’s if they are within the configured BD
Subnet subnet for local learns.
Enable IP learning as well as routing (if a BD
Unicast Routing
subnet is configured)
Disable IP Dataplane Only for PBR! Only local MAC’s are learned via
Learning DP. IP’s and remote macs learned via ARP.
When disabled, ARP is unicast routed based on
ARP Flooding
the Target IP (if known)
leaf# show endpoint ip 192.168.100.11 Proxy!

Who has leaf# show ip route 192.168.100.11 vrf CL2022:vrf1
192.168.100.11?
192.168.100.0/24, ubest/mbest: 1/0, direct, pervasive
*via 10.0.176.66%overlay-1, [1/0], 01w00d, static
EPG Level Forwarding Options
Feature is enabled for just the EPG (rather than all
Flood in Encapsulation epg’s in the BD). Requires proxy arp for L2 traffic
between encaps.
Designed for Direct Server Return flows. This
L4-L7 Virtual IP’s disables dataplane learning per IP. IP is learned
by ARP/ND.
Disable DP Learning Disables dataplane learning for non DSR
Per-IP/Prefix scenarios. More specific than VRF-level option
New in 5.2
Global Forwarding Options

Don’t learn an IP (both local and remote) if it is
Enforce Subnet Check
not within a configured BD subnet in the VRF.
Remote IP learning is disabled for Unicast flows
Disable Remote EP
on a leaf in a specific VRF if an l3out exists in the
Learning on BL’s
same VRF
Multicast sources are still learned
Also implicitly disabled when

intersite l3out is configured
The Anatomy
of an ACI
Switch
※ LST: Local Station Table, GST: Global Station Table
LEAF ASIC Generations ※ FP Tile: Forwarding and Policy Tile
1st generation 2nd generation (or later) Tile X: IP

Dest EP Remote Tile Y: MAC
Lookup EP Learn etc.
To SPINE To SPINE
GST GST
Cisco ASIC
ingress egress
Cloud Scale FP
LST LST CPU Tiles
ASIC
ingress egress
CPU Broadcom
Local EP Dest EP
Learn Lookup
N9K-C9332PQ N9K-C9396PX • Complete separation of N9K-C*-EX N9K-C*-FXP • More flexible/scalable

N9K-C9372PX N9K-C9396TX + Ingress and Egress N9K-C*-FX N9K-C*-GX with configurable tiles
N9K-C9372PX-E N9K-C93120TX + Source Learn and N9K-C*-FX2 N9K-C*-GX2 • Abstracted with HAL
N9K-C9372TX N9K-C93128TX Destination Lookup N9K-C*-FX3 • Tile X for both source
N9K-C9372TX-E • Separate GST/LST for learn and destination
IP and MAC lookup
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
※ number of ASIC per card depends on model
SPINE ASIC Generations
1st generation 2nd generation (or later)
SUP Fabric card SUP Fabric card

COOP Database COOP Database
CPU Broadcom CPU Cisco ASIC
Line card Line card
Cloud Cloud
Cisco Cisco TEP Information TEP Information
Scale Scale
ASIC ASIC
ASIC ASIC
Line card Fabric card Line card Fabric card

N9K-X9736PQ N9K-C9504-FM N9K-*X N9K-C*FM-E
N9K-C9508-FM N9K-C*FM-E2
Box spine N9K-C9516-FM Box spine N9K-C*FM-G
N9K-C9336PQ N9K-*C
N9K-*X
Inside an ACI Switch ASIC (Gen 2 and Later)
Lookup Destination
Evaluate Determine VRF, Based on FP result,
IP / MAC; FPC
frame format VLAN, EPG, etc re-evaluate LU Result
determines contract
Phys Parser Lookup Block Forwarding Block Lookup

Port Block (PRX) (LUA – LUB) (FPA – FPC) Block (LUC)
Evaluate any
matching ACL entries
ACL Engine
(ACA – ACC)
Build the final frame Apply QoS and Calaculate Load-
(egress qtag, etc) Buffering Policies Balance Hashing
Revaluate LU Result
based on ACL hits
Rewrite Block Buffering and
Phys Load Balancing Lookup
(RWX) Queueing Engine
Port (LBX) Block (LUD)
(BMX / QSX / BAX)
※ number of ASIC per card depends on model
Inside an ACI Modular Spine

What are the strange IP’s on
the Fabric Modules?
Fabric Module
Fabric Module
Fabric Module
Fabric Module
sp# vsh -c "slot 26 show plat internal hal l3 routes”
ASIC 0
ASIC 0
ASIC 0
ASIC 0
40.0.99.139/ 32
3.124.199.13/ 32
0.156.151.177/ 32
Where are the linecard

Internal PC’s (2 forwarding tables?
ports per)
ACI Line Card sp# vsh -c "slot 2 show plat internal hal l3 routes”
<no output>
ASIC 0 ASIC 1 ASIC 2 ASIC 3
Ports A-B Ports C-D Ports E-F Ports G-H
How is traffic forwarded?
For Proxied Traffic

• Depending on if the dest IP is the L2 or L3 Proxy TEP the VRF
VNID + Dest IP OR BD VNID + Dest MAC is used to hash a
synthetic Dest IP and VRF ID
• Synthetic information is used on LC to hash the uplink port to
FM
• FM routing lookup is based on Synthetic IP
• Each Synthetic IP is owned by two FM’s
• FM uses vnTag to tell egress LC which front panel port to use
How is traffic forwarded?
For Transit Traffic

• Line card hashes across ALL FM uplinks
• ALL FM’s have overlay TEP routes
• FM uses vnTag to tell egress LC which front panel port to use
sp# vsh -c "slot 26 show forwarding route platform" | grep -A 10
Fabric Module
Fabric Module
Fabric Module
Fabric Module
"15.180.164.253"
!
Table: (IN-HW) Type=100 Vrf=750 Synth=0
!
ASIC 0
ASIC 0
ASIC 0
ASIC 0
FC Cards/ASICs : FC22/ASIC-0 FC26/ASIC-0
Synthetic info programmed

on FM’s 22 and 26
sp# moquery -c coopIpv4Rec -f 'coop.Ipv4Rec.addr=="10.0.0.10"'

Internal PC’s (2 # coop.Ipv4Rec
ports per) addr : 10.0.0.10
synthIp : 15.180.164.253
ACI Line Card synthVrf : 250
ASIC 0 ASIC 1 ASIC 2 ASIC 3 1

Ingress LC hashes Synth IP and VRF
Ports A-B Ports C-D Ports E-F Ports G-H based on Real Tenant IP + VRF
DIPO is IPv4 Proxy TEP

Ingress Traffic: DIPI is 10.0.0.10
VRF Vnid is 111111 BRKDCN-3900 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
Fabric Module
Fabric Module
Fabric Module
Fabric Module
vnTag is derived from route lookup,
FM forwards to egress LC which
forward only based on vnTag
4
ASIC 0
ASIC 0
ASIC 0
ASIC 0
Fabric Module does route lookup
based on Synthetic IP and VRF
3
Internal PC’s (2
ports per) Traffic Hashes across either FM 22
or 26 uplinks
2
ACI Line Card
Egress LC knows exactly which front-panel port to
Ports A-B Ports C-D Ports E-F Ports G-H forward out of based on the received vntag
5
DIPO is IPv4 Proxy TEP
Ingress Traffic: DIPI is 10.0.0.10
VRF Vnid is 111111 BRKDCN-3900 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
Understanding
the Tools
Start with High-level Tools
Use Endpoint Tracker for Building a Topology
EP Locally Learned on
pod 2, nodes 401-402
No EP Learn, is this an
L3out?
Use Atomic Counters to Check for Overlay Drops and Latency (PTP)
Use Atomic Counters to Check for Overlay Drops and Latency (PTP)
104 Microseconds of
delay in overlay No overlay drops!
Use Tenant Visibility tools to check for Contract Drops
This flow is being

contract dropped
apic4# show acllog deny l3 pkt tenant common vrf CORE

srcIp dstIp protocol srcPort dstPort node srcIntf vrfEncap
----- ----- -------- ------- ------- ---- ------- --------
<EMPTY>
Port Counters are as Useful as Ever
leaf1# show interface eth1/8
Ethernet1/8 is up
Frames received
with bad FCS Indicates a previously
admin state is up, Dedicated Interface
Last link flapped 03:07:41 stomped frame was received
RX
3527922 unicast packets !ommitted What is a Stomp?
4041582 input packets 609518993 bytes • When a frame is received
12 jumbo packets 0 storm suppression bytes with a bad FCS and/or is
0 runts 0 giants 0 CRC 0 Stomped CRC 0 no buffer
malformed
0 input error 0 short frame 0 overrun !ommitted
0 watchdog 0 bad etype drop 0 bad proto drop !ommitted AND
0 input with dribble 0 input discard • The frame is cut-through
0 input buffer drop 0 input total drop Frame transmitted switched
TX with stomped CRC
32262479565 unicast packets !ommitted The switch will invert the
32395063346 output packets 49034781261665 bytes
Buffer drops, sign new CRC to tell the first
32249687943 jumbo packets
of congestion
0 output error 0 collision 0 deferred 0 late collision store-and-forward device to
0 lost carrier 0 no carrier 0 babble 0 output discard drop it
0 output buffer drops 0 output total drops
Using moquery to check port counters fabric-wide
#Check Fabric-wide for FCS Errors

moquery -c rmonDot3Stats -f 'rmon.Dot3Stats.fCSErrors>="1"' | egrep "dn|fCSErrors”
#Check Fabric-wide for total CRC Stomp + FCS Errors

moquery -c rmonEtherStats -f 'rmon.EtherStats.cRCAlignErrors>="1"' | egrep "dn|cRCAlignErrors”
#Check Fabric-wide for Output Buffer Drops

moquery -c rmonEgrCounters -f 'rmon.EgrCounters.bufferdroppkts>="1"' | egrep "dn|bufferdroppkts”
#Check Fabric-wide Output Errors

moquery -c rmonIfOut -f 'rmon.IfOut.errors>="1"' | egrep "dn|errors”
ELAM – Embedded
Logic Analyzer Module
Dst – TCP 10.0.0.1:3000
Dst – TCP 10.0.0.1:3001
• It is a tripwire in hardware Dst – TCP 10.0.0.1:3002
• The first frame to match a vsh_lc

specified condition ‘trips’ it debug platform internal
trigger reset
tah elam asic 0
trigger init in-select 6 out-select 1

• Report is created with vast set outer ipv4 dst_ip 10.0.0.1
amount of data regarding set outer l4 dst-port
start
3001
asic decisions
module-1(DBG-elam-insel6)# stat
Matching frame was
ELAM STATUS
caught!
===========
Frame was not Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Triggered
dropped in lookups!
module-1(DBG-elam-insel6)# ereport | grep "drop reason"
RW drop reason : no drop
LU drop reason : no drop
What ASIC should be vsh_lc
set in the ELAM? debug platform internal <asic> elam asic 0
Model Role Asic for Elam
N9K-C*C Fixed Spine roc

N9K-C*GX Fixed Spine app
N9K-C*-EX Leaf tah
N9K-C*-FX/FXP/FX2 Leaf roc
N9K-C*-GX Leaf app
N9K-C*-GX2 Leaf cho
N9K-X97*-EX Spine LC tah
N9K-X97*-FX Spine LC roc
N9K-X97*-GX Spine LC app
N9K-C95*-FM-E Spine FM tah
N9K-C950*-FM-E2 Spine FM roc
N9K-C95*-FM-G Spine FM app
Steps to Using Elam on Gen2+ Leaf or Fixed Spine
Elams are run from Refer to “What ASIC should
Leafs and fixed spines are single
the line card shell be set in the ELAM” slide
asic switches. Always use asic 0
vsh_lc
debug platform internal tah elam asic 0
trigger reset
Failing to reset the trigger
can cause past elam set outer ipv4 dst_ip 10.0.0.1
Use 0 or 1
configurations to take effect. set outer l4 dst-port 3001
Always reset the trigger!
start
module-1(DBG-elam)# trigger init in-select ?
!ommitted
14 Outer(l2(vntag)|l3|l4)-inner(l2|l3|l4)-ieth
6 Outerl2-outerl3-outerl4
7 Innerl2-innerl3-innerl4 Determines which headers conditions
!ommitted can be matched in. Use 14 or 7 when
matching vxlan encapsulated headers.
Steps to Using Elam on Gen2+ Leaf or Fixed Spine
Which headers to match

Use “set outer” or conditions for?
“set inner” depending vsh_lc
on in-select and if
matching outer or inner
debug platform internal tah elam asic 0
headers in vxlan packet trigger reset
set outer ipv4 dst_ip 10.0.0.1
set outer l4 dst-port 3001 What to match in the
header?
start
Finally enable the elam!
When running stat if Triggered is seen, this

means a matching packet was received
ereport available since 4.2
Reading an Elam
At a high-level…
module-1(DBG-elam-insel6)# ereport
!ommitted
----------------------------------------------------------------- • ereport provides a simple,
Outer L3 Header
----------------------------------------------------------------- human-readable report output
L3 Type : IPv4
IP Version : 4 • ereport requires >= 5.2 code
DSCP : 0 for modular spines
IP Packet Length : 84 ( = IP header(28 bytes) + IP payload )
Don't Fragment Bit : set • Groups data into outer/inner,
TTL : 64
IP Protocol Number : ICMP
headers, and lookup results
Destination IP : 192.168.200.11
Source IP : 192.168.100.10
!omitted
Contract Result
Contract Drop : no
Contract Logging : no
Contract Applied : yes
Contract Hit : yes
ereport available since 4.2
Reading an Elam
At a low-level…
report detail | grep –F "---------" | grep -v VECTOR | grep -v end • An elam report provides a
LU BEGIN ------------------------------
LUA ------------------------------ walkthrough of each ASIC
LUB ------------------------------ block
LUC ------------------------------
LUD ------------------------------ • Each decision in each block is
LU END ------------------------------ recorded
*** FP latch results ------------------------------
*** LBX latch results ------------------------------
• Refer to “Inside an ACI Switch
*** ACX latch results ------------------------------
RW BEGIN ------------------------------ ASIC” from part 1 for more
RW END ------------------------------ details
• All output is in HEX
What if Elam Shows a Drop? ereport available since 4.2
ereport
Lookup Drop
--------------------------------------
LU drop reason : SECURITY_GROUP_DENY
Common Drop Reasons
Drop Code What Does it Mean? What to Do?

For traffic destined to the CPU on an FX switch
it is expected and cosmetic. Also seen when Ignore if its an FX switch and destined to local
ACL_DROP traffic was received from a fabric port and the switch IP/process. Otherwise, check for
leaf has a remote EP learn with no bounce incorrect EP learn.
flag.
For multisite / remote-leaf, there was no Check contracts between local and remote
DCI_*_XLATE_MISS
matching vnid or pctag translation found. resources.
No route and/or tunnel found back to the outer Check for a tunnel pointing back to the outer
INFRA_ENCAP_SRC_TEP_MISS
source IP source IP. Also, check for a route in overlay.
Make sure a contract is configured to allow
SECURITY_GROUP_DENY Frame was contract dropped
the flow.
Check if the frame was correct
Received vlan not programmed on ingress
SRC_VLAN_MBR tagged/untagged. Make sure no invalid-path
port.
faults exist for the epg.
UC_PC_CFG_TABLE_DROP No route was found for the destination. Check the routing table for the destination.
Check if the frame is tagged with correct
VLAN_XLATE_MISS Received vlan doesn’t exist on the switch.
vlan. Check for invalid-path faults on the epg.
Steps to Using Elam on Gen2+ Modular Spine
Challenges of Modular Spines
• Line cards (and potentially FM’s) have multiple asics
• Elam must specify asic number
• Ingress/Egress ports may be internal LC – FM connections
• ereport only available in 5.2 and later
Fortunately, spine elams aren’t needed as commonly as leaf elams!
Steps to Using Elam on Gen2+ Modular Spine Ingress LC
Determine the Asic, Slice, and Srcid of the ingress port
FM23
FM26
FM22
FM24
ASIC 0
ASIC 0
ASIC 0
ASIC 0
1
sp# vsh
sp# attach mod 2
module-2# show plat internal hal l2 port gpd
======================================================
Uc Uc
I PC Pc
ACI Line Card (slot 2) IfId Ifname P Cfg MbrID As AP Sl Sp Ss Ovec
======================================================
!ommitted
ASIC 0 ASIC 1 ASIC 2 ASIC 3 1a09f000 Eth2/32 0 b9 38 3 31 1 8 10 90
eth2/32
Eth2/32 is on Asic 3,
Inner Headers - Slice 1, with srcid 0x10.
Ingress Src – 10.10.10.10 Use for Elam!
Traffic: Dst – 10.10.11.11
Asic and slice of eth1/32

(see last slide)
FM23
FM26
FM22
FM24
ASIC 0
ASIC 0
ASIC 0
ASIC 0
2
sp# vsh
sp# attach mod 2
debug plat internal tah elam asic 3 slice 1
trigger reset
set srcid 0x10
set inner ipv4 src_ip 10.10.10.10 dst_ip 10.10.11.11
ACI Line Card (slot 2) start
Source ID value of
ASIC 0 ASIC 1 ASIC 2 ASIC 3 eth1/32 (see last slide)
eth2/32
3
ELAM STATUS
===========
Inner Headers - Asic 3 Slice 1 Status Triggered
Ingress Src – 10.10.10.10
Traffic: Dst – 10.10.11.11 Packet was matched!
FM23
FM26
FM22
FM24
5
ASIC 0
ASIC 0
ASIC 0
ASIC 0
module-2# show plat internal hal l2 internal-port pi
=============================================
IfId IfName As Ovec
=============================================
96 lc(0)-fc(0):22:pc2:p1 0 b8
98 lc(1)-fc(0):22:pc2:p1 1 b8
9a lc(2)-fc(0):22:pc2:p1 2 b8
9c lc(3)-fc(0):22:pc2:p1 3 b8
ACI Line Card (slot 2)
Packet forwarded to FM
ASIC 0 ASIC 1 ASIC 2 ASIC 3 23! (output is zero-based)
Ovector indicates the
eth2/32 egress port to FM
4 report | egrep "drop\_vec|ovec|asic"
Dumping report for asic inst 3 slice 1 insel 14 outsel 1
*_sidebnd_no_spare_vec.ovector_idx: 0xB8
Inner Headers - *_vec.pbx_header_sidebnd_drop_vec.lux_drop_vec: 0x0000000
Traffic: Dst – 10.10.11.11 Packet wasn’t dropped in lookups!
Steps to Using Elam on Gen2+ Modular Spine Fabric Mod
9508 and 9516 FM’s have 2

asics; if no trigger on 0, try 1.
FM23
FM26
FM22
FM24
ASIC 0
ASIC 0
ASIC 0
ASIC 0
6
sp# vsh
sp# attach mod 23
debug plat internal tah elam asic 0
trigger reset
start
7
ASIC 0 ASIC 1 ASIC 2 ASIC 3 module-23(DBG-elam-insel14)#
stat
eth2/32 ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Triggered Packet was matched!
Inner Headers - Asic 0 Slice 2 Status Armed
Ingress Src – 10.10.10.10 Asic 0 Slice 3 Status Armed
Traffic: Dst – 10.10.11.11 Asic 0 Slice 4 Status Armed
Steps to Using Elam on Gen2+ Modular Spine Fabric Mod
FM23
FM26
FM22
FM24
ASIC 0
ASIC 0
ASIC 0
ASIC 0
9 ========================================
IfId Ifname As Ovec
============================
f5 fc0-lc1:3-1 0 58
Packet forwarded to LC 2
(zero based – Asic 3, Slice 1)
eth2/32 egress port to LC
report | egrep "drop\_vec|ovec|asic"

8 Dumping report for asic inst 0 slice 1 insel 14 outsel 1
*_sidebnd_no_spare_vec.ovector_idx: 0x58
Traffic: Dst – 10.10.11.11 Packet wasn’t dropped in lookups!
Steps to Using Elam on Gen2+ Modular Spine Egress LC
Asic 3 / slice 1 as seen

on last slide
FM23
FM26
FM22
FM24
ASIC 0
ASIC 0
ASIC 0
ASIC 0
10
sp# vsh
sp# attach mod 2
debug plat internal tah elam asic 3 slice 1
trigger reset
set outer l2 vntag_vld 1
ACI Line Card (slot 2) start
Vntag present only
ASIC 0 ASIC 1 ASIC 2 ASIC 3 coming from FM
eth2/27
11
ELAM STATUS
===========
Inner Headers - Asic 3 Slice 1 Status Triggered
Egress Traffic: Src – 10.10.10.10
Dst – 10.10.11.11 Packet was matched!
Steps to Using Elam on Gen2+ Modular Spine Egress LC
13
FM23
FM26
FM22
FM24
ASIC 0
ASIC 0
ASIC 0
=======================================
ASIC
IfId Ifname As AP Sl Sp Ss Ovec
0
=======================================
1a08a000 Eth2/11 1 5 0 4 8 8
1a09a000 Eth2/27 3 5 0 4 8 8
Spine forwards out
front-panel Eth2/27!
eth2/27 egress port to Leaf
12 report | egrep "drop\_vec|ovec|asic"
Dumping report for asic inst 3 slice 1 insel 14 outsel 1
*_sidebnd_no_spare_vec.ovector_idx: 0x8
Egress Traffic: Src – 10.10.10.10
Dst – 10.10.11.11 Packet wasn’t dropped in lookups!
Automating Modular Spine ELAMs
CLI-based Modular Spine Elam tool available at – EasySpineElam
Easily Set Conditions on
All or Some Modules
spine1#./easy-spine-elam.sh -m all -d ingress
Final module list is:
2 23 26 3
2022-06-08T14:55:57 In-select - 14 and out-select - 0 are being used.
!ommitted
70. inner ipv4 destination ip > Format : d.d.d.d
71. inner ipv4 protocol > Format : 0-255
73. inner ipv4 source ip > Format : d.d.d.d
91. inner l4 dest port > Format : 0-65535
Select corresponding numbers of conditions to set. Separate numbers with commas.

Ex: 1,2,3,4,5
Which conditions to match?
Enter selections: 70,73,71,91
Enter inner ipv4 destination ip > Format : d.d.d.d: 80.0.0.1
Enter inner ipv4 source ip > Format : d.d.d.d: 150.0.0.100 Set conditions
Enter inner ipv4 protocol > Format : 0-255: 6
Enter inner l4 dest port > Format : 0-65535: 8989
Automating Modular Spine ELAMs
CLI-based Modular Spine Elam tool available at – EasySpineElam
2022-06-08T14:56:28 Checking elam status for module 2 Generate and view ereport
2022-06-08T14:56:28 Checking elam status for module 23
2022-06-08T14:56:28 Checking elam status for module 26 from all Triggered Modules!
2022-06-08T14:56:28 Checking elam status for module 3
ELAM TRIGGERED on module 26:

ASIC: 0 SLICE: 1 ELAM triggered on
LC and FM!
ELAM TRIGGERED on module 2:
ASIC: 3 SLICE: 1
Type "status" to check elam status again. Type "ereport", "report" or "report detail"
to collect all reports: ereport
2022-06-08T14:57:36 Collecting report for module 26 asic 0...
2022-06-08T14:57:36 Collecting report for module 2 asic 3... Locally view or copy
2022-06-08T14:57:46 Converting reports to ereport format! off the final ereports
The following decoded elams are available -
/data/techsupport/mod26-asic0-elamreport-2022-06-08T14-57-36-EREPORT
/data/techsupport/mod2-asic3-elamreport-2022-06-08T14-57-36-EREPORT
2022-06-08T14:57:49 FINISHED!
Shouldn’t ELAM be More Simple?
Elam Assistant in DCAppCenter
https://dcappcenter.cisco.com
ELAM (Embedded Logic Analyzer Module)
• Perform an ASIC level packet capture
ELAM Assistant
• You can perform ELAM like a TAC
engineer!
• With a nicely formatted result report
Detail Explanations:
• https://dcappcenter.cisco.com/elam-assistant.html
• How to use video, pictures
➢ A download link for ELAM Assistant
ELAM Assistant in ACI AppCenter (example)
1. Perform an Elam
Triggered!! Set Parameters

and
Report is Ready
ELAM Assistant in ACI AppCenter (example)
2. Read a Report
Click to see
report
Report shows
up here
Scroll Down
FTRIAGE – Automating Elams
Orchestrate End-to-End
ELAMs from the APIC!
apic1# ftriage route -ii LEAF:101,102 -dip 10.99.99.100 -sip 192.168.100.10
20:19:54 INFO main:1295 L3 packet Seen on leaf102 Ingress: Eth1/34 (Po5) Egress: Eth1/54 Vnid: 2523136
20:19:55 INFO main:1364 leaf102: Packet's egress outer [SIP:10.0.176.67, DIP:10.0.64.70]
20:19:55 INFO main:1371 leaf102: Outgoing packet's Vnid: 2523136
20:19:56 INFO main:353 Computed ingress encap string vlan-3501
20:20:03 INFO main:464 Ingress BD(s) CL2022:bd1
20:20:03 INFO main:476 Ingress Ctx: CL2022:vrf1 Vnid: 2523136
!
20:21:46 INFO main:1295 L3 packet Seen on spine1005 Ingress: Eth1/1 Egress: Eth1/3 Vnid: 2523136
20:22:38 INFO fib:737 spine1005: Transit in spine
20:23:32 INFO main:1295 L3 packet Seen on leaf103 Ingress: Eth1/29 Egress: Eth1/27/4 Vnid: NULL
!
20:24:02 INFO fib:219 leaf103: L3 out interface Ethernet1/27/4
20:24:10 INFO main:781 Computed egress encap string vlan-1055
20:24:17 INFO main:1796 Packet is Exiting fabric with peer-device: N3K-1 and peer-port: Ethernet1/31
SPAN / ERSPAN
Don’t neglect old friends!
• Both local span and erspan supported
• ERSPAN requires an l3 endpoint learned anywhere in the fabric
• Still the best tool for checking –

• Packet contents
• Frame format
• Retransmissions
• …and anything else that can be seen in a pcap
Other Tools Requiring External Resources
Netflow
• Captures flow information based on specified criteria
• Useful for troubleshooting packet loss and latency
Flow Telemetry
• Hardware directly streams flow data to Nexus Dashboard Insights
• Useful for troubleshooting packet loss and latency
• Latency measurements leverage PTP for additional accuracy
• NDI can perform additional flow analytics
Debugging ACI
BUM Flows
Bridge Domain Settings:
ARP – Ingress Leaf Unicast Routing Disable
ARP Flooding Enabled
Spine Spine
Leaf Floods ARP in BD GIPO on
selected FTAG tree with BD
1 vnid set Check GIPO Route
show ip mroute 225.0.2.128 vrf overlay-1
IP Multicast Routing Table for VRF "overlay-1"
(*, 225.0.2.128/32), uptime: 22w2d, isis

Incoming interface: Null, RPF nbr: 0.0.0.0
Leaf Leaf Leaf Outgoing interface list: (count: 2)
Ethernet1/29.9, uptime: 8w2d
Ethernet1/30.10, uptime: 22w2d
ARP: Who has

192.168.100.10?
EP1 EP2
192.168.100.10/24 192.168.100.11/24
0000.cccc.dddd 0000.aaaa.bbbb
ARP – How to Find the GiPo
From the GUI…
From the APIC CLI…
moquery -c fvBD -f 'fv.BD.dn*"tn-CL2022/BD-bd1"'
# fv.BD
arpFlood : yes
bcastP : 225.0.2.128
dn : uni/tn-CL2022/BD-bd1
From the Switch CLI…

moquery -c l2BD -f 'l2.BD.name=="CL2022:bd1"' -x rsp-subtree=full rsp-subtree-class=fmcastGrp
# fmcast.Grp
addr : 225.0.2.128
dn : sys/ctx-[vxlan-2523136]/bd-[vxlan-14811121]/fmgrp-[225.0.2.128]
rn : fmgrp-[225.0.2.128]
ARP – Ingress Leaf Unicast Routing Disable
Spine Spine ELAM the ARP request!

Leaf Floods ARP in BD GIPO on
selected FTAG tree with BD vsh_lc
1 vnid set debug plat internal app elam asic 0
trigger reset
set outer arp source-ip 192.168.100.11
set outer arp target-ip 192.168.100.10
start
Leaf Leaf Leaf !
stat
ELAM STATUS
===========
ARP: Who has Asic 0 Slice 1 Status Armed
192.168.100.10? Asic 0 Slice 2 Status Triggered
EP1 EP2
192.168.100.10/24 192.168.100.11/24
ARP – Ingress Leaf Elam Results (ereport) Unicast Routing Disable
Outer L2 Header
------------------------------------ Make sure this matches
Access Encap VLAN : 3502( 0xDAE ) what is expected
Outer L3 Header
------------------------------------
ARP Opcode : Request( 0x1 )
ARP Sender IP : 192.168.100.11
ARP Target IP : 192.168.100.10
Contract Result
------------------------
Contract Drop : no
Contract Applied : no Frame is flooded in the Bridge Domain!
FINAL FORWARDING LOOKUP
------------------------------------------------------------------------------------
Bits set in Final Forwarding Block: : IFABRIC_IG MC TENANT MYTEP BRIDGE MISS FLOOD
Lookup Drop
----------------------------- Not Dropped in lookups!
ARP – How to Find the FTAG
No other way than Elam…
Selected ftag is 0x8

module-1(DBG-elam-insel6)# ereport | grep "nopad.ftag"
wol_lu2ba_sb_info.mc_info.mc_info_nopad.ftag: 0x8
• Leaf forwards to root port and OIF’s for ftag 8
• Since GIPO is 225.0.2.128, Dest multicast address is 225.0.2.136 (gipo + ftag)

• Check ftag topology with show isis internal mcast routes ftag
leaf103# show isis internal mcast routes ftag

IS-IS process: isis_infra
VRF : default
FTAG Routes
====================================
FTAG ID: 8 [Enabled] Cost:( 1/ 6/ 0)
----------------------------------
Leaf appends ftag to gipo and
Root port: Ethernet1/29.9 forwards out Eth1/29 to spine
OIF List:
ARP - Spine 2
Unicast Routing Disable
Root spine for ftag 8 ARP Flooding Enabled
Root for forwards out OIFs
Ftag 8
This spine is the root!
Spine Spine
spine1005# show isis internal mcast routes ftag
IS-IS process: isis_infra
VRF : default
FTAG Routes
====================================
FTAG ID: 8 [Root] [Enabled] Cost:( 0/ 0/ 0)
----------------------------------
Root port: -
Leaf Leaf Leaf OIF List:
Ethernet1/1.20
Ethernet1/2.21
Ethernet1/3.19
ARP: Who has

192.168.100.10?
Spine forwards out OIFs
EP1 EP2
192.168.100.10/24 192.168.100.11/24
ARP – Egress Leaf Unicast Routing Disable
leaf102# show vlan | grep CL2022:bd1

Spine Spine 70 CL2022:bd1 active
leaf102# show vlan id 70 extended
Egress leaf(s) floods out VLAN Name Encap Ports

3 ---- ------------ ---------------- ----------------------
all ports in the BD (except
with flood in encap) 70 CL2022:bd1 vxlan-14811121 Eth1/33, Eth1/34,
Eth1/43, Eth1/48, Po5,
Po6, Po8, Po9
Leaf Leaf Leaf
ARP: Who has

192.168.100.10?
Flood out these ports
EP1 EP2
192.168.100.10/24 192.168.100.11/24 and encaps in this BD
ARP – Egress Leaf Unicast Routing Disable
Spine Spine ELAM the ARP request!

vsh_lc
Egress leaf(s) floods out trigger reset
3 trigger init in-select 14 out-select 1
all ports in the BD (except
with flood in encap) set inner arp source-ip 192.168.100.11
set inner arp target-ip 192.168.100.10
set inner l2 dst_mac ffff.ffff.ffff
Leaf Leaf Leaf start
stat
ELAM STATUS
===========
ARP: Who has Asic 0 Slice 0 Status Triggered
192.168.100.10? Asic 0 Slice 1 Status Armed
EP1 EP2
192.168.100.10/24 192.168.100.11/24
ARP – Egress Leaf Elam Bridge Domain Settings:
Results (ereport) ARP Flooding Enabled
Outer L3 Header
-------------------------------
Destination IP : 225.0.2.136 Destination is GIPO
(225.0.2.128) + FTAG (0x8)
Inner L3 Header
----------------------------------
ARP Sender IP : 192.168.100.11
ARP Target IP : 192.168.100.10
Outer L4 Header
---------------------------------------
VRF or BD VNID : 14811121( 0xE1FFF1 )
Contract Result
----------------------
Contract Drop : no
Frame is flooded in the Bridge Domain!
----------------------------------------------------------------------------------------
Bits set in Final Forwarding Block: : IFABRIC_EG MC INFRA ENCAP MYTEP BRIDGE MISS FLOOD
Lookup Drop
Not Dropped in lookups!
---------------------------
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
ARP – Egress Leaf Port is VPC ARP Flooding Enabled
• Both VPC members receive a flooded copy
• One VPC member is the Designated Forwarder (DF) for the flow
• DF is hashed per flow
• Only DF floods out VPC interfaces Non-DF Leaf

module-1(DBG-elam-insel14)# ereport | grep df | grep vpc
sug_lub_latch_results_vec.lub4_1.vpc_df: 0x0
sug_fpx_lookup_vec.lkup.dciptvec.pt.vpc_df: 0x0
sug_fpc_lookup_vec.fplu_vec.lkup.dciptvec.pt.vpc_df: 0x0
DF Leaf
module-1(DBG-elam-insel14)# ereport | grep df | grep vpc
sug_lub_latch_results_vec.lub4_1.vpc_df: 0x1
sug_fpx_lookup_vec.lkup.dciptvec.pt.vpc_df: 0x1
Debugging
ACI Bridged
Flows
Known Unicast – Ingress Leaf Unicast Routing Disable
Unknown Unicast Flood
Lookup dst mac in ingress BD
leaf103# show endpoint mac 0000.cccc.dddd
Spine Spine
+---------------+---------------+-----------------+-------------+
VLAN/ Encap MAC Address Interface
Domain VLAN IP Address
+---------------+---------------+-----------------+-------------+
32/CL2022:vrf1 vxlan-14811121 0000.cccc.dddd tunnel1
Since dst mac is not the leaf103# show int tun1

2 router (GW) mac, leaf does Tunnel destination 10.0.176.67
mac lookup in BD (usually
Leaf Leaf Leaf determined by ingress vlan)
1
Leaf looks at the dst mac to
Ping 192.168.100.10 determine if it should route or switch
EP1 EP2
192.168.100.10/24 192.168.100.11/24
Spine Spine ELAM

vsh_lc
debug plat internal app elam asic 0
trigger reset
set outer ipv4 src_ip 192.168.100.11 dst_ip 192.168.100.10
start
Leaf Leaf Leaf
Ping 192.168.100.10
EP1 EP2
192.168.100.10/24 192.168.100.11/24
Forwarding Verifications Dest mac that is looked up within BD
Outer L2 Header
-------------------------------------
Make sure this is the expected vlan
Destination MAC : 0000.cccc.dddd
Source MAC : 0000.aaaa.bbbb
Access Encap VLAN : 3502( 0xDAE )
Outer L3 Header
------------------------------------- Dest is tunnel
IP Protocol Number : ICMP
Source IP : 192.168.100.11
show plat internal hal tunnel rtep apd
Other Forwarding Information =================================
--------------------------------------- ifId IP RwEncapIdx
Encap Index is valid : yes =================================
Encap Index : 34( 0x22 ) 18010001 10.0.176.67 22
FINAL FORWARDING LOOKUP Forward to this overlay TEP

------------------------------------------------------------------------
Bits set in Final Forwarding Block: IFABRIC_IG UC TENANT MYTEP BRIDGE HIT
Lookup Drop
-------------------------- Not Dropped in lookups! Unicast + Bridge (L2 lookup) +
LU drop reason : no drop Destination Known
Forwarding Verifications Unknown Unicast Flood
ereport | grep "ovector "

ovector : 152( 0x98 )
show platform internal hal l2 port gpd
=========================================

=========================================
1a01c000 Eth1/29 0 59 2 18 18 98
Traffic is forwarded out Eth1/29!
Contract Verification Unknown Unicast Flood
Contract Lookup Key

------------------------------------------------
IP Protocol : ICMP( 0x1 ) Source and Dest EPG is the
L4 Src Port : 2048( 0x800 ) same. Implicitly permit!
L4 Dst Port : 35914( 0x8C4A )
sclass (src pcTag) : 49154( 0xC002 )
(unless isolation enabled)
dclass (dst pcTag) : 49154( 0xC002 )
src pcTag is from local table : yes
Unknown Unicast / Flood Packet : no
Contract Result Contract Applied and

---------------------------------------------- no Drop!
Contract Drop : no
Contract Hit : yes
Contract Aclqos Stats Index : 131025
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 131025" )
Known Unicast – Egress Leaf Unicast Routing Disable
leaf101# show endpoint mac 0000.cccc.dddd

Spine Spine
+---------------+---------------+-----------------+-------------+
3
Since VNID is the BD VNID, forward +---------------+---------------+-----------------+-------------+
based on dest endpoint mac 3/CL2022:vrf1 vlan-3501 0000.cccc.dddd po5
4
Policy was applied by ingress
Leaf Leaf Leaf leaf. Don’t apply contracts!
Forward out portchannel
5 in vlan 3501!
Ping 192.168.100.10
EP1 EP2
192.168.100.10/24 192.168.100.11/24
Spine Spine
Elam
trigger reset
start
Leaf Leaf Leaf
Ping 192.168.100.10
EP1 EP2
192.168.100.10/24 192.168.100.11/24
Inner L2 Header Unknown Unicast Flood
---------------------------------------
Inner Destination MAC : 0000.cccc.ddddContracts have already been
applied. No need to check.
Inner L3 Header
---------------------------------------
Destination IP : 192.168.100.10 Mac lookup done in bridge
domain with this VNID
Outer L4 Header
---------------------------------------
L4 Type : iVxLAN
Src Policy Applied Bit : 1 show platform internal hal l2 port gpd
Dst Policy Applied Bit : 1
VRF or BD VNID : 14811121( 0xE1FFF1 ) =========================================
Sideband Information =========================================
----------------------------------
ovector : 146( 0x92 )
1a021000 Eth1/34 0 32 1 9 12 92

-------------------------------------------------------------------------
Forward out Eth1/34!
Bits set in Final Forwarding Block: IFABRIC_EG UC INFRA ENCAP MYTEP BRIDGE HIT
Lookup Drop Unicast + Bridge (L2 lookup) +

------------------------
LU drop reason : no drop Destination Known
Debugging ACI
Routed Flows
Known Unicast – Ingress Leaf Unicast Routing Enabled
Lookup dst IP in ingress VRF
leaf103# show endpoint ip 192.168.100.10
Spine Spine +---------------+-----------------+-------------+
VLAN/ MAC Address Interface
Domain IP Address
+---------------+-----------------+-------------+
CL2022:vrf1 192.168.100.10 tunnel1
leaf103# show int tun1

Tunnel destination 10.0.176.67
Leaf Leaf Leaf 2

Since dst mac is the router (GW) mac, leaf
does IP lookup in VRF of source IP
1
Ping 192.168.100.10 determine if it should route or switch
EP1 EP2
192.168.100.10/24 192.168.200.11/24
Get Sclass
4 103# show sys internal epm endpoint ip
Leaf forwards packet to remote 192.168.200.11
Spine Spine TEP with VRF VNID set !omitted
BD vnid : 16613259 ::: VRF vnid : 2523136
sclass : 32771
Get Dclass
3 103# show sys internal epm endpoint ip
Leaf does contract
lookup based on src 192.168.100.10
and dst pcTag values !omitted
BD vnid : 0 ::: VRF vnid : 2523136
Leaf Leaf Leaf sclass : 49154
Check Contract
103# show zoning-rule src-epg 32771
dst-epg 49154 scope 2523136
+---------+------------------+--------+
Ping 192.168.100.10 | RuleID | Name | Action |
+---------+------------------+--------+
EP1 EP2 | 4209 | CL2022:allow-all | permit |
192.168.100.10/24 192.168.200.11/24 +---------+------------------+--------+
Spine Spine ELAM

vsh_lc
trigger reset
set outer ipv4 src_ip 192.168.200.11
start
stat
Leaf Leaf Leaf ELAM STATUS
===========
Ping 192.168.100.10
EP1 EP2
192.168.100.10/24 192.168.200.11/24
Forwarding Verifications ACI Router Mac. Route this packet!
Outer L2 Header
-------------------------------------
Destination MAC : 0022.BDF8.19FF
Access Encap VLAN : 3769( 0xEB9 )
Outer L3 Header
------------------------------------- Dest is tunnel
Source IP : 192.168.200.11
Encap Index : 34( 0x22 )
18010001 10.0.176.67 22
------------------------------------------------------------------------ Forward to this overlay TEP
Bits set in Final Forwarding Block: IFABRIC_IG UC TENANT MYTEP ROUTE HIT
Lookup Drop Unicast + Route (L3 lookup) +

-------------------------- Not Dropped in lookups! L3 Route Found
Forwarding Verifications

ovector : 152( 0x98 )
=========================================

=========================================
1a01c000 Eth1/29 0 59 2 18 18 98
Contract Verification
Contract Lookup Key
------------------------------------------------
IP Protocol : ICMP( 0x1 ) Source and Dest EPG used
L4 Src Port : 2048( 0x800 ) for contract lookup
L4 Dst Port : 31219( 0x79F3 )
sclass (src pcTag) : 32771( 0x8003 )

---------------------------------------------- no Drop!
Contract Drop : no
Contract Hit : yes
But how do I know which

contract this is actually hitting?
Contract Result
-------------------------------- Hardware Index of
Contract Drop : no matching contract
Contract Hit : yes
Run this from vsh_lc
show sys int aclqos zoning-rules | grep -B 9 "Idx: 130974"
===========================================
Rule ID: 4163 Scope 8 Src EPG: 32771 Dst EPG: 49154 Filter 532
Zoning-rule ID
Curr TCAM resource:
=============================
=== SDK Info ===
Result/Stats Idx: 130974
Run this from normal shell

show zoning-rule rule-id 4163
Traffic
+---------+--------+--------+----------+---------+------------------+--------+
hit this contract!
| Rule ID | SrcEPG | DstEPG | FilterID | Scope | Name | Action |
+---------+--------+--------+----------+---------+------------------+--------+
| 4163 | 32771 | 49154 | 532 | 2523136 | CL2022:allow-all | permit |
+---------+--------+--------+----------+---------+------------------+--------+
Known Unicast – Egress Leaf Unicast Routing Enabled

Spine Spine
+---------------+---------------+-----------------+-------------+
Since VNID is the VRF VNID, forward +---------------+---------------+-----------------+-------------+
based on dest endpoint IP 3 vlan-3501 0000.cccc.dddd po5
5 CL2022:vrf1 vlan-3501 192.168.100.10 po5
Policy was applied by ingress

Leaf Leaf Leaf 6 leaf. Don’t apply contracts! Forward out portchannel
5 in vlan 3501!
Ping 192.168.100.10
EP1 EP2
192.168.100.10/24 192.168.200.11/24
Spine Spine
Elam
trigger reset
set inner ipv4 src_ip 192.168.200.11
set inner ipv4 dst_ip 192.168.100.10
start
stat
===========
Ping 192.168.100.10
EP1 EP2
192.168.100.10/24 192.168.200.11/24
Inner L2 Header
---------------------------------------
Inner Destination MAC : 000C.0C0C.0C0CContracts have already been
applied. No need to check.
Inner L3 Header
---------------------------------------
Destination IP : 192.168.100.10 IP lookup done in VRF with this VNID
Outer L4 Header
---------------------------------------
L4 Type : iVxLAN
Dst Policy Applied Bit : 1 =========================================
VRF or BD VNID : 2523136( 0x268000 )
---------------------------------- 1a021000 Eth1/34 0 32 1 9 12 92
ovector : 146( 0x92 )
FINAL FORWARDING LOOKUP Forward out Eth1/34!

-------------------------------------------------------------------------
Bits set in Final Forwarding Block: IFABRIC_EG UC INFRA ENCAP MYTEP ROUTE HIT

------------------------
LU drop reason : no drop L3 Route Found
Proxied Unicast – Ingress Leaf Unicast Routing Enabled
1a
show endpoint ip 192.168.100.10
Spine Spine +---------------+-----------------+-------------+ No endpoint learn,
VLAN/ MAC Address Interface check route table!
Domain IP Address
+---------------+-----------------+-------------+
<empty> 1b show ip route 192.168.100.10 vrf CL2022:vrf1
192.168.100.0/24, attached, direct, pervasive
*via 10.0.176.66%overlay-1, [1/0], static
Send to
1c show isis dtep vrf overlay-1
spine proxy!
Leaf Leaf Leaf DTEP-Address Role Type
10.0.176.66 SPINE PHYSICAL,PROXY-ACAST-V4
1
Dst mac is router mac so Route!
Ping 192.168.100.10
Lookup dest IP in vrf of source IP.
EP1 EP2
192.168.100.10/24 192.168.200.11/24
Spine Spine
3
Leaf forwards packet to spine
proxy TEP with VRF VNID set
2
Leaf Leaf Leaf Contract not applied for proxy lookups.
Ping 192.168.100.10
EP1 EP2
192.168.100.10/24 192.168.200.11/24
Spine Spine ELAM

vsh_lc
trigger reset
start
stat
===========
Ping 192.168.100.10
EP1 EP2
192.168.100.10/24 192.168.200.11/24
Outer L2 Header
-------------------------------------
Access Encap VLAN : 3769( 0xEB9 )
Outer L3 Header
------------------------------------- Dest is tunnel
Source IP : 192.168.200.11
18010007 10.0.176.66 1


ovector : 152( 0x98 )
=========================================

=========================================
1a01c000 Eth1/29 0 59 2 18 18 98
Contract Lookup Key
------------------------------------------------
IP Protocol : ICMP( 0x1 ) Dest EPG is 1 for fabric
L4 Src Port : 2048( 0x800 ) owned subnets
L4 Dst Port : 31219( 0x79F3 )
dclass (dst pcTag) : 1( 0x1 )
Contract Result Contract not applied

---------------------------------------------- since this is proxied!
Contract Drop : no
Contract Applied : no
Contract Hit : yes
Proxied Unicast – Spine Unicast Routing Enabled
4 Since this is proxied, Spine does
COOP lookup. Since VRF vnid is set,
Spine looks up IP rather than mac
Spine Spine spine# show coop internal info ip-db | grep -B 1 -A 15 192.168.100.10
------------------------------
IP address : 192.168.100.10
Vrf : 2523136
Num tunnels : 1
Tunnel address : 10.0.176.67
Tunnel ref count : 1
Dest TEP of Leaf(s) that
Leaf Leaf Leaf own this Endpoint
apic1# moquery -c ipv4Addr -f 'ipv4.Addr.addr=="10.0.176.67"'
*node-101/*dom-overlay-1/if-[lo1]/addr-[10.0.176.67/32]
*node-102/*dom-overlay-1/if-[lo1]/addr-[10.0.176.67/32]
Ping 192.168.100.10
EP1 EP2
192.168.100.10/24 192.168.200.11/24
Proxied Unicast – Egress Leaf Unicast Routing Enabled

Spine Spine
+---------------+---------------+-----------------+-------------+
5
Since VNID is the VRF VNID, forward +---------------+---------------+-----------------+-------------+
based on dest endpoint IP 3 vlan-3501 0000.cccc.dddd po5
CL2022:vrf1 vlan-3501 192.168.100.10 po5
6
Policy was NOT applied by
Leaf Leaf Leaf ingress leaf. Apply contracts!
Forward out portchannel
5 in vlan 3501!
Ping 192.168.100.10
EP1 EP2
192.168.100.10/24 192.168.200.11/24
Spine Spine
Elam
trigger reset
start
stat
===========
Ping 192.168.100.10
EP1 EP2
192.168.100.10/24 192.168.200.11/24
Inner L3 Header Contracts have not been applied yet!
---------------------------------------
IP lookup done in VRF with this VNID
Outer L4 Header
---------------------------------------
L4 Type : iVxLAN
Dst Policy Applied Bit : 0 =========================================
VRF or BD VNID : 2523136( 0x268000 ) IfId Ifname As AP Sl Sp Ss Ovec
=========================================
Sideband Information
1a021000 Eth1/34 0 32 1 9 12 92
----------------------------------
ovector : 146( 0x92 )
Forward out Eth1/34!
-------------------------------------------------------------------------

------------------------ Not Dropped in lookups! L3 Route Found
Contract Lookup Key
------------------------------------------------
IP Protocol : ICMP( 0x1 )
L4 Src Port : 2048( 0x800 ) Source and Dest EPG used
L4 Dst Port : 33226( 0x81CA ) for contract lookup.
src pcTag is from local table : no

---------------------------------------------- no Drop!
Contract Drop : no
Contract Hit : yes

Contract Result
Contract Hit : yes
===========================================
Rule ID: 4234 Scope 16 Src EPG: 32771 Dst EPG: 49154 Filter
Zoning-rule ID
532
=============================
=== SDK Info ===

show zoning-rule rule-id 4234 Traffic hit this contract!
+---------+--------+--------+----------+---------+------------------+--------+
+---------+--------+--------+----------+---------+------------------+--------+
| 4163 | 32771 | 49154 | 532 | 2523136 | CL2022:allow-all | permit |
+---------+--------+--------+----------+---------+------------------+--------+
L3Out Destination – Ingress Leaf Unicast Routing Enabled
2a
Domain IP Address
+---------------+-----------------+-------------+
<empty> show ip route 10.99.99.100 vrf CL2022:vrf1
2
Since dst mac is the router (GW) mac, 10.99.99.0/24, ubest/mbest: 1/0
2b *via 10.0.64.70%overlay-1, [200/20], bgp-65100
leaf does IP lookup in VRF of source IP
Send to BL
acidiag fnvread | grep 10.0.64.70 PTEP!
Leaf Leaf Leaf Name IP Address Role
2c -----------------------------
leaf103 10.0.64.70/32 leaf
1
Ping determine if it should route or switch
10.99.99.100
EP1
External IP
192.168.100.10/24
10.99.99.100
0000.cccc.dddd
Spine Spine
3
Leaf derives dest pcTag for contract
lookup based on l3out policy prefix 4 Leaf forwards packet to
remote TEP with VRF VNID set
Leaf Leaf Leaf

leaf# vsh_lc -c "show forwarding route 10.99.99.100 platf vrf CL2022:vrf1"
!
!
Ping vrf: 16(0x10), routed_if: 0x0 epc_class: 32772(0x8004)
10.99.99.100
EP1
External IP
192.168.100.10/24
10.99.99.100
0000.cccc.dddd
ELAM
Spine Spine vsh_lc
trigger reset
start
stat
ELAM STATUS
Leaf Leaf Leaf ===========
Ping
10.99.99.100
EP1
External IP
192.168.100.10/24
10.99.99.100
0000.cccc.dddd
Outer L2 Header
-------------------------------------
Access Encap VLAN : 3501( 0xDAD )
Outer L3 Header
------------------------------------- Dest is tunnel
Source IP : 192.168.100.10
18010004 10.0.64.70 25


ovector : 48( 0x30 )
=========================================

=========================================
1a035000 Eth1/54 0 19 0 18 30 30
Contract Lookup Key
------------------------------------------------
IP Protocol : ICMP( 0x1 ) Source and Dest EPG used
L4 Src Port : 2048( 0x800 ) for contract lookup
L4 Dst Port : 12063( 0x2F1F )
sclass (src pcTag) : 49154( 0xC002 )
dclass (dst pcTag) : 32772( 0x8004 )

---------------------------------------------- no Drop!
Contract Drop : no
Contract Hit : yes

Contract Result
Contract Hit : yes
===========================================
Rule ID: 4248 Scope 16 Src EPG: 0 Dst EPG: 32772 Filter 532
Zoning-rule ID
Curr TCAM resource:
=============================
=== SDK Info ===

show zoning-rule rule-id 4248 Traffic hit this contract!
+---------+--------+--------+----------+---------+-----------------------+--------+
+---------+--------+--------+----------+---------+-----------------------+--------+
| 4248 | 0 | 32772 | 532 | 2523136 | CL2022:l3out-allow-all| permit |
+---------+--------+--------+----------+---------+-----------------------+--------+
L3Out Destination – Egress Leaf Unicast Routing Enabled
Lookup dst IP in received VRF
5a
Domain IP Address
+---------------+-----------------+-------------+
<empty> show ip route 10.99.99.100 vrf CL2022:vrf1
5
Since received VNID is the VRF VNID, 5b 10.99.99.0/24, ubest/mbest: 1/0
forward based on dest endpoint IP *via 10.55.0.100, vlan25, [110/20], ospf, type-2
show ip arp 10.55.0.100 vrf CL2022:vrf1
5c Address MAC Address Interface
Leaf Leaf Leaf 10.55.0.100 0005.73ff.593c vlan25
show mac address addr 0005.73ff.593c vl 25
5d VLAN MAC Address Ports
---------+----------------+----------
Ping * 25 0005.73ff.593c eth1/27/4
10.99.99.100
Forward based on ARP
EP1 6 and MAC Adajcencies
External IP Policy was applied by ingress
192.168.100.10/24
10.99.99.100 leaf. No need to apply contracts
0000.cccc.dddd
Spine Spine
Elam
trigger reset
start
stat
===========
Ping
10.99.99.100
EP1
External IP
192.168.100.10/24
10.99.99.100
0000.cccc.dddd
Inner L2 Header
---------------------------------------
Inner Destination MAC : 000C.0C0C.0C0C
Contracts have already been
Inner L3 Header applied. No need to check.
---------------------------------------
IP lookup done in VRF with this VNID
Outer L4 Header
---------------------------------------
L4 Type : iVxLAN
Dst Policy Applied Bit : 1
=========================================
VRF or BD VNID : 2523136( 0x268000 )
---------------------------------- 4301a000 Eth1/27/4 0 54 2 13 13 93
ovector : 147( 0x93 )
FINAL FORWARDING LOOKUP Forward out Eth1/27/4!

-------------------------------------------------------------------------

------------------------ L3 Route Found
L3Out Source – Ingress Border Leaf Unicast Routing Enabled
Spine Spine
2b
If dest IP is not learned endpoint and
subnet is BD subnet, proxy!
Forward based on longest prefix-match within

source VRF. EP learns are always longest.
2a
Leaf Leaf Leaf
1
If VRF is in ingress mode, BL
doesn’t apply policy
ICMP Reply
EP1
192.168.100.10/24
External IP Refer back to the Routed Known Unicast
10.99.99.100
0000.cccc.dddd and Proxied Unicast for more verifications
Complete your Session Survey
• Please complete your session survey
after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (open from Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events Mobile App or
by logging in to the Session Catalog and clicking the
"Attendee Dashboard” at
https://www.ciscolive.com/emea/learn/sessions/session-
catalog.html
Continue Your Education
Visit the Cisco Showcase for related demos.
Book your one-on-one Meet the Engineer meeting.
Attend any of the related sessions at the DevNet,

Capture the Flag, and Walk-in Labs zones.
Visit the On-Demand Library for more sessions

at ciscolive.com/on-demand.
Thank you
Early Access.
Yes, please.

BRKDCN 3900

Uploaded by

Copyright:

Available Formats

BRKDCN 3900

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKDCN 3900

Uploaded by

Copyright:

Available Formats

A Network Engineer’s

Blueprint for ACI Forwarding

Joe Young, ACI Technical Leader, Customer Experience

4 Enter messages/questions in the Webex space

Webex spaces will be moderated

COOP Council of Oracle Protocol GIPo Outer Multicast Group IP

VxLAN Virtual eXtensible LAN VNID Virtual Network Identifier

• Makes forwarding decisions (policy

• ACI can classify traffic based on its

• Traffic inherits the forwarding and

Endpoints Policy-Prefixes PcTags Contracts

• Used by App EPG’s • Used by External • The security ID of • Defines security

VLAN ID for external devices Internal ID on LEAF For forwarding

Access Encap VLAN PI-VLAN VxLAN ID PI-VLAN Access Encap VLAN

BD1 For BD SVI

EPG1 vxlan- vxlan-

At the Switch level an Endpoint is a Mac address OR an IP/IPv6

Most Common TEP Types

TEP Type What is it? What is it for?

a-leaf101# show ip interface loopback0

leaf# moquery -c tunnelIf -f 'tunnel.If.id=="tunnel1"'

leaf# moquery -c tunnelIf -f 'tunnel.If.id=="tunnel1"'

Leaf Leaf Leaf

leaf103# show vlan encap-id 100

#Check object model for IP Address Endpoint

Reference commands can be run from leafs only

#Check hardware level endpoint process directly

Query COOP for l3 entry and get parent l2 entry:

Query COOP for l3 only entry (such as an SVI IP):

Query COOP for l3 ep:

leaf101# show ip route 192.168.200.20 vrf CL2022:vrf1

192.168.200.0/24, ubest/mbest: 1/0, attached, direct, pervasive “Pervasive” indicates this is a

leaf101# vsh_lc -c "show forwarding route 192.168.200.20 platform vrf CL2022:vrf1"

Don’t apply policy, Forward to proxy Anycast!

Contracts and Forwarding Set policy-applied bits in

Policy enforcement affects only traffic to or from the L3Out.

• FTAGs are loop-free trees Spine Spine

from values 0 – 0xc

• Outgoing interfaces calculated

Check FTAG tree Check FTAG tree

FTAG Routes FTAG Routes This spine is the

FTAG ID: 0 [Enabled] Cost:( 1/ 7/ 0) FTAG ID: 0 [Root] [Enabled] Cost:( 0/ 0/ 0)

Endpoint 1 Only Leaf 1 and Spines have to

Three types of Spine Proxy TEP

4 No COOP entry Anycast 6 Flood this “Glean” to reserved multicast group

Is Dst MAC on Is Dst IP on LEAF has

LEAF knows LEAF knows Dst IP

If ARP Flooding is OFF, ARP

1st Gen Leaf • For traffic going to the cpu

EX (or Later) Leaf

knet1 would show Tx traffic

No decode necessary for kpm_inb (cpu) interface…Gleans aren’t easily readable

Legacy Behavior ACI Behavior

vsh_lc -c "show plat internal hal ep l2 mac <addr>"

vsh_lc -c "show plat internal

vsh_lc -c "show forwarding route <ip/pfx len> vrf <name>“

vsh_lc -c "show platform internal hal l3 routes vrf <name>”

When Troubleshooting Layer 3 Flows Always…

3) Check if there is an External Route

N9K-C9332PQ N9K-C9396PX • Complete separation of N9K-C-EX N9K-C-FXP • More flexible/scalable