BRKSPG-3012

BRKSPG-3012
SP Security
Leveraging BGP FlowSpec to protect
your infrastructure
Nicolas Fevrier, Technical Leader Engineering

@CiscoIOSXR
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
cs.co/ciscolivebot#BRKSPG-3012
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
What We Hope To Achieve With This Session
• Introduce BGP Flowspec
• Clarify what it can do and where it fits
• DDoS Mitigation is not the only use-case in production
• Provide one more tool to your networking belt
Me ?
• Nicolas Fevrier
• TL / Technical Marketing Engineer based in Paris
• Service Providers BU
• In Cisco since 2004
• Worked on all IOS XR Platforms
• from CRS-1 to NCS5500
• Worked in Services/Deployment and BU
BRKSPG-3012 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
You ?
CiscoLive attendees registered to this session
Agenda
• Introduction
• BGP FlowSpec Protocol Description
• Use-cases, Demo
w/ DDoS Mitigation
• Configuring the Protocol
• Caveats and Limitations
• Conclusion
Acknowledgements
• Andy Karch
• Bertrand Duvivier
• Gunter Van de Velde
• Brian Prater
• Kirill Kasavchenko
• Tomas Sundstrom
Another 180+ Pages Slidedeck ?
• 90 Minutes
• Large “Back Up Slides” section
• Use of “For your reference” logo
For Your
Reference
Introduction
Introduction
• August 2009, IETF ratified “Dissemination of Flow Specification Rules”
• Separation of controlling and forwarding plane. Sounds familiar ?
• A powerful tool in the SP Security toolbox but Use-cases are expending way
beyond Security
DP CP
Introduction
BGP FlowSpec is not:
• Netflow
• Sample traffic and generate records from local table collector
• Openflow
• But similarities exist
• Microflow Policing
• Per user rate-limiting, some overlap
Introduction
• A Controller programs remotely how packets
should be treated when received on Clients interfaces
• Remote PBR: redirect packet in VRF X
• Remote PBR: redirect packet to @IP X BGP FS
• Remote QoS: DSCP Marking client
• Remote QoS: Policing (rate-limiter)
• Remote ACL: Policing to 0 bps
BGP FS BGP
client
Data Redirect
UDP/TCP/ICMP
IPv4/v6 Remark
L2
BGP
RL / Drop
BGP FS
controller
Introduction: Rule is Description and Action
• BGP is used to program remotely a rule made of:
• A traffic description (v4/v6 L3/L4)
• An action Traffic Description Action
dst:2001:4:5::23/128 redirect-in-VRF Dirty
• Traffic received on client (ingress only today)
UDP:123 Size: 800-1500 rate-limit 0 bps
matching the Description will be applied
dst:1.2.3.4 SYN redirect-to-IP 20.2.3.4
the Action
src:4.0.0.1 TCP80 mark DSCP ef
BGP FS
client
Data
UDP/TCP
IPv4/v6 BGP FS
L2 controller
BGP FS
rule
BGP FlowSpec Components
Controller
• Injects rules remotely in the clients
• Needs to implement Control Plane (CP) at the minimum
• Examples of BGP FS Controllers:
• router (ASR9000, CRS, NCS 6000, XR 12000, …)
• server (ExaBGP, YABGP, Open Day Light, Arbor SP, …)
• virtual router (XRv 9000)
CP CP
DP BGP FS
BGP FS controller
client
Client
• Receives rules from Controller(s) and programs the match/actions in hardware
• Needs to implement both Control Plane (CP) and Data Plane (DP)
• Examples of BGP FS Clients:
• router (ASR 9000, CRS, NCS 6000, ASR 1000, CSR 1000v…)
CP CP
DP BGP FS
BGP FS controller
client
L2
Route-Reflector
• Receives rules from Controller(s) and distributes them to Clients
• Usually Control Plane only, doesn’t (need to) program the rules locally
BGP FS
• Examples of BGP FS Router-Reflector: RR
• ASR 9000, CRS, NCS 6000 or XRv 9000
CP
• ASR 1000, CSR 1000v
BGP FS
client
CP
DP
CP
CP CP BGP FS
DP DP controller
BGP FS BGP FS
client client
BGP FlowSpec
Uni-Directional
• BGP FS is not bi-directional
• One way arrow from Controller to Client  no feedback loop
• Need other mechanism to collect counters / stats and measure the impact of the
rule on traffic
CP CP
DP
BGP FS BGP FS
client controller
NC/XML
Show commands
BGP FlowSpec Session
Internal / External
• BGP FlowSpec follows the same rules than “traditional” BGP
• Rules received from eBGP are sent to other eBGP peers
• Rules received from eBGP are sent to iBGP peers
• Rules received from iBGP are sent to eBGP peers
• Rules received from iBGP are not sent to other iBGP peers unless the router is
configured as a route-reflector
iBGP iBGP
eBGP eBGP
CP
BGP FS BGP FS BGP FS BGP FS BGP FS
client client client client controller
AS X AS Y AS Z
BGP FlowSpec
Protocol Description
RFC 5575
Dissemination of Flow Specification Rules
• Why using BGP?
• Simple to extend by adding a new NLRI
• MP_REACH_NLRI / MP_UNREACH_NLRI
• Already used for every other kind of technology
• IPv4, IPv6, VPN, Multicast, Labels, MAC addresses, EVPN, …
• Point to multipoint with Route-Reflectors
• Inter-domain support
• Networking engineers and architects understand perfectly BGP
• Why not Openflow or direct NetConf to the router ?

• Strong framework exists with RR architecture, policies, HA, LLGR
• Data can be spread at scale and beyond the AS boundaries
RFC 5575
Dissemination of Flow Specification Rules: Traffic Matching
• NLRI defined (AFI=1, SAFI=133) to describe the traffic of interest
1. Destination IP Address
2. Source IP Address
3. IP Protocol Type Length
4. Port Address Family Identifier (AFI) 2 octets
5. Destination port Subsequent Address Family Identifier (SAFI) 1 octet
6. Source Port Length of Next Hop Network Address 1 octet
7. ICMP Type Network Address of Next Hop Variable
8. ICMP Code Reserved 1 octet
9. TCP Flags
Network Layer Reachability Information (NLRI) Variable
10. Packet length
11. DSCP
The MP_REACH_NLRI – RFC 4760
12. Fragment
RFC 5575
Dissemination of Flow Specification Rules: Traffic Matching
IPv4 Version IHL ToS Total Length
Not matched:
Identification Flags Frag Offset
- MPLS labels number
TTL Protocol Header Checksum
- MAC address
Source Address
- L5-7 data like
- HTTP URL
Destination Address
- Cookie
Options Padding
- DNS requests…
TCP Source Port Destination Port UDP Source Port Destination Port
Sequence Number Length Checksum
Ack Number Data
H lgh Res C bit Window
Checksum Urgent ICMP Type Code Checksum
Options Quench
Data Data
RFC 5575
Dissemination of Flow Specification Rules: Traffic Actions
• Traffic Action is defined in extended-communities (RFC4360)
Type Description Encoding

0x8006 Traffic-rate 2 bytes ASN; 4 bytes as float
0x8007 Traffic-action Bitmask
0x8008 Redirect 6 bytes RT (Route Target)
0x8009 Traffic-marking DSCP Value
RFC 7674
Clarification of the Flowspec Redirect Extended Community
• Following Redirect actions are supported since IOS XR 5.2.0
Type Description Encoding

0x8008 Redirect 2B ASN RT 2 Octets ASN , 4 Octets Value
0x8108 Redirect IPv4 RT 4 Octets IPv4 address, 2 Octets Value
0x8208 Redirect 4B ASN RT 4 Octets ASN, 2 Octets Value
Note: the IPv4 RT ( a.b.c.d : value ) is not the the redirect to IP action
IETF Drafts
Extensions for RFC5575: IETF Drafts
• On top of the RFC implementation, IOS XR supports:
• IPv6 extensions: draft-ietf-idr-flow-spec-v6-03
• Redirect IP extension: draft-simpson-idr-flowspec-redirect-02
• IBGP extension: draft-ietf-idr-bgp-flowspec-oid-01
• Persistence Support: draft-uttaro-idr-bgp-persistence-02 (in IOS XR5.2.2)
• HA/NSR Support
• Max-prefix
IETF Drafts
Extensions for RFC5575: IETF Drafts
• On top of the former list, IOS XE supports:
•draft-ietf-idr-flowspec-interfaceset-03
New Extended community to inform remote router where (interface) to apply the rule
Not supported on XR
For Your
Reference
IETF Drafts
Extensions for RFC5575: Unsupported IETF Drafts
• Other drafts are under work in the IDR group but not supported in IOS XR:
• Carrying Label Information for BGP FlowSpec: draft-ietf-idr-bgp-flowspec-label-01
• Dissemination of Flow Specification Rules for L2 VPN: draft-ietf-idr-flowspec-l2vpn-05
• BGP Flow Specification Filter for MPLS Label: draft-ietf-idr-flowspec-mpls-match-01
• BGP Flow Specification Packet-Rate Action: draft-ietf-idr-flowspec-packet-rate-01
• Flowspec Indirection-id Redirect: draft-ietf-idr-flowspec-path-redirect-01
• Dissemination of Flow Specification Rules: draft-ietf-idr-rfc5575bis-01
• Inter-provider Propagation of BGP Flow specification Rules:
draft-bashir-idr-inter-provider-flowspec-actions-00
• Populate to FIB Action for FlowSpec: draft-li-idr-flowspec-populate-to-fib-00
For Your
Cisco Routers BGP FS Implementation Reference
Platform Hardware Support in Data Plane

ASR 9k – Typhoon LC (MOD80/160, 24-36x10G, 1-2x100G) XR 5.2.0
ASR 9k – SIP700 XR 5.2.2
ASR 9001(-S) XR 5.2.2
ASR 9k – Tomahawk (MOD200/400, 4-8-12x100G) XR 5.3.0
CRS-3 (Taiko) LC (1x100G, 14-20x10G, Flex) XR 5.2.0
CRS-X (Topaz) LC (4x100G, 40x10G, Flex) XR 5.3.2
NCS 6000 XR 5.2.4 / 6.2.2 / roadmap*
XRv 9000 5.4.0 CP only / DP later
NCS 5000 / NCS 5500 In the roadmap
ASR 1000 IOS XE 3.15
CSR 1000v IOS XE 3.15
NCS 5500 (Jericho+ w/ eTCAM) XR 6.5.1
Note: IOS XE introduced the support of BGP FS in 3.15 (but not as a controller role)
For Your
Reference
Cisco IOS XR Routers BGP FS Implementation

Value input
NLRI type Match fields method
XR PI ASR9000 CRS NCS6000
Type 1 IPv4 Destination address Prefix length    
Type 2 IPv4 Source address Prefix length    
Type 3 IPv4 protocol Multi value range    
Type 4 IPv4 source or destination port Multi Value range    
Type 5 IPv4 destination port Multi Value range    
Type 6 IPv4 Source port Multi Value range    
Type 7 IPv4 ICMP type Multi value range    
Type 8 IPv4 ICMP code Multi value range    
Only Lower byte Only Lower byte Only Lower byte
IPv4 TCP flags (2 bytes
Type 9
include reserved bits)
Bit mask  Reserved and NS bit
not supported
Reserved and NS bit
not supported
Reserved and NS bit
not supported
Type 10 IPv4 Packet length Multi value range    
Type 11 IPv4 DSCP Multi value range    
Type 12 IPv4 fragmentation bits Bit mask  Only indication
of fragment  
For Your
Reference
Cisco IOS XR Routers BGP FS Implementation

Value input
NLRI type Match fields method
XR PI ASR9000 CRS NCS6000
Type 1 IPv6 Destination address Prefix length    
Type 2 IPv6 Source address Prefix length    
Type 3 IPv6 Next Header Multi value range    
Type 4 IPv6 source or destination port Multi Value range    
Type 5 IPv6 destination port Multi Value range    
Type 6 IPv6 Source port Multi Value range    
Type 7 IPv6 ICMP type Multi value range    
Type 8 IPv6 ICMP code Multi value range    
Only Lower byte Only Lower byte Only Lower byte
Type 9
Bit mask  Reserved and NS bit
not supported
Reserved and NS bit
not supported
Reserved and NS bit
not supported
Type 10 IPv6 Packet length Multi value range    
Type 11 IPv6 Traffic Class Multi value range    
Type 12 Reserved N/A N/A N/A N/A N/A
Type 13 IPv6 Flow Based (20 bytes) Multi value range  BRKSPG-3012
  
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
IOS XR Implementation
Resource Usage
• BGP Flowspec entries are stored in TCAM
• Up to 3000 simple rules per line card (limited on the controller today)
• Resource is finite and shared with other protocols
BGP FS
TCAM entries
Client QoS
ACL
• What is YOUR scale requirement ?
IOS XR Implementation Data
Intf x
Application on Interface IPv4/v6
• In current implementation, rules are applied:

Data
• in ingress Intf x
IPv4/v6
• on physical or logical interfaces (Link-bundles and dot1q) BE100
intf y
• but not on tunnels
• with IPv4 and IPv6 traffic
Data
Intf1.x
IPv4/v6
Data
IPv4/v6
Data
BE100.x
IPv4/v6
BGP FlowSpec with 6PE
• Network with legacy devices not supported dual-stack are leveraging 6PE to
transport IPv6 over MPLS
• When packets are received on PE routers, they are encapsulated in MPLS labels
• ASR9000 will be able to apply BGP FS rules on the P-PE interface receiving 6PE
labelled packets and match in the IPv6 Header (L3 and L4)
• Works also with VPNv4 and VPNv6 IPv6
PE payload
IPv6 hdr
IPv6 IPv6
payload payload
BGP FS
IPv6 hdr IPv6 hdr rule
6PE label 6PE label
LSP label
Internet v6 PE P PE Access
IOS XR Interface Disabled
rtrA BGP FlowSpec Enabled
BGP FlowSpec Disabled
Packet to 10.0.16.51 rtrB

10.0.16.0/20
rtrA BGP FS is applied to

Controller Rule: dst-IP: 10.0.16.0/20
Action: IP NH rtrA
the whole router but
can be activated or
Packet to 10.0.16.51 rtrB deactivated on
10.0.16.0/20
particular interfaces via
rtrA CLI configuration.
Controller Rule: dst-IP: 10.0.16.0/20 Particularly useful in
Action: IP NH rtrA
Distributed DDoS
rtrB mitigation architecture.
Packet to 10.0.16.51
10.0.16.0/20
IOS XE Implementation
• Implementation on IOS XE is very similar than the IOS XR one (sharing a lot of
code), hence the features are almost identical but with a different scale support
ASR1000 CSR1000v ISR4400
Max rules per system 4000 250 4000
Max rules per VRF 1000 32 250
Use-cases:
DDoS Mitigation
DDoS Attacks
• No longer necessary to explain the risk
• Distributed Denial of Service (DDoS) is a lucrative activity for attackers
• ISP, Hosting Services, Enterprises: it can jeopardize your business
Everyone is at risk
• 2017:
• More sophisticated
• Less volumetric
• But still very high
Source: http://www.digitalattackmap.com/
DDoS Attacks
• Denial of Service attacks are of different natures:
• Application-layer attacks
• Detected and handled by Firewalls, IDS or at the Server level
• Volumetric attacks (including Protocols attacks)
• Can NOT be mitigated in datacenter or server farm (too late)
• Should be handled in the backbone or at the border
Web Web
Server Cache
IPS/IDS Firewall
Edge
The Internet
Core
Peering DC
Transit
DPI Database
DDoS Attacks Mitigation
• BGP FS was initially designed with DDoS Mitigation use-case in mind
• Distributed attack received from all transit and peering points
• We can use a mitigation system in a ASR9000/VSM card or an appliance
connected to your IOS-XR router
• We differentiate arbitrarily three DDoS attack families:
• Stateless Amplification
• Stateless L3 / L4 / others
• Stateful / up-to-L7 on application resources
Different Business, Different Targets
DataCenter and Hosting DataCenter
Web Web
Server Cache
! !
!
DC ! !
Edge Firewall
The Internet
Peering !
Transit Core Database
Enterprise
Agg PE Fw IPS/IDS DNS, Mail,
! ! ! ERP, SAN, …
!
! !
! Residential
! ! ! ! DPI
! !
! ! ! ! ! !
! ! ! ! ! !
! ! ! ! ! !
Use-cases:
DDoS Mitigation
Amplification Attacks
DDoS Mitigation with BGP FS
Amplification Attacks 101
• Stateless attacks are not using a full handshake and are based on spoofed
source addresses
• Amplification attacks using vulnerable protocols on high bandwidth servers
– DNS
Much larger reply
– NTP
– CharGen
– SNMP
2.1.1.1
– SSDP
– RIPv1
– Port Mapper
Small request
Amplification Attacks Always Trendy
• 2015/2016  raise of Amp attacks

• 2016/2017  botnets (Mirai, Satori, …)
• Victims
• #1 Online Gaming
• #2 Criminal demonstration
• #3 Extortion
Source: Arbor WISR 2018
Amplification Attacks Always Trendy Source: Akamai State of the Internet 2017
• But Amplification attacks didn’t

disappear
• UDP Frag, DNS and NTP still
in the top 3
Source: Arbor WISR 2018
Amplification Attacks Always Trendy
Source: https://www.shadowserver.org/
Rate-limiting / Filtering Amplification Attacks
• Amplification attacks, example NTP

• Don’t need to be handled by a “sophisticated” scrubbing system to be mitigated
• Can be filtered at the router line card level  much higher performance
• Identified by precisely matching the traffic pattern and filtered at the edge router level
<100Gbps
800Gbps / LC
Data
Length
2.1.1.1
UDP 123 Match: dest-IP: 2.1.1.1

+ src-port: UDP 123
IPv4 + size <1000B
Action: rate-limit 0bps
BGP FS
controller 47
BRKSPG-3012 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fragments
• Very often seen with amplification attacks (packets larger than the path MTU)
Data
2000B
UDP 123 Data 500B

IPv4 IPv4
Frag
2.1.1.1
Data
1500B Match: dest-IP: 2.1.1.1
+ frag field set
UDP 123
IPv4 BGP FS
controller
Use-cases:
DDoS Mitigation
L3/L4 Attacks
Rate-limiting / Filtering Stateless Attacks: L3/L4 Protocol Attacks
• Generic family covering non-amplified stateless streams like ICMP flood
• Source address could be forged or not (botnet members are corrupted hosts)
2.1.1.1
Rate-limiting / Filtering Stateless Attacks: L3/L4 Protocol Attacks
• L3/L4 attacks can be also filter at the edge router via BGP FS
• Same principles than previous use-case
2.1.1.1
Match: dest-IP: 2.1.1.1

+ ICMP 0/9
BGP FS
controller
Use-cases:
DDoS Mitigation
Stateful Attacks
Addressing Stateful Attacks
• More advanced attacks using Botnets or even real users (LOIC) needs to be
addressed differently by a specific scrubbing device. Examples:
• HTTP: bots mimicking the behavior of a real web browser
• TCP SYN
• SSL
• SIP
• … Requests
2.1.1.1
Replies
Addressing Stateful Attacks
• BGP FlowSpec will be used to program a different action here
• Diversion to a next-hop address
• Diversion to a different VRF
2.1.1.1

+ dest-port: 80
Action: NH @TMS
BGP FS
controller
DDoS Mitigation Demo
with Arbor Solution
Demo
Rate-limiting and Redirect Attacks Traffic w/ BGP FlowSpec
• Edited version of a recording from Tomas Sundstrom

• Using Arbor TMS as a controller and ASR9000 as client
192.168.9.2
7.7.7.7
• Detection of the attack itself is out of the scope of this short demo
Demo
• First attack is identified as a TCP 80 SYN with very large packet size
• We will use BGP FS to divert the TCP 80 traffic targeted to 7.7.7.7 into the TMS
192.168.9.2
TCP 80 SYN
7.7.7.7

+ src-port: TCP 80
Action:
NH 192.168.9.2
BGP FS
controller
Demo
• Second attack is identified as a NTP Amplification (abnormal packet size)

• We will use BGP FS to drop UDP 123 packets from 300 to 1000 bytes
192.168.9.2
7.7.7.7
UDP 123
Match:
src-port: UDP 123
+ size 300-B1000B
BGP FS
controller
Demo
Rate-limiting and Redirect Attacks w/
BGP FlowSpec
http://bit.ly/2rYSKY9
Arbor SP Solution
Dynamic Black-list Offload with BGP FlowSpec
1• A countermeasure is activated
and detects an offender
2• TMS instructs the ASR9000 via 1
Flowspec to program an ACL for offender victim
the src-@ or the pair src-@+dst-@
 For one minute
src-@ dst-@
3• After 1min, the ACL is removed.
If the offender is seen by the
countermeasure again, ACL will be
programmed for 5min, and then 5
Match: src-IP: 2.1.1.1
min, again and again 3
Action: drop 2
• No “drop” in BGP Flowspec actions, just a policer to 0 bps
• In DDoS attack context what could be the benefits of rate-limiting to X bps
instead of 0 bps
• X bps will drop packets randomly (legitimate or malicious ones equally), creating
difficult troubleshooting situation
• 0 bps is advised
Benefits
• Single point of control to program rules in many clients

• Granularity: Allows a very precise description/matching of the attack traffic
• Can be used for both mitigation and diversion of the attack traffic, without
impact the course of the rest of the traffic targeted to the victim
• Off-Load Mitigation system: Filtering stateless attacks on the edge route
permits mitigation of millions of PPS of dirty traffic while liberating precious CPU
cycle on the scrubbing device for more advanced mitigation needs
• Useful but not “Magic”: can not be do much for stateful attacks (Mirai, etc)
Improving Existing DDoS
Mitigation Models
DDoS Mitigation Models
Network Design
• Several approaches exist in the design of a DDoS mitigation solution

• No real “best practices” in this field, it mainly depends on
• The topology
• The protocols and services: IP only, MPLS transport, L2/L3VPN
• They all consist in:

• Diverting the traffic targeted to the victim to push it into scrubbing devices
• Performing an analysis of the packets to discriminate legit packets from attack packets
• Re-injecting the legit traffic into the network
• Following examples are real-case used in very large production networks
IP-only Network w/ Distributed TMS
Currently deployed
• A static route for 10.51.51.51 is defined on routers M and J pointing to local TMS
static
10.51.51.51/32
10.2.1.2
Te0/0/0/0.1
.2
2.1.1.0/24 10.2.1.0/30
IGP J
SA1 SA2 rtrA nh:rtrI J Te0/0/0/0.2
10.1.1.0/30 10.1.2.0/30 10.2.2.0/30 .2
M
I F Victim
10.51.51.51/32
10.1.1.2 2.1.1.1
static L H E C
M
2.1.1.0/24
2.1.1.0/24 K G D B A
rtrA nh:rtrK
IGP IP Network
Currently deployed
BGP BGP
2.1.1.1/32 BGP 2.1.1.1/32
10.51.51.51 10.51.51.51
2.1.1.1/32
Te0/0/0/0.1
.2 nh:10.51.51.51 10.2.1.0/30
J J
SA1 SA2 Te0/0/0/0.2
10.1.1.0/30 10.1.2.0/30 10.2.2.0/30 .2
Collector
M
I F Victim
2.1.1.1
L H E C
M
2.1.1.0/24
K G D B A
10.51.51.51 is a dummy route, advertised to trigger the redirection

Currently deployed
BGP BGP
2.1.1.1/32 2.1.1.1/32
10.51.51.51 10.51.51.51
static static Te0/0/0/0.1
.2 10.2.1.0/30
10.51.51.51/32 10.51.51.51/32
10.1.1.2
J 10.2.1.2 J
SA1 SA2 Te0/0/0/0.2
10.1.1.0/30 10.1.2.0/30 10.2.2.0/30 .2
2.1.1.1/32
M
10.51.51.51
nh:10.2.1.2
I F Victim
2.1.1.1
L H E C
M
2.1.1.0/24
2.1.1.1/32 K G D B A
10.51.51.51
nh:10.1.1.2
Currently deployed
• With the specific route received we now have to deal with a routing loop for the
legit traffic going out of the TMS device. We need solutions to prevent it
BGP static
2.1.1.1/32 10.51.51.51/32
10.51.51.51 10.2.1.2
BGP
2.1.1.1/32
10.51.51.51 Te0/0/0/0.1 .2
.2
static 10.2.1.0/30
10.51.51.51/32 J
10.1.1.2 BE1.1 BE1.2 Te0/0/0/0.2
10.1.1.0/30 10.1.2.0/30 10.2.2.0/30
L
M
K I
Solution to Avoid the Routing Loop (without BGP FS)
• Define an VRF-Lite Clean and assigned the egress TMS interfaces to it
• We need two sub-interfaces to the core, one in GRT, one in the clean VRF
• In the clean VRF, to pick the best path to the destination, we need the full IGP table
IGP
BGP 10.1.1.2/30 10.1.2.2/30 .2

2.1.1.1/32
10.51.51.51
Te-0/0/0/0.2
static
BE1.3
L BE1.2 BE1.3 Te-0/0/0/0.2
L
10.51.51.51/32 BE1.2
Te-0/0/0/0 Te-0/0/0/0
10.1.1.2
M M
Te-0/1/0/1.1 Te-0/1/0/1.1
Te-0/1/0/1.2
Te-0/1/0/1.2
Te-0/0/0/0.1 Te-0/0/0/0.1
Te-0/0/0/0.2 Te-0/0/0/0.2
K K
BGP FlowSpec Improvement: Granularity
• BGP FS defines precisely the flow to divert to the local scrubbing device
Rule: dest-IP: 2.1.1.1
+ dest-port: 80
+ dest-port: 80
Action: NH: 10.1.1.2
Action: NH: 10.2.1.2
BGP FS Te0/0/0/0.1
advertisement 10.2.1.0/30
J J
SA1 SA2 Te0/0/0/0.2
10.1.1.0/30 10.1.2.0/30 10.2.2.0/30 .2
FS
M controller
I F Victim
2.1.1.1
L H E C
M
2.1.1.0/24
K G D B A
BGP FlowSpec Improvement: No VRF-Lite needed
• BGP FlowSpec is activated on Te0/0/0/1, dirty traffic targeted to 2.1.1.1:80 is
forwarded to the scrubbing device address 10.2.1.2
• BGP FlowSpec is deactivated on port te0/0/0/0.2, clean traffic from the
scrubbing device is routed naturally via IGP route 2.1.1.0/24 to router I
2.1.1.0/24
.2 rtrA nh:rtrI
BGP FlowSpec
BGP FlowSpec Te0/0/0/0.1
Te0/0/0/1 10.2.1.0/30 .2 Match: dest-IP: 2.1.1.1
Match: dest-IP: 2.1.1.1 + dest-port: 80
+ dest-port: 80 J Action: NH: 10.1.1.2
BE1.1
10.1.1.0/30
BE1.2
10.1.2.0/30
Action: NH: 10.2.1.2 Te0/0/0/0.2 L
10.2.2.0/30
Te0/0/0/3
2.1.1.0/24 M
rtrA nh:rtrI Te0/0/0/1
I K
Other Use-Cases
Other BGP FS Use-Cases
Unequal Load-Balancing
• Different peering / transit points

• Different NATing points with different performances / capabilities
CGSE+
B
10.0.16.0/20 30Gbps
10.0.1.0/24 10Gbps
A
10.0.0.0/24 10Gbps
The Internet
ISM
• Based on the source ranges,

we will divert traffic to one CGN
Rule: src-IP: 10.0.16.0/20
engine or another Action: IP NH CGSE+
Rule: src-IP: 10.0.1.0/24
Action: IP NH ISM BGP FS
RR
Rule: src-IP: 10.0.0.0/24 controller
Action: IP NH CGSE+
CGSE+
B
10.0.16.0/20 30Gbps
10.0.1.0/24 10Gbps
A
10.0.0.0/24 10Gbps
The Internet
ISM
• This approach allows fine tuning of the traffic in the NAT engines, advertising
one prefix with one NH or another
Rule: src-IP: 10.0.16.0/20
Action: IP NH CGSE+
Rule: src-IP: 10.0.1.0/24

Action: IP NH ISM
CGSE+ Rule: src-IP: 10.0.0.0/24

Action: IP NH CGSE+
B
10.0.16.0/20 30Gbps
10.0.1.0/24 10Gbps
A
10.0.0.0/24 10Gbps
The Internet
ISM
Low QoS Priority Traffic for DDoS Attacks
• Important back-up is using 3 Gbps of traffic
• Simultaneously, a DDoS attack starts and is Scrubbing
diverted to the scrubbing center Center
S
• Links are not saturated
6G
L H C 3G Victim
2G M 3G 2.1.1.1
2G K G B A
2G
3G
• The attack intensity increases
• Links are congested, it impacts internal traffic Scrubbing
Center
S
8G
10G L H C 2G Victim
4G M 2G 2.1.1.1
4G K G B A
4G
3G
Low QoS Priority Traffic for DDoS Attacks w/ FlowSpec
• BGP FS rule forces the route leaking in VRF-Dirty and positioning a DSCP field
Scrubbing
Center
BGP FS
Rule: dest-IP: 2.1.1.1 S

Action: Set DSCP X
+ NH: VRF Dirty
7G
10G L H C 3G Victim
4G M 3G 2.1.1.1
4G K G B A
4G
3G
 Routers will dropped attack traffic in priority in
case of congestion based on this DSCP field
Transit AS Policing
• A transit provider is offering to AS Customer a 10GE connectivity to the Internet

• An asset in AS Customer is under a heavy DDoS attack of 50Gbps
• It’s pointless for AS Transit to transport the 50Gbps in it’s infra to drop it on the
last router connecting to AS Customer
Victim
20G 2.1.1.1
10G
20G
AS Transit AS Customer
Transit AS Policing
• AS Transit programs a BGP FS rule to rate-limit traffic targeted to the victim IP

address at the level of the committed bandwidth (10Gbps here)
 Impact is the same for AS Customer
but this approach offloads AS Transit’s backbone
BGP FS

Action: police 10G
Victim
20G 2.1.1.1
10G
20G
Give the Power to the Victim
• Rule disseminated upstream to tackle the attack as early as possible

• Not popular today, but may change in the future
UDP 123 Size<500B
Action: Drop
eBGP FS
iBGP eBGP iBGP
Victim
20G 2.1.1.1
10G
20G
Enterprise PBR
• Used to redirect traffic

through security devices
A
• Used to select a transport

(MPLS or Internet)
MPLS Internet
C D
Enterprise PBR: Security Classification
• Based on source addresses

and application ports
Rule: dst-port: mail
• Packets diverted to specific Src-Add: y.y.y.0/24
Action: NH B
security appliances Rule: dst-port: http
(proxy, antispam, waf, fw, …) Src-Add: z.z.z.0/24
Action: NH B
A
Enterprise PBR: “SD-WAN”
• Based on dstination addresses Rule: dst-port: AAA

Dst-Add: x.x.x.0/24
and application ports Action: NH D
Rule: dst-port: BBB
• Packets use different Dst-Add: x.x.x.0/24
Action: NH C
transport
MPLS Internet
C D
BGP FlowSpec Configuration
Configuring BGP FlowSpec on IOS XR Routers
Overview of the Configuration Steps
• On both Client and Controller

address-family flowspec
BGP Peer
IPv4/IPv6/vpnv4/vpnv6
• On Client
disable
local-install
flowspec specific
interface-all
interfaces
• On Controller
flowspec
class-map policy-map
service-policy pbr
C3PL model
Note: all examples in following slides are equally valid for IPv4 and IPv6
Signalization: Use of a new Address-Family flowspec
Controller Client
router bgp 1 router bgp 1
bgp router-id 6.6.6.6 bgp router-id 3.3.3.3
address-family ipv4 flowspec address-family ipv4 flowspec
! !
neighbor-group ibgp-flowspec neighbor-group ibgp-flowspec
remote-as 1 remote-as 1
update-source loopbook0 update-source loopback0
! !
! neighbor 25.2.1.11
neighbor 25.2.1.3 use neighbor-group ibgp-flowspec
use neighbor-group ibgp-flowspec !
! !
neighbor 25.2.1.4 flowspec Install all rules
use neighbor-group ibgp-flowspec local-install interface-all
! ! on all interfaces
!
flowspec
address-family ipv4
service-policy type pbr FS
Advertise
policy FS
Configuring Rules on the Controller
class-map type traffic match-all match-UDP53
match destination-port 53
match protocol udp
end-class-map
!
class-map type traffic match-all match-src-ipv4-addr
match destination-address ipv4 25.1.104.0 255.255.255.0
end-class-map
!
policy-map type pbr FS

class type traffic match-src-ipv4-addr
police rate 100000 bps
!
!
class type traffic match-UDP53 flowspec
redirect next 192.42.52.125 address-family ipv4
! service-policy type pbr FS
!
class type traffic class-default
!
end-policy-map
For Your
Reference

Configuring Rules on the Controller
class-map type traffic match-all MATCH-UDP123 class-map type traffic match-all MATCH-UDP123
match destination-port 123 match destination-port 123
match protocol udp match protocol udp
end-class-map end-class-map
! !
class-map type traffic match-all MATCH-SRCv4 class-map type traffic match-all MATCH-SRCv4
match destination-address ipv4 2.1.1.0/24 match destination-address ipv4 2.1.1.0/24
! !
policy-map type pbr FS1 policy-map type pbr FS
class type traffic MATCH-SRCv4 class type traffic MATCH-SRCv4
police rate 100000 bps police rate 100000 bps
! !
end-policy-map class type traffic MATCH-UDP123
! redirect nexthop 192.168.2.5
policy-map type pbr FS2 !
class type traffic MATCH-UDP123 end-policy-map
redirect nexthop 192.168.2.5
! flowspec
end-policy-map address-family ipv4
service-policy type pbr FS
flowspec
address-family ipv4
service-policy type pbr FS1
service-policy type pbr FS2
Configuration Demo
http://bit.ly/bgpfs-config
Configuring a Type 1 Match “Destination Address”
RP/0/0/CPU0:Ctrl(config)#class-map type traffic match-all MATCHING-RULE

RP/0/0/CPU0:Ctrl(config-cmap)#match destination-address ipv4 81.253.193.0/24
RP/0/0/CPU0:Ctrl(config-cmap)#
RP/0/RP0/CPU0:Client#sh flowspec ipv4 detail

Type Prefix length Prefix
AFI: IPv4
Flow :Dest:81.253.193.0/24 1 byte 1 byte Variable
Actions :Traffic-rate: 100000 bps (bgp.1)
Statistics (packets/bytes) 1 /24 81.253.193
Matched : 0/0
Transmitted : 0/0
0 x01 0x18 0x 51 fd c1
Dropped : 0/0
RP/0/RP0/CPU0:Client#sh flowspec ipv4 nlri 0x011851fdc1
AFI: IPv4
NLRI (Hex dump) : 0x011851fdc1
RP/0/RP0/CPU0:Client#
Mixing Several Matching Statements
class-map type traffic match-all MATCHING-RULE1
match source-port 10 20 30-40 50-52 60-70
match protocol udp
match dscp ef
match packet length 10-100 102-200 202-400 402-1500
match destination-address ipv4 11.200.4.0 255.255.255.0
end-class-map
RP/0/RSP0/CPU0:Client#sh flowspec afi-all detail

AFI: IPv4
Flow
:Dest:11.200.4.0/24,Proto:=17,DPort:=80,SPort:=10|=20|>=30&<=40|>=50&<=52|>=60&<=70,Length:>=10&<
=100|>=102&<=200|>=202&<=400|>=402&<=1500,DSCP:=46
Statistics (packets/bytes)
Matched : 0/0
Dropped : 0/0
RP/0/RSP0/CPU0:Client#sh flowspec afi-all nlri
AFI: IPv4
NLRI (Hex dump) :
0x01180bc80403811105815006010a0114031e452803324534033cc5460a030a4564036645c803ca550190130192d505d
c0b812e
RP/0/RSP0/CPU0:Client# BRKSPG-3012 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
Configuring an Action: Police
RP/0/0/CPU0:Ctrl(config)#policy-map type pbr FS
RP/0/0/CPU0:Ctrl(config-pmap)# class type traffic MATCHING-RULE1
RP/0/0/CPU0:Ctrl(config-pmap-c)#police ?
rate Committed Information Rate
RP/0/0/CPU0:Ctrl(config-pmap-c)#police rate ?
<1-4294967295> Committed Information Rate
RP/0/0/CPU0:Ctrl(config-pmap-c)#police rate 1000 ?
bps Bits per second (default)
cellsps Cells per second
gbps Gigabits per second
kbps Kilobits per second
mbps Megabits per second
<cr>
RP/0/0/CPU0:Ctrl(config-pmap-c)#police rate 1000
RP/0/0/CPU0:Ctrl(config-pmap-c)#
Hex 4a3ebc20 = 31,125,000 Bytes/sec

= 25 Mbps
Configuring an Action: Police
Application FS Rule
Drop = Police at 0bps
Configuring an Action: Redirection
• If the ingress interface is in the Global Routing Table, the flowspec rule should
be advertised via an “address-family IPv4 flowspec”
• Redirection to an NH address implies the egress interface is in the GRT too
• Redirection to a different VRF can not specify the destination address, a second
lookup in this target VRF will happen to the destination address of the packet
AF ipv4 FS rtrA AF ipv4 FS rtrA
1.2.3.2/30 1.2.3.2/30
Action: Action:
Redirect 1.2.3.2 Redirect VRF Blue
VRF
GRT
Blue
GRT GRT
Configuring an Action: Example of a Redirection to an IP Address
Controller Configuration Client View
RP/0/RSP0/CPU0:Client#show bgp ipv4 flowspec
policy-map type pbr TEST <SNIP>
class type traffic MATCHING-RULE1 Status codes: s suppressed, d damped, h history, * valid, > best
redirect nexthop 25.3.9.3 i - internal, r RIB-failure, S stale, N Nexthop-
! discard
class type traffic class-default Origin codes: i - IGP, e - EGP, ? - incomplete
! Network Next Hop Metric LocPrf Weight
end-policy-map Path
! *>iDest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550/128
traffic MATCHING-RULE1 25.3.9.3 100 0 i
class-map type traffic match-all MATCHING-RULE1
match protocol udp Processed 1 prefixes, 1 paths
match packet length 500-1550
match destination-address ipv4 25.1.102.1 RP/0/RSP0/CPU0:Client#show flowspec afi-all detail
255.255.255.255
end-class-map AFI: IPv4
! Flow :Dest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550
Actions :Nexthop: 25.3.9.3 (bgp.1)
Matched : 0/0
Dropped : 0/0
RP/0/RSP0/CPU0:Client#
For Your
Reference
Action Redirect: Digging Deeper

Controller Configuration
policy-map type pbr test
class type traffic test
redirect ipv4 nexthop 25.3.9.3
!
end-policy-map
Client View (Debug all Flowspec Events)

bgp[1052]: FlowSpec: Updating NLRI Proto:=6,DPort:=80 for TBL default:IPv4.
flowspec_mgr[1094]: FlowSpec: Client bgp.1 NLRI Proto:=6,DPort:=80 Update for TBL default:IPv4.
flowspec_mgr[1094]: FlowSpec: Added client bgp.1 flow active Proto:=6,DPort:=80 with actions IP-25.3.9.3 from TBL
default:IPv4.
flowspec_mgr[1094]: FlowSpec: Finished receving 1 IPC msgs for conn 0x20000099, 0:No error.
In this case, we used an IPv4 address for the Next-Hop.

It’s transported as a BGP attribute and no longer as an Extended Community
Gotchas with Redirect Action
• A rule is advertised from controller only if the configured NH is reachable
• Not necessary reachable on the client side but mandatory on the controller side
RP/0/0/CPU0:Ctrl#sh route 25.1.102.1 RP/0/RSP0/CPU0:Client#sh bgp ipv4 flowspec
RP/0/RSP0/CPU0:Client#sh bgp ipv4 flowspec sum

% Network not in table
Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
25.2.1.11 0 1 16488 16457 596 0 0 00:32:57 0
RP/0/0/CPU0:Ctrl#
RP/0/0/CPU0:Ctrl#sh run router static RP/0/RSP0/CPU0:Client#show bgp ipv4 flowspec

router static
Status codes: s suppressed, d damped, h history, * valid, > best
address-family ipv4 unicast
i - internal, r RIB-failure, S stale, N Nexthop-discard
25.3.9.3/32 GigabitEthernet0/0/0/0 Origin codes: i - IGP, e - EGP, ? - incomplete
! Network Next Hop Metric LocPrf Weight Path
! *>iDest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550/128
25.3.9.3 100 0 i
Processed 1 prefixes, 1 paths

Gotchas with Redirect Action
• If the NH is not reachable in the Client, the rule will be ignored

RP/0/RSP0/CPU0:Client#sh route 11.22.33.44 RP/0/RSP0/CPU0:Client#show bgp ipv4 flowspec
Dest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550/128 detail
% Network not in table BGP routing table entry for
Dest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550/128
RP/0/RSP0/CPU0:Client# <SNIP>
Last Modified: Feb 8 12:55:45.095 for 00:01:19
RP/0/0/CPU0:Ctrl#sh run policy-map type pbr TEST Paths: (1 available, no best path)
policy-map type pbr TEST Not advertised to any peer
class type traffic MATCHING-RULE1 Path #1: Received by speaker 0
redirect nexthop 11.22.33.44 Flags: 0x4000000000020005, import: 0x20
! Not advertised to any peer
class type traffic class-default Local
! 11.22.33.44 (inaccessible) from 25.2.1.11 (6.6.6.6)
end-policy-map Origin IGP, localpref 100, valid, internal
! Received Path ID 0, Local Path ID 0, version 0
RP/0/0/CPU0:XRv-service#sh run router static Extended community: FLOWSPEC Redirect-IP:0
router static
address-family ipv4 unicast RP/0/RSP0/CPU0:Client#show flowspec afi-all detail
11.22.33.44/32 GigabitEthernet0/0/0/0
! RP/0/RSP0/CPU0:Client#
!
RP/0/0/CPU0:Ctrl# No blackhole
Mixing Multiple Actions
• We can mix several Actions:

• Rate-limit + Redirect VRF/IP DSCP
Rate-limit Redirect
• Rate-limit + DSCP Marking
Marking
• Redirect VRF/IP + DSCP Marking
• Rate-limit + Redirect VRF/IP + DSCP Marking
• It’s not possible to mix:

• Redirect VRF + Redirect NH IP
• Redirect NH IP@A + Redirect NH IP@B
AFI: IPv4
Flow :Dest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550
Actions :Traffic-rate: 100000 bps DSCP: ef Nexthop: 25.3.9.3 (bgp.1)
Matched : 75899782/106259694800
Dropped : 75686514/105961119600
BGP Persistence
• In IOS XR 5.2.2 we introduced the support of the LLGR draft
draft-uttaro-idr-bgp-persistence-02
• Both sides need to negotiate this capability when establishing the session
neighbor-group ibgp-flowspec neighbor-group ibgp-flowspec
remote-as 1 remote-as 1
update-source GigabitEthernet0/0/0/0 update-source GigabitEthernet0/0/0/0
long-lived-graceful-restart capable long-lived-graceful-restart capable
long-lived-graceful-restart stale-time send 360 accept 360 long-lived-graceful-restart stale-time send 360 accept 360
! !
neighbor 10.0.0.2 neighbor 10.0.0.1
use neighbor-group ibgp-flowspec use neighbor-group ibgp-flowspec
Contr Client
iBGP FS
RP/0/0/CPU0:Client#sh bgp ipv4 flowspec neighbors 10.0.0.1 detail | i Long-lived

Long-lived Graceful Restart Capability advertised
Advertised Long-lived Stale time 360 seconds
Long-lived Graceful Restart Capability received
RP/0/0/CPU0:Client#
BGP Persistence
• We cut the link between the Client and Controller
FS entries
Client Ctrl
✖FS ✖
iBGP
RP/0/0/CPU0:May 11 16:01:53.980 : bgp[1052]: %ROUTING-BGP-5-ADJCHANGE : neighbor 10.0.0.1 Down - BGP

Notification sent, hold time expired (VRF: default) (AS: 1)
RP/0/0/CPU0:May 11 16:01:53.980 : bgp[1052]: %ROUTING-BGP-5-NSR_STATE_CHANGE : Changed state to NSR-Ready
RP/0/0/CPU0:Client#sh bgp ipv4 flowspec sum
BGP router identifier 2.2.2.2, local AS number 1
<SNIP>
10.0.0.1 0 1 7508 7513 0 0 0 00:00:29 Active
RP/0/0/CPU0:Client#sh flowspec ipv4 detail

AFI: IPv4
Flow :Dest:11.200.4.0/24,Proto:=6,DPort:=80|=443|=8080,Length:>=1000&<=1500
Actions :Traffic-rate: 10000000 bps DSCP: ef Nexthop: 25.3.9.3 (bgp.1)
RP/0/0/CPU0:Client#sh bgp ipv4 flowspec neighbors 10.0.0.1 detail | i "(Long|LLGR)”
Remaining LLGR stalepath time 320
RP/0/0/CPU0:Client#
BGP Persistence
• When the timer expires, the associated BGP FS entries are removed
FS entries
Client Ctrl
✖ ✖
iBGP FS
✖
RP/0/0/CPU0:XRv2-demo#sh bgp ipv4 flowspec neighbors 10.0.0.1 detail | i "(Long|LLGR)"
Mon May 11 16:07:53.845 UTC
Remaining LLGR stalepath time 2
RP/0/0/CPU0:XRv2-demo#sh bgp ipv4 flowspec neighbors 10.0.0.1 detail | i "(Long|LLGR)"
Mon May 11 16:08:01.285 UTC
Long-lived Graceful Restart not in effect as Graceful Restart capability not received
RP/0/0/CPU0:XRv2-demo#sh flowspec ipv4 detail
Mon May 11 16:08:04.615 UTC
RP/0/0/CPU0:XRv2-demo#
BGP FS Controller Redundancy
• No Controller to Controller protocol to sync the rules advertisement
rule A
rule B
Ctrl rule C
rule D
rule A
rule B
rule C
Client ?
rule D
rule B
Ctrl rule D
• You need manual config or scripting to align config on each Controller
BGP FS Controller Redundancy
• If a controller is lost, the rules are not temporarily removed and re-installed
rule A
rule B
Ctrl rule C
rule D
rule A
rule B
rule C
Client
rule D
✖
Ctrl
For Your
Reference
Configuring BGP FlowSpec

Order of Matching Types
NLRI type Match fields
• Not dependent on the arrival order of Type 1 IPv4 Destination address
the flow specification's rules Type 2 IPv4 Source address
Order of preference
Type 3 IPv4 protocol
• The algorithm starts by comparing Type 4 IPv4 source or destination port
the left-most components of the Type 5 IPv4 destination port
rules. Type 6 IPv4 Source port
Type 7 IPv4 ICMP type
• If the types differ, the rule with lowest
Type 8 IPv4 ICMP code
numeric type value has higher
precedence (and thus will match Type 9
before) than the rule that doesn't Type 10 IPv4 Packet length
contain that component type. Type 11 IPv4 DSCP
Type 12 IPv4 fragmentation bits
For Your
Reference

Order of Matching Types
• If the component types are the same, then a type-specific comparison is

performed.
• For IP prefix values (IP destination and source prefix) precedence is given
to the lowest IP value of the common prefix length; if the common prefix is
equal, then the most specific prefix has precedence.
• For all other component types, unless otherwise specified, the comparison
is performed by comparing the component data as a binary string using the
memcmp() function as defined by the ISO C standard.
• For strings of different lengths, the common prefix is compared. If equal,
the longest string is considered to have higher precedence than the shorter
one.
For Your
Reference

class-map type traffic match-all MATCHING-RULE1 RP/0/RSP0/CPU0:Client#show flowspec afi-all detail
match protocol udp
match packet length 500-1550 AFI: IPv4
match destination-address ipv4 25.1.102.1 255.255.255.255 Flow
end-class-map
:Dest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550
!
class-map type traffic match-all MATCHING-RULE2 Actions :Nexthop: 25.4.9.3 (bgp.1)
match protocol udp Statistics (packets/bytes)
match packet length 500-1550 Matched : 304006799/425609518600
match destination-address ipv4 25.1.102.0 255.255.255.0 Dropped : 0/0
end-class-map Flow
! :Dest:25.1.102.0/24,Proto:=17,Length:>=500&<=1550
policy-map type pbr TEST1 Actions :Nexthop: 25.3.9.3 (bgp.1)
class type traffic MATCHING-RULE1 Statistics (packets/bytes)
Matched : 0/0
! Dropped : 0/0
end-policy-map
! RP/0/RSP0/CPU0:Client#
policy-map type pbr TEST2
class type traffic MATCHING-RULE2
Client
! 25.1.102.1/32 more specific than 25.1.102.0/24
end-policy-map
flowspec
address-family ipv4
service-policy type pbr TEST1
service-policy type pbr TEST2
! Controller BRKSPG-3012 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 109
NLRI Filtering
“Safety Net”
• We don’t want any user or operator to accidentally blackhole important traffic

• DNS servers (8.8.8.8)
• Infrastructure addresses (routers, tacacs/radius, netflow collectors, snmp, …)
• Addresses of other customers
• Local definition of prefixes / protocols which can NOT be overruled by BGP

FlowSpec
NLRI Filtering
Configuration
prefix-set ALLOW-FLOW
1.1.1.0/24 ge 32 • Server advertises two BGP FS rules:
end-set • Destination 1.1.1.1/32
!
route-policy ALLOW-FLOW-POLICY • Destination 1.1.2.1/32
if destination-prefix in ALLOW-FLOW then
pass
endif RP/0/RP0/CPU0:Client#sh flowspec ipv4 detail
end-policy AFI: IPv4
! Flow :Dest:1.1.1.1/32
router bgp 65117 Actions :Traffic-rate: 0 bps (bgp.1)
neighbor 25.2.1.14 RP/0/RP0/CPU0:Client#
remote-as 65117
update-source GigabitEthernet0/0/0/0
address-family ipv4 flowspec
route-policy ALLOW-FLOW-POLICY in
 Only the 1.1.1.1/32 rule is accepted
! and configured.
Consistency Checking
Example: TCP with ICMP Code
class-map type traffic match-all c21

match protocol tcp
match ipv4 icmp-type 10
end-class-map
RP/0/0/CPU0:CONTROLLER#show flowspec vrf foo1 ipv4 internal

VRF: foo1 AFI: IPv4
Flow :Proto:=6,ICMPType:=10
Actions :DSCP: af11 (policy.1.p21.c21)
<... SNIP ...>
Sequence: 1024
Match Unsupported: ICMP type/code with non-ICMP protocol
Synced: FALSE
<... SNIP ...>
Matched : 0/0
Transmitted : 0/0
Dropped : 0/0
RP/0/0/CPU0:CONTROLLER#
For Your
Reference
Consistency Checking
Other Examples
class-map type traffic match-all c22 class-map type traffic match-all c23
match protocol icmp match protocol icmp
match tcp-flag 16 match destination-port 10
RP/0/0/CPU0:CONTROLLER#show flowspec vrf foo2 ipv4 internal RP/0/0/CPU0:CONTROLLER#show flowspec vrf foo3 ipv4 internal
VRF: foo2 AFI: IPv4 VRF: foo3 AFI: IPv4
Flow :Proto:=1,TCPFlags:=0x10 Flow :Proto:=1,DPort:=10
Actions :DSCP: af11 (policy.1.p22.c22) Actions :DSCP: af11 (policy.1.p23.c23)
<... SNIP ...> <... SNIP ...>
Sequence: 1024 Sequence: 1024
Match Unsupported: TCP flags with non-TCP protocol Match Unsupported: Port with non-TCP/UDP protocol
Synced: FALSE Synced: FALSE
<... SNIP ...> <... SNIP ...>
Statistics (packets/bytes) Statistics (packets/bytes)
Matched : 0/0 Matched : 0/0
Transmitted : 0/0 Transmitted : 0/0
Dropped : 0/0 Dropped : 0/0
RP/0/0/CPU0:CONTROLLER# RP/0/0/CPU0:CONTROLLER#
Checking Counters with Netconf/XML
<<<SNIP>>>
<FlowTable>
• Proprietary models are available <Flow>
<Naming>
for configuration and monitoring <FlowNotation>
Dest:25.1.104.0/24
</FlowNotation>
</Naming>
<FlowStatistics>
<Classified>
<rpc message-id="101" <Packets>
xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> 21946725652
<get> </Packets>
<filter> <Bytes>
<Operational> 13958117514672
<FlowSpec></FlowSpec> </Bytes>
</Operational> </Classified>
</filter> <Dropped>
</get> <Packets>
</rpc>]]>]]> 21946488774
</Packets>
<Bytes>
13957966860264
</Bytes>
</Dropped>
</FlowStatistics>
</Flow>
<<</SNIP>>>
Netflow Sampling vs BGP flowspec
• Even if a BGP flowspec rule drops the packets, they are sampled and handled
by the linecard CPU.
Attack still detected
RP/0/RSP0/CPU0:Client#sh run int hundredGigE 0/0/0/0
interface HundredGigE0/0/0/0
description *** to Boca ***
cdp
ipv4 address 25.1.9.4 255.255.255.0
load-interval 30
flow ipv4 monitor MON-MAP-IP sampler SAM-MAP ingress
!
RP/0/RSP0/CPU0:Client#sh flowspec ipv4 detail

AFI: IPv4
Flow :Proto:=17,Length:>=500&<=1550
Matched : 146077011/182594343700
Dropped : 146077011/182594343700
• Before applying the BGP FlowSpec rules, we check the NF cache:
RP/0/RSP0/CPU0:Client#sh flow monitor MON-MAP-IP cache location 0/0/CPU0
Cache summary for Flow Monitor MON-MAP-IP:
Cache size: 1000000
Current entries: 164916
Flows added: 2043769
<SNIP>
Flows exported 1878853
IPV4SrcAddr IPV4DstAddr L4SrcPort L4DestPort BGPDstOrigAS BGPSrcOrigAS BGPNextHopV4 IPV4DstPrfxLen

IPV4SrcPrfxLen IPV4Prot IPV4TOS InputInterface OutputInterface L4TCPFlags ForwardStatus FirstSwitched
LastSwitched ByteCount PacketCount Dir SamplerID InputVRFID OutputVRFID
100.102.8.178 11.200.0.2 123 123 0 0 0.0.0.0 24
0 udp 0 Hu0/0/0/0 Te0/2/0/1 0 Fwd 12 15:47:40:093
12 15:47:40:093 1402 1 Ing 1 default default
100.2.42.67 11.200.0.2 123 123 0 0 0.0.0.0 24
0 udp 0 Hu0/0/0/0 Te0/2/0/1 0 Fwd 12 15:47:51:618
100.77.86.28 11.200.0.2 123 123 0 0 0.0.0.0 24
0 udp 0 Hu0/0/0/0 Te0/2/0/1 0 Fwd 12 15:48:31:530
• After applying the BGP FlowSpec rules, we check the NF cache:
RP/0/RSP0/CPU0:Client#sh flow monitor MON-MAP-IP cache location 0/0/CPU0
Cache summary for Flow Monitor MON-MAP-IP:
Cache size: 1000000
Current entries: 12706
Flows added: 1467559
<SNIP>
Flows exported 1454853
IPV4SrcAddr IPV4DstAddr L4SrcPort L4DestPort BGPDstOrigAS BGPSrcOrigAS BGPNextHopV4 IPV4DstPrfxLen

IPV4SrcPrfxLen IPV4Prot IPV4TOS InputInterface OutputInterface L4TCPFlags ForwardStatus FirstSwitched
LastSwitched ByteCount PacketCount Dir SamplerID InputVRFID OutputVRFID
100.37.17.132 11.200.0.2 123 123 0 0 0.0.0.0 24
0 udp 0 Hu0/0/0/0 0 0 DropACLDeny 12 15:45:00:310
12 15:45:00:310 1362 1 Ing 1 default 0
100.47.47.62 11.200.0.2 123 123 0 0 0.0.0.0 24
12 15:45:01:850 1122 1 Ing 1 default 0
100.11.100.55 11.200.0.2 123 123 0 0 0.0.0.0 24
12 15:45:00:947 1462 1 Ing 1 default 0
ACL vs BGP flowspec
• It’s important that ACL is applied before the BGP FlowSpec action.
RP/0/RSP0/CPU0:Client#sh int hundredGigE 0/0/0/1 accounting rates
HundredGigE0/0/0/1
Ingress Egress
Protocol Bits/sec Pkts/sec Bits/sec Pkts/sec
IPV4_UNICAST 5065311000 458150 1000 2

AFI: IPv4
Flow :Dest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550
Actions :Nexthop: 25.3.9.3 (bgp.1)
Matched : 0/0
Dropped : 0/0
RP/0/RSP0/CPU0:Client#sh access-lists ipv4 INFRA-ACL hardware ingress location 0/0/CPU0
ipv4 access-list INFRA-ACL

10 deny udp any host 25.1.102.1 counter INFRA-ACL-COUNT (230292976 hw matches)
20 permit ipv4 any any
ACL-Based Fwd (PBR) vs BGP flowspec
• Which one will take precedence ?
Before applying the BGP FS rule, on the Client side:
interface HundredGigE0/0/0/1
ipv4 address 25.1.104.4 255.255.255.0
ipv6 address 2001:25:1:104::4/64
AS 2 AS 1 load-interval 30
ipv4 access-group ABF ingress
XR-Services !
ipv4 access-list ABF
25.2.1.11
10 permit udp any host 25.1.102.1 nexthop1 ipv4 25.3.9.3
20 permit ipv4 any any
!
25.1.9.3 25.1.9.4
.1 Hu0/1/0/0 Hu0/0/0/0
102
25.1.102.3 Boca 25.3.9.3 25.3.9.4 Inter 25.1.104.4
104
Hu0/1/0/0.2 Hu0/0/0/0.2 Hu0/0/0/1
25.4.9.3 25.4.9.4
Hu0/1/0/0.3 Hu0/0/0/0.3
Traffic to 25.1.102.1
ACL-Based Fwd (PBR) vs BGP flowspec
• BGP FlowSpec action takes precedence over ABF/PBR
After applying the rule, traffic follows the BGP FlowSpec Redirect action.
AFI: IPv4
AS 2 AS 1 Flow
:Dest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550
XR-Services Actions :Nexthop: 25.4.9.3 (bgp.1)
25.2.1.11 Matched : 2217686/3104760400
Dropped : 0/0
25.1.9.3 25.1.9.4
.1 Hu0/1/0/0 Hu0/0/0/0
102
25.1.102.3 Boca 25.3.9.3 25.3.9.4 Inter 25.1.104.4
104
Hu0/1/0/0.2 Hu0/0/0/0.2 Hu0/0/0/1
25.4.9.3 25.4.9.4
Hu0/1/0/0.3 Hu0/0/0/0.3
Traffic to 25.1.102.1
Caveats and Limitations
Too Late ?
Upstream Link Saturated
• Using BGP FS in the limit of your AS only can be too late
Victim
50G 10G
TenGE
Upstream Customer
Provider
Controller
FS rule
Configuring a Type 4 Match “Source or Dest Ports”
• We can receive Type4 messages on client but can not generate it on the
controller due to C3PL limitation
RP/0/0/CPU0:Ctrl(config)#show config failed
<SNIP>
class-map type traffic match-any MATCH-TYPE-4
match source-port 123
end-class-map
!
!!% Policy manager does not support this feature: Match all is the only mode supported
for match type "source-port" in class-map type "traffic"
End
Rate-limiter Shared per NPU
• A policer action will be applied at the NPU level and not at the port level
• Ex: you receive a 50Mbps police action, and FS is activated on three ports
• Te0/1/0/18 is assigned to one NPU
• Te0/1/0/10 and Te0/1/0/11 are assigned to a different NPU
• We apply the policer per NPU

• Traffic on Te0/1/0/18 is rate limited to 50Mbps
• Total traffic on Te0/1/0/10+Te0/1/0/11 is rate-limit to 50Mbps, hence 25Mbps each
• Not relevant if the action is drop

Description of Fragmentation
• ASR9000 only matches traffic on the indication of the fragmentation:
• With first-fragment and is-fragment
• Not with last-fragment nor do-not-fragment
Frag Description Supported?

1 Don’t Fragment No
2 Is a Fragment Yes
4 First Fragment Yes
8 Last Fragment No
ICMP Lists and Ranges
• FlowSpec rules for ICMP can only support one type and code
• No support for lists or ranges
• Decoded but not programmed in hardware
RP/0/RSP0/CPU0:Client#sh bgp ipv4 flowspec
Network Next Hop Metric LocPrf Weight Path
*>iICMPType:=1|=2|=3|=4|=5,ICMPCode:=1/112
0.0.0.0 100 0 i

RP/0/RSP0/CPU0:Client#show policy-map transient type pbr pmap-name __bgpfs_default_IPv4
policy-map type pbr __bgpfs_default_IPv4
handle:0x36000002
table description: L3 IPv4 and IPv6
class handle:0xf6000002 sequence 4294967295 (class-default)
!
end-policy-map
RP/0/RSP0/CPU0:Client#sh flowspec ipv4 internal | i Match Unsupported
Match Unsupported: ICMP type count exceeded
Workaround: configure multiple rules

Filter with Inter-AS BGP FlowSpec
• We support NLRI filtering on source and destination
• But we don’t filter on action type
• Customer could potentially
• Redirect their traffic to a NH address or force the leaking into a different VRF
• Remark all their traffic to an EF class, potentially giving them higher priority in case of
congestion
Per Interface Selection
• Today implementation is binary, BGP FS applied or not applied on an interface
• XR: No current way to decide which FS rule should be applied on which
interface
• XE: interface-set draft is supported
Conclusion
BGP FlowSpec in SP Security
• Very powerful addition to your countermeasure tools
• Large adoption now in the industry
• Interoperable, Standard-based solution to remotely program actions on precisely
identified flows
• Particularly useful in DDoS mitigation architectures
• Filtering the stateless attacks on the Edge router, it offloads the scrubbing devices
• Allow redirection of only the attack traffic into the scrubbing device
• But new use-cases are emerging

• You can start learning right now in a virtual environment:
• XRv 9000 can be used as a controller, CSR1000v can be used as a client
Cisco Spark
Questions?
Use Cisco Spark to communicate
with the speaker after the session
How
1. Find this session in the Cisco Live Mobile App
2. Click “Join the Discussion”
3. Install Spark or go directly to the space
4. Enter messages/questions in the space
cs.co/ciscolivebot#BRKSPG-3012
• Please complete your Online Complete Your Online
Session Evaluations after each
session
Session Evaluation
• Complete 4 Session Evaluations
& the Overall Conference
Evaluation (available from
Thursday) to receive your Cisco
Live T-shirt
• All surveys can be completed via
the Cisco Live Mobile App or the
Communication Stations
Don’t forget: Cisco Live sessions will be available
for viewing on-demand after the event at
www.ciscolive.com/global/on-demand-library/.
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
Thank you
Back-Up Slides
Other Use-Cases
Centralized
• A central point in the network is dedicated for hosting scrubbing devices
Peering
Scrubbing Center
Victim
Transit
Core
Centralized
• Traffic target to the victim is diverted to this place for analysis
Peering
Scrubbing Center
Victim
Transit
Core
Note: asymmetric traffic, i2o traffic doesn’t go through the scrubbing center
Distributed
• We install scrubbers at the edge of the backbone to tackle the attack as early as
possible Peering
Victim
Transit
Core
Mixed
• Specific attacks can be handled in the central point or to off-load the edge
systems Peering
Scrubbing Center
Victim
Transit
Core
L3VPN Network w/ Scrubbing Center
Currently deployed
• 2.1.1.1 is victim of a large size SYN attack. Traffic is transported in the GRT
or in a VRF “Internet”
2.1.1.0/24 J
IGP rtrA nh:rtrI
S
Collector
RR I F Victim
2.1.1.1
L H E C
M
2.1.1.0/24
2.1.1.0/24 K G D B A
rtrA nh:rtrK
L3VPN Network
IGP
Currently deployed
IGP
2.1.1.0/24 @rtrA 0.0.0.0/0
MP-BGP
• VRF Dirty is configured on J and M nh:rtrI nh Router S
M
VRF
GRT
• MP-BGP is configured too, Dirty
default route is advertised static 10.51.51.51/32
VRF dirty @TMS
from @TMS in VRF Dirty J
• On edge routers J and M, S

we configure static entries MP-BGP VRF Dirty
RR
for a dummy host route IPv4
RR 0.0.0.0/0 nh: Rtr S
(10.51.51.51/32) with vpnv4
a NH in VRF Dirty.
If matched, traffic will M
M
leak into this VRF Dirty 0.0.0.0/0
MP-BGP
nh Router S
• Now, traffic to 2.1.1.1 static 10.51.51.51/32
GRT
VRF
VRF dirty @TMS Dirty
uses the IGP route
IGP
2.1.1.0/24 @rtrA
2.1.1.0/24 nh:rtrK
Currently deployed
0.0.0.0/0
MP-BGP
• A more specific 2.1.1.1/32 route nh Router S
M
BGP
is advertised via BGP and learnt 2.1.1.1/32
GRT
VRF
10.51.51.51 Dirty
in the GRT with NH the dummy
10.51.51.51/32
route 10.51.51.51 static VRF dirty @TMS
J
• A recursive lookup triggers the
S
leaking into VRF Dirty
MP-BGP VRF Dirty
RR RR 0.0.0.0/0 nh: Rtr S
IPv4 vpnv4
2.1.1.1/32
nh:10.51.51.51
Collector
M
M
BGP
2.1.1.1/32 0.0.0.0/0
10.51.51.51 nh Router S
MP-BGP
• Now attack traffic is
static 10.51.51.51/32 VRF
in VRF Dirty and VRF dirty @TMS GRT
Dirty
attracted to Router S
Currently deployed
• CP advertises a BGP route for 2.1.1.1/32 with next-hop the dummy 10.51.51.51
BGP
2.1.1.1/32
10.51.51.51
@TMS
static
10.51.51.51/32
Collector J VRF dirty
nh: @TMS
BGP RR S
IPv4
2.1.1.1/32 I F Victim
nh:10.51.51.51
2.1.1.1
L H E C
M
K G D B A
BGP
2.1.1.1/32
10.51.51.51
static
10.51.51.51/32
VRF dirty
nh: @TMS
Currently deployed
• Traffic with a VRF label Dirty is dragged to router S
• Router S is pushing unlabeled traffic to the TMS via an interface in VRF Dirty
@TMS
• Clean traffic is received in Te0/0/0/0.1: dirty
GRT and naturally routed S
Te0/0/0/0.2: clean
to the victim
1 label
F
2 labels 2 labels
E C Victim
2.1.1.1
2.1.1.0/24
D B A
Te0/1/0/0
Improved with BGP FlowSpec
• BGP FlowSpec inject rules to redirect attack traffic into VRF dirty
• No more dummy route needed Collector
VRF dirty
• Only a default route in dirty VRF SrcIP:*
0.0.0.0/0
TCP SYN
is needed to reach the scrubber Dst:2.1.1.1/32 nh: @TMS
Size > 200B
J
• More granular “matching” parameters: Redirect
VRF dirty
only the packets with specific
RR
protocol/port/packet-size/etc IPv4 I
VRF dirty
are diverted in Dirty VRF 0.0.0.0/0
nh: @TMS
L H
M
K G
Improved with BGP FlowSpec
static
VRF dirty
0.0.0.0/0
BGP FS nh: @TMS
TCP SYN > 200B @TMS
Controller Action: NH: VRF Dirty
J
RR S
BGP FS
IPv4
TCP SYN > 200B
I F Victim
Action: NH: VRF Dirty
2.1.1.1
L H E C
M
K G D B A
static
VRF dirty
0.0.0.0/0
nh: @TMS
• Important back-up is using 3 Gbps of traffic
Scrubbing Center
• Simultaneously, a DDoS attack starts and is
diverted to the scrubbing center S
• Links are not saturated
6G
L H C 3G Victim
2G M 3G 2.1.1.1
2G K G B A
2G
3G
• The attack intensity increases
Scrubbing Center
• Links are congested, it impacts internal traffic
S
8G
10G L H C 2G Victim
4G M 2G 2.1.1.1
4G K G B A
4G
3G
Low QoS Priority Traffic for DDoS Attacks w/ Flowspec
• BGP FS rule forces the route leaking in VRF-Dirty and positioning a DSCP field
BGP FS
Rule: dest-IP: 2.1.1.1 S

Action: Set DSCP X
+ NH: VRF Dirty
7G
10G L H C 3G Victim
4G M 3G 2.1.1.1
4G K G B A
4G
3G
 Routers will dropped attack traffic in priority in
case of congestion based on this DSCP field
Back-Up Slides
Configuration
For Your
Reference

Configuring a Type 1 Match “Destination Address”

RP/0/0/CPU0:Ctrl(config-cmap)#match destination-address ipv4 81.253.193.0/24
RP/0/RP0/CPU0:Client#show contr pse tcam summary location 0/0/CPU0
<SNIP>
TCAM Device Information for Ingress PSE, CAM bank 1:

Device size: 20M (256K array entries of 80-bits), 261122 available
Current mode of operation: Turbo
<SNIP>
Feature specific information:
<SNIP>
FlowSpec IPv4 (id 32):
Owner client id: 20. Limit 245760 cells
Total 1 regions using 4 CAM cells
<SNIP>
For Your
Reference

Configuring a Type 2 Match “Source Address”

RP/0/0/CPU0:Ctrl(config-cmap)#match source-address ipv4 2.2.0.0/16

Type Prefix length Prefix
AFI: IPv4
Flow :Source:2.2.0.0/16 1 byte 1 byte Variable
Statistics (packets/bytes) 2 /16 2.2
Matched : 0/0
Transmitted : 0/0
0x 02 0x 10 0x 02 02
Dropped : 0/0
RP/0/RP0/CPU0:Boca#sh flowspec ipv4 nlri 0x02100202
AFI: IPv4
NLRI (Hex dump) : 0x02100202
RP/0/RP0/CPU0:Boca#
For Your
Reference

Configuring a Type 3 Match “IPv4 Protocol Type” / “IPv6 Next Header”
RP/0/0/CPU0:Ctrl(config-cmap)#match protocol udp tcp

Option Byte
AFI: IPv4
Flow :Proto:=0|=17|=6 End And Len 0 Lt “<“ Gt “>” Eq “=“
Statistics (packets/bytes) 1b 1b 2b 1 1b 1b 1b
Matched : 0/0
Transmitted : 0/0
b
Dropped : 0/0 0x03010001118106
RP/0/RP0/CPU0:Client#sh flowspec ipv4 nlri
AFI: IPv4
NLRI (Hex dump) : 0x03010001118106
Type Option1 IP Option2 IP proto2 Option3 IP

proto1 proto3
1 byte 1 byte 1 byte 1 byte 1 byte 1 byte 1 byte
1 0b00000001 0x00 0b00000001 17 = 0x11 0b10000001 0x06
0x 03 01 00 01 11 81 06
For Your
Reference

Configuring a Type 5 Match “Destination Port”
RP/0/0/CPU0:Ctrl(config)#class-map type traffic match-all MATCHING-TYPE5
RP/0/0/CPU0:Ctrl(config-cmap)#match destination-port 80 443 8080
RP/0/RP0/CPU0:Client#show flowspec afi-all detail

AFI: IPv4 Type (1B) Option x (1B) Dest Port (1B or 2B)
Flow :DPort:=80|=443|=8080
Actions :Traffic-rate: 314152 bps (bgp.1) 5 equal/length=0 d80 = x50
Matched : 0/0 Not last
Transmitted : 0/0
Dropped : 0/0 0 x05 0x01 0x50
RP/0/RP0/CPU0:Client#show flowspec ipv4 nlri
AFI: IPv4
NLRI (Hex dump) : 0x0501501101bb911f90
- equal/length=1 d443 = x1BB
Actions :Traffic-rate: 314152 bps (bgp.1) Not last
- 0x11 0x01BB
Option Byte
- equal/length=1 d8080 = x1F90
End And Len 0 Lt “<“ Gt “>” Eq “=“
last
01 0 0 00 0 0 0 1
- 0x91 0x1F90
11 0 0 01 0 0 0 1
0x0501501101bb911f90
91 1 0 01 0 0 0 1
For Your
Reference

Configuring a Type 6 Match “Source Port”
RP/0/0/CPU0:Ctrl(config-cmap)#match source-port 80-100

AFI: IPv4 Type (1B) Option 1 (1B) Dest Port
Flow :SPort:>=80&<=100
Actions :Traffic-rate: 314152 bps (bgp.1) 6 0000 0011 80
Matched : 0/0
greater+equal/le=0/not last
Transmitted : 0/0
Dropped : 0/0 0 x06 0x03 0x50
AFI: IPv4 - 1100 0101 100
NLRI (Hex dump) : 0x060350c564 lower+equal/le=0/last
- 0xc5 0x64
Option Byte
End And Len 0 Lt “<“ Gt “>” Eq “=“ 0x060350c564
03 0 0 00 0 0 1 1
c5 1 1 00 0 1 0 1
For Your
Reference

Configuring a Type 7+8 Match “ICMP Type” + “ICMP Code”
RP/0/0/CPU0:Ctrl(config-cmap)# match ipv4 icmp-type 3
RP/0/0/CPU0:Ctrl(config-cmap)# match ipv4 icmp-code 13
RP/0/0/CPU0:Ctrl(config-cmap)#commit
RP/0/RSP0/CPU0:Client#show flowspec afi-all detail

AFI: IPv4 Type (1B) Option 1 (1B) ICMP
Flow :ICMPType:=3,ICMPCode:=13
Matched : 0/0 0 x07 0x81 0x03
Dropped : 0/0
RP/0/RSP0/CPU0:Client#show flowspec ipv4 nlri 8 100 0001 13
AFI: IPv4
NLRI (Hex dump) : 0x07810308810d 0 x08 0x81 0x0d
RP/0/RSP0/CPU0:Client# 0x07810308810d
Option Byte
81 1 0 00 0 0 0 1
For Your
Reference

Configuring a Type 9 Match “TCP Flag Component”
RP/0/0/CPU0:Ctrl(config-cmap)#match tcp-flag 2

AFI: IPv4 Type (1B) Option 1 (1B) Flag
Flow :TCPFlags:=0x02
Actions :Traffic-rate: 314152 bps (bgp.1) 9 1000 0001 x02
Matched : 8/496 0 x09 0x81 0x02
Dropped : 0/0
AFI: IPv4
0x098102
RP/0/RP0/CPU0:Client# Option Byte
e bit a bit Len 0 0 Not bit m bit
• Ex: http://rapid.web.unc.edu/resources/tcp-flag-key/ 81 1 0 00 0 0 0 1
• 0x02: SYN • 0x01: FIN
• 0x12: SYN-ACK • 0x04: RST
• 0x10: ACK
For Your
Reference

Configuring a Type 10 Match “Packet Length”
RP/0/0/CPU0:Ctrl(config-cmap)#match packet length 100

AFI: IPv4 Type (1B) Option 1 (1B) Pkt Length
Flow :Length:=100
Matched : 0/0 0 x0a 0x81 0x64
Transmitted : 0/0
Dropped : 0/0
0x0a8164
AFI: IPv4
NLRI (Hex dump) : 0x0a8164
Actions :Traffic-rate: 314152 bps (bgp.1) Option Byte
81 1 0 00 0 0 0 1
For Your
Reference

Configuring a Type 11 Match “IPv4/IPv6 DSCP”
RP/0/0/CPU0:Ctrl(config-cmap)#match dscp ef

AFI: IPv4 Type (1B) Option 1 (1B) DSCP
Flow :DSCP:=46
Actions :Traffic-rate: 314152 bps (bgp.1) 11 1000 0001 ef
Matched : 0/0 0 x0b 0x81 0x2e
Transmitted : 0/0
Dropped : 0/0
RP/0/RP0/CPU0:Client#show flowspec afi-all nlri
0x0b812e
AFI: IPv4
NLRI (Hex dump) : 0x0b812e
Actions :Traffic-rate: 314152 bps (bgp.1) Option Byte
81 1 0 00 0 0 0 1
For Your
Reference

Configuring a Type 12 Match “IPv4 Fragment”
RP/0/0/CPU0:Ctrl(config-cmap)#match fragment-type is-fragment last-fragment

AFI: IPv4
Type (1B) Option 1 (1B) Pkt Length
Flow :Frag:=LF:IsF
Actions :Traffic-rate: 314152 bps (bgp.1) 11 1000 0001 LF + IsF
Matched : 0/0 0 x0c 0x81 0x0a
Transmitted : 0/0 0x0c810a
Dropped :
0/0RP/0/RP0/CPU0:Client#sh flowspec ipv4 nlri
AFI: IPv4
NLRI (Hex dump) : 0x0c810a Option Byte
81 1 0 00 0 0 0 1
Bitmask
0 0 0 lf ff isf df
0a 0 0 0 1 0 1 0
For Your
Reference

redirect nexthop route-target 1:1
!
end-policy-map

flowspec_mgr[1094]: FlowSpec: Registered for AFI IPv4 RT ASN2-1:1.
flowspec_mgr[1094]: FlowSpec: Added client bgp.1 flow active Proto:=6,DPort:=80 with actions RT-ASN2-1:1 from TBL
default:IPv4.
bgp[1052]: FlowSpec: Notifying client bgp.1 for Register RT ASN2-1:1 (AFI IPv4).
In this case, we used 2-byte long ASN for the Route Target definition.
It’s transported with extended community 0x8008
For Your
Reference

redirect nexthop route-target 123456789:1
!
end-policy-map

flowspec_mgr[1094]: FlowSpec: Registered for AFI IPv4 RT ASN4-123456789:1.
flowspec_mgr[1094]: FlowSpec: Added client bgp.1 flow active Proto:=6,DPort:=80 with actions RT-ASN4-123456789:1
from TBL default:IPv4.
bgp[1052]: FlowSpec: Notifying client bgp.1 for Register RT ASN4-123456789:1 (AFI IPv4).
In this case, we used 4-byte long ASN for the Route Target definition.
It’s transported with extended community 0x8208
Back-Up Slides
Monitoring
Show Commands to Check BGP Flowspec Operation
• First, we verify the BGP session for the address-family Flowspec
RP/0/RP0/CPU0:Client#show bgp ipv4 flowspec

BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 16
BGP main routing table version 16
BGP NSR Initial initsync version 0 (Reached)
BGP NSR/ISSU Sync-Group versions 16/0
BGP scan interval 60 secs
Status codes: s suppressed, d damped, h history, * valid, > best

i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> SPort:=80/24 0.0.0.0 0 1 i

Verifying the Session Establishment (on Client)
RP/0/RP0/CPU0:Client#sh bgp ipv4 flowspec summary

BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0x0 RD version: 7072
BGP main routing table version 7072
BGP NSR Initial initsync version 0 (Reached)
BGP NSR/ISSU Sync-Group versions 7072/0
BGP scan interval 60 secs
BGP is operating in STANDALONE mode.
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer

Speaker 7072 7072 7072 7072 7072 7072

25.2.1.11 0 1 106269 105679 7072 0 0 1w1d 1001
Show Commands
• Then, we can get more details for this particular rule
RP/0/RP0/CPU0:Client#show bgp ipv4 flowspec SPort:=80/24 detail
BGP routing table entry for SPort:=80/24

NLRI in Hex: 068150/24
Versions:
Process bRIB/RIB SendTblVer
Speaker 16 16
Flags: 0x04001001+0x00000000;
Paths: (1 available, best #1)
Not advertised to any peer
Path #1: Received by speaker 0
Flags: 0x4000000001060001, import: 0x20
1
0.0.0.0 from 25.2.1.11 (6.6.6.6)
Origin IGP, localpref 100, valid, external, best, group-best
Received Path ID 0, Local Path ID 1, version 16
Extended community: FLOWSPEC Traffic-rate:1,39269
Show Commands
• Globally, we verify which interfaces are enable for FlowSpec
RP/0/RP0/CPU0:Client#show policy-map transient targets type pbr
1) Policymap: __bgpfs_default_IPv4 Type: pbr

Targets (applied as main policy):
HundredGigE0/1/0/0 input
HundredGigE0/0/0/0 input
ServiceInfra7 input
TenGigE0/2/0/5 input
Total targets: 6
Show Commands
• We verify also how are reconstructed these policies
RP/0/RP0/CPU0:Client#show policy-map transient type pbr pmap-name
__bgpfs_default_IPv4
policy-map type pbr __bgpfs_default_IPv4

handle:0x36000002
table description: L3 IPv4 and IPv6
class handle:0x7600000a sequence 1024
match source-port 80
police rate 314152 bps
conform-action transmit
exceed-action drop
!
!
class handle:0xf6000002 sequence 4294967295 (class-default)
!
end-policy-map
!
Show Commands
• Globally, we verify which interfaces are enable for FlowSpec
AFI: IPv4
Flow :SPort:=80
Matched : 0/0
Transmitted : 0/0
Dropped : 0/0
AFI: IPv4
Show Commands
RP/0/RP0/CPU0:Client#show flowspec ipv4 internal
AFI: IPv4
Flow :SPort:=80
Client Version: 0
Unsupported: FALSE
RT:
VRF Name Cfg: 0x00
RT Cfg: 0x00
RT Registered: 0x00
RT Resolved: 0x00
Class handles:
Handle [0]: 300000007600000a
Class Handle Version: 1
Sequence: 1024
Synced: TRUE
Match Unsupported: None
Ref Count: 1
Last Error: 0:No error
Last Batch: 9
Matched : 0/0
Transmitted : 0/0
Dropped : 0/0
RP/0/RP0/CPU0:Client# © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 171
Show Commands
• On a CRS client, we check the TCAM usage on the linecard
RP/0/RP0/CPU0:CRS-3#show contr pse tcam summary location 0/0/CPU0
<SNIP>
TCAM Device Information for Ingress PSE, CAM bank 1:

Device size: 20M (256K array entries of 80-bits), 261122 available
Current mode of operation: Turbo
<SNIP>
Feature specific information:
<SNIP>
Flowspec IPv4 (id 32):
Owner client id: 20. Limit 245760 cells
Total 1 regions using 4 CAM cells
<SNIP>
Show Commands
• On a ASR9000 client, we can also check the TCAM entries in some extend
RP/0/RSP0/CPU0:ASR9000#sh prm server tcam summary all PBR np0 location 0/0/CPU0
Node: 0/0/CPU0:
----------------------------------------------------------------
TCAM summary for NP0:
TCAM Logical Table: TCAM_LT_L2 (1)

Partition ID: 0, priority: 2, valid entries: 1, free entries: 2047
TCAM Logical Table: TCAM_LT_ODS2 (2), free entries: 89723, resvd 128
ACL Common Region: 448 entries allocated. 448 entries free
Application ID: NP_APP_ID_PBR (5)
Total: 0 vmr_ids, 0 active entries, 0 allocated entries.
TCAM Logical Table: TCAM_LT_ODS8 (3), free entries: 15204, resvd 127
ACL Common Region: 448 entries allocated. 448 entries free
Application ID: NP_APP_ID_PBR (5)
Total: 1 vmr_ids, 2 active entries, 2 allocated entries.
RP/0/RSP0/CPU0:ASR9000#
Show Commands
• On a NCS6000 client too
attach location 0/1/CPU0
pbtm_show -n 0 -s
NPU:0 Dev:0 Num Cblks:64 InUse:Y Num SubCblks:128 SubCblks Used:3
Idx Idx Sub In Unit Alloc Res Num Num Use

HW cblk use size feature Size Cells Free %
=== === ==== === ==== ========== ==== ===== ===== ====
0 0 0 Y 160b ACLv4 16B 2048 1974 4%
--- --- ---- --- ---- ---------- ---- ----- ----- ----
1 1 0 Y 640b ACLv6 16B 2048 2040 1%
--- --- ---- --- ---- ---------- ---- ----- ----- ----
63 63 1 Y 160b 16B 2048 2044 1%
--- --- ---- --- ---- ---------- ---- ----- ----- ----
NPU:0 Dev:1 Num Cblks:64 InUse:Y Num SubCblks:128 SubCblks Used:2
Idx Idx Sub In Unit Alloc Res Num Num Use

HW cblk use size feature Size Cells Free %
=== === ==== === ==== ========== ==== ===== ===== ====
0 128 0 Y 160b ACLv4 16B 2048 2046 1%
--- --- ---- --- ---- ---------- ---- ----- ----- ----
1 129 0 Y 640b ACLv6 16B 2048 2040 1%
--- --- ---- --- ---- ---------- ---- ----- ----- ----
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 174
Show Commands
• To help TAC progress faster to identify a problem
On the Controller:
- show run class-map
- show class-map
On the Client:
- debug flowspec all
- show flowspec trace manager event error
- show flowspec trace client event error
- show flowspec client internal
- show logging | inc FLOW
- show flowspec vrf all afi-all summary internal
- show flowspec vrf all afi-all internal
- show tech flowspec
Show Commands
• To measure the traffic matched, no SNMP but CLI and Netconf/XML.
RP/0/RP0/CPU0:Client#show flowspec ipv4 detail
AFI: IPv4
Flow :Dest:25.1.104.0/24
Matched : 21946725652/13958117514672
Transmitted : 236878/150654408
Dropped : 21946488774/13957966860264
Flow :Proto:=17,DPort:=53
Matched : 0/0
Transmitted : 0/0
Dropped : 0/0
Counters for each rule are available per VRF / address-family, not per interface.
eBGP FlowSpec router bgp 1
neighbor 25.2.1.3 Controller
remote-as 2
route-policy pass-all in
route-policy pass-all out
next-hop-unchanged
!
neighbor 25.2.1.4
remote-as 1
Client eBGP address-family ipv4 flowspec
Client iBGP
router bgp 2
XR-Services router bgp 1
!
!
neighbor 25.2.1.11
neighbor 25.2.1.10
remote-as 1
remote-as 1
update-source TenGigE0/2/0/8
update-source TenGigE0/2/0/6
!
!
route-policy pass-all in
route-policy pass-all out Client Client !
!
validation disable
!
!
!
AS 2 AS 1
eBGP FlowSpec: Validate Disable
Without the “Validate disable”, a check on AS Path is done and the route is not
accepted.
RP/0/RP0/CPU0:Client#sh bgp ipv4 flowspec Dest:25.1.104.1/32,Proto:=17,Length:>=500&<=1550/128 detail
BGP routing table entry for Dest:25.1.104.1/32,Proto:=17,Length:>=500&<=1550/128

NLRI in Hex: 0120190168010381110a1301f4d5060e/128
Versions:
Process bRIB/RIB SendTblVer
Speaker 8 8
Flags: 0x04000001+0x00000200;
Paths: (1 available, no best path)
Path #1: Received by speaker 0
Flags: 0x4000080000020001, import: 0x20
1
0.0.0.0 from 25.2.1.11 (6.6.6.6)
Origin IGP, localpref 100, valid, external, invalid flowspec-path
Received Path ID 0, Local Path ID 0, version 0
Extended community: FLOWSPEC Traffic-rate:1,12500
eBGP FlowSpec: Next-Hop Unchanged
• Without the “NH unchanged” configuration, the NH action will not work on eBGP
• NH will be, by default, positioned as the peer address
XR-Services
25.2.1.11
25.2.1.3
Client Client
102 104
Hu0/0/0/0 Hu0/1/0/0 Hu0/0/0/0 Hu0/0/0/1
25.1.9.3 25.1.9.4
AS 2 AS 1
eBGP FlowSpec: Next-Hop Unchanged
Controller eBGP Client
policy-map type pbr TEST RP/0/RP0/CPU0:Client#sh bgp ipv4 flowspec
class type traffic MATCHING-RULE1 <SNIP>
redirect nexthop 25.3.9.4 Network Next Hop Metric LocPrf Weight Path
! *> Dest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550/128
class type traffic class-default 25.2.1.11 0 1 i
!
end-policy-map Processed 1 prefixes, 1 paths
We configure next-hop-unchanged on the controller:

RP/0/0/CPU0:Ctrl#conf RP/0/RP0/CPU0:Client#sh bgp ipv4 flowspec
Tue Feb 10 03:55:22.423 UTC <SNIP>
RP/0/0/CPU0:Ctrl(config)#router bgp 1 Network Next Hop Metric LocPrf Weight Path
RP/0/0/CPU0:Ctrl(config-bgp)#neighbor-group ebgp- *> Dest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550/128
flowspec 25.3.9.4 0 1 i
RP/0/0/CPU0:Ctrl(config-bgp-nbrgrp)#address-family RP/0/RP0/CPU0:Client#sh flows ipv4 det
ipv4 flowspec AFI: IPv4
RP/0/0/CPU0:Ctrl(config-bgp-nbrgrp-af)#next-hop- Flow :Dest:25.1.102.1/32,Proto:=17,Length:>=500&<=1550
unchanged Actions :Nexthop: 25.3.9.4 (bgp.1)
RP/0/0/CPU0:Ctrl(config-bgp-nbrgrp-af)#commit Statistics (packets/bytes)
RP/0/0/CPU0:Ctrl(config-bgp-nbrgrp-af)# Matched : 10964755/15306797980
Dropped : 0/0
IOS XR Implementation
Application on Interface
• Uses the PBR infrastructure with similar performance penalty than other PBR
features like ABF. Performance cost will vary depending upon the action
• DSCP marking will be least expensive
• redirect action pointing to recursive TE tunnel path being most expensive
• Can coexist with other features like QoS or ACL

• Interface can be in the Global Routing Table or on a VRF (L3VPN or VRF-Lite)
Back-Up Slides
3rd Party
Controller
BGP FlowSpec with 3rd Party Apps
• BGP FlowSpec is based on IETF standard
• It can interoperate with non-Cisco devices compliant to the standards
• Following list in offering a few controllers examples and is non-exhaustive
• Arbor SP
• ExaBGP
• YABGP
• Open Day Light
Using Arbor SP
Using Arbor SP
Using Arbor SP
Using Arbor SP
Using Arbor SP
For Your
Reference
Using ExaBGP
flow {
route name-of-the-route {
match {
source 10.0.0.1/32;
destination 192.168.0.1/32;
neighbor 10.0.0.1 { port =80 =8080;
description "xrv 5.2.0"; destination-port >8080&<8088 =3128;
router-id 192.168.2.26; source-port >1024;
local-address 192.168.2.26; protocol [ tcp udp ];
local-as 65000; packet-length >200&<300 >400&<500;
#fragment not-a-fragment;
peer-as 65000;
fragment [ first-fragment last-fragment ];
graceful-restart 5; icmp-type [ unreachable echo-request echo-reply ];
icmp-code [ host-unreachable network-unreachable ];
flow { tcp-flags [ urgent rst ];
route name-of-the-route { dscp [ 10 20 ];
match { ...
<<<description>>> }
} then {
then { ... #rate-limit 9600;
#discard;
<<<action>>>
redirect 65500:12345;
} #redirect 1.2.3.4:5678;
} community [30740:0 30740:30740];
#extended-community [ origin:2345:6.7.8.9 origin:2.3.4.5:6789 ];
}
}
}
<flowspec> For Your
<source-ports> Reference
Using Open Day Light <op>greater-than end-of-list</op>
<value>1024</value>
<flowspec-route xmlns="urn:opendaylight:params:xml:ns:yang:bgp-flowspec"> </source-ports>
<route-key>flow1</route-key> </flowspec>
<flowspec> <flowspec>
<destination-prefix>192.168.0.1/32</destination-prefix> <types>
</flowspec> <op>equals end-of-list</op>
<flowspec> <value>0</value>
<source-prefix>10.0.0.1/32</source-prefix> </types>
</flowspec> </flowspec>
<protocol-ips> <codes>
<op>equals end-of-list</op> <op>equals end-of-list</op>
<value>6</value> <value>0</value>
</protocol-ips> </codes>
<ports> <tcp-flags>
<op>equals end-of-list</op> <op>match end-of-list</op>
</ports> </tcp-flags>
<destination-ports> <packet-lengths>
<op>greater-than</op> <op>greater-than</op>
</destination-ports> </packet-lengths>
<destination-ports> <packet-lengths>
<op>and-bit less-than end-of-list</op> <op>and-bit less-than end-of-list</op>
</destination-ports> </packet-lengths>

BRKSPG-3012 - SP Security Leveraging BGP FlowSpec To Protect Your Infrastructure

Uploaded by

Copyright:

Available Formats

BRKSPG-3012 - SP Security Leveraging BGP FlowSpec To Protect Your Infrastructure

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

BRKSPG-3012 - SP Security Leveraging BGP FlowSpec To Protect Your Infrastructure

Uploaded by

Copyright:

Available Formats

Nicolas Fevrier, Technical Leader Engineering

• Why not Openflow or direct NetConf to the router ?

Type Description Encoding

Type Description Encoding

Platform Hardware Support in Data Plane

Cisco IOS XR Routers BGP FS Implementation

Cisco IOS XR Routers BGP FS Implementation

• In current implementation, rules are applied:

Packet to 10.0.16.51 rtrB

rtrA BGP FS is applied to

ASR1000 CSR1000v ISR4400

Max rules per system 4000 250 4000

Max rules per VRF 1000 32 250

• 2015/2016  raise of Amp attacks

Source: Arbor WISR 2018

• But Amplification attacks didn’t

Source: Arbor WISR 2018

• Amplification attacks, example NTP

UDP 123 Match: dest-IP: 2.1.1.1

UDP 123 Data 500B

Match: dest-IP: 2.1.1.1

Match: dest-IP: 2.1.1.1

• Edited version of a recording from Tomas Sundstrom

Match: dest-IP: 7.7.7.7

• Second attack is identified as a NTP Amplification (abnormal packet size)

• Single point of control to program rules in many clients

• Several approaches exist in the design of a DDoS mitigation solution

• They all consist in:

• Following examples are real-case used in very large production networks

10.51.51.51 is a dummy route, advertised to trigger the redirection

BGP 10.1.1.2/30 10.1.2.2/30 .2

• Different peering / transit points

• Based on the source ranges,

Rule: src-IP: 10.0.1.0/24

CGSE+ Rule: src-IP: 10.0.0.0/24

Rule: dest-IP: 2.1.1.1 S

• A transit provider is offering to AS Customer a 10GE connectivity to the Internet

• AS Transit programs a BGP FS rule to rate-limit traffic targeted to the victim IP

Rule: dest-IP: 2.1.1.1

• Rule disseminated upstream to tackle the attack as early as possible

• Used to redirect traffic

• Used to select a transport

• Based on source addresses

• Based on dstination addresses Rule: dst-port: AAA

• On both Client and Controller

policy-map type pbr FS

Configuring BGP FlowSpec on IOS XR Routers

RP/0/0/CPU0:Ctrl(config)#class-map type traffic match-all MATCHING-RULE

RP/0/RP0/CPU0:Client#sh flowspec ipv4 detail

RP/0/RSP0/CPU0:Client#sh flowspec afi-all detail

Hex 4a3ebc20 = 31,125,000 Bytes/sec

Action Redirect: Digging Deeper

Client View (Debug all Flowspec Events)

In this case, we used an IPv4 address for the Next-Hop.

RP/0/0/CPU0:Ctrl#sh route 25.1.102.1 RP/0/RSP0/CPU0:Client#sh bgp ipv4 flowspec

RP/0/RSP0/CPU0:Client#sh bgp ipv4 flowspec sum

RP/0/0/CPU0:Ctrl#sh run router static RP/0/RSP0/CPU0:Client#show bgp ipv4 flowspec

Processed 1 prefixes, 1 paths