LARGE SCALE DYNAMIC MULTIPOINT VPN

NOVEMBER 2004

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

INTRODUCTION

Large Scale DMVPN, 11/04 Presentation_ID

2004, Cisco Systems, Inc. All rights reserved. 2004 Cisco Systems, Inc. All rights reserved.

Dynamic Multipoint VPN Facts

Dynamic Multipoint VPN (DMVPN) can work with static routes but shows its power with routing protocols

The routing protocol consumes a lot of CPU with so many neighbors

Resource consumption increases with the number of tunnels

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

IPsec facts

IPsec maximum throughput is better with large packets

On medium and low platforms, CPU is impacted by large SADB

Cisco recommends that users keep a DMVPN hub within reasonable limits

Consult your Account Team about platform details

Mbps

64 bytes

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

1400 bytes

Packet size
4

Example Cisco 7200 Series/VAM2

The Cisco 7200 Series Router is a popular platform for DMVPN

It can accept a maximum of 375 tunnels without particular attention (EIGRP)

In that case, the max throughput would be

42,000 pps for 64 bytes packets

22,000 pps for 1400 bytes packets

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

Scaling the Cisco 7200 Series/VAM2 Further

If a second mGRE interface is set up on the Cisco 7200 Series Router, it can accept a maximum of 350 tunnels per interface (700 total)

In that case the max throughput is:

40,000 pps for 64 bytes packets

22,000 pps for 1400 bytes packets

A third interface does not improve things

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

Is This Low?

Yes and no

The theoretical maximum number of tunnels (Cisco 7200 Series / VAM2) is 5,000 so DMVPN looks bad

The theoretical max speed is 250Mbps so DMVPN looks the same

250Mbps/700 = 350Kbps per spoke

Not very useful below that throughput anyway

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

Remarks

This presentation describes current performance

Performances change every day and protocols evolve

Check with your account team to evaluate the best DMVPN platform for your needs

It is possible to scale DMVPN very high

Just wait for the next chapter

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

Summary on DMVPN Fitness

If many spokes with very low IPsec throughput, DMVPN may not be a good fit

DMVPN starts to become useful at the edge between remoteaccess and lan-to-lan

DMVPN works best for spokes that need statistically constant equal access to central resources

Small offices, branch offices, hot-spots, administrations, schools

Many existing remote-access or LAN to LAN solutions should actually be DMVPN like networks

DMVPN shows a network with integrated security

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

APPLICATION TO LARGE SCALE IPSEC

Large Scale DMVPN, 11/04 Presentation_ID

2004, Cisco Systems, Inc. All rights reserved. 2004 Cisco Systems, Inc. All rights reserved.

10

Problem description

Need to deploy a large DMVPN network

Any number 700+ ; tens of thousands allowed

More than just basic connectivity needed

Limited to hub and spoke

Spoke to spoke via the hub is allowed

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

11

Requirements

Constraints

LAN to LAN

Dynamic IP addresses

Solution must:

Be easy to manage (deployment and monitoring)

Recover by itself

Scale to thousands of spokes

Allow Cisco rich features (ie: Cisco IOS Intrusion Prevention System (IPS), Cisco IOS Firewall)

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

12

Overall Solution
HQ

Edge of HQ Cluster of DMVPN hubs Aggregates user tunnels Cluster can be heterogeneous GRE/IPsec tunnels IGP + NHRP

SLB balances connections Owns virtual IP address

Spokes (83x) DMVPN based Provide QoS And Firewalling

No special software needed on PC IP phones work out of the box

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

The Load Balancer In General

Load Balancer owns a Virtual IP Address (VIP)

When IKE or ESP packets are targeted at the VIP, the LB chooses a hub

The hub choice is policy (predictor) based:

Weighted round-robin

Least-connections

Once a decision is made for a tunnel, all subsequent packets go to the same hub (stickyness)

Once a decision is made for IKE, the same is made for ESP (buddying)

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

14

High Level Description

Spokes think there is a single hub

They have an NHRP map pointing to the Load Balancers Virtual IP Address

The Load Balancer is configured in forwarding mode (no NAT)

All the hubs have the same configuration

Same Tunnel interface address

Same Loopback address (= VIP)

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

15

Topology with Addresses

.2 10.1.1.0/24 10.1.0.0/24 .3 .1

.1 .3

Loopback: 172.17.0.1 Tunnel0: 10.0.0.1/16

Loopback: 172.17.0.1 Tunnel0: 10.0.0.1/16

.2

Load Balancer VIP: 172.17.0.1 (no tunnel)

Physical: (dynamic)172.16.1.1 Tunnel0: 10.0.0.11 Spoke A

Physical: (dynamic)172.16.2.1 Tunnel0: 10.0.0.12

192.168.1.1/24

Spoke B

192.168.2.1/24

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

16

Spoke Configuration

The spoke configuration is the same as with a single hub

It has an NHRP map

ip nhrp map 10.0.0.1 172.17.0.1

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

17

Load Balancer

We will study Cisco IOS Software SLB

Runs on most Cisco IOS Software platforms, including the Cisco Catalyst 6500 Series Switch

Opt for Releases 12.2S or 12.1E

CSM 3.1 or above should work too but we do not need most of its features (useless)

Load balancing must be able to do Layer 3 and 4 load balancing

Upper layers are useless (encrypted)

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

18

Cisco IOS Software SLB performances

Cisco IOS Software SLB on a Cisco Catalyst 6500 Series Switch (MSFC-2)

Can manage 1M connections w/ 128MB RAM

Can create 20,000 connections per second

Switches packets at 10Gbps (64 bytes)

Cisco IOS Software SLB on a Cisco 7200 Series Router (NPE400)

Can create 5,000 connections per second

Switches packets at the Cisco Express Forwarding rate (depending on other features)

Should not be a bottleneck

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

19

Cisco IOS Software SLB cluster definition

ip slb probe PINGREAL ping faildetect 2

ip slb serverfarm HUBS failaction purge probe PINGREAL ! predictor round-robin

Weighted round-robin This is the default

real 10.1.0.2 weight 4 inservice

real 10.1.0.3 weight 4 inservice

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

20

Cisco IOS Software SLB VIP definition

Same farm

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

21

Monitoring and managing

SLB-7200#sh ip slb connections

vserver prot client real state nat ------------------------------------------------------------------------------IKESLB UDP 64.103.8.8:500 10.1.0.2 ESTAB none ESPSLB ESP 217.136.116.189:0 10.1.0.2 ESTAB none IKESLB UDP 213.224.65.3:500 10.1.0.2 ESTAB none ESPSLB ESP 80.200.49.217:0 10.1.0.2 ESTAB none ESPSLB ESP 217.136.132.202:0 10.1.0.3 ESTAB none connections connections connections connections ? for a firewallfarm for a specific serverfarm for a specific virtual server

SLB-7200#clear ip slb firewallfarm Clear serverfarm Clear vserver Clear <cr>

SLB-7200#sh ip slb reals

real farm name weight state conns ------------------------------------------------------------------10.1.0.2 HUBS 4 OPERATIONAL 4 10.1.0.3 HUBS 4 OPERATIONAL 1
22

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

Hub Tunnel configuration

interface Tunnel0 interface Loopback0 bandwidth 10000 ip address 172.17.0.1 255.255.255.255 ip address 10.0.0.1 255.255.0.0 end no ip redirects Must be same on all ip mtu 1350 Mask is /32 ip nhrp map multicast dynamic Must be same on all ip nhrp network-id 1 Mask allows 2^16-2 nodes ip nhrp holdtime 3600 no ip split-horizon interface FastEthernet0/0 no ip mroute-cache ip address 10.1.0.{2,3} 255.255.255.0 tunnel source Loopback0 interface FastEthernet0/1 tunnel mode gre multipoint ip address 10.2.0.{2,3} 255.255.255.0 tunnel key 1 tunnel protection ipsec profile tp end

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

23

Routing protocols
HQ Speaks EIGRP 2 Redistribute EIGRP 1 into BGP (with filtering) Redistribute BGP(summarized) into EIGRP 1

Redistribute EIGRP 2 into BGP (summary) Redistribute floating static (Null0) into EIGRP2

Spokes are EIGRP 1 stubs They speak to hubs thru GRE/IPsec tunnel
24

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

Hub Routing protocol configuration

router eigrp 1 redistribute bgp 1 metric 1 0 255 20 1400 network 10.0.0.0 0.0.255.255 default-metric 64 2000 255 1 1400 no auto-summary router bgp 1 bgp router-id 10.2.0.{2,3} bgp log-neighbor-changes neighbor 10.0.0.1 remote-as 1

address-family ipv4 redistribute eigrp 1 route-map <IGPREDIST> neighbor 10.2.0.1 activate neighbor 10.2.0.1 next-hop-self no auto-summary no synchronization bgp redistribute-internal exit-address-family
25

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

Edge router BGP configuration

router bgp 1 no synchronization bgp log-neighbor-changes aggregate-address 10.0.0.0 255.0.0.0 summary-only aggregate-address 192.168.0.0 255.255.0.0 summary-only redistribute eigrp 2 neighbor HUB peer-group neighbor HUB remote-as 1 neighbor HUB next-hop-self neighbor 10.0.0.2 peer-group HUB neighbor 10.0.0.3 peer-group HUB no auto-summary

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

26

Edge router EIGRP configuration

EIGRP 2 attracts spoke subnets to the edge router

Floating static route to Null0 discards packets to unconnected spokes

ip route 192.168.0.0 255.255.255.127 Null0 254

router eigrp 2 redistribute static network 192.168.1.0 0.0.0.128 no auto-summary no eigrp log-neighbor-changes

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

27

Packet Flow
Corporate
4

2 1 PC1

6 7 PC2
28

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

Result

Tunnels reconnect automatically

Working sessions are not lost

QoS allocates bandwidth to voice

All other features are available

No need to touch the hubs while adding a spoke

New hubs can be added/removed on the fly

Simple to deploy

Leverages monitoring infrastructure (interfaces, CDP)

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

29

Support

Each feature has plenty of nerd knobs for tuning

Each feature has advanced debugging capabilities

Each feature can be troubleshot independently

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

30

IGP choices

BGP between Hubs and Edge is good due to number of prefixes and flexibility

Scaling the IGP between hubs and spokes is the hardest part

A distance vector is recommended

EIGRP shows best results so far but ODR is under test (lightweight)

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

31

Positioning

The main advantages of the solution are:

Virtually limitless scaling

Can be deployed in zero touch with ISC and Intelligent Engine

Automatic load management

Load balancing AND resilience

Multiply performances by number of hubs (creation rate, speed, max SAs)

No forklift when upgrading

Resilience in N+1

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

32

Improvements

It is possible to collapse the Load balancer and the edge router (hubs in lollipop)

If the load balancer is a Cisco Catalyst 6500 Series Switch, this is even recommended as Layer 3 switching will accelerate spoke to spoke traffic

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

33

Routing Protocols
HQ Speaks EIGRP 2 Redistribute EIGRP 1 into BGP (with filtering) Redistribute BGP(summarized) into EIGRP 1

Redistribute EIGRP 2 into BGP (summary) Redistribute floating static (Null0) into EIGRP2

Spokes are EIGRP 1 stubs They speak to hubs thru GRE/IPsec tunnel
34

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

Large Scale DMVPN, 11/04

2004 Cisco Systems, Inc. All rights reserved.

35