Saturday, April 13, 2013

Configuring vPC (Virtual Port Channel) with Cisco Nexus


In this scenario, I have two Cisco Nexus 5548s, two Cisco Nexus 2248 FEXes, and a Windows Server with 4 Ethernet interfaces where two interfaces connect to FEX 100 and the other two connect to FEX 101. I want to configure Virtual Port Channels in such a way that if there is a loss of connectivity from either 5K, the two FEXes will still service clients.



To start, I'll configure the management interface address (10.2.8.13 and .14), and default gateways for both 5Ks (N5K-1 is shown).

N5K-1# conf t
N5K-1(config)# interface mgmt0
N5K-1(config-if)# ip address 10.2.8.13/24
N5K-1(config-if)# exit
N5K-1(config)# ip route 0.0.0.0/0 10.2.8.1

Note that the management interface is in the management VRF. I will use the management network as the keep-alive management link for the vPCs I'll create.

N5K-1(config)# show vrf
VRF-Name                           VRF-ID State   Reason                        
default                                 1 Up      --                            
management                              2 Up      --                            
N5K-1(config)# ping 10.2.8.1 vrf management count 2
PING 10.2.8.1 (10.2.8.1): 56 data bytes
64 bytes from 10.2.8.1: icmp_seq=0 ttl=254 time=0.969 ms
64 bytes from 10.2.8.1: icmp_seq=1 ttl=254 time=0.667 ms

--- 10.2.8.1 ping statistics ---
2 packets transmitted, 2 packets received, 0.00% packet loss
round-trip min/avg/max = 0.667/0.817/0.969 ms
N5K-1(config)# 

Now I'll create uplink trunks to the core network for access to VLAN 930 for both N5Ks (N5K-1 is shown).

N5K-1(config)# interface e1/15-16
N5K-1(config-if-range)# no shut
N5K-1(config-if-range)# switchport mode trunk
N5K-1(config-if-range)# switchport trunk allowed vlan 1,930
N5K-1(config-if-range)# vlan 930
N5K-1(config-vlan)# sh vlan b

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Eth1/1, Eth1/2, Eth1/3, Eth1/4
                                                Eth1/5, Eth1/6, Eth1/7, Eth1/8
                                                Eth1/9, Eth1/10, Eth1/11
                                                Eth1/12, Eth1/13, Eth1/14
                                                Eth1/15, Eth1/16, Eth1/17
                                                Eth1/18, Eth1/19, Eth1/20
                                                Eth1/21, Eth1/22, Eth1/23
                                                Eth1/24, Eth1/25, Eth1/26
                                                Eth1/27, Eth1/28, Eth1/29
                                                Eth1/30, Eth1/31, Eth1/32
930  VLAN0930                         active    Eth1/15, Eth1/16

N5K-1(config-vlan)# show interface trunk 

--------------------------------------------------------------------------------
Port          Native  Status        Port
              Vlan                  Channel
--------------------------------------------------------------------------------
Eth1/15       1       trunking      --
Eth1/16       1       trunking      --

--------------------------------------------------------------------------------
Port          Vlans Allowed on Trunk
--------------------------------------------------------------------------------
Eth1/15       1,930
Eth1/16       1,930

(...output omitted...)
N5K-1(config)# 

Each N5K will have two links to each FEX which I will configure as port-channels. I'll enable the features for fex and lacp for both 5Ks (N5K-1 is shown).

N5K-1(config)# show feature | in enabled|fex|lacp|vpc
fex                   1         disabled
lacp                  1         disabled
lldp                  1         enabled 
sshServer             1         enabled 
vmfex                 1         disabled
vpc                   1         disabled
N5K-1(config)# 
N5K-1(config)# feature fex
N5K-1(config)# feature lacp
N5K-1(config)# 
N5K-1(config)# show feature | in enabled
fex                   1         enabled 
lacp                  1         enabled 
lldp                  1         enabled 
sshServer             1         enabled 
vpc                   1         enabled
N5K-1(config)# 

Now I'll stage connectivity to the two FEX units; 100, and 101 for each 5K. Since I will be creating port-channels afterward, the FEX associations will only require 1 link for pinning. (N5K-1 is shown).

N5K-1(config)# fex 100
N5K-1(config-fex)# pinning max-links 1
Change in Max-links will cause traffic disruption.
N5K-1(config-fex)#
N5K-1(config-fex)# fex 101
N5K-1(config-fex)# pinning max-links 1
Change in Max-links will cause traffic disruption.
N5K-1(config-fex)#

Now I'll create port-channels 78 and 91 (N5K-1 is shown).

N5K-1(config-fex)# interface e1/7-8
N5K-1(config-if-range)# channel-group 78
N5K-1(config-if-range)# no shut
N5K-1(config-if-range)#
N5K-1(config-if-range)# interface e1/9-10
N5K-1(config-if-range)# channel-group 91
N5K-1(config-if-range)# no shut
N5K-1(config-if-range)#

At this point, I should be able to see that the two FEXes have been discovered (N5K-1 is shown)

N5K-1(config-if-range)# show fex
  FEX         FEX           FEX                       FEX               
Number    Description      State            Model            Serial     
------------------------------------------------------------------------
---       --------            Discovered     N2K-C2248TP-1GE   SSI141904QR
---       --------            Discovered     N2K-C2248TP-1GE   SSI14250839
N5K-1(config-if-range)#

Now I'll associate the port-channel interfaces with their respective FEXes (N5K-1 is shown).

N5K-1(config-if-range)# int po78
N5K-1(config-if)# switchport mode fex-fabric
N5K-1(config-if)# fex associate 100
N5K-1(config-if)# 
N5K-1(config-if)# int po91
N5K-1(config-if)# switchport mode fex-fabric 
N5K-1(config-if)# fex associate 101
N5K-1(config-if)# exit
N5K-1(config)#

From N5K-1 I see the two FEXes online (since I configured this one before N5K-2), but for N5K-2, I see them as "offline". This should change once I complete the vPC configuration.

N5K-1(config-if)# show fex
  FEX         FEX           FEX                       FEX               
Number    Description      State            Model            Serial     
------------------------------------------------------------------------
100        FEX0100                Online     N2K-C2248TP-1GE   SSI14250839
101        FEX0101                Online     N2K-C2248TP-1GE   SSI141904QR
N5K-1(config-if)# 


N5K-2# sho fex
  FEX         FEX           FEX                       FEX               
Number    Description      State            Model            Serial     
------------------------------------------------------------------------
---       --------               Offline     N2K-C2248TP-1GE   SSI14310232
---       --------               Offline     N2K-C2248TP-1GE   SSI14270CLA
N5K-2# 

Since the FEXes are online for N5K-1, I should be able to see all interfaces available on both FEXes.

N5K-1(config-if)# show interface brief

--------------------------------------------------------------------------------
Ethernet      VLAN    Type Mode   Status  Reason                   Speed     Por
t
Interface                                                                    Ch 
#
--------------------------------------------------------------------------------
(...output omitted...)

Eth1/7        1       eth  fabric up      none                        10G(D) 78
Eth1/8        1       eth  fabric up      none                        10G(D) 78
Eth1/9        1       eth  fabric up      none                        10G(D) 91
Eth1/10       1       eth  fabric up      none                        10G(D) 91

(...output omitted...)

--------------------------------------------------------------------------------
Port-channel VLAN    Type Mode   Status  Reason                    Speed   Proto
col
Interface                                                                  
--------------------------------------------------------------------------------
Po78         1       eth  fabric up      none                       a-10G(D)  no
ne
Po91         1       eth  fabric up      none                       a-10G(D)  no
ne

--------------------------------------------------------------------------------
Port   VRF          Status IP Address                              Speed    MTU
--------------------------------------------------------------------------------
mgmt0  --           up     10.2.8.53                               1000     1500

--------------------------------------------------------------------------------
Ethernet      VLAN    Type Mode   Status  Reason                   Speed     Por
t
Interface                                                                    Ch 
#
--------------------------------------------------------------------------------
Eth100/1/1    1       eth  access down    Administratively down      auto(D) --
Eth100/1/2    1       eth  access down    Administratively down      auto(D) --
Eth100/1/3    1       eth  access down    Administratively down      auto(D) --

(...output omitted...)

Eth100/1/1    1       eth  access down    Administratively down      auto(D) --
Eth100/1/2    1       eth  access down    Administratively down      auto(D) --

(...output omitted...)

Now I'll establish the vPC configuration. I'll create vPC domain 55, and use the management network for the peer-keepalives. I'll then create a port-channel between the N5Ks, and enable that for vPC peer-linking.

N5K-1(config)# vpc domain 55
N5K-1(config-vpc-domain)# peer-keepalive destination 10.2.8.14
Note: 
 --------:: Management VRF will be used as the default VRF ::--------
N5K-1(config-vpc-domain)# exit
N5K-1(config)# 
N5K-1(config)# int e1/17-18
N5K-1(config-if-range)# channel-group 1718 mode active
N5K-1(config-if-range)# interface po1718
N5K-1(config-if)# switchport mode trunk
N5K-1(config-if)# switchport trunk allow vlan all
N5K-1(config-if)# vpc peer-link
Please note that spanning tree port type is changed to "network" port type on vPC peer-link.
This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge Assurance
(which is enabled by default) is not disabled.
N5K-1(config-if)# 

Now I'll verify that the vPC is operational.

N5K-1(config-if-range)# sh vpc brief
Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                   : 55  
Peer status                     : peer adjacency formed ok      
vPC keep-alive status           : peer is alive                 
Configuration consistency status: success 
Per-vlan consistency status     : success                       
Type-2 consistency status       : success 
vPC role                        : secondary                     
Number of vPCs configured       : 0   
Peer Gateway                    : Disabled
Dual-active excluded VLANs      : -
Graceful Consistency Check      : Enabled

vPC Peer-link status
---------------------------------------------------------------------
id   Port   Status Active vlans    
--   ----   ------ --------------------------------------------------
1    Po1718 up     1,930        
N5K-1(config-if-range)# exit
N5K-1(config)# 

On both N5Ks, I'll create port-channel 78 and 91, and create vPC 78 and 91.

N5K-1(config-if)# int po78
N5K-1(config-if)# vpc 78
N5K-1(config-if)# int po91
N5K-1(config-if)# vpc 91
N5K-1(config-if)# 
N5K-1(config-if)# show fex
  FEX         FEX           FEX                       FEX               
Number    Description      State            Model            Serial     
------------------------------------------------------------------------
100        FEX0100                Online     N2K-C2248TP-1GE   SSI14270CLA
101        FEX0101                Online     N2K-C2248TP-1GE   SSI14310232
N5K-1(config-if)# 


N5K-2(config-if)# 
N5K-2(config-if)# show fex
  FEX         FEX           FEX                       FEX               
Number    Description      State            Model            Serial     
------------------------------------------------------------------------
100        FEX0100                Online     N2K-C2248TP-1GE   SSI14270CLA
101        FEX0101                Online     N2K-C2248TP-1GE   SSI14310232
N5K-2(config-if)# 

At this point, I'll enable the access ports for both FEXes to the server. (N5K-1 is shown).

N5K-1(config)# int e100/1/1-2 , e101/1/1-2
N5K-1(config-if-range)# switchport mode access
N5K-1(config-if-range)# switchport access vlan 930
N5K-1(config-if-range)# no shut
N5K-1(config-if-range)# 
N5K-1(config-if-range)# show interface brief

(...output omitted...)

--------------------------------------------------------------------------------
Ethernet      VLAN    Type Mode   Status  Reason                   Speed     Por
t
Interface                                                                    Ch 
#
--------------------------------------------------------------------------------
Eth100/1/1    930     eth  access up      none                       1000(D) --
Eth100/1/2    930     eth  access up      none                       1000(D) --

(...output omitted...)

Eth101/1/1    930     eth  access up      none                       1000(D) --
Eth101/1/2    930     eth  access up      none                       1000(D) --

On the server, I have an Intel Quad port card which I've teamed together.


I've assigned the address of 100.1.1.7/24.


 I'll test connectivity by pinging 10.1.1.21. This is a machine in the core network on VLAN 930.


Note that although the ports are teamed, I've left the adapter in fault tolerance mode. This means that one port will be active, and the rest will remain in standby. 


I'll set a continuous ping, and then reload N5K-1.

N5K-1# copy run start
[########################################] 100%
Copy complete, now saving to disk (please wait)...
N5K-1# reload
WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y

Note that a ping packet was lost, but the adapter remained the same. This means that FEX 100 was able to continue providing access to the server through the vPC through N5K-2.