In this scenario, I have two Cisco Nexus 5548s, two Cisco Nexus 2248 FEXes, and a Windows Server with 4 Ethernet interfaces where two interfaces connect to FEX 100 and the other two connect to FEX 101. I want to configure Virtual Port Channels in such a way that if there is a loss of connectivity from either 5K, the two FEXes will still service clients.
To start, I'll configure the management interface address (10.2.8.13 and .14), and default gateways for both 5Ks (N5K-1 is shown).
N5K-1# conf tN5K-1(config)# interface mgmt0N5K-1(config-if)# ip address 10.2.8.13/24N5K-1(config-if)# exitN5K-1(config)# ip route 0.0.0.0/0 10.2.8.1Note that the management interface is in the management VRF. I will use the management network as the keep-alive management link for the vPCs I'll create.
N5K-1(config)# show vrfVRF-Name VRF-ID State Reason default 1 Up -- management 2 Up -- N5K-1(config)# ping 10.2.8.1 vrf management count 2PING 10.2.8.1 (10.2.8.1): 56 data bytes64 bytes from 10.2.8.1: icmp_seq=0 ttl=254 time=0.969 ms64 bytes from 10.2.8.1: icmp_seq=1 ttl=254 time=0.667 ms
--- 10.2.8.1 ping statistics ---2 packets transmitted, 2 packets received, 0.00% packet lossround-trip min/avg/max = 0.667/0.817/0.969 msN5K-1(config)# Now I'll create uplink trunks to the core network for access to VLAN 930 for both N5Ks (N5K-1 is shown).
N5K-1(config)# interface e1/15-16N5K-1(config-if-range)# no shutN5K-1(config-if-range)# switchport mode trunkN5K-1(config-if-range)# switchport trunk allowed vlan 1,930N5K-1(config-if-range)# vlan 930N5K-1(config-vlan)# sh vlan b
VLAN Name Status Ports---- -------------------------------- --------- -------------------------------1 default active Eth1/1, Eth1/2, Eth1/3, Eth1/4 Eth1/5, Eth1/6, Eth1/7, Eth1/8 Eth1/9, Eth1/10, Eth1/11 Eth1/12, Eth1/13, Eth1/14 Eth1/15, Eth1/16, Eth1/17 Eth1/18, Eth1/19, Eth1/20 Eth1/21, Eth1/22, Eth1/23 Eth1/24, Eth1/25, Eth1/26 Eth1/27, Eth1/28, Eth1/29 Eth1/30, Eth1/31, Eth1/32930 VLAN0930 active Eth1/15, Eth1/16
N5K-1(config-vlan)# show interface trunk
--------------------------------------------------------------------------------Port Native Status Port Vlan Channel--------------------------------------------------------------------------------Eth1/15 1 trunking --Eth1/16 1 trunking --
--------------------------------------------------------------------------------Port Vlans Allowed on Trunk--------------------------------------------------------------------------------Eth1/15 1,930Eth1/16 1,930
(...output omitted...)N5K-1(config)# Each N5K will have two links to each FEX which I will configure as port-channels. I'll enable the features for fex and lacp for both 5Ks (N5K-1 is shown).
N5K-1(config)# show feature | in enabled|fex|lacp|vpcfex 1 disabledlacp 1 disabledlldp 1 enabled sshServer 1 enabled vmfex 1 disabledvpc 1 disabledN5K-1(config)# N5K-1(config)# feature fexN5K-1(config)# feature lacpN5K-1(config)# N5K-1(config)# show feature | in enabledfex 1 enabled lacp 1 enabled lldp 1 enabled sshServer 1 enabled vpc 1 enabledN5K-1(config)# Now I'll stage connectivity to the two FEX units; 100, and 101 for each 5K. Since I will be creating port-channels afterward, the FEX associations will only require 1 link for pinning. (N5K-1 is shown).
N5K-1(config)# fex 100N5K-1(config-fex)# pinning max-links 1Change in Max-links will cause traffic disruption.N5K-1(config-fex)#N5K-1(config-fex)# fex 101N5K-1(config-fex)# pinning max-links 1Change in Max-links will cause traffic disruption.N5K-1(config-fex)#Now I'll create port-channels 78 and 91 (N5K-1 is shown).
N5K-1(config-fex)# interface e1/7-8N5K-1(config-if-range)# channel-group 78N5K-1(config-if-range)# no shutN5K-1(config-if-range)#N5K-1(config-if-range)# interface e1/9-10N5K-1(config-if-range)# channel-group 91N5K-1(config-if-range)# no shutN5K-1(config-if-range)#At this point, I should be able to see that the two FEXes have been discovered (N5K-1 is shown)
N5K-1(config-if-range)# show fex FEX FEX FEX FEX Number Description State Model Serial --------------------------------------------------------------------------- -------- Discovered N2K-C2248TP-1GE SSI141904QR--- -------- Discovered N2K-C2248TP-1GE SSI14250839N5K-1(config-if-range)#Now I'll associate the port-channel interfaces with their respective FEXes (N5K-1 is shown).
N5K-1(config-if-range)# int po78N5K-1(config-if)# switchport mode fex-fabricN5K-1(config-if)# fex associate 100N5K-1(config-if)# N5K-1(config-if)# int po91N5K-1(config-if)# switchport mode fex-fabric N5K-1(config-if)# fex associate 101N5K-1(config-if)# exitN5K-1(config)#From N5K-1 I see the two FEXes online (since I configured this one before N5K-2), but for N5K-2, I see them as "offline". This should change once I complete the vPC configuration.
N5K-1(config-if)# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------100 FEX0100 Online N2K-C2248TP-1GE SSI14250839101 FEX0101 Online N2K-C2248TP-1GE SSI141904QRN5K-1(config-if)# N5K-2# sho fex FEX FEX FEX FEX Number Description State Model Serial --------------------------------------------------------------------------- -------- Offline N2K-C2248TP-1GE SSI14310232--- -------- Offline N2K-C2248TP-1GE SSI14270CLAN5K-2# Since the FEXes are online for N5K-1, I should be able to see all interfaces available on both FEXes.
N5K-1(config-if)# show interface brief
--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------(...output omitted...)
Eth1/7 1 eth fabric up none 10G(D) 78Eth1/8 1 eth fabric up none 10G(D) 78Eth1/9 1 eth fabric up none 10G(D) 91Eth1/10 1 eth fabric up none 10G(D) 91
(...output omitted...)
--------------------------------------------------------------------------------Port-channel VLAN Type Mode Status Reason Speed ProtocolInterface --------------------------------------------------------------------------------Po78 1 eth fabric up none a-10G(D) nonePo91 1 eth fabric up none a-10G(D) none
--------------------------------------------------------------------------------Port VRF Status IP Address Speed MTU--------------------------------------------------------------------------------mgmt0 -- up 10.2.8.53 1000 1500
--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth100/1/1 1 eth access down Administratively down auto(D) --Eth100/1/2 1 eth access down Administratively down auto(D) --Eth100/1/3 1 eth access down Administratively down auto(D) --
(...output omitted...)
Eth100/1/1 1 eth access down Administratively down auto(D) --Eth100/1/2 1 eth access down Administratively down auto(D) --
(...output omitted...)Now I'll establish the vPC configuration. I'll create vPC domain 55, and use the management network for the peer-keepalives. I'll then create a port-channel between the N5Ks, and enable that for vPC peer-linking.
N5K-1(config)# vpc domain 55N5K-1(config-vpc-domain)# peer-keepalive destination 10.2.8.14Note: --------:: Management VRF will be used as the default VRF ::--------N5K-1(config-vpc-domain)# exitN5K-1(config)# N5K-1(config)# int e1/17-18N5K-1(config-if-range)# channel-group 1718 mode activeN5K-1(config-if-range)# interface po1718N5K-1(config-if)# switchport mode trunkN5K-1(config-if)# switchport trunk allow vlan allN5K-1(config-if)# vpc peer-linkPlease note that spanning tree port type is changed to "network" port type on vPC peer-link.This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge Assurance(which is enabled by default) is not disabled.N5K-1(config-if)# Now I'll verify that the vPC is operational.
N5K-1(config-if-range)# sh vpc briefLegend: (*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 55 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status: success Per-vlan consistency status : success Type-2 consistency status : success vPC role : secondary Number of vPCs configured : 0 Peer Gateway : DisabledDual-active excluded VLANs : -Graceful Consistency Check : Enabled
vPC Peer-link status---------------------------------------------------------------------id Port Status Active vlans -- ---- ------ --------------------------------------------------1 Po1718 up 1,930 N5K-1(config-if-range)# exitN5K-1(config)# On both N5Ks, I'll create port-channel 78 and 91, and create vPC 78 and 91.
N5K-1(config-if)# int po78N5K-1(config-if)# vpc 78N5K-1(config-if)# int po91N5K-1(config-if)# vpc 91N5K-1(config-if)# N5K-1(config-if)# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------100 FEX0100 Online N2K-C2248TP-1GE SSI14270CLA101 FEX0101 Online N2K-C2248TP-1GE SSI14310232N5K-1(config-if)# N5K-2(config-if)# N5K-2(config-if)# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------100 FEX0100 Online N2K-C2248TP-1GE SSI14270CLA101 FEX0101 Online N2K-C2248TP-1GE SSI14310232N5K-2(config-if)# At this point, I'll enable the access ports for both FEXes to the server. (N5K-1 is shown).
N5K-1(config)# int e100/1/1-2 , e101/1/1-2N5K-1(config-if-range)# switchport mode accessN5K-1(config-if-range)# switchport access vlan 930N5K-1(config-if-range)# no shutN5K-1(config-if-range)# N5K-1(config-if-range)# show interface brief
(...output omitted...)
--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth100/1/1 930 eth access up none 1000(D) --Eth100/1/2 930 eth access up none 1000(D) --
(...output omitted...)
Eth101/1/1 930 eth access up none 1000(D) --Eth101/1/2 930 eth access up none 1000(D) --On the server, I have an Intel Quad port card which I've teamed together.
I've assigned the address of 100.1.1.7/24.
I'll test connectivity by pinging 10.1.1.21. This is a machine in the core network on VLAN 930.
Note that although the ports are teamed, I've left the adapter in fault tolerance mode. This means that one port will be active, and the rest will remain in standby.
I'll set a continuous ping, and then reload N5K-1.
N5K-1# copy run start[########################################] 100%Copy complete, now saving to disk (please wait)...N5K-1# reloadWARNING: This command will reboot the systemDo you want to continue? (y/n) [n] yNote that a ping packet was lost, but the adapter remained the same. This means that FEX 100 was able to continue providing access to the server through the vPC through N5K-2.
