In this scenario, I have two Cisco Nexus 5548s, two Cisco Nexus 2248 FEXes, and a Windows Server with 4 Ethernet interfaces where two interfaces connect to FEX 100 and the other two connect to FEX 101. I want to configure Virtual Port Channels in such a way that if there is a loss of connectivity from either 5K, the two FEXes will still service clients.
To start, I'll configure the management interface address (10.2.8.13 and .14), and default gateways for both 5Ks (N5K-1 is shown).
N5K-1# conf tN5K-1(config)# interface mgmt0N5K-1(config-if)# ip address 10.2.8.13/24N5K-1(config-if)# exitN5K-1(config)# ip route 0.0.0.0/0 10.2.8.1Note that the management interface is in the management VRF. I will use the management network as the keep-alive management link for the vPCs I'll create.
N5K-1(config)# show vrfVRF-Name VRF-ID State Reason default 1 Up -- management 2 Up -- N5K-1(config)# ping 10.2.8.1 vrf management count 2PING 10.2.8.1 (10.2.8.1): 56 data bytes64 bytes from 10.2.8.1: icmp_seq=0 ttl=254 time=0.969 ms64 bytes from 10.2.8.1: icmp_seq=1 ttl=254 time=0.667 ms
--- 10.2.8.1 ping statistics ---2 packets transmitted, 2 packets received, 0.00% packet lossround-trip min/avg/max = 0.667/0.817/0.969 msN5K-1(config)# Now I'll create uplink trunks to the core network for access to VLAN 930 for both N5Ks (N5K-1 is shown).
N5K-1(config)# interface e1/15-16N5K-1(config-if-range)# no shutN5K-1(config-if-range)# switchport mode trunkN5K-1(config-if-range)# switchport trunk allowed vlan 1,930N5K-1(config-if-range)# vlan 930N5K-1(config-vlan)# sh vlan b
VLAN Name Status Ports---- -------------------------------- --------- -------------------------------1 default active Eth1/1, Eth1/2, Eth1/3, Eth1/4 Eth1/5, Eth1/6, Eth1/7, Eth1/8 Eth1/9, Eth1/10, Eth1/11 Eth1/12, Eth1/13, Eth1/14 Eth1/15, Eth1/16, Eth1/17 Eth1/18, Eth1/19, Eth1/20 Eth1/21, Eth1/22, Eth1/23 Eth1/24, Eth1/25, Eth1/26 Eth1/27, Eth1/28, Eth1/29 Eth1/30, Eth1/31, Eth1/32930 VLAN0930 active Eth1/15, Eth1/16
N5K-1(config-vlan)# show interface trunk
--------------------------------------------------------------------------------Port Native Status Port Vlan Channel--------------------------------------------------------------------------------Eth1/15 1 trunking --Eth1/16 1 trunking --
--------------------------------------------------------------------------------Port Vlans Allowed on Trunk--------------------------------------------------------------------------------Eth1/15 1,930Eth1/16 1,930
(...output omitted...)N5K-1(config)# Each N5K will have two links to each FEX which I will configure as port-channels. I'll enable the features for fex and lacp for both 5Ks (N5K-1 is shown).
N5K-1(config)# show feature | in enabled|fex|lacp|vpcfex 1 disabledlacp 1 disabledlldp 1 enabled sshServer 1 enabled vmfex 1 disabledvpc 1 disabledN5K-1(config)# N5K-1(config)# feature fexN5K-1(config)# feature lacpN5K-1(config)# N5K-1(config)# show feature | in enabledfex 1 enabled lacp 1 enabled lldp 1 enabled sshServer 1 enabled vpc 1 enabledN5K-1(config)# Now I'll stage connectivity to the two FEX units; 100, and 101 for each 5K. Since I will be creating port-channels afterward, the FEX associations will only require 1 link for pinning. (N5K-1 is shown).
N5K-1(config)# fex 100N5K-1(config-fex)# pinning max-links 1Change in Max-links will cause traffic disruption.N5K-1(config-fex)#N5K-1(config-fex)# fex 101N5K-1(config-fex)# pinning max-links 1Change in Max-links will cause traffic disruption.N5K-1(config-fex)#Now I'll create port-channels 78 and 91 (N5K-1 is shown).
N5K-1(config-fex)# interface e1/7-8N5K-1(config-if-range)# channel-group 78N5K-1(config-if-range)# no shutN5K-1(config-if-range)#N5K-1(config-if-range)# interface e1/9-10N5K-1(config-if-range)# channel-group 91N5K-1(config-if-range)# no shutN5K-1(config-if-range)#At this point, I should be able to see that the two FEXes have been discovered (N5K-1 is shown)
N5K-1(config-if-range)# show fex FEX FEX FEX FEX Number Description State Model Serial --------------------------------------------------------------------------- -------- Discovered N2K-C2248TP-1GE SSI141904QR--- -------- Discovered N2K-C2248TP-1GE SSI14250839N5K-1(config-if-range)#Now I'll associate the port-channel interfaces with their respective FEXes (N5K-1 is shown).
N5K-1(config-if-range)# int po78N5K-1(config-if)# switchport mode fex-fabricN5K-1(config-if)# fex associate 100N5K-1(config-if)# N5K-1(config-if)# int po91N5K-1(config-if)# switchport mode fex-fabric N5K-1(config-if)# fex associate 101N5K-1(config-if)# exitN5K-1(config)#From N5K-1 I see the two FEXes online (since I configured this one before N5K-2), but for N5K-2, I see them as "offline". This should change once I complete the vPC configuration.
N5K-1(config-if)# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------100 FEX0100 Online N2K-C2248TP-1GE SSI14250839101 FEX0101 Online N2K-C2248TP-1GE SSI141904QRN5K-1(config-if)# N5K-2# sho fex FEX FEX FEX FEX Number Description State Model Serial --------------------------------------------------------------------------- -------- Offline N2K-C2248TP-1GE SSI14310232--- -------- Offline N2K-C2248TP-1GE SSI14270CLAN5K-2# Since the FEXes are online for N5K-1, I should be able to see all interfaces available on both FEXes.
N5K-1(config-if)# show interface brief
--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------(...output omitted...)
Eth1/7 1 eth fabric up none 10G(D) 78Eth1/8 1 eth fabric up none 10G(D) 78Eth1/9 1 eth fabric up none 10G(D) 91Eth1/10 1 eth fabric up none 10G(D) 91
(...output omitted...)
--------------------------------------------------------------------------------Port-channel VLAN Type Mode Status Reason Speed ProtocolInterface --------------------------------------------------------------------------------Po78 1 eth fabric up none a-10G(D) nonePo91 1 eth fabric up none a-10G(D) none
--------------------------------------------------------------------------------Port VRF Status IP Address Speed MTU--------------------------------------------------------------------------------mgmt0 -- up 10.2.8.53 1000 1500
--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth100/1/1 1 eth access down Administratively down auto(D) --Eth100/1/2 1 eth access down Administratively down auto(D) --Eth100/1/3 1 eth access down Administratively down auto(D) --
(...output omitted...)
Eth100/1/1 1 eth access down Administratively down auto(D) --Eth100/1/2 1 eth access down Administratively down auto(D) --
(...output omitted...)Now I'll establish the vPC configuration. I'll create vPC domain 55, and use the management network for the peer-keepalives. I'll then create a port-channel between the N5Ks, and enable that for vPC peer-linking.
N5K-1(config)# vpc domain 55N5K-1(config-vpc-domain)# peer-keepalive destination 10.2.8.14Note: --------:: Management VRF will be used as the default VRF ::--------N5K-1(config-vpc-domain)# exitN5K-1(config)# N5K-1(config)# int e1/17-18N5K-1(config-if-range)# channel-group 1718 mode activeN5K-1(config-if-range)# interface po1718N5K-1(config-if)# switchport mode trunkN5K-1(config-if)# switchport trunk allow vlan allN5K-1(config-if)# vpc peer-linkPlease note that spanning tree port type is changed to "network" port type on vPC peer-link.This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge Assurance(which is enabled by default) is not disabled.N5K-1(config-if)# Now I'll verify that the vPC is operational.
N5K-1(config-if-range)# sh vpc briefLegend: (*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 55 Peer status : peer adjacency formed ok vPC keep-alive status : peer is alive Configuration consistency status: success Per-vlan consistency status : success Type-2 consistency status : success vPC role : secondary Number of vPCs configured : 0 Peer Gateway : DisabledDual-active excluded VLANs : -Graceful Consistency Check : Enabled
vPC Peer-link status---------------------------------------------------------------------id Port Status Active vlans -- ---- ------ --------------------------------------------------1 Po1718 up 1,930 N5K-1(config-if-range)# exitN5K-1(config)# On both N5Ks, I'll create port-channel 78 and 91, and create vPC 78 and 91.
N5K-1(config-if)# int po78N5K-1(config-if)# vpc 78N5K-1(config-if)# int po91N5K-1(config-if)# vpc 91N5K-1(config-if)# N5K-1(config-if)# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------100 FEX0100 Online N2K-C2248TP-1GE SSI14270CLA101 FEX0101 Online N2K-C2248TP-1GE SSI14310232N5K-1(config-if)# N5K-2(config-if)# N5K-2(config-if)# show fex FEX FEX FEX FEX Number Description State Model Serial ------------------------------------------------------------------------100 FEX0100 Online N2K-C2248TP-1GE SSI14270CLA101 FEX0101 Online N2K-C2248TP-1GE SSI14310232N5K-2(config-if)# At this point, I'll enable the access ports for both FEXes to the server. (N5K-1 is shown).
N5K-1(config)# int e100/1/1-2 , e101/1/1-2N5K-1(config-if-range)# switchport mode accessN5K-1(config-if-range)# switchport access vlan 930N5K-1(config-if-range)# no shutN5K-1(config-if-range)# N5K-1(config-if-range)# show interface brief
(...output omitted...)
--------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed PortInterface Ch #--------------------------------------------------------------------------------Eth100/1/1 930 eth access up none 1000(D) --Eth100/1/2 930 eth access up none 1000(D) --
(...output omitted...)
Eth101/1/1 930 eth access up none 1000(D) --Eth101/1/2 930 eth access up none 1000(D) --On the server, I have an Intel Quad port card which I've teamed together.
I've assigned the address of 100.1.1.7/24.
I'll test connectivity by pinging 10.1.1.21. This is a machine in the core network on VLAN 930.
Note that although the ports are teamed, I've left the adapter in fault tolerance mode. This means that one port will be active, and the rest will remain in standby.
I'll set a continuous ping, and then reload N5K-1.
N5K-1# copy run start[########################################] 100%Copy complete, now saving to disk (please wait)...N5K-1# reloadWARNING: This command will reboot the systemDo you want to continue? (y/n) [n] yNote that a ping packet was lost, but the adapter remained the same. This means that FEX 100 was able to continue providing access to the server through the vPC through N5K-2.
QQ- when the traffic gets distributed on the uplinks from the FEX to Switch1 and Switch2 how does it get to its default gateway?
ReplyDeleteLet's assume we use the 3:1 oversubscribed module in the 5548 wouldn't ~50% of the traffic go to Switch2 and then have to cross over the VPC link between the 5548s to get to Switch1 where the HSRP/VRRP MAC/IP pair of the active default gateway was residing?
This seems like it is sub-optimal and will cause significant traffic loss.
Hi Alan,
DeletevPC configurations are optimized to help ensure that traffic is symmetric. For example, a flow reaching a Nexus switch from the core is forwarded toward the access layer without traversing the peer Nexus switch. Similarly, traffic from the server directed to the core is forwarded so that the Nexus routes this traffic directly to the core without unnecessarily passing it to the peer device. This process occurs regardless of which Cisco Nexus device is the primary HSRP device for a given VLAN.
Here are a couple of links to documentation that elaborates much more on the subjects:
Cisco NX-OS Virtual PortChannel: Fundamental DesignConcepts with NXOS 5.0
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/design_guide_c07-625857.pdf
Cisco Nexus 7000 Series NX-OS Unicast Routing Configuration Guide, Release 5.x - (FHRP - Configuration Guide)
http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/unicast/configuration/guide/l3_glbp.html
I hope this helps!
From the core to access I agree with you. The flow will go from 'the network' to Switch1 and assuming you are running L3 on the N5k and the ARP cache is populated it will direct the traffic to the appropriate host regardless of whether the traffic from the network gees to switch1 or Switch2. (The assumption that the ARP cache is populated can cause a lot of flooding if CAM ages and ARP is still populated btw)
ReplyDeleteHowever I don't think the reverse path is correct - Traffic from your host that goes to FEX1 will then choose to go to either Switch1 or Switch2 based on a hashing algorithm. If you are running GLBP as your config guide indicates GLBP will 'salt and pepper' ARP responses so that some hosts direct their traffic to MAC1 and some MAC2 for the default gateway. (MAC1 is on Switch1 and MAC2 is on Switch2 to keep it simple) Since the EtherChannel hash is not aware of which MAC is active on which N5k in any fashion 50% of the traffic going 'up' will go to Switch1 and 50% will go to Switch2, however 50% of the traffic arriving at Switch1 will be destined for the MAC served on Switch2 and vice versa. This will have to be carried across the VPC spine link between the two N5ks causing ~50% of the traffic going 'up' to cross the VPC link - I think you'd need to plan for 16x10Gb x-linked for this design to work marginally well in production.
Now the ARP table in the N5k is only capable of 8k entries to the best of my understanding, so I would be VERY VERY careful about ever running this config on VMware - as each vNIC takes a MAC/IP pair in the IP hosts table. If these fill up it will be packets everywhere and they are already 3:1 oversubscribed.
Hello again - Alan Harris, I presume? Your name now shows as "Unknown"
DeleteAlan, I agree with your GLBP assessment. However, when I linked the documentation for you, GLBP just happened to be at the top of the FHRP list. In my response, I addressed HSRP, as you indicated in your initial response. GLBP doesn't appear have the same interoperability with vPCs as HSRP/VRRP. At any rate, vPC configurations will ensure traffic does not unnecessarily traverse the peer-link with HSRP/VRRP. In that regard, the reverse path is indeed correct.
Regarding your understanding of the MAC address table, the Cisco Nexus 5548P, which is what I had access to in my lab, provides an address table for 32,000 MAC addresses. That doesn’t mean one wouldn't still be VERY VERY careful about ever running this config on VMware, but then again, aren't we all VERY VERY careful about ever running anything in production? What I mean is; any investment in technology is driven by the return the business expects. Situations vary, and some technologies are better suited for some scenarios, and other technologies are better suited for other scenarios. The business need is going to drive what design I deploy in the real world.
If you read this post or any of the other posts I've done, you'll find that I'm not selling something. I'm documenting the hands on configurations of scenarios I come up with. Why? http://www.stupidroutertricks.com/p/about.html <- My about page :)
That being said, I hope you can agree that vPC is still a viable technology that combines the benefits of hardware redundancy with the benefits of port-channel loop management, and faster failure recovery than STP.
The other thing to think about here is the sheer amount of oversubscription this design introduces and how will you measure and monitor the oversubscription to know if you are creating congestion scenarios that are negatively affecting overall network and application performance.
ReplyDeleteWith the new Intel servers we are deploying all 10Gb to the server now, so I would assume this would be a FEX 2232 in practice. Then those connect to a Nexus 5548 with Integrated L3 and those uplink to a N7k in the core, also running L3.
Thinking through this:
The FEX can be configurable oversubscribed bu best case is 32 ports facing hosts and 8 ports uplinked: 4:1 oversubscribed.
The Nexus 5548 is wirespeed, albeit a bit low on buffers for real world applications, but it has a 3:1 oversubscribed L3 module meaning I can use max 16 uplinks. So another 3:1 oversubscribed here.
The Nexus 7000 has only 80gb per slot on the 10Gb L3 modules that have decent buffering and routing table sizes (M1 modules) so those are either 64 ports wirespeed (meaning only 8 5548s can be wired up, or 4:1 oversubscribed if you use all 32 ports.
4:1 * 3:1 * 4:1 = 48:1 oversubscribed.
So given 48:1 oversubscription how would we determine what is dropping, what is causing drops, what applications are being impacted by congestion? It seems easier to reduce the FEX oversubscription with a real switch, and use a wirespeed box like a Nexus 3064 or Arista 7050 for the mid-tier and eliminate the management headache. You'd still have to deal with oversubscription between boxes based on links, but at least then we'd have the intelligence in the network to know what was being dropped.
To answer your question of how one would determine what is dropping, what's causing drops, what applications are being impacted? Hmm... "I'll take network management for $200, Alex!"
DeleteJoking of course!! :)
I agree that it's difficult to glean that kind information from CAM tables and ASICs, but you'll probably agree that using management tools is the best method to begin determining issues on how network and applications are performing on a network.
Valid as you may be in a particular situation, see my response to your last comment design points.
Thank you for your comments.
This comment has been removed by the author.
ReplyDeleteHello.
ReplyDeleteExcuse my English.
If i change the port vlan on nexus 2000 in N5K-1 also going to have to change the configuration in the N5K-2 too?
Hello Francisco,
DeleteYour English is fine :)
You are correct. The port on the FEX that links to the two N5Ks should be configured the same on each N5K.
I want to have Nexus home lab with VPC capable.
ReplyDelete?
1. can I just use 2x Nexus 5k without FEX
2. any cheaper alternative other than 5k for VPC
3. can 1x 5k doing like VRF, so I don't have to buy 2x 5k
tq
Hello.
ReplyDeleteExcuse my English.
If i change the port vlan on nexus 2000 in N5K-1 also going to have to change the configuration in the N5K-2 too? access router
Configuring Virtual Port Channel (vPC) on Cisco Nexus switches involves creating a mechanism that allows a device to use a Port Channel across two switches, providing redundancy and increasing bandwidth. Here’s a step-by-step guide on how to configure vPC on Cisco Nexus switches:
Deletedomains in computer science
Prerequisites:
Before configuring vPC, ensure the following prerequisites are met:
Nexus Switches: Ensure your Cisco Nexus switches are configured and connected correctly.
Compatibility: Verify that the Nexus switches support vPC and are running a compatible NX-OS version.
Network Topology: Understand your network topology and ensure all physical connections are correctly established.
Configuration Steps:
1. Configure vPC Domain:
Define a unique vPC domain number that will be used across the vPC peers (Nexus switches).
Networking Projects For Final Year
Network Security Projects For Final Year Students
bash
Copy code
config t
vpc domain 10
ReplyDeleteThis is the great post and I hope more different ideas from your post. Really I enjoy to visit your post and keep posting...
Pega Training in Chennai
Pega Course in Chennai
Oracle Training in Chennai
Oracle DBA Training in Chennai
Excel Training in Chennai
Embedded System Course Chennai
Tableau Training in Chennai
Unix Training in Chennai
Power BI Training in Chennai
ReplyDeleteAll are saying the same thing repeatedly, but in your blog I had a chance to get some useful and unique information, I love your writing style very much, I would like to suggest your blog in my dude circle, so keep on updates.
AWS Training in Chennai
AWS Course in Chennai
Big Data Training in Chennai
Web Designing Course in chennai
PHP Training in Chennai
AWS Training in Porur
AWS Training in Tambaram
Great Article
ReplyDeleteNetwork Security Final Year Project Ideas
Project Centers in Chennai
JavaScript Training in Chennai
JavaScript Training in Chennai
Thanks for your insight for your fantastic posting. I’m exhilarated I have taken the time to see this. It is not enough; I will visit your site every day. https://192-168-i-i.com/
ReplyDeleteNice article, its very informative content..thanks for sharing...Waiting for the next update...
ReplyDeleteManual Testing Training in Chennai
Manual Testing course in Chennai
Manual Testing Training institute in Chennai
Manual Testing Training in Velachery
Manual Testing Training in Tambaram
Mobile Testing Training in Chennai
core java training in chennai
DOT NET Training in Chennai
Hibernate Training in Chennai
Html5 Training in Chennai
Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
ReplyDeleteDot Net Training in Chennai | Dot Net Training in anna nagar | Dot Net Training in omr | Dot Net Training in porur | Dot Net Training in tambaram | Dot Net Training in velachery
"This blog is very nice and the author written way was very good with a brief explanation. Well done...!
ReplyDelete.
Digital Marketing Training Course in Chennai | Digital Marketing Training Course in Anna Nagar | Digital Marketing Training Course in OMR | Digital Marketing Training Course in Porur | Digital Marketing Training Course in Tambaram | Digital Marketing Training Course in Velachery
"
Thanks for your informative blog. Your post helped me to understand the future and career prospects. Keep on updating your blog with such awesome blog.
ReplyDeleteDigital Marketing Training Course in Chennai | Digital Marketing Training Course in Anna Nagar | Digital Marketing Training Course in OMR | Digital Marketing Training Course in Porur | Digital Marketing Training Course in Tambaram | Digital Marketing Training Course in Velachery
Your style is very unique compared to other folks I've read stuff from. Many thanks for posting when you have the opportunity, blog Guess I will just book mark this site.
ReplyDeleteIELTS Coaching in chennai
German Classes in Chennai
GRE Coaching Classes in Chennai
TOEFL Coaching in Chennai
spoken english classes in chennai | Communication training
This post is so interactive and informative.keep update more information...
ReplyDeleteDigital Marketing Courses in Bangalore
Digital Marketing Course in Pune
Thanks for sharing this blog. It was so informative.
ReplyDeletePython training institute in chennai
Python institute in chennai
Nice blog! Thanks for sharing this valuable information
ReplyDeleteCyber Security Course in Bangalore
Cyber Security Training in Bangalore
sage 50 wont update
ReplyDeleteprint sage 50 w2 forms
sage error fix runtime error 482 printing error in windows 10
change the fiscal year without adding historical transactions in sage
convert quickbooks online to sage 50
install sage 50 on a shared server
setup recurring entry in sage 50