Cisco NX VLAN to VLAN F5 Bypass

If you recall in my previous blog "Cisco IOS VLAN to VLAN F5 Bypass"  we dealt with a specific scenario between F5 and Cisco IOS.   In this new scenerio we are now dealing with Cisco NX operating system.

 

“I have a pair of F5 ADC in an Internet DMZ, where nodes behind the load balancer need to access NAS system(s) on a VLAN located on a separate VLAN that is not behind the load balancer. The problem is that in my current design I have to route through the F5 Load balancer to access the NAS system(s).  Unfortunately the amount of bandwidth it takes supersedes the F5 ADC’s total throughput.  I would like to bypass this without adding extra network cards or recreating a new VLAN and would like preserve the IP addresses as much as possible.  Also the F5 ADC is sitting on a network design that participates in vPC within Cisco Nexus Data center gear"

Based on this description above you extrapolate a high-level logical network design as shown in Figure 1 ( I have removed vPC design for now as you read on you will see it introduced into the article

Figure 1

 

In the figure 1, we VIP VLAN which is a routable VLAN. Node VLAN is a non-routable VLAN, which is strictly Layer 2.  Since the VLAN is non-routable no external devices except the F5 can access the Nodes directly.  Finally we have Server VLAN Z which is where the NAS system is connected to.  In order to have communication between Server VLAN Z and Node VLAN, the traffic must route through the F5 via VIP VLAN. This is done by a static route pointing to .11 on VIP VLAN which is the F5 floating address on VIP VLAN to reach node VLAN address block. In figure 1 you also have all servers in Node VLAN pointing to .1 as their default gateway which is the floating address of the F5. The F5’s default gateway is .1 on VIP VLAN. Now that we have described the current behavior of Figure 1, we can start looking at making some changes.
So how do we change the network to accommodate the result that is being looked for? It is actually much easier then you might think.
The first item you want to remove is the static route on the switch pointing to point to .11 on VIP VLAN to access NODE VLAN. You will not need this since the end result is to allow SERVER VLAN and NODE VLAN to communicate directly via the Cisco Nexus Switch router.


Next you will need to change NODE VLAN from a non-routable network to a routable network. Thus, NODE VLAN will have a gateway of .1 on the switch router. The F5 will then change its own floating address to say .11 and subsequently change the self-addresses to .12 and .13.  All the servers in NODE VLAN will continue to use .1  as the default gateway.

Figure 2

 

At this point, you are thinking how is the traffic going to return to F5 load balancer when it’s traffic via VIP. The easy way is to apply SNAT Automap. Which works, but then you run into another problem where you lose the client IP address. Normally this might be work, but will make tracking clients more difficult especially around traffic that is not HTTP based.
The short answer to this is utilizing a Cisco’s Policy Based Route. How does that work?
On a Cisco switch router, you can do the following configuration (NX OS Syntax)

ip access-list from_node_vlan_deny

 10 permit ip y.y.y.0/24 z.z.z.0/24

ip access-list from_node_vlan_allow

 10 permit ip y.y.y.0/24 any

route map to_node_vlan deny 10

  match ip address from_node_vlan_deny

route map to_node_vlan permit 10

 match ip address from_node_vlan_allow

 set ip next-hop y.y.y.11

interface VIP_VLAN

 ip policy route-map to_node_vlan

NOTE: You must have feature pbr enabled.

If you are a student of Cisco IOS you might notice that IP access-list does not contain deny statements.  This is because PBR statements in the Nexus OS was designed to ignore the deny statements within IP access-lists.  I haven’t received an official reason of why this happened, but the best case was that they wanted  to make the ultimate PERMIT/DENY decision at the route map level.   The good news is that this new behavior only exists when applied to the pBR. Meaning Deny statements within an IP access-list will not be ignored when applying as a standard ACL for security access.   Also you can use the same access-list for security access and route-maps so just keep in mind that that DENY statements will be ignored by the route-maps ONLY.
Looking at the configuration above, the behavior is that if the NODE VLAN traffic is destined to the SERVER VLAN, skip the route-map statement and use the internal routing table of the switch. This allows NODE VLAN to communicate directly to SERVER VLAN and vice versa. Subsequently, if traffic from NODE VLAN is attempting to reach to the internet then it will match the IP access-list “from_node_vlan_allow” within route map “to_node_vlan permit 10”.  Once that is done it will process the next command, which is sending the traffic to the next hop of y.y.y.11 (Floating address of the F5) within NODE VLAN.

If we left everything alone, this story would be complete.   Unfortunately, the network example I used is also using vPC, which adds another layer of complexity, which needs to be accounted for in our Cisco configuration.   Figure 3 shows us what a vPC topology would look like with an F5 attached.

Figure 3

 

You see F5 along with other vendors decided to optimize the packets by ignoring the arp reply given by the HSRP primary and instead forwards Ethernet frames to whichever MAC address it receives frames from the result is a faster response time.   Unfortunately, this is a nonstandard behavior.   If you are well versed enough on the F5 you would immediately think to turn off the auto Last hop feature would counteract this behavior.  Unfortunately, this does not work in Cisco Nexus OS world.  Cisco recognized that many vendors had this same issue so they introduced the command “peer-gateway” command. This command in affect disabled the optimization.


Therefore, you would introduce the command in the following configuration example, in our diagram it would be on Nexus 7010 MDF A and MDF B

vpc domain 1

  role priority 10

  peer-keepalive destination 10.1.1.2 source 10.1.1.1 vrf VPC-KeepAlive

  peer-gateway

Of course, this is still not end of the story because peer-gateway has a caveat as stated in the Nexus OS Layer 2 guide

Packets arriving at the peer-gateway vPC device will have their TTL decremented, so packets carrying TTL = 1 may be dropped in transit due to TTL expire. This needs to be taken into account when the peer-gateway feature is enabled and particular network protocols sourcing packets with TTL = 1 operate on a vPC VLAN.

This means that the traffic will be treated like a layer 3 hop which means we need to make small adjustment in our access list

From

ip access-list from_node_vlan_deny

 10 permit ip y.y.y.0/24 z.z.z.0/24

ip access-list from_node_vlan_allow

 10 permit ip y.y.y.0/24 any

route map to_node_vlan deny 10

  match ip address from_node_vlan_deny

route map to_node_vlan permit 10

 match ip address from_node_vlan_allow

 set ip next-hop y.y.y.11

interface VIP_VLAN

 ip policy route-map to_node_vlan

to

ip access-list from_node_vlan_deny

5 permit ip y.y.y.0/24 y.y.y.0/24

10 permit ip y.y.y.0/24 z.z.z.0/24

ip access-list from_node_vlan_allow

 10 permit ip y.y.y.0/24 any

route map to_node_vlan deny 10

  match ip address from_node_vlan_deny

route map to_node_vlan permit 10

 match ip address from_node_vlan_allow

 set ip next-hop y.y.y.11

interface VIP_VLAN

 ip policy route-map to_node_vlan

If you have been following closely on the difference you might be wondering, why should you have a permit for traffic between NODE VLAN to NODE VLAN?  After all the access-list looks at Layer 3, not Layer 2 traffic.   As I mentioned above “Packets arriving at the peer-gateway vPC device will have their TTL decremented…”  Which means that Layer 2 traffic under vPC Peer Gateway will treat any traffic within that VLAN as a layer 3 hop and it will be processed within the access-list.


Conclusion
If you are running a F5 ADC which routes through F5 Nexus devices, then you don’t need peer-gateway , but you will if you are directly attached to a Nexus Device that is configured to use vPC.


I have yet to face any issues with this configuration so it might be a good idea to add Peer-gateway into your vPC configuration as a default.

Cisco IOS VLAN to VLAN F5 Bypass

From time to time, I usually receive a request that goes something like this.

“I have a pair of F5 ADC in an Internet DMZ, where the servers behind the load balancer need to access NAS system(s) on a VLAN located in the same network on another VLAN that is not behind the load balancer.
The problem is that in my current design I have to route through the F5 Load balancer to access the NAS system(s).  Unfortunately the amount of bandwidth it takes supersedes the F5 ADC’s total throughput.  I would like to by pass this without adding extra network cards or recreating a new VLAN and would like preserve the IP addresses as much as possible.”




For the purposes of the blog we will call the person requesting this Keyser Söze.   Yes I have to call him Keyser.




Based on this description above you can extrapolate a high-level logical network design as shown in Figure 1.


Figure 1

In the figure 1, we have the following VLANS

• VLAN 10 - Which is for VIPs (Virtual IP addresses).  The VLAN is routable via Cisco HSRP.   Gateway is .1.  The F5's floating address will be .11.  The floating address is used as the NEXT HOP to get to VLAN 11.
• VLAN 11 - Where the Servers that will be load balanced -  is a non Cisco routable VLAN.  From Cisco's perspective it's simply a VLAN without HSRP.   The gateway for this segment is on the F5 which is .1
• VLAN 12 - This is a routable VLAN similar to VLAN 10, except there are physical servers on this segment.  In our case this is where the NAS will sit.  The gateway is .1 for the NAS.

Here are some facts we know about this design

• VLAN 12 is accessible by any part of this network
• VLAN 10 has a floating IP address which is the shared address between F5.  It is used in scenario to be the gateway to the segment in VLAN 11
• In order for a Web server in VLAN 11 to access the NAS server in VLAN12 and vice versa, the F5 is the router.

There is nothing entirely wrong with this done and it most cases this works.  However, Keyser is worried that multiple web servers in VLAN 11 will request so much content from the NAS server in VLAN 12 that there is a throughput concern.


So how do we change the network to accommodate the result that Kyser is looking for?


It is actually much easier then you might think.


For the purposes of this explanation, let us assume the switches are connected on Cisco Switch routers


The first item you want to remove is the the static route on the switch pointing to point to .11 on VLAN10 to access VLAN12. You will not need this since the end result is to allow VLAN 12 and VLAN 11 to communicate directly via the Cisco Switch router.
Next you will need to change VLAN11 from a non-routable network to a routable network. Thus, VLAN 11 will have a gateway of .1 on the switch router. The F5 will then change its own floating address to say .11 and subsequently change the self-addresses. All the servers will continue to use .1 on VLAN11 as their default gateway.


Thus the network will now look more like Figure 2
Figure 2

At this point, you are thinking well if that is the case then how do we get traffic back to the F5 for Load balancing traffic. Well the easy way is to apply SNAT Automap across all the Virtual addresses. Which works, but then you run into another problem where you lose the client IP address. Normally this might be work, BUT if you are tracking clients for statistical purposes, this is not going to work.
The short answer to this is utilizing a Cisco’s Policy Based Route. How does that work?
On a Cisco switch you can do the following configuration (IOS Syntax):


ip access-list extended from_vlan11
Deny y.y.y.0 0.0.0.255 z.z.z.0 0.0.0.255
Permit y.y.y.0 0.0.0.255 any
route map to_lb_vlan11
Match ip address from_vlan11
ip default next-hop y.y.y.11
interface Vlan11
ip policy route-map to_lb_vlan11



What these statements mean is that any traffic from VLAN11 is destined to addresses on VLAN12, skip the route-map statement and use the internal routing table of the switch. Thus allowing VLAN11 to communicate directly to VLAN12 and vice-versa.


Subsequently, if traffic from VLAN11 is attempting to talk to the internet then it will match the permit statement in the IP access list “from_vlan11” then apply the route map statement and thus your next hope is .11, which is hosted on VLAN11.
That pretty much sums up how to use the switches throughput for VLAN-to-VLAN traffic and the F5 ADC continues to do what it does best while Kyser can go home happy.