Cisco NX VLAN to VLAN F5 Bypass

If you recall in my previous blog "Cisco IOS VLAN to VLAN F5 Bypass"  we dealt with a specific scenario between F5 and Cisco IOS.   In this new scenerio we are now dealing with Cisco NX operating system.

 

“I have a pair of F5 ADC in an Internet DMZ, where nodes behind the load balancer need to access NAS system(s) on a VLAN located on a separate VLAN that is not behind the load balancer. The problem is that in my current design I have to route through the F5 Load balancer to access the NAS system(s).  Unfortunately the amount of bandwidth it takes supersedes the F5 ADC’s total throughput.  I would like to bypass this without adding extra network cards or recreating a new VLAN and would like preserve the IP addresses as much as possible.  Also the F5 ADC is sitting on a network design that participates in vPC within Cisco Nexus Data center gear"

Based on this description above you extrapolate a high-level logical network design as shown in Figure 1 ( I have removed vPC design for now as you read on you will see it introduced into the article

Figure 1

 

In the figure 1, we VIP VLAN which is a routable VLAN. Node VLAN is a non-routable VLAN, which is strictly Layer 2.  Since the VLAN is non-routable no external devices except the F5 can access the Nodes directly.  Finally we have Server VLAN Z which is where the NAS system is connected to.  In order to have communication between Server VLAN Z and Node VLAN, the traffic must route through the F5 via VIP VLAN. This is done by a static route pointing to .11 on VIP VLAN which is the F5 floating address on VIP VLAN to reach node VLAN address block. In figure 1 you also have all servers in Node VLAN pointing to .1 as their default gateway which is the floating address of the F5. The F5’s default gateway is .1 on VIP VLAN. Now that we have described the current behavior of Figure 1, we can start looking at making some changes.
So how do we change the network to accommodate the result that is being looked for? It is actually much easier then you might think.
The first item you want to remove is the static route on the switch pointing to point to .11 on VIP VLAN to access NODE VLAN. You will not need this since the end result is to allow SERVER VLAN and NODE VLAN to communicate directly via the Cisco Nexus Switch router.


Next you will need to change NODE VLAN from a non-routable network to a routable network. Thus, NODE VLAN will have a gateway of .1 on the switch router. The F5 will then change its own floating address to say .11 and subsequently change the self-addresses to .12 and .13.  All the servers in NODE VLAN will continue to use .1  as the default gateway.

Figure 2

 

At this point, you are thinking how is the traffic going to return to F5 load balancer when it’s traffic via VIP. The easy way is to apply SNAT Automap. Which works, but then you run into another problem where you lose the client IP address. Normally this might be work, but will make tracking clients more difficult especially around traffic that is not HTTP based.
The short answer to this is utilizing a Cisco’s Policy Based Route. How does that work?
On a Cisco switch router, you can do the following configuration (NX OS Syntax)

ip access-list from_node_vlan_deny

 10 permit ip y.y.y.0/24 z.z.z.0/24

ip access-list from_node_vlan_allow

 10 permit ip y.y.y.0/24 any

route map to_node_vlan deny 10

  match ip address from_node_vlan_deny

route map to_node_vlan permit 10

 match ip address from_node_vlan_allow

 set ip next-hop y.y.y.11

interface VIP_VLAN

 ip policy route-map to_node_vlan

NOTE: You must have feature pbr enabled.

If you are a student of Cisco IOS you might notice that IP access-list does not contain deny statements.  This is because PBR statements in the Nexus OS was designed to ignore the deny statements within IP access-lists.  I haven’t received an official reason of why this happened, but the best case was that they wanted  to make the ultimate PERMIT/DENY decision at the route map level.   The good news is that this new behavior only exists when applied to the pBR. Meaning Deny statements within an IP access-list will not be ignored when applying as a standard ACL for security access.   Also you can use the same access-list for security access and route-maps so just keep in mind that that DENY statements will be ignored by the route-maps ONLY.
Looking at the configuration above, the behavior is that if the NODE VLAN traffic is destined to the SERVER VLAN, skip the route-map statement and use the internal routing table of the switch. This allows NODE VLAN to communicate directly to SERVER VLAN and vice versa. Subsequently, if traffic from NODE VLAN is attempting to reach to the internet then it will match the IP access-list “from_node_vlan_allow” within route map “to_node_vlan permit 10”.  Once that is done it will process the next command, which is sending the traffic to the next hop of y.y.y.11 (Floating address of the F5) within NODE VLAN.

If we left everything alone, this story would be complete.   Unfortunately, the network example I used is also using vPC, which adds another layer of complexity, which needs to be accounted for in our Cisco configuration.   Figure 3 shows us what a vPC topology would look like with an F5 attached.

Figure 3

 

You see F5 along with other vendors decided to optimize the packets by ignoring the arp reply given by the HSRP primary and instead forwards Ethernet frames to whichever MAC address it receives frames from the result is a faster response time.   Unfortunately, this is a nonstandard behavior.   If you are well versed enough on the F5 you would immediately think to turn off the auto Last hop feature would counteract this behavior.  Unfortunately, this does not work in Cisco Nexus OS world.  Cisco recognized that many vendors had this same issue so they introduced the command “peer-gateway” command. This command in affect disabled the optimization.


Therefore, you would introduce the command in the following configuration example, in our diagram it would be on Nexus 7010 MDF A and MDF B

vpc domain 1

  role priority 10

  peer-keepalive destination 10.1.1.2 source 10.1.1.1 vrf VPC-KeepAlive

  peer-gateway

Of course, this is still not end of the story because peer-gateway has a caveat as stated in the Nexus OS Layer 2 guide

Packets arriving at the peer-gateway vPC device will have their TTL decremented, so packets carrying TTL = 1 may be dropped in transit due to TTL expire. This needs to be taken into account when the peer-gateway feature is enabled and particular network protocols sourcing packets with TTL = 1 operate on a vPC VLAN.

This means that the traffic will be treated like a layer 3 hop which means we need to make small adjustment in our access list

From

ip access-list from_node_vlan_deny

 10 permit ip y.y.y.0/24 z.z.z.0/24

ip access-list from_node_vlan_allow

 10 permit ip y.y.y.0/24 any

route map to_node_vlan deny 10

  match ip address from_node_vlan_deny

route map to_node_vlan permit 10

 match ip address from_node_vlan_allow

 set ip next-hop y.y.y.11

interface VIP_VLAN

 ip policy route-map to_node_vlan

to

ip access-list from_node_vlan_deny

5 permit ip y.y.y.0/24 y.y.y.0/24

10 permit ip y.y.y.0/24 z.z.z.0/24

ip access-list from_node_vlan_allow

 10 permit ip y.y.y.0/24 any

route map to_node_vlan deny 10

  match ip address from_node_vlan_deny

route map to_node_vlan permit 10

 match ip address from_node_vlan_allow

 set ip next-hop y.y.y.11

interface VIP_VLAN

 ip policy route-map to_node_vlan

If you have been following closely on the difference you might be wondering, why should you have a permit for traffic between NODE VLAN to NODE VLAN?  After all the access-list looks at Layer 3, not Layer 2 traffic.   As I mentioned above “Packets arriving at the peer-gateway vPC device will have their TTL decremented…”  Which means that Layer 2 traffic under vPC Peer Gateway will treat any traffic within that VLAN as a layer 3 hop and it will be processed within the access-list.


Conclusion
If you are running a F5 ADC which routes through F5 Nexus devices, then you don’t need peer-gateway , but you will if you are directly attached to a Nexus Device that is configured to use vPC.


I have yet to face any issues with this configuration so it might be a good idea to add Peer-gateway into your vPC configuration as a default.