Introduction
The purpose of this lab is to provide a potential solution for a problem that occurs when operating outside of ideal solutions. While it is nice to lab up an environment that represents having privately owned dark fiber, low latency connectivity all over the place, this is often not reality.Today's lab will build off the previous OSPF lab topology, but will be adding some interesting twists, so to speak. Referring to the topology below, for the sake of an interesting lab, we said the Fast Ethernet link between Branch1 (remote site) and Wan1 (head-end edge WAN router) was really a transparent point-to-point wireless bridge. The serial link between Branch1 and Wan2, to add some variety, is a privately leased T1 point-to-point circuit.
A problem appears...
With the network in its final configuration from the previous lab, users at Branch1 are complaining of quality problems with the Voice-over-IP system. However, all their web applications and file transfers seem to be performing within expectations. After digging in, it is apparent that there is some packet loss over the wireless point-to-point bridge, and the UDP-based real-time VoIP system is being impacted. Luckily, TCP is doing a decent enough job at 'hiding' the problem in the other applications.After looking at bandwidth counters, we can observe that during the peak business hours, the 100 Mbit link between Branch1 and Wan1 is operating with its 95th percentile around 40 Mbps. With that much bandwidth, using the T1 "backup" link as a primary path is not viable; the link would be far over-saturated.
In review, there's too much traffic to use the T1 as the primary routed link, and packet loss on the 100 Mbit link is causing issues with VoIP. Since we can't run all-or-nothing on either link, what if we can cherry-pick the VoIP traffic to send over the T1, and leave the rest of the traffic across the wireless bridge? To accomplish this, we can attempt to use Policy-Based Routing (PBR).
Alright! So we'll start by reviewing our relevant base configurations in the lab.
Base Lab Configuration
Branch1:
| ! interface Loopback0 ip address 172.16.255.3 255.255.255.255 ! interface FastEthernet1/0 description To-WAN1 ip address 172.16.11.1 255.255.255.254 ip ospf network point-to-point duplex auto speed auto ! interface Serial2/0 description To-WAN2 ip address 172.16.11.3 255.255.255.254 ip ospf network point-to-point serial restart-delay 0 ! interface GigabitEthernet3/0 description Branch-LAN ip address 172.16.10.1 255.255.255.0 negotiation auto ! router ospf 1 router-id 172.16.255.3 log-adjacency-changes auto-cost reference-bandwidth 100000 area 0.0.0.5 stub network 172.16.10.0 0.0.0.255 area 0.0.0.5 network 172.16.11.0 0.0.0.1 area 0.0.0.5 network 172.16.11.2 0.0.0.1 area 0.0.0.5 network 172.16.255.3 0.0.0.0 area 0.0.0.5 ! |
Wan1:
| ! interface Loopback0 ip address 172.16.255.1 255.255.255.255 ! interface Loopback1 description Fake branch ip address 172.16.12.1 255.255.255.0 ! interface GigabitEthernet1/0 description To-Core1 ip address 10.0.11.1 255.255.255.254 ip ospf network point-to-point negotiation auto ! interface FastEthernet2/0 description To-Branch1 ip address 172.16.11.0 255.255.255.254 ip ospf network point-to-point duplex auto speed auto ! interface GigabitEthernet3/0 no ip address negotiation auto ! interface GigabitEthernet3/0.1 encapsulation dot1Q 1 native ip address 10.0.11.6 255.255.255.254 ip ospf network point-to-point ! interface GigabitEthernet3/0.5 encapsulation dot1Q 5 ip address 172.16.11.4 255.255.255.254 ip ospf network point-to-point ! router ospf 1 router-id 172.16.255.1 log-adjacency-changes auto-cost reference-bandwidth 100000 area 0.0.0.5 stub no-summary area 0.0.0.5 range 172.16.0.0 255.255.240.0 network 10.0.11.0 0.0.0.1 area 0.0.0.0 network 10.0.11.6 0.0.0.1 area 0.0.0.0 network 172.16.11.0 0.0.0.1 area 0.0.0.5 network 172.16.11.4 0.0.0.1 area 0.0.0.5 network 172.16.12.0 0.0.0.255 area 0.0.0.5 network 172.16.255.1 0.0.0.0 area 0.0.0.5 ! |
Wan2:
| ! interface Loopback0 ip address 172.16.255.2 255.255.255.255 ! interface Loopback1 description Fake branch2 ip address 172.16.14.1 255.255.255.0 ! interface GigabitEthernet1/0 description To-Core2 ip address 10.0.11.3 255.255.255.254 ip ospf network point-to-point negotiation auto ! interface Serial2/0 description To-Branch1 ip address 172.16.11.2 255.255.255.254 ip ospf network point-to-point serial restart-delay 0 ! interface GigabitEthernet3/0 no ip address negotiation auto ! interface GigabitEthernet3/0.1 encapsulation dot1Q 1 native ip address 10.0.11.7 255.255.255.254 ip ospf network point-to-point ! interface GigabitEthernet3/0.5 encapsulation dot1Q 5 ip address 172.16.11.5 255.255.255.254 ip ospf network point-to-point ! router ospf 1 router-id 172.16.255.2 log-adjacency-changes auto-cost reference-bandwidth 100000 area 0.0.0.5 stub no-summary area 0.0.0.5 range 172.16.0.0 255.255.240.0 network 10.0.11.2 0.0.0.1 area 0.0.0.0 network 10.0.11.6 0.0.0.1 area 0.0.0.0 network 172.16.11.2 0.0.0.1 area 0.0.0.5 network 172.16.11.4 0.0.0.1 area 0.0.0.5 network 172.16.14.0 0.0.0.255 area 0.0.0.5 network 172.16.255.2 0.0.0.0 area 0.0.0.5 ! |
Core1:
| ! interface Loopback0 ip address 10.0.255.1 255.255.255.255 ! interface GigabitEthernet1/0 description To-Core2 ip address 10.0.11.4 255.255.255.254 ip ospf network point-to-point negotiation auto ! interface GigabitEthernet2/0 description To-Wan1 ip address 10.0.11.0 255.255.255.254 ip ospf network point-to-point negotiation auto ! router ospf 1 router-id 10.0.255.1 log-adjacency-changes auto-cost reference-bandwidth 100000 network 10.0.11.0 0.0.0.1 area 0.0.0.0 network 10.0.11.4 0.0.0.1 area 0.0.0.0 network 10.0.255.1 0.0.0.0 area 0.0.0.0 ! |
Core2:
| ! interface Loopback0 ip address 10.0.255.2 255.255.255.255 ! interface GigabitEthernet1/0 description To-Core1 ip address 10.0.11.5 255.255.255.254 ip ospf network point-to-point negotiation auto ! interface GigabitEthernet2/0 description To-Wan2 ip address 10.0.11.2 255.255.255.254 ip ospf network point-to-point negotiation auto ! router ospf 1 router-id 10.0.255.2 log-adjacency-changes auto-cost reference-bandwidth 100000 network 10.0.11.2 0.0.0.1 area 0.0.0.0 network 10.0.11.4 0.0.0.1 area 0.0.0.0 network 10.0.255.2 0.0.0.0 area 0.0.0.0 ! |
Implementing Policy-based Routing
For starters, I want to make sure that this blog post is in no way, shape, or form making it sound like I am giving a glowing endorsement of PBR (neither the routing technique nor the beer). Frankly, it is hard to support and manage, and it has a tendency to get out of control quickly. Whenever possible, I try to avoid using it. However, there are times when PBR is among a short list of viable solutions. In these cases, I believe it is more important to deploy it in such a way that it can be managed easier, even if this means that traffic is flowing across a non-optimal route.In order to successfully implement PBR, it is important to remember the basics about how PBR works. In order to influence the routed path beyond what the implemented routing protocols are doing, PBR allows intervention by effectively skipping the routing table lookup. The first step is to define a route-map, which contains two parts: a 'match' clause and a 'set' clause. The match clause defines which traffic will be routed according to PBR, and the set clause defines how the matched traffic should be routed. As PBR needs to 'intercept' the normal routing decision, it is applied to interfaces in the inbound direction. In other words, in order to policy-route traffic, the policy has to be applied to the interface on which the traffic is entering the said router. The PBR-selected outbound interface does not require the policy to be applied.
If we want to policy-route the VoIP traffic to use the T1 line, we need to determine at which points we need to manually intervene in the routed path. We can start by picking a direction and working out the details, and then once we are satisfied, we can move on to the opposite direction. I am going to recommend starting with policy-routing traffic sourcing from Branch1, as it is the simpler of the two. First, we will create the route-map that will be used to define the "policy" of our PBR, using an access-list as its criteria. Then, we will apply it to the inbound LAN interface to match appropriate traffic sourcing from the Branch1 local network.
| ! ip access-list extended PBR-VoIP-to-T1-ACL permit udp 172.16.10.0 0.0.0.255 10.0.11.0 0.0.0.255 range 16384 32768 ! route-map PBR-VoIP-to-T1 permit 10 match ip address PBR-VoIP-to-T1-ACL set ip next-hop 172.16.11.2 ! interface GigabitEthernet3/0 ip policy route-map PBR-VoIP-to-T1 ! end Branch1#show route-map route-map PBR-VoIP-to-T1, permit, sequence 10 Match clauses: ip address (access-lists): PBR-VoIP-to-T1-ACL Set clauses: ip next-hop 172.16.11.2 Policy routing matches: 0 packets, 0 bytes Branch1# |
Note that the output of 'show route-map' includes a line that shows counters for PBR matches. One of the problems with PBR is lack of transparency, as 'show ip route' is no longer 100% accurate. So, those counters in 'show route-map' - as well as 'show ip policy' to point out which interfaces have which route-maps associated with them - are pretty much the way to see what's happening. Now that we have the policy applied, we can test by injecting traffic in with our test PC off Branch1, and make sure it routes as expected.
| root@bt:~# hping3 10.0.11.10 --udp -p 16386 HPING 10.0.11.10 (tap0 10.0.11.10): udp mode set, 28 headers + 0 data bytes ICMP Host Unreachable from ip=172.16.11.2 name=UNKNOWN ICMP Host Unreachable from ip=172.16.11.2 name=UNKNOWN ICMP Host Unreachable from ip=172.16.11.2 name=UNKNOWN ^C --- 10.0.0.10 hping statistic --- 3 packets tramitted, 3 packets received, 0% packet loss round-trip min/avg/max = 0.0/0.0/0.0 ms |
Now that we've sent three UDP packets into the test network, which match the route-map's ACL, let's see if our PBR worked:
| Branch1#show ip policy Interface Route map Gi3/0 PBR-VoIP-to-T1 Branch1#show route-map route-map PBR-VoIP-to-T1, permit, sequence 10 Match clauses: ip address (access-lists): PBR-VoIP-to-T1-ACL Set clauses: ip next-hop 172.16.11.2 Policy routing matches: 3 packets, 126 bytes Branch1# |
Success! For kicks, I also injected TCP traffic to the same destination port, and confirmed that the packet count in 'show route-map' did not increase. So, we can safely say that VoIP traffic from the branch office will use the T1, while the TCP applications will go forth using the Fast Ethernet link.
Now that we've tackled the branch office, we can take on the head-end. This one takes a bit more planning as there are more routers and paths involved. With Branch1, all ingress traffic towards the WAN was sourced from one interface, and the outbound connections to the WAN were both directly connected. At the head-end side, there are two different WAN routers, so it's not as clear-cut.
This is probably a great time to point out something that should already be obvious -- even though the methods used to achieve the goals in this lab will work -- it does not mean it is the only way, nor possibly even the best way. It's just one way, and my mindset behind PBR is to limit its deployment to the smallest number of devices possible. This may lead to inefficient routing, but as long as it provides the application-specific routing that cannot be achieved otherwise, that's good enough. With all that being said, when considering the routing from the core to branch1, the traffic is guaranteed to hit either Wan1 or Wan2. Because of this, we'll configure PBR on Wan1 and Wan2 exclusively.
Wan1:
| ip access-list extended PBR-VoIP-to-T1-ACL permit udp 10.0.11.0 0.0.0.255 172.16.10.0 0.0.0.255 range 16384 32768 ! route-map PBR-VoIP-to-T1 permit 10 match ip address PBR-VoIP-to-T1-ACL set ip next-hop 10.0.11.7 ! interface GigabitEthernet1/0 description To-Core1 ip policy route-map PBR-VoIP-to-T1 ! |
Wan2:
| ip access-list extended PBR-VoIP-to-T1-ACL permit udp 10.0.11.0 0.0.0.255 172.16.10.0 0.0.0.255 range 16384 32768 ! route-map PBR-VoIP-to-T1 permit 10 match ip address PBR-VoIP-to-T1-ACL set ip next-hop 172.16.11.3 ! interface GigabitEthernet1/0 description To-Core2 ip policy route-map PBR-VoIP-to-T1 ! interface GigabitEthernet3/0.1 description To-Wan2-OSPF-0 ip policy route-map PBR-VoIP-to-T1 ! interface GigabitEthernet3/0.5 description To-Wan2-OSPF-5 ip policy route-map PBR-VoIP-to-T1 ! |
With this configuration of Wan1 and Wan2, policy-based routing has been configured to take traffic coming into either Wan router from the Core, and ensure it is sent out Wan2's T1 interface. To keep the amount of PBR involved to a minimum, we are allowing the inefficient route of traffic routing from Core1 to Wan1, then across to Wan2 so it can leave the T1. The following picture depicts a visual representation of where PBR is being used:
Note that however traffic enters Wan2, it will be forwarded out the T1 interface due to PBR. Now, PBR only works if the next-hop interface is in up/up state. Otherwise PBR is ignored and the routing table is followed. Why is this important? If the T1 is down, Wan2 will theoretically follow OSPF telling it to send traffic destined to Branch1 via its point-to-point interface with Wan1. It is because of this that we do not want to put an ingress PBR policy on Wan1's ingress interface from Wan2. If we did, we would risk a routing loop.
I will also take a moment to point out that this configuration, as it stands, is not really doing much to protect against potential black-hole scenarios. Part 2 of this post will spend more time in making PBR a bit more resilient and also giving a bit more visibility for troubleshooting.
As it stands, we still need to test the policy as it was implemented. What we really want is a test PC hanging off the "Core" network, sending traffic destined to the Branch1 subnet. Since we don't have that, we can 'hack' it by forcing the Core1 and Core2 routers to send traffic that matches our VoIP policy. We can add the following configuration to Core1 and Core2, and then test our PBR by generating syslogs. Is this considered cheating? I do believe so. :)
| ! ------------ Core1 ----------- logging source-interface GigabitEthernet1/0 logging host 172.16.10.10 transport udp port 16390 ! ! ! ------------ Core2 ----------- logging source-interface GigabitEthernet1/0 logging host 172.16.10.10 transport udp port 16392 ! ! ------------ Test Syslog from Core2 ----------- ! ! Core2#config t Enter configuration commands, one per line. End with CNTL/Z. Core2(config)#^Z Core2# *Jun 14 15:50:07.995: %SYS-5-CONFIG_I: Configured from console by console Core2#! ! ------------ Check Wan2 Routemap Counters ----------- ! ! Wan2#show route-map route-map PBR-VoIP-to-T1, permit, sequence 10 Match clauses: ip address (access-lists): PBR-VoIP-to-T1-ACL Set clauses: ip next-hop 172.16.11.3 Policy routing matches: 1 packets, 124 bytes Wan2# ! ! ------------ Test Syslog from Core1 ----------- ! ! Core1#config t Enter configuration commands, one per line. End with CNTL/Z. Core1(config)#^Z Core1# *Jun 14 15:52:56.763: %SYS-5-CONFIG_I: Configured from console by console Core1# ! ! ------------ Check Wan1 Routemap Counters ----------- ! ! Wan1#show route-map route-map PBR-VoIP-to-T1, permit, sequence 10 Match clauses: ip address (access-lists): PBR-VoIP-to-T1-ACL Set clauses: ip next-hop 10.0.11.7 Policy routing matches: 1 packets, 124 bytes Wan1# ! ! ------------ Check Wan2 Routemap Counters ----------- ! ! Wan2#show route-map route-map PBR-VoIP-to-T1, permit, sequence 10 Match clauses: ip address (access-lists): PBR-VoIP-to-T1-ACL Set clauses: ip next-hop 172.16.11.3 Policy routing matches: 2 packets, 248 bytes Wan2# |
As evidenced here, PBR is working as expected. We generated syslogs specially crafted to mimic a VoIP packet based on what our ACL considered VoIP, so that we could test PBR on each Wan router.
UDP "VoIP" traffic from Core2 was sent to Wan2 (following OSPF/routing table), where PBR intercepted it and sent it out the T1 link.
Furthermore, the syslog generated by Core1 was sent to Wan1 (following OSPF/routing table), where PBR intercepted it and instead of sending it directly to Branch1, it sent the "VoIP" traffic over to Wan2. Since Wan2 has an inbound PBR policy on that interface, it again intercepted that traffic and changed the next-hop to send it down the T1 towards Branch1.
Conclusion
With the configuration as it stands, PBR is allowing bidirectional VoIP traffic between the head-end and the branch office to flow across the T1, while all other traffic is going through the FastEthernet interface. This meets our lab criteria of allowing the bulk traffic to use the high-bandwidth point-to-point wireless link, since TCP is mitigating the packet loss. It also allows the VoIP traffic that had been impacted by the packet loss to use the lower bandwidth link that runs without error.Next time we will take this configuration further by making PBR a bit more dynamic with respect to its decision-making, which will allow for better failure handling. We will also look at ways to improve visibility of traffic with PBR, since the ip routing table is not used to decide how traffic is routed.
No comments:
Post a Comment