Thursday, July 25, 2013

Determining exact route when using ECMP

Many enterprise Cisco networks take advantage of multiple, redundant routed paths between routers or layer-3 switches. Certain routing protocols make it easy to not only achieve redundancy, but also higher throughput by utilizing both links via equal cost multipath (ECMP). This means that the routing table contains multiple entries, with same administrative distance and same metric, for the same network. The router then balances traffic between the two links using CEF (Cisco Express Forwarding). OSPF, paired with a symmetrical design, lends itself well to ECMP links, whereas EIGRP may require variance to be configured in order to allow for ECMP.

This works great, but because of this, the normal troubleshooting commands to find the deterministic path of a packet are somewhat ambiguous. Traditionally in a non-ECMP environment, even if there are multiple routes to a destination, the best route wins and is inserted into the routing table. Therefore, a simple "show ip route" can give you the path over which a particular flow is leaving the router. However, when there are more than one path in the routing table for a particular network, "show ip route" will show both. If you need to know for sure which link is being used, there is a way to do this.

In IOS, the way to determine the exact route a particular flow is taking, is to utilize the command "show ip cef exact-route". This command will take a source IP, destination IP, and give the outgoing interface from which the packet will leave. Below is an example:

TestLab-6500#show ip route 172.16.0.5
Routing entry for 172.16.0.5/32
  Known via "ospf 1", distance 110, metric 21, type intra area
  Last update from 10.0.35.74 on TenGigabitEthernet4/1, 11:16:48 ago
  Routing Descriptor Blocks:
  * 10.0.35.74, from 172.16.0.5, 11:16:48 ago, via TenGigabitEthernet1/1
      Route metric is 21, traffic share count is 1
    10.0.35.72, from 172.16.0.5, 11:16:48 ago, via TenGigabitEthernet2/1
      Route metric is 21, traffic share count is 1

TestLab-6500#
TestLab-6500#show ip cef exact-route 10.0.0.10 172.16.0.5
10.0.0.10 -> 172.16.0.5 => IP adj out of TenGigabitEthernet1/1, addr 10.0.35.74
TestLab-6500#


For those of you who have found the Nexus line to be fitting for L3 switching, the same basic functionality is there in NX-OS. The command syntax has changed slightly -- "show routing hash":

N7K-R1# show ip route 172.16.0.5
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]

172.16.0.5/32, ubest/mbest: 2/0
    *via 10.0.4.121, Eth2/1, [110/31], 5d05h, ospf-1, intra
    *via 10.0.8.121, Eth1/1, [110/31], 15w4d, ospf-1, intra
N7K-R1#
N7K-R1#
N7K-R1# show routing hash 10.0.0.10 172.16.0.5
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Universal-id seed: 0x9617c10
Hash for VRF "default" resulting hash: 0x01 path '>'

172.16.0.5/32 unicast forwarding path(s) 2
  *via 10.0.4.121%Ethernet2/1
> *via 10.0.8.121%Ethernet1/1

Route:
N7K-R1#


While most of the time, it's acceptable to just know that it is taking one of several redundant paths, sometimes it is necessary to know the specific path a packet is taking. Hopefully this will serve as a quick reference during such events.

Tuesday, July 9, 2013

Linux software for networking geeks

Sometimes I feel that without the ability to run Linux, my work life would be much harder. Not that I use Linux 100% of the time or anything, but there are just certain things that are simply much more efficient in Linux.

I may be biased, but I believe a number of tools stand out when it comes to network troubleshooting. While I am sure some of these are available in Windows as well, I know they're available for Linux. Heck, some of what I use most are straight up bash shell or common command-line tools. The following list is in no particular order and is definitely not all-inclusive. It's just the list of tools that are top of mind for me right now.

Also, I'm hoping that this can serve simply as an introduction/primer to these different tools. If these sound interesting to you, there are plenty of great resources online that dive deep into each of these tools, or you can take some time to experiment and play with them. Furthermore, I'd definitely appreciate others sharing their favorite network-centric Linux software in the comments -- I love to learn what else is out there!

GNU Screen (or tmux, for those screen haters)

There are very few programs that have made as big of an impact on my workflow as screen. Simply put, it allows you to run multiple persistent bash shells that can be attached and detached at will. This means that I can SSH to a Linux server from my desktop, and once connected, start screen. I can create several windows that I can switch between, without having to open a bunch of different terminal windows (or putty windows, etc).

When I get distracted from the task at hand, I feel like it is hard to get back into the same mindset. Screen does a great job of jump-starting you back into your work, as you can easily pick up right from where you left off. Let's say I'm in the midst of building a configuration, so I've got vim open in one window and I'm ssh'd to a lab router from another. I could detach from my screen session, close my SSH connection, go to a meeting, sit down at another PC, SSH back in and re-attach to screen. My vim would still be open in one window and I'd still have my router connection in the other (assuming I don't set timeouts on my lab gear).

There are other great possibilities that come from having persistent, always running sessions available via screen. Anyone still use instant messaging or IRC? There are several text-based IM/IRC clients that run fine within screen. This means that you can always be 'online' in some way, shape, or form, and then attach in every so often to catch any missed messages and/or engage in conversation.

Another smaller corner-case use of screen (sorry tmux folks, I don't believe this is supported) -- terminal emulator via serial. Sure, there's minicom and other useful tools, but screen has this function built right in. I can hop on a router console via my USB->Serial adapter by simply running 'screen /dev/ttyUSB0' on my laptop.

The following table contains the basics of using screen. I highly recommend reading man pages or some more detailed documentation on the Internet, but this should at least get you started. As with anything else, it may take some time to get used to, but it is absolutely worth it.

GNU Screen cheat sheet

// Launch screen into its default shell
user@host# screen

// Start a program in screen, i.e. irssi
user@host# screen irssi

// Check for active screen session
user@host# screen -list

// Reattach screen (detach first if needed)
user@host# screen -d -r

// Multiuser attach (aka attach without detaching other instance)
user@host# screen -x

// Multiuser attach (but detach other instance first)
user@host# screen -dx

// Control from within screen
[Ctrl-a + d] Detach from screen
[Ctrl-a + ?] Display key bindings
[Ctrl-a + c] Create new window
[Ctrl-a + A] Rename current window
[Ctrl-a + Ctrl-a] Switch to most recent window 
[Ctrl-a + #] (# = 1, 2, 3 etc) Switch to first, second, etc window
[Ctrl-a + p] Switch to previous window
[Ctrl-a + n] Switch to next window
[Ctrl-a + "] Bring up menu of windows to choose from
[Ctrl-a + k] Kill current window
[Ctrl-a + Esc] Enter copy mode (read up on this before using)



mtr

First called "Matt's Traceroute" (not me!) and now "My Traceroute" -- MTR is a super useful traceroute tool that beats the pants off the traditional 'traceroute/tracert' tools. Instead of just tracing the route and reporting three response times per hop, it continually pings each hop to measure latency and loss. While traceroutes are not always super accurate due to firewall filtering, this tool makes the best of it. Since mtr uses ICMP as opposed to UDP like the standard 'traceroute' tool, it often seems to return better results. Thanks to +Mike Neir for the reminder on that last bit!

blogger@bloggy:~$ mtr -n 8.8.8.8

                              My traceroute  [v0.80]
bloggy (0.0.0.0)                                       Wed Jul  3 04:25:50 2013
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 111.111.111.111                   0.0%    10    0.0   0.0   0.0   0.0   0.0
 2. 111.111.111.112                   0.0%    10    1.0   0.6   0.5   1.0   0.2
 3. 111.111.111.113                   0.0%    10    0.2   0.2   0.2   0.2   0.0
 4. 111.111.111.114                   0.0%    10   24.8  19.6  13.3  24.8   4.1
 5. 207.88.14.194                    55.6%    10   12.8  12.8  12.8  13.0   0.1
 6. 216.156.72.30                     0.0%    10   80.7  72.5  55.0  86.5  11.8
 7. 209.85.254.128                    0.0%    10   13.5  13.7  13.0  17.3   1.3
 8. 72.14.237.130                     0.0%    10   13.2  13.3  13.1  13.5   0.1
 9. 72.14.238.106                     0.0%    10   29.6  29.8  28.2  41.8   4.2
10. 216.239.46.191                    0.0%    10   39.4  38.0  25.9  56.3  11.8
11. ???
12. 8.8.8.8                           0.0%     9   26.0  26.0  25.9  26.0   0.0


hping3

The next tool I want to mention is somewhat of a swiss army knife of ping tools. It can operate in ICMP, TCP, UDP modes, as well as a fairly functional port scanning mode. It also gives plenty of customization options with regard to flags, spoofing addresses, etc. This is one of the more common tools I use in lab, but it can often come in handy in day-to-day troubleshooting as well. The ability to spoof source addresses makes it an invaluable tool to test firewall rules.

hping3 can also be used to test QoS policies, if policies match on DSCP values. The key to this, of course, is that the trust boundary needs to move down to the port. Otherwise the markings will be rewritten to 0 on ingress. See an example below of how to test QoS policy matching on DSCP value EF, typically used for VoIP payload. Along with the hping syntax, we'll also do some command line binary/hex/decimal conversion.

// hping3 takes ToS, not DSCP
// DSCP = 6 most significant bits of 8-bit ToS field
// Convert DSCP 46 to binary
blogger@bloggy:~# echo "obase=2; 46" | bc
101110

// Pad the DSCP out to find ToS value
blogger@bloggy:~# echo "ibase=2; 10111000" | bc
184

// Convert the ToS value to hex for use with hping3
blogger@bloggy:~# echo "obase=16; 184" | bc
B8

// Set up tshark to capture and validate DSCP from hping3
blogger@bloggy:~# tshark -c1 -i wlan0 -f 'host 8.8.8.8' -T fields -e ip.dsfield.dscp 2>/dev/null | awk '{ print strtonum($0) }'

// In another terminal, send packet with DSCP 46
blogger@bloggy:~# hping3 --udp -c 1 -o b8 -p 53 8.8.8.8 
HPING 8.8.8.8 (wlan0 8.8.8.8): udp mode set, 28 headers + 0 data bytes

--- 8.8.8.8 hping statistic ---

1 packets tramitted, 0 packets received, 100% packet loss
round-trip min/avg/max = 0.0/0.0/0.0 ms

// Check other terminal to see if captured DSCP value is 46
blogger@bloggy:~# tshark -c1 -i wlan0 -f 'host 8.8.8.8' -T fields -e ip.dsfield.dscp 2>/dev/null | awk '{ print strtonum($0) }' 
46


tcpdump / Wireshark / tshark

The next set of tools are a must-have in any networking toolkit. These three tools all allow for the capture and displaying of network traces. I find myself using tcpdump most often from the command-line to capture packets to a file, tshark if I need to search/manipulate/trigger off the results, and Wireshark as a general capture or diagnostic tool. Wireshark has a nice graphical user interface to assist in capturing, displaying, filtering, and analyzing results of a trace. It's probably the best place to start if looking to learn how to really dig into network conversations. Of course, it is also great for advanced analysis of captures.

tcpdump is tried and true, and it is often included in the base installation of most Linux distributions. The prevalence of tcpdump makes it extremely convenient to either snag a capture or to quickly analyze the results of one. It is fairly straightforward to use; typically all it takes is specifying an interface, deciding whether or not to write the capture to a file, and optionally specifying capture filters.

tshark is a lot like tcpdump, in that it's a CLI-based packet capture utility. However, it has many of Wireshark's expansive display filtering options available as well. An example of this is seen in the hping3 command output above, as we used tshark to validate whether hping3 was truly sending the DSCP value we expected. Note the syntax of the tshark command used:
  • "-c1" means stop after one packet
  • "-i wlan0" specifies wlan0 as the capture interface
  • "-f 'host 8.8.8.8'" specifies the tcpdump-syntax capture filter to use
  • "-T fields -e ip.dsfield.dscp" tells tshark to output using only specified fields, of which only the DSCP field is specified.
Since we're discussing that command output, note that we piped the output of tshark into awk in order to convert the hexadecimal output from tshark into a decimal number. And hence, we see the immediate value of tshark in certain scenarios; by having much more granular control over the output, tshark makes exporting certain pieces of information from a large capture file much easier.

netcat

It would be hard to call any networking toolkit complete without netcat. Again, like all other tools on this list, there are plenty of in-depth tutorials out there. I highly recommend taking some time to learn this tool, as it is quite handy in connectivity testing. netcat can be used to either initiate or listen for incoming TCP or UDP connections. On the surface, this alone makes netcat pretty useful. Previous posts have shown the use of netcat as a UDP listener for syslogs, although the command has not been explained.

To start netcat as a client initiating a connection, just specify the IP address and port of the server. If you're planning to use it in an interactive mode, I'd recommend throwing the -v or even -vv flags to increase verbosity of the output. To start as a listener, you'll want to specify the -l flag for listening and the -p flag to specify the listening port. Outside of these basics, netcat can become a much more flexible tool. Since it sets up the framework for basic network connectivity, it can be used as a conduit for transferring files, passing a bash shell, etc. While there are plenty of tools that can meet these needs, netcat can often fit the bill. In the example below, we'll use netcat on one machine without tshark to send a packet capture to another machine that has tshark.

// read pcap contents into netcat listen server
blogger@bloggy:~# nc -lp 3333 < file.pcap

// on second box, run netcat to connect to bloggy
// pipe output to tshark to see results of analysis
blogger@blogged:~# nc bloggy 3333 | tshark -r - -T fields -e ip.src -e udp.srcport -e ip.dst -e udp.dstport -e frame.time_relative

10.21.177.12 55306 10.21.177.223 69 0.000000000
10.21.177.223 39720 10.21.177.12 55306 0.000756000
10.21.177.12 55306 10.21.177.223 39720 0.004289000
10.21.177.12 2834 10.21.177.223 69 38.648645000
10.21.177.223 36855 10.21.177.12 2834 38.649119000
10.21.177.12 2834 10.21.177.223 36855 38.649273000
10.21.177.223 36855 10.21.177.12 2834 38.649354000
10.21.177.12 2834 10.21.177.223 36855 38.649692000
10.21.177.223 36855 10.21.177.12 2834 38.649747000
10.21.177.12 2834 10.21.177.223 36855 38.650081000
10.21.177.223 36855 10.21.177.12 2834 38.650128000
10.21.177.12 2834 10.21.177.223 36855 38.650458000
10.21.177.223 36855 10.21.177.12 2834 38.650511000


tcpreplay / Bit-Twist

The next set of tools are pretty unique and very good for network testing. Both tcpreplay and Bit-Twist allow you to place the contents of a packet capture back on the wire. While this is not something that is useful on a regular basis, when it's needed, it's very helpful. Personally I've used these tools most often as a method to test either firewall policies or quality of service policies.

QoS policy testing is actually a pretty neat scenario to discuss. Most of the time when building a QoS policy, it's somewhat hard to test the functionality prior to putting production traffic across it. Aside from marking traffic with DSCP values and shifting the trust boundary, it's not necessarily trivial. However, with Bit-Twist or tcpreplay, you can replay previously-captured traffic that includes known quantities of traffic types that match certain QoS queues. This means that if you have class-maps that match based on IP ACL matches, NBAR inspection, etc, you can actually test these features with "real" traffic.

tcpreplay, in its default mode, will replay a pcap file back onto the network with the same headers and payload as it was captured, at the same rate at which it was captured. Note that contrary to its name, it will have no problem replaying other protocols aside from TCP. There are options to increase speed and modify headers if necessary, as well.

Bit-Twist also allows for modification and replaying of packet captures, with a pretty heavy focus on the rewrite capabilities. I have used both Bit-Twist and tcpreplay several times over the years, and they both work well. I can't say that I would necessarily recommend one over the other.

Kismet and Aircrack-ng

Between these two software packages, you can gain a very good understanding of all things 802.11! Kismet is a passive wireless network scanner/sniffer that does a great job of mapping out the wireless environment. It will detect and report wireless access points, wireless clients, SSID's, etc. It will even write out packet capture files of all the data it collects! Since most wireless cards in Linux can be placed in monitor mode, these captures will include the 802.11 control frames that would normally be invisible to a Windows wireless packet capture.

Aircrack-ng is a software suite that provides a number of tools that can be used to test the security of a wireless network. I won't go too deep into this software package, but I feel that it provides several good functions for a network admin. First, it has some basic wireless scanning functions, with control/filtering over the scanning parameters. Furthermore, especially in a lab environment, it can provide a great platform for learning more about the way wireless authentication, authorization, and encryption work.

Conclusion

I am sure there are several more tools that went without mention, but hopefully this list sparks some interest. The only other 'toolset' that I really wanted to at least mention - is the Linux command line environment in general. The use of bash with programs like grep, awk, sed, and xargs allow for quick and simple scripting. The more I learn, the more I continue to find myself in bash to perform data manipulation.

Finally, it should go without saying, but many of these tools can impact your production network! Please use caution when experimenting.