Double-sided and Double-dumb Printing

Summit 09 Bonus: All Summit 09 attendees will receive a full licensed copy of NetScanTools Pro - a $249 value.

It's not easy going "green"... I mean - all we wanted to do was print the customized course manuals double-sided. What's the big problem?

WIRESHARK and I NEARLY BARFED!



First let me say I was printing to an HP OfficeJet Pro L7780 All-in-One printer - it supports auto-duplexing. The feature is implemented in the most funky way however! Watching the traffic live while the printer did it's 'dance of duplex' - I was shocked to see the lame process of auto-duplex printing.

I had to wonder if the HP printer group might be interested insome classes on networking.*

Ok... here's how this printer does duplex printing. It's automatic, so you don't have to do the old 'refeed the paper and hope it's in the right order' process. You just select auto-duplex and away it goes. The printer prints the first page and - while still holding the paper by the bottom edges, the console reads "Ink drying - please wait." After a while the paper is then sucked back into the printer (a moment that always makes my heart jump - I've heard the shredding sound of mangled paper too many times). The second side of the paper prints and we all sigh relief - another 2-sided page done!

The trace file showed that the printer creates the pause process by sending a Zero Window packet to the client - in essence "talk to the hand" (isn't that HP's little helper logo?). Each 'drying process' essentially was caused by halted flow of print data from my client to the printer. My client sent Zero Window Probe packets to ask 'What is wrong with you, buddy?" and the printer kept the traffic at bay by sending Zero Window Probe ACKs for about 30 seconds.

The printer's TCP window size did decrease down to an unacceptable zero so it wasn't lying about not having any receive buffers. It seems the printer doesn't clear out the buffer space consistently during the print process - it's buffer becomes depleted right at the point when the 'ink needs to dry" - perhaps it knows we'll all need to take a breath then so allowing the buffer to fill doesn't seem like a big deal - it's going to have to sit around idle anyway (anyone think about adding a nice 'ink fan' attachment to make my printing go faster?).

I decided to add a tcp.window_size column and filter on traffic coming from the printer. Using File > Export > File and choosing the displayed packets, I created a csv file with a column just depicting the TCP receive window sizes advertised by the printer. In Excel I selected only the tcp.win_size column and inserted a line graph.

BEAUTI-FRIGGIN-FUL!

Look at the intentional pattern of the window size values advertised with definite zero window conditions during the ink drying time. There's just got to be a better way!

What? Is my printer totally dumb? I didn't think so, but this is certainly dumb behavior. The file is only 13,296 KB! The printer has 64 MB memory standard. Why didn't the printer allow me to send the entire document to the printer, then buffer the data while the ink dried? Why did I have to sit around and wait for the ink to dry before being able to send the next page of data? The Wireshark summary of the data sent to the printer showed the transmission rate averaged around 0.431 Mbit/second - snooze...

I also look at the spooled file in my Windows\System32\spool\PRINTERS directory - holy schmoley! The spooled file (.spl) was 670,016 KB! Whazzup with that? The idea of researching the MS spooling process and .spl file format made my stomach turn again - but it did look like this Powerpoint Notes document was processed as a graphics file.

Then came the really interesting part - I printed twenty-five copies of the custom student manual. You'd think the printer would have buffered the first copy in memory and just pulled from that when it needed to make each successive copy - right? Nope. My client sent the entire file to the printer a second time, a third time, a fourth time... etc. Whoa... that's one dumb process! No turning off my system until the entire job was sent 25 times from the spool\PRINTERS directory. Barf.

Basic network analysis should be a mandatory requirement for any vendor making network-capable devices/applications - or else they might make a network-incapable devices/applications - oh wait - they do!



Enjoy life one bit at a time!
Laura

*Although this is the lamest way to get the job done, I swear by my HP printers - they take a beating and keep on spitting out surreal amounts of pages every month - I am a tactile, visual person - there's nothing like that printed version of the spec to snuggle up with at night! So much for going green. (The only green that will fly around here will be some big $ for printers that can buffer the entire doc once - without the 'talk to the hand process' - and then print extra copies from that!)

Tracing the Route

Summit 09 Bonus: All Summit 09 attendees will receive a full licensed copy of NetScanTools Pro - a $249 value.

During troubleshooting processes, a standard ping test is often used to check connectivity to a host and determine the round trip latency time. This process uses an ICMP Type 8 Echo Request and relies on an ICMP Type 0 Echo Reply.

Sometimes, however, the target won't respond to ICMP Echo Replies - either because it is configured to ignore these ICMP Echo Requests or because a device along the path filters these packets out so they don't reach the target.

I prefer to perform traceroute using NetScanTools' TCP option. Besides setting the TCP port and the sequence number settings, you can also set the MTU (Maximum Transmission Unit) to test the maximum packet size along a path.Another option available is to set the TCP window size - in our example I have set the window size field value to 16,384. In addition, you can define the payload - using a binary or text file.

Why would you use a big fat file for the test? Ahhh... my padawan - to test the MTU allowed through the path and consider putting a signature in the payload that should trigger an IDS or be logged by a firewall - multiple birds with one stone - connectivity testing, latency testing, IDS/firewall testing! Nice!

In the figure below, you can see my host 192.168.0.113 sending a series of TCP SYN packets - the target port is 79 (finger). The packets colored with a red background have an IP header Time-to-Live value less than 5 - a sure sign of a traceroute operation.

When we reach the target, a RST is generated in response. That's what gives us our round trip latency time.I appreciate why companies restrict ICMP-based traffic on their networks - and when I'm doing connectivity tests and latency tests, customizing my TCP-based traceroutes always sits on the top of my to-do list.

Enjoy life one bit at a time!
Laura

Storms Rip the Net

Summit 09 Bonus: Licensed NetScanTools Pro - All Summit 09 attendees will receive a full licensed copy of NetScanTools Pro – a $249 value.

I recognized the tone in the voice that day - the panicked sound of someone dealing with a non-functional network. In this case, the network was a critical one (I can't disclose the specific type of network or customer).

At approximately 3:34am, their critical network came crashing down - no connectivity for any hosts on the network. They'd placed Wireshark on the network and it too crashed within moments.

Ok... so there was something definitely cruising along the network wreaking havoc. I had to see those packets!With over 2,000 miles separating us, it would be a 'walk through the capture' process.

The first step - dump the GUI!

Wireshark comes with Tshark for command-line capture. The syntax used was:

tshark -c 100 -w gen1.pcap

The -c parameter indicates the number of packets to capture. The -w parameter is used to define the name of the trace file to create. Why only 100 packets? What? Well... if there is a catastrophic issue on the network that could kill systems that connect to it that quickly, it shouldn't take many packets to characterize that traffic.

Immediately upon capturing these 100 packets, I instructed the customer to disconnect from the network. You don't need network access to analyze captured traffic - trace files are processed through the Wiretap Library - directly off the disk.

The 100 packets told the story - an insane looping packet storming through the network at a blazingly rapid packets per second rate. When facing a traffic issue like this it is important to look at the IP header's Identification value. You need to differentiate between a looping packet or a series of individual packets sent from a 'killer host' (and I mean killer as in "network killer").

If the IP Identification field value is the same for all the packets, then the packet is looping somehow. If the packet has a different IP Identification field then the packets have each passed through the IP protocol separately from a host. It's an amazingly simple differentiation - and an important one.If the packets had unique IP Identification field values, we'd be looking at a single host causing the problems. We'd be delving into the MAC header of the packet to identify the location of the lousy host. (Having a master list of MAC addresses for all hosts on the network is imperative in that situation. Mark that down as something to do this week!)

In this case, all the IP Identification field contained the same value - this was a looping packet. We had an infrastructure issue. On this heavily switched network it seemed spanning tree was not doing its job. Poor spanning tree - no one really pays attention to it until it has a problem.

Being remote to the customer location, I could not look over their shoulder as they walked the network and shut down one switch at a time. It was in their hands now. I sat waiting for their call - waiting to hear if they'd found the culprit. I didn't wait long.

I waited 30 looooong minutes for the call even though hit had taken the client less than 5 minutes to find a switch that was looping traffic back through the network. They spent the other 25 minutes starting up hosts on the network to ensure all was well. The switch was configured properly - so it would be replaced with another switch while they played with the problem switch in the lab (someday... someday).This offsite analysis hits a key point in troubleshooting - the devastating failures are typically easier to spot. They scream at us. They stomp their feet and throw things. All they want is to have someone listen.

Enjoy life one bit at a time...
Laura

Join us at Summit 09 on December 7-9th! You'll get a copy of NetScanTools Pro and 3 full days of hands-on individual and group labs focused on troubleshooting and security. Download the Summit Information Guide. All Access Pass members receive a 50% discount to Chappell Summit 09. Don't miss it!

SNMP Snooping

Summit 09 Bonus: Licensed NetScanTools Pro - a $249 ValueAll Summit 09 attendees will receive a full licensed copy of NetScanTools Pro - a $249 value.

One of the labs for Summit 09 deals with SNMP snooping - locating information about a device by taking advantage of available MIB (Management Information Base) data through SNMP walking. Networks abound with SNMP-based devices - we can use the Port Scanner tool to generate a simple UDP scan for port 161 to discover those SNMP devices.

In NetScanTools, I discovered a few network printers supporting SNMP. I entered the IP address of one of the printers and selected the WALK action for the Object ID (OID) .iso.org.dod.internet. I left the community string at the default as I was certain no one had changed it since the printer was plugged in.

The result - a 24-page document filled with information about that printer and the other devices on the wired and wireless networks. The standard printer information was puked out as expected, but this SNMP snoop yielded loads more information:

• ARP table listing devices on the wired and wireless network
• MAC layer In/Out statistics (including errors)
• TCP In/Out statistics (including errors)
• UDP In/Out statistics (including errors)
• ICMP In/Out statistics (including errors)
• Routing table
• List of all received/transmitted ICMP packets
• SSIDs, channel numbers and signal strength of local WLANs - not just the WLAN that the printer was on and not just on the channel the printer was on

As I started playing a bit more and finding other unique SNMP devices, I realized I needed to load some new MIBs - a MIB is a database of objects. I found hundreds of MIBs available online at www.oidview.com/mibs/detail.html.

One of the coolest features in NetScanTools' SNMP tool is the ability to determine listening ports on the target without using a port scan. By generating udpLocalPort and tcpConnState queries, I could get the list of open UDP and TCP ports directly from the source. Using NetScanTools we can discover SNMP devices on the network, load an unlimited number of additional MIBs and perform a dictionary attack to identify the community string used by SNMP devices.

Join us at Summit 09 on December 7-9th! You'll get a copy of NetScanTools Pro and 3 full days of hands-on individual and group labs focused on troubleshooting and security. Don't miss it! Download the Summit Information Guide from www.chappellU.com. All Access Pass members receive a 50% discount to Chappell Summit 09.

Enjoy life one bit at a time!
Laura

The Game of VoIP

I always get that tingly excited feeling when I open up new trace files.

It's not unlike the feeling I have as I pick up the pile of cards dealt to me in a game of gin. What will the cards hold? What will I see in them? That is what analysis is to me - a card game. Last week I began my slide preparation for an upcoming set of projects related to VoIP analysis. I find VoIP traffic fascinating - its duality - signaling separate from voice - its trusting nature - using UDP - its frailty.

This VoIP project came up first in preparation for the VoIP labs for the Summit 09 training event. I wanted to include a lecture portion dealing with SIP and RTP behavior and point out the cool new organization of telephony-related options in Wireshark. I wanted to build some hands-on labs to allow students to identify the likely points on a network that they'd look to isolate problems with the calls. I'm energized with the process of preparing new lectures, new trace files and new hands-on exercises for Summit 09.But Summit 09 had to go on the back burner for a bit... another hand had been dealt - by Microsoft.

The Microsoft Webinars (Oct 6th and Oct 28th)
I have presented two webinars for Microsoft to date - one wildly popular (where we pulled in Microsoft's largest registration and attendance counts) and the other nearly empty (seems the old Microsoft marketing engine had stalled on that one and we didn't push it because it was open to partners only). If you want to join me at the Summit 09 event on December 7-9th, check out the Summit 09 Information Document at www.chappellseminars.com/summit09.html.

No RTP? A Lousy Deal?!
When I began systematically opening the VoIP files in my folder, I hit the first stumbling block of VoIP analysis. One of the first trace files I opened up did not contain the SIP packets for a call setup so Wireshark didn't apply an RTP dissector to the voice traffic - it just dumped me off at UDP. Hmmm... seems I have a hand with no runs or matches.

I've talked about the process of "Decode As" in the past for interpreting traffic traveling over non-standard ports. The same steps are quite common to use with VoIP traffic. I right clicked on a packet in the trace file and selected Decode As. On the right side, I chose one of the transport port numbers and selected RTP in the protocol list.

Voila! It's decoded as RTP! Another option is to enable the RTP Preferences "Try to decode RTP outside of conversations". The image at left shows the traffic of an RTP stream that Wireshark decodes as just UDP. Wireshark didn't see the SIP call setup process and it decodes the voice traffic as just UDP.



Viewing the RTP Streams
When Wireshark reached version 1.2, the menu bar was revised to include a Telephony item. Once you force the RTP decode (if required), you can select Telephony > RTP > Show All Streams.

Wireshark detected two RTP streams in my trace file once I forced the RTP decode on the traffic.

As you can see from the figure at left, Wireshark can now show me the RTP streams in the trace file - one stream going in each direction. We can also see that one of the streams has a 2.1% packet loss. Packet loss is ugly in VoIP communications - when a call experiences serious packet loss, the conversation quality suffers.There's an example of a G.711 audio file with 10% packet loss at VoIPTroubleshooter.com.

Playing the Hand That Was Dealt
Unlike gin or poker, we have to play the hand we were dealt in network analysis. We can only make the hand stronger by filtering out traffic that is not relevant and interpreting the possible causes for the packets that remain.

In the area of VoIP analysis, my favorite filter is sip || rtp and my favorite feature is the RTP playback feature. I like to set the time column to Seconds Since Previous Displayed Packet and export the trace file to csv format.

In Excel I sort on the source column first, the number column second and then graph out the time column to show me the variance between packets (the jitter value). I could use Wireshark's graphing capabilities to give me some of this information, but this is one of the areas where a formal spreadsheet program really shines. I exported the trace file to a csv format to graph out a portion of the jitter values.

VoIP at Summit 09
I'm anxious to get back to the Summit 09 preparations - to document all the tips and tricks of VoIP analysis. I've put together a few lab exercises already, but I'm keeping my eyes open for new hands to play - if you have any VoIP traces that I could use in Summit 09, please send them to me (laura@chappellU.com). Download the Summit Information Guide. All Access Pass members receive a 50% discount to Chappell Summit 09.

Enjoy life one bit at a time.
Laura

Summit 09 Registration Opens

It's a Geekfest Training Event! Last year's Summit was a great success with a room full of folks hunched over their laptops for three-days of labs and training. Since that time I've been researching and developing new materials - they are ready to go into Summit 09. Summit 09 will be another BYOL (Bring Your Own Laptop) event - filled with hands-on labs. This time around we will offer both individual and group labs.



Hot Tools and Key Tasks
We will focus on which tools are best for each task, such as the following:

• traffic redirection and interception
• IP address sanitizing
• locating firewalled hosts
• throughput testing
• jitter measurement
• identifying blocked/filtered ports
• WLAN RF analysis

Troubleshooting Hot Spots and Testing
On day two, we will delve into a corporate network that is consistently garnering complaints. Working from the network diagram, we'll devise a plan to isolate the cause of network problems and perform proactive throughput testing and live RF analysis. We'll address both wired and wireless network problems.

Security and Forensics
On the third and final day, we focus on security. Starting with the TCP vulnerabilities and the SMB2 vulnerability announced this month, we will dissect the key issues and build a malicious/suspicious traffic profile. This profile will save you lots of time and help you spot security flaws on the network.

During the Summit you will be working on network diagrams to pinpoint where to capture traffic and follow the path of communications.The schedule is packed with the latest techniques for catching network problems, identifying suspicious traffic, testing network throughput, analyzing WLAN traffic and discovering network devices and services.

Download the Summit Information Guide. All Access Pass members receive a 50% discount to Chappell Summit 09. Enjoy life one bit at a time. Lauraw

WLAN Profiling: It's a Good Thing

Speed up your WLAN Analysis Processes

This weekend I recorded the WLAN Analysis 101 course (available to All Access Pass members already). I spent a fair amount of time customizing a profile for Wireshark to include columns, display filters and color filters focused on WLAN traffic.

What should your customized WLAN profile include?

Three Columns to Add
Consider adding columns for Frequency/Channel information, RSSI (Receive Signal Strength Indicator), and transmit rate. Once you have these columns added to Wireshark, you can sort on the columns and even use them when exporting to a spreadsheet program for further graphing.

Hot Display Filters
Add display filters to quickly view all traffic on a specific channel - for example, radiotap.channel.freq == 2412 will display all Channel 1 traffic. You might also want to create filters to display all beacons for a specific SSID. The syntax for that would be wlan.fc.type_subtype == 0x08 && frame contains "wsu" to see all the beacons related to the wsu WLAN network.

Wild Color Filters
Besides creating color filters for the various channels (which use the same syntax as the display filters), consider coloring traffic that has the retry bit set to 1 (for example, wlan.fc.retry == 1) , Probe Requests/Responses and Associations/Reassociations/Disassociations. I create 'butt ugly' color filters for frames that I consider a problem, such as retry frames.

Customized profiles can include your column settings, font settings and capture filters as well. You could create a custom profile for the corporate office, a specific client or a specific network type.Customizing Wireshark to fit the network you are working on helps you sort out the traffic and spot problems faster!

The "WLAN Analysis 101" course released to All Access Pass members on September 13, 2009. This course includes color filters, display filters, WLAN trace files, Chanalyzer recordings and a 29-page course handout. This course is available for individual purchase at www.chappellseminars.com/s-wlan1.html.

Enjoy life... one bit at a time.

Laura