Double-sided and Double-dumb Printing

Summit 09 Bonus: All Summit 09 attendees will receive a full licensed copy of NetScanTools Pro - a $249 value.

It's not easy going "green"... I mean - all we wanted to do was print the customized course manuals double-sided. What's the big problem?

WIRESHARK and I NEARLY BARFED!



First let me say I was printing to an HP OfficeJet Pro L7780 All-in-One printer - it supports auto-duplexing. The feature is implemented in the most funky way however! Watching the traffic live while the printer did it's 'dance of duplex' - I was shocked to see the lame process of auto-duplex printing.

I had to wonder if the HP printer group might be interested insome classes on networking.*

Ok... here's how this printer does duplex printing. It's automatic, so you don't have to do the old 'refeed the paper and hope it's in the right order' process. You just select auto-duplex and away it goes. The printer prints the first page and - while still holding the paper by the bottom edges, the console reads "Ink drying - please wait." After a while the paper is then sucked back into the printer (a moment that always makes my heart jump - I've heard the shredding sound of mangled paper too many times). The second side of the paper prints and we all sigh relief - another 2-sided page done!

The trace file showed that the printer creates the pause process by sending a Zero Window packet to the client - in essence "talk to the hand" (isn't that HP's little helper logo?). Each 'drying process' essentially was caused by halted flow of print data from my client to the printer. My client sent Zero Window Probe packets to ask 'What is wrong with you, buddy?" and the printer kept the traffic at bay by sending Zero Window Probe ACKs for about 30 seconds.

The printer's TCP window size did decrease down to an unacceptable zero so it wasn't lying about not having any receive buffers. It seems the printer doesn't clear out the buffer space consistently during the print process - it's buffer becomes depleted right at the point when the 'ink needs to dry" - perhaps it knows we'll all need to take a breath then so allowing the buffer to fill doesn't seem like a big deal - it's going to have to sit around idle anyway (anyone think about adding a nice 'ink fan' attachment to make my printing go faster?).

I decided to add a tcp.window_size column and filter on traffic coming from the printer. Using File > Export > File and choosing the displayed packets, I created a csv file with a column just depicting the TCP receive window sizes advertised by the printer. In Excel I selected only the tcp.win_size column and inserted a line graph.

BEAUTI-FRIGGIN-FUL!

Look at the intentional pattern of the window size values advertised with definite zero window conditions during the ink drying time. There's just got to be a better way!

What? Is my printer totally dumb? I didn't think so, but this is certainly dumb behavior. The file is only 13,296 KB! The printer has 64 MB memory standard. Why didn't the printer allow me to send the entire document to the printer, then buffer the data while the ink dried? Why did I have to sit around and wait for the ink to dry before being able to send the next page of data? The Wireshark summary of the data sent to the printer showed the transmission rate averaged around 0.431 Mbit/second - snooze...

I also look at the spooled file in my Windows\System32\spool\PRINTERS directory - holy schmoley! The spooled file (.spl) was 670,016 KB! Whazzup with that? The idea of researching the MS spooling process and .spl file format made my stomach turn again - but it did look like this Powerpoint Notes document was processed as a graphics file.

Then came the really interesting part - I printed twenty-five copies of the custom student manual. You'd think the printer would have buffered the first copy in memory and just pulled from that when it needed to make each successive copy - right? Nope. My client sent the entire file to the printer a second time, a third time, a fourth time... etc. Whoa... that's one dumb process! No turning off my system until the entire job was sent 25 times from the spool\PRINTERS directory. Barf.

Basic network analysis should be a mandatory requirement for any vendor making network-capable devices/applications - or else they might make a network-incapable devices/applications - oh wait - they do!



Enjoy life one bit at a time!
Laura

*Although this is the lamest way to get the job done, I swear by my HP printers - they take a beating and keep on spitting out surreal amounts of pages every month - I am a tactile, visual person - there's nothing like that printed version of the spec to snuggle up with at night! So much for going green. (The only green that will fly around here will be some big $ for printers that can buffer the entire doc once - without the 'talk to the hand process' - and then print extra copies from that!)

Tracing the Route

Summit 09 Bonus: All Summit 09 attendees will receive a full licensed copy of NetScanTools Pro - a $249 value.

During troubleshooting processes, a standard ping test is often used to check connectivity to a host and determine the round trip latency time. This process uses an ICMP Type 8 Echo Request and relies on an ICMP Type 0 Echo Reply.

Sometimes, however, the target won't respond to ICMP Echo Replies - either because it is configured to ignore these ICMP Echo Requests or because a device along the path filters these packets out so they don't reach the target.

I prefer to perform traceroute using NetScanTools' TCP option. Besides setting the TCP port and the sequence number settings, you can also set the MTU (Maximum Transmission Unit) to test the maximum packet size along a path.Another option available is to set the TCP window size - in our example I have set the window size field value to 16,384. In addition, you can define the payload - using a binary or text file.

Why would you use a big fat file for the test? Ahhh... my padawan - to test the MTU allowed through the path and consider putting a signature in the payload that should trigger an IDS or be logged by a firewall - multiple birds with one stone - connectivity testing, latency testing, IDS/firewall testing! Nice!

In the figure below, you can see my host 192.168.0.113 sending a series of TCP SYN packets - the target port is 79 (finger). The packets colored with a red background have an IP header Time-to-Live value less than 5 - a sure sign of a traceroute operation.

When we reach the target, a RST is generated in response. That's what gives us our round trip latency time.I appreciate why companies restrict ICMP-based traffic on their networks - and when I'm doing connectivity tests and latency tests, customizing my TCP-based traceroutes always sits on the top of my to-do list.

Enjoy life one bit at a time!
Laura

Storms Rip the Net

Summit 09 Bonus: Licensed NetScanTools Pro - All Summit 09 attendees will receive a full licensed copy of NetScanTools Pro – a $249 value.

I recognized the tone in the voice that day - the panicked sound of someone dealing with a non-functional network. In this case, the network was a critical one (I can't disclose the specific type of network or customer).

At approximately 3:34am, their critical network came crashing down - no connectivity for any hosts on the network. They'd placed Wireshark on the network and it too crashed within moments.

Ok... so there was something definitely cruising along the network wreaking havoc. I had to see those packets!With over 2,000 miles separating us, it would be a 'walk through the capture' process.

The first step - dump the GUI!

Wireshark comes with Tshark for command-line capture. The syntax used was:

tshark -c 100 -w gen1.pcap

The -c parameter indicates the number of packets to capture. The -w parameter is used to define the name of the trace file to create. Why only 100 packets? What? Well... if there is a catastrophic issue on the network that could kill systems that connect to it that quickly, it shouldn't take many packets to characterize that traffic.

Immediately upon capturing these 100 packets, I instructed the customer to disconnect from the network. You don't need network access to analyze captured traffic - trace files are processed through the Wiretap Library - directly off the disk.

The 100 packets told the story - an insane looping packet storming through the network at a blazingly rapid packets per second rate. When facing a traffic issue like this it is important to look at the IP header's Identification value. You need to differentiate between a looping packet or a series of individual packets sent from a 'killer host' (and I mean killer as in "network killer").

If the IP Identification field value is the same for all the packets, then the packet is looping somehow. If the packet has a different IP Identification field then the packets have each passed through the IP protocol separately from a host. It's an amazingly simple differentiation - and an important one.If the packets had unique IP Identification field values, we'd be looking at a single host causing the problems. We'd be delving into the MAC header of the packet to identify the location of the lousy host. (Having a master list of MAC addresses for all hosts on the network is imperative in that situation. Mark that down as something to do this week!)

In this case, all the IP Identification field contained the same value - this was a looping packet. We had an infrastructure issue. On this heavily switched network it seemed spanning tree was not doing its job. Poor spanning tree - no one really pays attention to it until it has a problem.

Being remote to the customer location, I could not look over their shoulder as they walked the network and shut down one switch at a time. It was in their hands now. I sat waiting for their call - waiting to hear if they'd found the culprit. I didn't wait long.

I waited 30 looooong minutes for the call even though hit had taken the client less than 5 minutes to find a switch that was looping traffic back through the network. They spent the other 25 minutes starting up hosts on the network to ensure all was well. The switch was configured properly - so it would be replaced with another switch while they played with the problem switch in the lab (someday... someday).This offsite analysis hits a key point in troubleshooting - the devastating failures are typically easier to spot. They scream at us. They stomp their feet and throw things. All they want is to have someone listen.

Enjoy life one bit at a time...
Laura

Join us at Summit 09 on December 7-9th! You'll get a copy of NetScanTools Pro and 3 full days of hands-on individual and group labs focused on troubleshooting and security. Download the Summit Information Guide. All Access Pass members receive a 50% discount to Chappell Summit 09. Don't miss it!

SNMP Snooping

Summit 09 Bonus: Licensed NetScanTools Pro - a $249 ValueAll Summit 09 attendees will receive a full licensed copy of NetScanTools Pro - a $249 value.

One of the labs for Summit 09 deals with SNMP snooping - locating information about a device by taking advantage of available MIB (Management Information Base) data through SNMP walking. Networks abound with SNMP-based devices - we can use the Port Scanner tool to generate a simple UDP scan for port 161 to discover those SNMP devices.

In NetScanTools, I discovered a few network printers supporting SNMP. I entered the IP address of one of the printers and selected the WALK action for the Object ID (OID) .iso.org.dod.internet. I left the community string at the default as I was certain no one had changed it since the printer was plugged in.

The result - a 24-page document filled with information about that printer and the other devices on the wired and wireless networks. The standard printer information was puked out as expected, but this SNMP snoop yielded loads more information:

• ARP table listing devices on the wired and wireless network
• MAC layer In/Out statistics (including errors)
• TCP In/Out statistics (including errors)
• UDP In/Out statistics (including errors)
• ICMP In/Out statistics (including errors)
• Routing table
• List of all received/transmitted ICMP packets
• SSIDs, channel numbers and signal strength of local WLANs - not just the WLAN that the printer was on and not just on the channel the printer was on

As I started playing a bit more and finding other unique SNMP devices, I realized I needed to load some new MIBs - a MIB is a database of objects. I found hundreds of MIBs available online at www.oidview.com/mibs/detail.html.

One of the coolest features in NetScanTools' SNMP tool is the ability to determine listening ports on the target without using a port scan. By generating udpLocalPort and tcpConnState queries, I could get the list of open UDP and TCP ports directly from the source. Using NetScanTools we can discover SNMP devices on the network, load an unlimited number of additional MIBs and perform a dictionary attack to identify the community string used by SNMP devices.

Join us at Summit 09 on December 7-9th! You'll get a copy of NetScanTools Pro and 3 full days of hands-on individual and group labs focused on troubleshooting and security. Don't miss it! Download the Summit Information Guide from www.chappellU.com. All Access Pass members receive a 50% discount to Chappell Summit 09.

Enjoy life one bit at a time!
Laura