Do You Know Where Your Throughput is Today?

iPerf Might Scream "This Stinks!"

Last night I was laughing and crying at the same time while reading a local small town newspaper and the dissertation on bandwidth problems at the local library. They actually shut off half the computers to aleviate the problem. One quote really grabbed me - "We don't notice the bandwidth problem when no one is using the computers." Hunh?

There were a few puzzling comments in the article such as the comment that the bandwidth problem "started 3 weeks ago." That sudden onset of the problem feels like something else to me. It feels like a device along the path that is causing the problems. A quick look at the traffic would validate that.

The article tied in nicely to the iPerf Throughput Testing materials I am finishing up. The reaction to the live iPerf testing done during the Analyze and Improve Throughput course and numerous requests for more information on iPerf prompted me to start developing a course that shows how to perform a series of throughput tests for UDP and TCP traffic.

iPerf is simple and complex at the same time. The one application can be run either in client mode or server mode.

Tests can be run in one direction (from the client to the server, which is the default) or bi-directionally. Here is one of my favorite tests for iPerf:

Client: -c 10.1.1.1 -u -t 60 -i 5
Server: iperf -s -u -i 5

This test enables me to locate jitter and packet loss along a path using a UDP stream sent over a 60 second time with results displayed every five seconds at both the server and the client.As you can see from the screenshot above, the path suffers packet loss reaching 39%. We were sending a steady stream at 1.05 Mbit/second specifically to identify packet loss. Well... we found it. Running the test for a full 24 hours would help us identify specific times of the day when packet loss is at its highest.

I hope to get into the local library and run Wireshark and iPerf on the network soon. I have no idea if the systems are locked down - a slight problem that might require some workaround. Meanwhile I'll peruse the shelves for those network troubleshooting books.

The "iPerf Throughput Testing" modules will be included in the Summit 09 course in December (www.chappellseminars.com/summit09.html) - the "Analyze and Improve Throughput" course is available now at www.chappellseminars.com.

Enjoy life one bit at a time.
Laura

Enough is Enough! No More Broken Windows

No... I'm not Microsoft-bashing (today)... not really. After all, this issue is seen on other operating systems as well. I recorded information about this in the things that perplexes many new and experienced analysts.

You may be aware that Wireshark has an Expert Info Composite entry for "Window is Full" and "Frozen Window" but unfortunately, this condition can be occurring on your network without Wireshark catching it.

You can set up a butt-ugly color filter and a display filter to alert you to this condition. Let me explain...











In the picture above, I've added column for the receive window size value set in the TCP headers of each packet. It's a custom column using the syntax tcp.window_size. I also added a column for the tcp.len value so I can see how much data is contained in each packet.

Notice in packet 361 that 10.0.52.164 is advertising a window size of 2,920 bytes - enough for two 1460-byte segments to fill as Wireshark notes in packet 363 [TCP Window Full]. The full receive buffer leads the client to begin advertising a receive window size of 0. Ok... duh... We can spot that one easily!









Now look at this screenshot. This delay is caused by a window sized problem as well - but this time the window size field didn't go alt the way down to zero - its at 536 (packet 374). That's too small for the queued up TCP segment at the other side so you might as well have said "shut up" with a window zero setting.

So what can we do about this? How can we easily see that we are having this problem when Wireshark doesn't have an Expert Notification for this? Aha! Here's where your butt-uglies come into play. Make a butt ugly filter for:

(tcp.window_size < reset ="="">










Check out the Trace File Analysis: TCP In-Depth course for more information on working with TCP traffic!

Laura
Enjoy life one bit at a time!

Sexy Spread Spectrum Signals

In the WLAN Analysis 101 course last month, I showed the effects of a cheap 2.4GHz phone on the wireless network by knocking myself off the network during my live video feed. Duh... I hope it made a point.

If I hadn't been picking up the RF signals around me, the death of my network connection would have been a mystery. After all, the cutoff was so sudden and folks in other locations around weren't having any problems at all.

The live course viewers saw the sudden spike in the signal as I'd told them to watch the Chanalyzer Spectral View. begin to climb near channel 1 and then SCREECH! The video came to a halt and my voice (fed through VoIP on my end) became scratchy and my words almost impossible to decipher.










The figure above shows their view at the time I attacked myself! Wow! What a hot, my connection to the online seminar engine, it felt like real life - this is what really happens in the WLAN world - and we got to experience it together.

I love looking at the Chanalyzer Spectral View - it consists of time across the X axis and frequency/channel across the Y axis. The color coding is based on signal amplitude. The closer to red, the stronger the signal. Vertical stripping indicates a consistent signal on a specific frequency. Manipulating the time controller at the bottom of the Chanalyzer window enables me to focus in on a specific area of time for a clearer picture.

The Chanalyzer/Wi-Spy Adapter products are some of the sexiest products that have come around in the industry in a long time. Displaying the live RF signals around me prior to making a presentation at a conference is like wearing a hot pair of steel stilettos. Attention-getting and very sexy (in a sick and twisted geeky way).

We've now partnered with the Metageek folks on the upcoming WLAN Analysis 101 course on September 10th - if you purchase the 2.4x or DBx Wi-Spy adapters, you'll get into the live class for free. If you already own their products, you should receive a 50% off coupon via their newsletter. As soon as we record the course, you'll also receive one-week unlimited access to the recorded course.

It's a good time to get the adapter... c'mon... you know you want one! You can order the products at www.metageek.net.

Laura
Enjoy life one bit at a time!

Ethereal is Dead!

Gerald Combs created Ethereal over 11 years ago when his boss wouldn't buy him a brand spanking new Sniffer box - something about budgets and all... so Gerald told his Sniffer rep that he was going to write his own packet sniffing tool. While that Sniffer rep was still rolling around laughing, Gerald started working on Ethereal.

The name? Yeah - the name Ethereal was always an issue - how do you pronounce it? Ethereal (play wav) or Ethereal (play wav)? Many a late night has been spent huddled over pizzas in the cabling closet debating that issue. The answer - Ethereal (play wav).

Notes:

• Download the latest version of Wireshark
  
• Watch a video on how to set up the GeoIP feature at SecuityTube.
  
• Peruse through the videos, trace files and podcasts on our media roll.
  
• Register for the live or recorded Wireshark Jumpstart course.
  

It surprises me to find many folks haven't moved up to Wireshark - it is, after all, the successor to Ethereal. The same developers, the same creator, the same base code set, the same development directory structure. I can only assume those folks also have 8-track tape players and beam with pride when talking about their 'vinyl collection.







For fun, I went to visit the old ethereal.com website - I thought the old Ethereal website was taken down ages ago, but imagine that NIS is still reaping some benefit from all the misguided hits. Looking at the stats in Alexa was pretty interesting - you can see the dramatic move to Wireshark at the end of the first quarter of 2008 - but what the heck is happening with Ethereal.com in 2009?















Why are people still even hitting that site? Is everyone writing a blog entry about 'dead' software projects? Did some of my old articles and courses get reissued? Who are these Neanderthals walking among us?

It's time to upgrade to Wireshark folks. Wireshark v1.2.1 was released just a few weeks ago and fixed numerous bugs in the v1.2 release. There are still a few uglies in there, but would you rather be in a car that has a window that slowly rolls up or take a bicycle on that long drive along the network analysis road?

So perhaps today is the day to throw away those old bell bottom jeans and that mood ring (and perhaps dump those Shaper Image gift cards and Clear cards as well).

Come on - get with the times! Oh... one more thing - and you pronounce Wireshark like this (play mp3).

Laura
Enjoy life one bit at a time!

Out of Sight, Out of Mind?

Embedded OS Security Issues
This month seems to be "medical industry month" around here. My email has been loaded up with various hospitals and medical facilities. One of the topics that is hot right now is 'embedded OS' security issues. For example, the three devices shown in the image above all contain Microsoft embedded operating systems - Windows Embedded CE. (See http://www.microsoft.com/windowsembedded/en-us/default.mspx)

How many hosts on your network support an embedded OS? Is the vendor keeping those hosts up-to-date with patches and security fixes? An interesting question... this is a great reason to run OS fingerprinting against the range of IP addresses supported on your network (with permission of course) to find out where the addressable devices are. Listen to the network traffic and check out the endpoint listing that Wireshark provides. Any unusual devices around?

Some of our office printers have embedded OSes in them and can tell you they've never been updated by the vendor. What outdated OS is hanging around on those boxes? We're tapping into the nets now and doing some OS fingerprinting to see what we're up against - I suggest you do the same!

Laura
Have fun one bit at a time...

One Key Sign of QoS Problems

There are some trace files that SCREAM at you! If you stand too closely you can feel spit hitting your face!

In the "Top 10 Reasons Your Network is Slow" online course (course abstract), we examine one of the causes of slow network performance. We look at a trace file of traffic that has passed through a router set up with QoS. You may not be aware how obvious QoS issues can be when analyzing traffic - feed a nice steady stream through that puppy and catch the traffic on the other side to see how it performed its duties.

Look for an EKG Pattern
In a datastream that is 'steady' - as in the video streaming example shown in the picture, we look for an "EKG pattern" in data coming through the router. This pattern is seen when data is held in the queue temporarily and then released (causing the sudden jump in the IO). As you can see in the image above, we can also spot packets that are droped by the queue. (Make sure you take a trace on the other side of the router to compare the IO graphs - you want to be certain a steady stream of data is traveling towards the QoS device and any alteration in the IO pattern has not already occurred.)

Get the Trace File
Go ahead - try checking it out yourself. Open up mcaststream-queued2.pcap in Wireshark. Select Statistics > IO Graph.

What? It's not screaming at you? Aha! That is because the X axis is too large - you are looking at ants from space! Change the X axis value to 0.01 seconds.

SCREAM!!!!
Do you see it? Right around 1.10 seconds into the trace - the EKG pattern! If users are not complaining about performance then dont' sweat it. Keep an eye on times when the line drops and doesn't jump up above the average point - those are dropped packets.

I'll be teaching the "Top 10 Reasons Your Network is Slow" on July 30th - it's a fun class to teach (although last time I was demonstrating the process of jamming a wireless network and nearly killed my own seminar hosting connection - duh). Register here.

Enjoy the trace! See you online!

Laura

Brad and the Top-Secret Bl-Ear Project

Brad Pitt on the cover of wired poo-pooing the bluetooth look? No way! They aren't going pre-announce an invention that I already pre-announced at TechEd?! I quickly blew through the pages of Wired Magazine's August issue to find a picture of Brad texting at the urinals with a bourbon close by (page 89).

Whew! No mention of the Bl-Ear - the exciting beta-phase invention in bluetooth beauty and buffness. It's tough to stay ahead of the game (and game mags) in technology. Sometimes you have to be... well... inventive.

Let's face it - there are tons of products we'd love to see out there - the Bl-Ear fills a need to reduce the high Nerdlook-Factor (NF) of walking around with that bluetooth device hanging off your head - don't even start spewing the "jawbone is sexy" defense with me. No one (not even Brad) looks good with electronics hanging off their aural lobes.

Bluetooth devices are the new pocket-protectors, folks. And you need to admit it.

As you may have missed the TechEd presentation in May, I've put up a short video showing the Bl-ear over at the Chappell Seminars Projects page.

Before you go out the door today, look in the mirror. Laptop - check. iPhone - check. Starbucks card - check. Bluetooth adapter - check. Now remember - accessorize, then minimize - take off the ear-tech that screams "I hope someone wants to talk to me today".

Sign up for the Bl-Ear and watch your NF drop to near-normal levels. Oh... and just wait 'til you see their upcoming Ear-Bluds! I can hardly wait.

Laura
The Bl-ear and Blear Corporation are bunk. All rights reserved.