The Game of VoIP

I always get that tingly excited feeling when I open up new trace files.

It's not unlike the feeling I have as I pick up the pile of cards dealt to me in a game of gin. What will the cards hold? What will I see in them? That is what analysis is to me - a card game. Last week I began my slide preparation for an upcoming set of projects related to VoIP analysis. I find VoIP traffic fascinating - its duality - signaling separate from voice - its trusting nature - using UDP - its frailty.

This VoIP project came up first in preparation for the VoIP labs for the Summit 09 training event. I wanted to include a lecture portion dealing with SIP and RTP behavior and point out the cool new organization of telephony-related options in Wireshark. I wanted to build some hands-on labs to allow students to identify the likely points on a network that they'd look to isolate problems with the calls. I'm energized with the process of preparing new lectures, new trace files and new hands-on exercises for Summit 09.But Summit 09 had to go on the back burner for a bit... another hand had been dealt - by Microsoft.

The Microsoft Webinars (Oct 6th and Oct 28th)
I have presented two webinars for Microsoft to date - one wildly popular (where we pulled in Microsoft's largest registration and attendance counts) and the other nearly empty (seems the old Microsoft marketing engine had stalled on that one and we didn't push it because it was open to partners only). If you want to join me at the Summit 09 event on December 7-9th, check out the Summit 09 Information Document at www.chappellseminars.com/summit09.html.

No RTP? A Lousy Deal?!
When I began systematically opening the VoIP files in my folder, I hit the first stumbling block of VoIP analysis. One of the first trace files I opened up did not contain the SIP packets for a call setup so Wireshark didn't apply an RTP dissector to the voice traffic - it just dumped me off at UDP. Hmmm... seems I have a hand with no runs or matches.

I've talked about the process of "Decode As" in the past for interpreting traffic traveling over non-standard ports. The same steps are quite common to use with VoIP traffic. I right clicked on a packet in the trace file and selected Decode As. On the right side, I chose one of the transport port numbers and selected RTP in the protocol list.

Voila! It's decoded as RTP! Another option is to enable the RTP Preferences "Try to decode RTP outside of conversations". The image at left shows the traffic of an RTP stream that Wireshark decodes as just UDP. Wireshark didn't see the SIP call setup process and it decodes the voice traffic as just UDP.



Viewing the RTP Streams
When Wireshark reached version 1.2, the menu bar was revised to include a Telephony item. Once you force the RTP decode (if required), you can select Telephony > RTP > Show All Streams.

Wireshark detected two RTP streams in my trace file once I forced the RTP decode on the traffic.

As you can see from the figure at left, Wireshark can now show me the RTP streams in the trace file - one stream going in each direction. We can also see that one of the streams has a 2.1% packet loss. Packet loss is ugly in VoIP communications - when a call experiences serious packet loss, the conversation quality suffers.There's an example of a G.711 audio file with 10% packet loss at VoIPTroubleshooter.com.

Playing the Hand That Was Dealt
Unlike gin or poker, we have to play the hand we were dealt in network analysis. We can only make the hand stronger by filtering out traffic that is not relevant and interpreting the possible causes for the packets that remain.

In the area of VoIP analysis, my favorite filter is sip || rtp and my favorite feature is the RTP playback feature. I like to set the time column to Seconds Since Previous Displayed Packet and export the trace file to csv format.

In Excel I sort on the source column first, the number column second and then graph out the time column to show me the variance between packets (the jitter value). I could use Wireshark's graphing capabilities to give me some of this information, but this is one of the areas where a formal spreadsheet program really shines. I exported the trace file to a csv format to graph out a portion of the jitter values.

VoIP at Summit 09
I'm anxious to get back to the Summit 09 preparations - to document all the tips and tricks of VoIP analysis. I've put together a few lab exercises already, but I'm keeping my eyes open for new hands to play - if you have any VoIP traces that I could use in Summit 09, please send them to me (laura@chappellU.com). Download the Summit Information Guide. All Access Pass members receive a 50% discount to Chappell Summit 09.

Enjoy life one bit at a time.
Laura

Summit 09 Registration Opens

It's a Geekfest Training Event! Last year's Summit was a great success with a room full of folks hunched over their laptops for three-days of labs and training. Since that time I've been researching and developing new materials - they are ready to go into Summit 09. Summit 09 will be another BYOL (Bring Your Own Laptop) event - filled with hands-on labs. This time around we will offer both individual and group labs.



Hot Tools and Key Tasks
We will focus on which tools are best for each task, such as the following:

• traffic redirection and interception
• IP address sanitizing
• locating firewalled hosts
• throughput testing
• jitter measurement
• identifying blocked/filtered ports
• WLAN RF analysis

Troubleshooting Hot Spots and Testing
On day two, we will delve into a corporate network that is consistently garnering complaints. Working from the network diagram, we'll devise a plan to isolate the cause of network problems and perform proactive throughput testing and live RF analysis. We'll address both wired and wireless network problems.

Security and Forensics
On the third and final day, we focus on security. Starting with the TCP vulnerabilities and the SMB2 vulnerability announced this month, we will dissect the key issues and build a malicious/suspicious traffic profile. This profile will save you lots of time and help you spot security flaws on the network.

During the Summit you will be working on network diagrams to pinpoint where to capture traffic and follow the path of communications.The schedule is packed with the latest techniques for catching network problems, identifying suspicious traffic, testing network throughput, analyzing WLAN traffic and discovering network devices and services.

Download the Summit Information Guide. All Access Pass members receive a 50% discount to Chappell Summit 09. Enjoy life one bit at a time. Lauraw

WLAN Profiling: It's a Good Thing

Speed up your WLAN Analysis Processes

This weekend I recorded the WLAN Analysis 101 course (available to All Access Pass members already). I spent a fair amount of time customizing a profile for Wireshark to include columns, display filters and color filters focused on WLAN traffic.

What should your customized WLAN profile include?

Three Columns to Add
Consider adding columns for Frequency/Channel information, RSSI (Receive Signal Strength Indicator), and transmit rate. Once you have these columns added to Wireshark, you can sort on the columns and even use them when exporting to a spreadsheet program for further graphing.

Hot Display Filters
Add display filters to quickly view all traffic on a specific channel - for example, radiotap.channel.freq == 2412 will display all Channel 1 traffic. You might also want to create filters to display all beacons for a specific SSID. The syntax for that would be wlan.fc.type_subtype == 0x08 && frame contains "wsu" to see all the beacons related to the wsu WLAN network.

Wild Color Filters
Besides creating color filters for the various channels (which use the same syntax as the display filters), consider coloring traffic that has the retry bit set to 1 (for example, wlan.fc.retry == 1) , Probe Requests/Responses and Associations/Reassociations/Disassociations. I create 'butt ugly' color filters for frames that I consider a problem, such as retry frames.

Customized profiles can include your column settings, font settings and capture filters as well. You could create a custom profile for the corporate office, a specific client or a specific network type.Customizing Wireshark to fit the network you are working on helps you sort out the traffic and spot problems faster!

The "WLAN Analysis 101" course released to All Access Pass members on September 13, 2009. This course includes color filters, display filters, WLAN trace files, Chanalyzer recordings and a 29-page course handout. This course is available for individual purchase at www.chappellseminars.com/s-wlan1.html.

Enjoy life... one bit at a time.

Laura

Cloud Concerns

Are We Going to Look Back and Say..."What Were We Smoking?"

You've got a blazin' fast gig network running like Usain Bolt on Jolt. You've tweaked every inch of the network and spent hours explaining that you "don't control the Internet" when their web browsing sessions slow to a crawl because some blowhard ISP decided to cap your bandwidth. The winds have turned cold and a chill is in the air as management performs rain dances that bring ominous cloud computing towards the horizon. Now you need to watch for latency along the ISPs path to your servers... your data. What about packet loss dragging your throughput rates through the mud? Oh... don't even get me started about rate throttling! Makes me want to throttle someone! Let's ponder this case study... A customer of mine complained that communications between their offices seemed rather slow lately. They asked me to take a look at the traffic and evaluation the throughput rates. There seemed to be a consensus that some device was dropping packets along the path.

The IO graph screamed a story.The graph above is my rendition of what I'd seen that day - the trace files were to remain at the client.Do you see how that line seems to hit a ceiling? Oh yeah... a ceiling by the name of ***** (name hidden to avoid lawsuits). The ISP had promised them (and charged them)the world. In reality, however, they had put a cap on my customer's traffic that was way below what my customer paid for. Now... imagine that this is your corporate network and your service response times are dependent upon some ISP! Do you love your ISP that much?

Cloud computing brings up so many issues - the idea of putting the network health in the hands of the ISPs triggers my gag reflexes. Have you ever tried to work with an ISP to find out why your round trip latency times are so high? What are the bandwidth requirements of your apps? What will the end-to-end (client to cloud-based server) throughput be? Where will your data really be? What path will it take... today...? Who will pick up the phone when you call to say "the cloud is slow?"

Perhaps Lena Horne said it best... "Don't know why... there's no sun up in the sky... stormy weather... when my SERVER and I ain't together...". Lena knew networking! The Analyze and Improve Network Throughput and Top 10 Reasons Your Network is Slow seminars explain throughput testing and the most common causes of poor performance.

Enjoy life - one bit at a time...
aura

Do You Know Where Your Throughput is Today?

iPerf Might Scream "This Stinks!"

Last night I was laughing and crying at the same time while reading a local small town newspaper and the dissertation on bandwidth problems at the local library. They actually shut off half the computers to aleviate the problem. One quote really grabbed me - "We don't notice the bandwidth problem when no one is using the computers." Hunh?

There were a few puzzling comments in the article such as the comment that the bandwidth problem "started 3 weeks ago." That sudden onset of the problem feels like something else to me. It feels like a device along the path that is causing the problems. A quick look at the traffic would validate that.

The article tied in nicely to the iPerf Throughput Testing materials I am finishing up. The reaction to the live iPerf testing done during the Analyze and Improve Throughput course and numerous requests for more information on iPerf prompted me to start developing a course that shows how to perform a series of throughput tests for UDP and TCP traffic.

iPerf is simple and complex at the same time. The one application can be run either in client mode or server mode.

Tests can be run in one direction (from the client to the server, which is the default) or bi-directionally. Here is one of my favorite tests for iPerf:

Client: -c 10.1.1.1 -u -t 60 -i 5
Server: iperf -s -u -i 5

This test enables me to locate jitter and packet loss along a path using a UDP stream sent over a 60 second time with results displayed every five seconds at both the server and the client.As you can see from the screenshot above, the path suffers packet loss reaching 39%. We were sending a steady stream at 1.05 Mbit/second specifically to identify packet loss. Well... we found it. Running the test for a full 24 hours would help us identify specific times of the day when packet loss is at its highest.

I hope to get into the local library and run Wireshark and iPerf on the network soon. I have no idea if the systems are locked down - a slight problem that might require some workaround. Meanwhile I'll peruse the shelves for those network troubleshooting books.

The "iPerf Throughput Testing" modules will be included in the Summit 09 course in December (www.chappellseminars.com/summit09.html) - the "Analyze and Improve Throughput" course is available now at www.chappellseminars.com.

Enjoy life one bit at a time.
Laura