Tim – the network tool man – Taylor
First Published Thursday, 16th July 2009 12:32 am from Real-Time Innovations (RTI) : Jan Van Bruaene
The opinions expressed by this blogger and those providing comments are theirs alone, this does not reflect the opinion of Automated Trader or any employee thereof. Automated Trader is not responsible for the accuracy of any of the information supplied by this article.
When I moved to the US a decade
ago, href="http://en.wikipedia.org/wiki/Home_Improvement">Home
Improvement was a popular television sitcom, starring
Tim Allen as Tim the tool man Taylor, a host of a made up home
improvement show.
Among the support team, when
we learn about a new debugging tool, we frequently joke about
'our network debugging tool belt'. Over the
years, this tool belt has grown to a nice variety of debugging
tools, especially when you include platform specific debugging
tools. We currently support a great variety of platforms: from
the enterprise platforms (Linux, Solaris, Windows) to various
embedded platforms, (including VxWorks, LynxOS and
Integrity).
Below is a list of the tools we
frequently use on support. I am leaving out platform specific
tools, tools to monitor how a particular network stack is
processing network traffic (e.g href="http://en.wikipedia.org/wiki/netsta">netstat)
and general operating system debugging tools ( href="http://en.wikipedia.org/wiki/Vmstat">vmstat,
cpustat, href="http://en.wikipedia.org/wiki/Lsof">lsof, href="http://en.wikipedia.org/wiki/Gdb">gdb, href="http://en.wikipedia.org/wiki/Strace">strace
etc.).
One of the
most common questions on support is: why are my two DDS
applications not communicating? The answer to this
question falls into two categories: it is either a general
networking / connectivity problem, or it is a configuration
problem. Let's take a look at the various tools we use
to debug.
Tools to verify
connectivity
1. href="http://en.wikipedia.org/wiki/Ping">ping
verifies whether you have basic connectivity between two hosts.
Not commonly known is that ping can also be used to generate
multicast data. This is especially useful when your DDS discovery
is using multicast and you are suspecting a multicast or href="http://en.wikipedia.org/wiki/Time_to_live">time-to-live
(ttl) problem.
For example, you can
try to ping 224.0.0.1 which is the all-hosts group. If you ping
that group, all multicast capable hosts on the network should
answer, as every multicast capable host must join that group at
start-up on all it's multicast capable interfaces.
There is one gotcha, by default when running a Linux kernel 2.6,
the capability of ICMP to reply to broadcast/multicast ping
message is disabled. Whereas, it is enabled by default in Linux
kernel 2.4. You can enable it by disabling
icmp_echo_ignore_broadcasts (reference:
href="http://kerneltrap.org/node/16225">http://kerneltrap.org/node/16225)
If you want to experiment with the time-to-live value
many ping implementations allow you to configure this via -t or
-T.
2. href="http://en.wikipedia.org/wiki/Traceroute">traceroute
(unix) or href="http://en.wikipedia.org/wiki/Tracert">tracert
(Windows) is a tool to visualize packet path used when sending a
packet. It shows you the different hops / routers a packet
traverses.
3. rtiddsping
is a command line utility found in the scripts directory of your
RTI Data Distribution Service installation. It basically does the
same thing as ping, but not by using the ICMP protocol, but by
using DDS. If you were able to ping the hosts and wonder whether
the issue is a problem in how your DDS application is configured,
then rtiddsping is a neutral application to
verify whether the DDS ping /pong topic is working. Take a look
at the various flags to modify the peerlist, ttl, multicast and
reliability qos parameters.
Tools to analyze the data on the
wire
4. One of my favorite network
tools is probably href="http://www.wireshark.org">Wireshark.
Wireshark, previously called Ethereal, is a free packet sniffer
and allows you to inspect the data on the wire. It will decode
the packets based upon various protocol dissectors. The DDS wire
protocol is commonly known as RTPS - real-time publish
subscribe wire protocol. RTPS dissectors in earlier versions of
Wireshark supported an early non-standard version of the
protocol. RTPS2 is the dissector which supports the href="http://www.omg.org/technology/documents/dds_spec_catalog.htm">OMG
standard of the wire protocol. RTI contributed the
source of this dissector to the Wireshark community. We also
offer it as part of our href="http://www.rti.com/products/developer_platform/#PROTOCOL.">RTI
Protocol Analyzer product. If you are using RTI Data
Distribution Service 4.2 or later, use the RTPS2
dissector.
We use RTI Protocol Analyzer to
debug discovery misconfiguration. You can observe which IP
addresses are used during discovery announcements or which ports
a particular node is contacting. You can also look at whether the
reliability protocol needs to repair samples and how many. You
are able to filter on Hearthbeats as well as corresponding
ACK/NACK packets. Lastly the nice thing about it is that you can
save it to a file to share with the RTI support team. We can
partake in the analysis of why something isn't working
as expected.
Make sure you are monitoring the
data both at the send and receive side. RTI Protocol Analyzer can
only capture what is on its wire. If you can not install RTI
Protocol Analyzer on the publisher or subscriber, hooking up a
simple hub is a good option. A switch, unless you enable port
mirroring, won't do the job. (Doh, where did I leave my
old hub now?)
5. href="http://en.wikipedia.org/wiki/Snoop_(software)">snoop
(Solaris) is a packet sniffer for Solaris.
Unfortunately it does not dissect the RTPS protocol.
6. rtiddsspy is a command line
utility, found in the scripts directory of your RTI Data
Distribution Service installation, which allows you to
automatically subscribe to publications in a specific DDS domain.
You can observe both the discovery data as well as the user data.
Take a look at some of the command line options to filter out
particular topics.
7. href="http://www.rti.com/products/data_distribution/rtirecorder.html">RTI
Recorder is a more complete data recorder. It can be
configured to record specific topics and fields to a database
which lateron can be queried using sql commands, or can be
converted into XML, HTML or Excel. Since we often do not have
access to your data source, this is best used in your test lab.
You can then share the database file or converted file with us.
The nice thing of recorder is that it records a lot more that the
user data: also meta information such as time stamps are
recorded.
Tools to analyzer the
state or configuration of the system
8. We use href="http://www.rti.com/products/developer_platform/#ANALYZER">RTI
Analyzer when we suspect a configuration issue: e.g. a
quality of service (QoS) mismatch. Another good use case is when
you have typo in the topicname. DDS will consider this a
different topic. Analyzer will quickly show you all the topics in
your system, as well as the qos parameters used by the different
DDS entities (reader, writer, publisher, subscriber). The match
analysis shows you in one screen which reader/writers are
compatible.
One common misunderstanding is
that Analyzer can not show you all quality of service paramters
of a specific entity. For example resource limits or batching
configuration are not shown. This is because those are local
configuration parameters and are not sent out during discovery.
If the configuration is not sent out during discovery, RTI
Analyzer does not know about it.
If you want
to want to verify whether you configured your batching correctly,
I recommend to use RTI Protocol Analyzer as it will distinguish
between regular data and batched data samples. It will even show
the indivual batched samples.
href="http://feeds.wordpress.com/1.0/gocomments/rtidds.wordpress.com/116/">
alt="" border="0"
src="http://feeds.wordpress.com/1.0/comments/rtidds.wordpress.com/116/"
href="http://feeds.wordpress.com/1.0/godelicious/rtidds.wordpress.com/116/">
alt="" border="0"
src="http://feeds.wordpress.com/1.0/delicious/rtidds.wordpress.com/116/"
href="http://feeds.wordpress.com/1.0/gostumble/rtidds.wordpress.com/116/">
alt="" border="0"
src="http://feeds.wordpress.com/1.0/stumble/rtidds.wordpress.com/116/"
href="http://feeds.wordpress.com/1.0/godigg/rtidds.wordpress.com/116/">
alt="" border="0"
src="http://feeds.wordpress.com/1.0/digg/rtidds.wordpress.com/116/"
href="http://feeds.wordpress.com/1.0/goreddit/rtidds.wordpress.com/116/">
alt="" border="0"
src="http://feeds.wordpress.com/1.0/reddit/rtidds.wordpress.com/116/"
/>
src="http://stats.wordpress.com/b.gif?host=blogs.rti.com&blog=7350090&post=116&subd=rtidds&ref=&feed=1"
/>
