Be your support engineer's best friend
from Real-Time Innovations (RTI) : Jan Van Bruaene - 31st December 1969
The opinions expressed by this blogger and those providing comments are theirs alone, this does not reflect the opinion of Automated Trader or any employee thereof. Automated Trader is not responsible for the accuracy of any of the information supplied by this article.
In my previous post, I detailed two resources for answers to commonly asked questions or to look up specific known issues. Of course that requires you to at last know what to ask or look for. Before getting there you probably want to find out more about how the middleware is doing, or in the event of a defect, contextual information to provide to the RTI support team. Other than a reproducer, the following configuration, error and status information makes life for a support engineer so much easier.
RTI Data Distribution Service provides a great amount of information about the status of your real-time distributed system. Being able to collect and log this information is essential to start debugging a DDS based application. There are several ways to implement a logging system which has minimal impact on performance and does not require a recompile. Before we get to that, let's first take a look at what information is available and what will typically be requested by the support engineer.
The RTI middleware is configured through various Quality of Service (QoS) parameters. QoS parameters can be specified for different DDS entities: domain participant, publisher/subscriber, and writer/reader. Easy access to the QoS configuration of each DDS entity within an application is a must. At a minimum the list should include QoS parameters which have been changed from the default values. When using version 4.3 or later of RTI's middleware QoS parameters can be loaded from an XML file. The XML file is an easy way to change the QoS parameters at start up time, but also to share the configuration.
Part of how each DDS application is configured is how it will discover other applications and matching DDS topics. Gathering the following discovery information is key:
- What is my initial peer list of each domain participant, used to initially announce myself to other applications?
- What is multicast receive address used by various domain participants? Is multicast enabled in my DDS application?
- What transport are enabled (See: DDS_TransportBuiltinKindMask) and what are the transport properties used by each domain participant? These properties configure the UDP and shared memory transports at a low level.
Error logging and status information
By default RTI Data Distribution Service will log error messages using the NDDSConfigLogger class to the console. The default verbosity configuration is NDDS_CONFIG_LOG_VERBOSITY_ERROR. We encourage you to log this information to a file, and to allow the verbosity level to be modified either at run time or at start up to include warnings (NDDS_CONFIG_LOG_VERBOSITY_WARNING ) or overall status information (NDDS_CONFIG_LOG_VERBOSITY_STATUS_ALL).
DDS status information - DDS data readers and writers maintain status information which can be queried or can be made available through corresponding callback routines when a change in status occurs. This status information provides a great deal of insight into each entity. Not all of these status fields are indicative of an error. However they will help in debugging a problem or a mis-configuration. It is strongly recommended to instrument the corresponding callbacks and log changes to the following data reader and data writer status fields:
· DDS Data Reader status info includes: DDS_LIVELINESS_CHANGED_STATUS, DDS_REQUESTED_DEADLINE_MISSED_STATUS, DDS_REQUESTED_INCOMPATIBLE_QOS_STATUS, DDS_SAMPLE_LOST_STATUS, DDS_SAMPLE_REJECTED_STATUS, DDS_SUBSCRIPTION_MATCHED_STATUS
· DDS Data Writer status info includes: DDS_LIVELINESS_LOST_STATUS, DDS_OFFERED_DEADLINE_MISSED_STATUS, DDS_OFFERED_INCOMPATIBLE_QOS_STATUS, DDS_PUBLICATION_MATCHED_STATUS, DDS_RELIABLE_READER_ACTIVITY_CHANGED_STATUS
The DDS Data Writer DDS_RELIABLE_WRITER_CACHE_CHANGED_STATUS provides a summary of the state of a data writer's cache of unacknowledged samples written. Logging this continuously will likely have an impact on performance as the callback will be triggered each time all samples have been acknowledged. We recommend to implement the ability to turn this on at start up or at run time as needed.
DDS Data Reader and Writer Cache and Protocol status - Version 4.3e and later adds statistics information about what is going on at the protocol level. Making this information selectively available will speed up isolation of the problem. The protocol status provides information such as the number of samples pushed, filtered and the status of wire protocol traffic. The cache status provides information about the queue: e.g. number of samples in the reader or writer queue.