Ad-hoc Interoperability Test: An Interesting Experiment Indeed

On Tuesday, August 17, the UCIF held an ad-hoc interoperability test, involving multiple protocols and modes of communication. This included testing of audio, video, Web conferencing and instant messaging, presence and multi-user chat interoperability. It was an interesting experience indeed!

Thank you for all those who participated. We ended up attracting many individuals on multiple continents; speaking different languages written in Latin, as well as non-Latin scripts such as Chinese, Arabic, Hebrew and Greek. Since this was ad-hoc, there was no formalized agenda or comprehensive testing plan. The tests were solely based on the interests and curiosity of the participants. Given the background of these attendees, a considerable amount of time was spent in testing internationalization; a topic that we realized needs much more attention in the UC community.

We set out to get a general sense of where interoperability is today. Imagine a scale that goes up to 10. Where do you think we are even with all the new technologies and breakthroughs from the past few years? Would it be different for audio, video, Web conferences or instant messaging? We initiated this experiment to find out the answers to these questions, and quite possibly raise other ones.

 

Testing Example

What Was Tested?

• Audio: An audio bridge (hosted by a major carrier) was available and was tested using multiple VoIP implementations. Quality was very good. Audio was also available on the Web conferencing platforms, though audio quality was noticeably worse than on the bridge. The best audio quality was observed during a successful Jingle audio test (wideband codec was negotiated). The biggest problem during the audio tests related to devices (sound cards, headsets and microphones), as well as volume adjustment and silence detection algorithms.
• Web conferencing: Two different Web conferencing systems were tested with various Web browsers on multiple operating systems: Windows, Mac, Linux Mozilla, Chrome, Safari, IE, Opera. One conferencing platform provided all features on all browsers and PC platforms (with a plug-in), but others had issues on some platform/browser combinations. None of the web conferencing solutions was usable on a mobile handset, though neither conferencing implementation claimed this as a feature. Issues were observed with parsing of some file formats (e.g. pptx vs. ppt).  In some cases cryptic error messages were generated and the presentations failed to display. Internationalization was tested to a limited extent; characters from non-Latin scripts were entered and displayed correctly.  Also most browsers now support IDNA [Internationalized Domain Names] to varying degrees.  Both Web conferencing systems appeared to be capable of sending and receiving basic audio and “postage stamp” video. Support for mobile platforms is an area which appears to require more attention from the industry.
• Instant Messaging, Presence and Multi-User Chat (MUC): A wide range of XMPP clients was tested on multiple OS platforms and services. Basic IM&P operations such as subscription, rosters and instant messaging appeared to work satisfactorily, but some discovery and setup operations could not be carried out by all clients (e.g. creating a new conference and setting conference parameters). In a number of situations, security settings required adjustment before connectivity could be achieved. This resulted in some cryptic error messages. MUC was not completely supported on some clients (e.g. one client could accept an invite to a multi-user chat, but could not send an invitation or join from the native UI).
• With respect to internationalization, some basic operations were successful. For example, we were able to create a user with a username and password in Arabic, and were able to login to that account by cutting and pasting the username and password. However, since the username involved a right-to-left script and the domain name involved a left-to-right script, the username/domain name combination did not always display correctly; at times the domain name and user name were reversed. Some major issues were discovered, such as inability to correctly display or enter characters on some implementations. In some cases, cut/paste of non-Latin characters worked, but attempting to enter them on a keyboard did not. There is no question that internationalization is an area which appears to require more attention if the UC industry is to reach its potential.
• Jingle: Interoperability of Jingle was tested between multiple XMPP clients across multiple operating systems and services. Several failure modes were observed, including inability to negotiate video codecs, and in one situation, Jingle sessions could only be initiated in one direction, but not the other (potentially due to NAT traversal issues which could benefit from more testing of protocols such as STUN/TURN/ICE). Jingle audio operation was demonstrated in one direction with excellent sound quality between multiple platforms, services and clients, but attempts to upgrade to video generated unusual failure modes (e.g. video appeared to be sent but not received, receiver received no indication of video being sent, sender received no indication that video was not being received). Jingle appears to be a feature that could benefit from an increased focus on compliance and interoperability.

 

Observations:

Remember that scale of 1 to 10? Our results varied quite a bit, ranging from 2 to 8 depending on the implementation. Here are the overall findings:

• Internationalization is an area that appears to require more attention if the UC industry is to reach its potential. The issues encountered (such as the mixing of RTL and LTR scripts) appeared to involve more than implementation bugs; some standardization work will probably be necessary. 
• Mobility is an area that also requires more attention. The rapid growth in smartphone shipments implies that a significant fraction of UC endpoints will be mobile. Yet, Web conferencing solutions can be difficult to utilize on a mobile device or may be dependent on plug-ins not yet available on popular mobile platforms as well as browser compatibility.
• Audio, while generally more mature than other UC modes, still has it challenges. We encountered issues with devices such as headsets and microphones.   
• Instant messaging and presence clients may not necessarily support the entire set of core capabilities needed to carry out common tasks. Profiling the required capabilities may therefore provide value. Multi-User-Chat could benefit from an “end to end” approach to testing, to make sure that all elements required for a decent user experience are available.
• Jingle, though promising, requires additional interoperability testing. 

 

Check out the No Jitter.com interview on this topic with Bernard Aboba of Microsoft and chair of UCIF Board of Directors.

If you want to know more about UC interoperability or more testing events by the UCIF, fill-out the Interest Sign-up Form.