Patent Application
20140016487
1. Technical Field
The present invention relates to a configuration of a test system to evaluate impairments, including losses, errors, noise and jitter, in a network wireless communication signal to enable estimation of degradation in voice or video quality.
2. Related Art
Several test system configurations exist to estimate the quality of uplink and downlink media connections or uplink and downlink media links of networked devices, the network devices including fixed, nomadic and mobile phones, tablets, notebooks, laptops and computers. The systems provide estimates for speech and/or video. The connections tested may be dedicated circuit-switched connections or may be Voice over IP (VoIP). Existing test methods may base the estimate of quality on information about connection characteristics, such as delay, packet loss and jitter. An example of a system that uses this type of estimation is the Anritsu CMA 3000 All-In-One Field Tester for Fixed and Mobile Networks.
These conventional methods base the estimate on a comparison of speech or video media before and after the media have been passed through the test system from a sending user to a receiving user. An example of a system that uses this type of estimation is the Anritsu ME7834 Mobile Device Test Platform. This platform emulates a wireless network and an air interface to provide realistic testing of advanced mobile User Equipment (UE).
In a conventional test system shown in
To make the comparison, typically speech or video degradation was estimated by a group of people who listen to the speech or look at videos to form their opinion about the quality. The people rated the quality on a scale of 1 to 5 and the average of the scores was calculated to get a Mean Opinion Score (MOS). Currently, standardized software and algorithms are available to derive the MOS. Example software tools are PESQ—which stands for “Perceptual Evaluation of Speech Quality” (standardized by ITU-T Recommendation P.862); POLQA—which stands for “Perceptual Objective Listening Quality Analysis” (standardized by ITU-T Recommendation P.863); PEVQ—which stands for “Perceptual Evaluation of Video Quality”, the Peak-Signal-to-Noise-Ratio (PSNR) algorithm; and the structural similarity (SSIM) algorithm.
In addition to testing speech and video quality in the laboratory, operators and equipment vendors also want to test the media quality in the field, preferably using a large number of UEs at a wide variety of locations. Media quality testing in the field is much harder, because it is non-trivial to access both the Reference Media and the Degraded Media at a single location. Several topologies have been used, and a few examples are given below.
It is well known that reference media files provided by subjects with different dialects or speech patterns can produce rather different MOS values. Apparently the MOS depends on subtle properties of the subjects used to generate the reference files. Thus, there always is a possibility that the MOS that is measured for one user does not correspond to the MOS measured for a user with a slightly different speech pattern, dialect or intonation.
Embodiments of the present invention provide a test system to test the quality of the voice and video connection of a large number of mobile phones or laptops from a central location. The quality can be expressed as a MOS and the uplink and downlink connection or link can be separately estimated. The system allows both audio speech phonemes that are characteristic of the end user and video data that are characteristic of the end user to be measured at the time that the user typically makes calls.
Embodiments of the present invention in particular provides one or more of the following features:
(1) The test system provides independent measurements of the quality of the downlink media link and the uplink media link.
(2) The MOS values determined by the test system are determined at times and locations that are representative for the times and locations where the user typically makes calls.
(3) A test system that is provided with a single license for media quality evaluation can evaluate the media quality for a large number of UEs.
(4) The user equipment (UE) for which the link quality is measured, which includes an end-user's own cell phone, tablet computer, or other transmission/reception device, only requires a minimal amount of modifications and remains fully available to the end-user.
(5) The system executes an evaluation with a minimal amount of time to reduce the impact on the end-users.
(6) The MOS values provided by the system can be determined from a wide variety of reference files, and the reference files can be representative of the language and speech patterns of the end-user and the people with whom the end-user communicates.
Further details of the present invention are explained with the help of the attached drawings in which:
The coupling between the TS 2 and Wireless System 4 can be very tight, as in a solution where the TS communicates directly with infrastructure elements like a Packet Data Gateway (PDG) and a Policy and Charging Rules Function (PCRF). The PDG provides packet data access to the UE, which in turn allows for the setup of data bearers for calls and for the downloading and uploading of data files. The connection to the PCRF allows the TS to get the necessary Quality of Service (QoS) for these data bearers.
The coupling between the TS 2 and the Wireless System 4 can also be rather loose. For example, the TS can be an Application Server in an IMS system and communicate with the Wireless System 4 via a Serving-Call Session Control Function (CSCF) and a Proxy-CSCF. In this case the TS 2 can still obtain the necessary QoS by communicating with the PCRF via the P-CSCF. The setup of the calls and packet data connections can use the Session Initiation Protocol (SIP) but may also use other methods.
Note that
For
For
Further for
The DUT 6 is programmed to store each downloaded file in such a way that it can later be retrieved. When multiple files are downloaded, the files are indexed or named such that a specific file can later be specified and selected. The downloaded samples are referenced as “U_R”, because they will be used by the TS 2 and the DUT 6 as the Uplink Reference sample. Storage of the sample U_Ri in the DUT is illustrated by box 58.
To start a test session, the Test System 2 initiates a call to the DUT 6 as illustrated by line 56. The call setup can be fully conventional, and no special provisions are needed in the communication system to establish the call. As part of the conventional call setup the communication system will establish uplink and downlink connections or bearers for the call that have the Quality of Service necessary to support the transmission of voice and/or video, as appropriate for the call that is being set up. Establishing the QoS is illustrated by line 57. Typically the QoS is established automatically as part of the call setup procedure. For example, the Gateway 50 or a Proxy-CSCF may request the QoS for the call via a PCRF 52.
The DUT 6 is programmed to recognize the call as belonging to a test session. The DUT 6 may do this by recognizing the caller-ID for the call as the ID of the TS 2. Alternatively, if SIP is used for call setup, there may be test-session-identifying information in one of the call setup messages for example in a SIP message header or in a SIP message body. When a DUT 6 recognizes the call as belonging to a test session, the DUT 6 preferably auto-answers the call. It suppresses all notifications to the end-user; i.e. the DUT 6 does not ring or vibrate when a test call comes in. That way, the test session can proceed without bothering the end user 7 (as illustrated by the “No ringing” label in
Note that in an alternative embodiment the test call can also be initiated by the DUT 6. For this variation the phone number or the Uniform Resource Indicator of the TS 2 is programmed into the DUT 6. The DUT 6 can then initiate a call to the TS 2 at any convenient time. The TS 2 is programmed to interpret any calls from a DUT 6 as the initiation of a test call.
The DUT 6 stores one or more samples and the test session setup may explicitly specify one of the samples, for example by including the sample file name in a SIP message header or in a message body. The session setup may also implicitly specify a media sample, for example by using the convention that media samples will be used in the order that they were downloaded. In this case the DUT 6 will use the media sample that follows the one that was used in the previous test session, and recommence from the first sample after the last one has been used. If the DUT 6 downloads only a single media sample as illustrated by line 55 in
When the DUT 6 recognizes the call as belonging to a test session it will play the specified Uplink Reference sample U_Ri over the corresponding uplink connection or uplink bearer as uplink media, as illustrated by line 60. Thus, the setup of a call 56 by the TS 2 and the specification of the sample will cause the DUT 6 to play uplink media corresponding to the specified uplink reference sample over the uplink connection. In case the sample contains both voice and video, the DUT 6 may play the sample over a single uplink multimedia bearer or play it over two uplink bearers; one bearer for voice media and one bearer for video media.
The communication system will automatically route the uplink media to the TS 2, because the TS 2 is the other party in the call. The media may suffer impairments at the DUT 6 or in the communication system link and these impairments may affect the media quality. In any case, the TS 2 will capture the uplink media stream and store it as a Degraded Uplink media sample or uplink degraded sample U_Di as illustrated by box 62 in
If the Test System has downloaded and specified multiple Uplink Reference samples to be played, the DUT plays each of the specified samples in the order they are specified. The TS 2 will capture each sample as a Degraded Uplink media sample U_Di. In
When the media for the test session have been played, the TS 2 or the DUT 6 end the call in the conventional way as illustrated by line 68. The TS 2 compares the captured sample(s) U_Di with the reference sample(s) U_Ri to estimate the media quality of the uplink as shown by box 72. The TS 2 may use standards-compliant software for the estimation, such as PEQ, POLQA or PEVQ and may express the result of the comparison as a Mean Opinion Score (MOS).
Also as illustrated by
The TS 2 may play and deliver multiple downlink samples during a single test session and the DUT 6 may be programmed to record all samples and upload them in the order they have been previously downloaded 55 or use another mechanism to specify the individual samples.
The Test System 2 compares the received sample D_Dj with the reference sample D_Rj to estimate the media quality of the downlink as illustrated by box 72. The TS may use software for the estimation, such as PEQ, POLQA or PEVQ and may express the result of the comparison as a Mean Opinion Score (MOS). Link quality estimation may be done before or after the call has been ended. For example, the TS 2 may defer the ending of the call until the uplink quality and/or the downlink quality have been estimated.
As shown in
Further, although
Test calls tend to be short because it may take only 10 or 15 seconds to play or deliver a media sample. To minimize end-user impact in one way, the call duration can be limited by simultaneously playing and delivering the uplink and downlink samples. This limits the impact on the end-user 7. However, there remains a possibility that the end-user 7 initiates or receives a call while a test call is in progress as illustrated in
Since the DUT does not notify the user at the start of a test call, the end-user typically is not aware that a test call is in progress. As a result, an end user may want to start a non-test call while the DUT is busy in the test call. In this case, the non-test call would fail. Steps to minimize user impact when a non-test call is initiated are illustrated by
Once a third party call interrupts a test call, the connectivity between the DUT and a third party continues as illustrated in
In the embodiment illustrated in
Once the bearer with the correct QoS has been established, the DUT plays 60 the specified Uplink Reference sample(s) U_Ri which are recorded by the TS as the uplink degraded samples (U_Di) 62 and/or the TS plays or delivers 64 the downlink reference sample(s) U_Di and which are recorded by the DUT as the corresponding downlink degraded media samples D_Dj 66. The DUT uploads 70 the recorded sample(s) D_Dj. This may implicitly end the session, or the session may be explicitly ended by the TS or the DUT by sending a message or a BYE 68. The TS must take care that the bearers are released, or the bearers may be automatically released upon expiration of a timer. As before, the TS compares 72 the reference and degraded samples to estimate the media quality of the uplink and/or downlink.
Note that the above description assumes that the TS does all the work to estimate the MOS values (i.e. comparing of samples). This assumption is being made because the estimation software provided by an outside service is expensive. However, it is also possible to let the DUTs do some or all of the comparing if the DUT includes MOS estimation software. The TS would have to download relevant reference and/or degrade samples. A DUT can then communicate the MOS results to the TS (for example in a SIP MESSGE or a text message).
Embodiment with User-Specific Samples
Accordingly, with the embodiment illustrated in
Media samples that are to be used for link quality tests need to have a high Signal to Noise Ratio (SNR). Therefore the DUT 6 may be programmed to collect the media fragments only while it measures a good SNR for the media fragments, for example when the user speaks relatively loudly from a relatively quiet environment, or when the user takes a video under well-lit conditions and with a steady hand.
The TS 2 may use said created user-specific reference samples U_Ri to test the uplink of the user's DUT 6. This makes sense, because this user-specific speech likely will appear on the uplink. In a further variation, the TS 2 learns the identities (e.g. phone numbers of URIs) of third party's UEs 82 with whom the user often communicates. The TS may learn this from the service provider's records, or the TS may obtain such information directly from the DUT, for example, if the DUT is programmed to send such information to the TS 2, e.g. in a SIP MESSAGE. In this variation, the TS checks if it has collected user-specific uplink samples U_Rx from any of the third party UEs 82 and uses such user-specific uplink samples to construct or create downlink reference samples D_Rj to test the downlink quality for DUT 6 (and vice-versa). The use of the third party's user-specific uplink samples U_Rx has the advantage that these samples have been collected by the third party peer devices before they were transmitted over a call connection 104 and should thus have a good fidelity or quality.
In another variation, the DUT is programmed to collect 105 media fragments (speech or video) that is spoken or filmed by the third party in a call and transmitted over the downlink connection of the call 104 from the third party 82 to the DUT 6. As above, the DUT 6 or the TS uses 112 these fragments to construct media reference samples. The TS can then use these samples (D_Rj) to test the downlink of the DUT. The DUT preferably collects the media fragments while there is a high SNR on the downlink connection with the peer device and while there is little background noise in the call. The DUT can thus collect the fragments while there is a good downlink connection for the user call and while the downlink bearer uses a codec with a higher bit rate which provide better quality media.
Embodiment with User-Specified Test Session Timing
While the above variations provide test media that is more relevant to the DUT 6, there is no correlation between the time at which test sessions that take place and the times at which the user of the DUT 6 typically makes calls.
Thus, in one embodiment a variation is provided where the DUT is programmed to initiate a test call with the TS at the end of a user call. When the application in the DUT observes that the user hangs up, or when it observes a SIP BYE, the DUT initiates a test call to the TS to exchange test media samples as described above. This way a tight correlation can be established between the times at which the user makes or receives calls; and the times at which MOS values are estimated. This will automatically also correlate the call locations. (In a variation of this embodiment, the DUT can monitor the locations at which the user makes calls and later initiate test calls with the TS when the DUT returns to those locations.)
The variation of this user specified test session timing may introduce a slight bias. Namely the user may sometimes end a user call because the connection becomes very bad. This bias would increase the number of MOS estimations under less favorable conditions. Some people may argue that that is a desirable bias; others may see it as detrimental.
In one embodiment of the present invention, the above-mentioned bias can be removed. In this variation the DUT is programmed to observe whether the user initiates or receives a user call. When the DUT notices the start of a user call, it sets up 92 a generic test session with the TS. The test session and the user call overlap in time, thereby establishing perfect correlation both in time and in location.
For each of the above embodiments, the system of the present invention will include conventional components to enable the tasks to be accomplished as would be understood by a person of ordinary skill. For example, both the TS and the DUT can include a communication transmit/receive interface, a processor for controlling the communication interface, and a memory for storing software to control the processor, the interface and programming necessary to accomplish steps performed in the present invention.
Although the present invention has been described above with particularity, this was merely to teach one of ordinary skill in the art how to make and use the invention. Many additional modifications will fall within the scope of the invention, as that scope is defined by the following claims.
