Patent Application
20070030815
The invention relates to data communication systems, and more particularly relates to a method and system for measuring end-to-end perceived data quality in wireless data communication systems.
With the introduction of wireless data networks it has become important to measure the end-to-end quality for the different types of services that can be used over a (wireless) data network, not only from a technical point of view but also from the point of view of the user, how does the user perceive the quality of these services. It is already common practice to measure the data quality of a network on the lower layers of the OSI model, for example on the physical, transport or network layer. But for the broad acceptance of wireless data networks, and for the resulting commercial success, it is necessary to assess how the user of these wireless services will perceive the quality of the service. To be able to assess the perceived quality based on tower layer measurements it is necessary to determine how lower layer measurements correlate with the perception of the quality.
The present invention relates to field measurements because these measurements provide the best correlation with user perception. The present invention particularly focuses on the end-to-end perceived quality for a service that will be broadly used on wireless data networks: downloading files using FTP (File Transfer Protocol). However, it will be clear to the person skilled in the art that end-to-end quality determination in other types of data transfer (browsing, audio/video data streaming, chatting) also may benefit from the present invention.
Tools for performing field measurements on the data quality on the lower layers of wireless communication networks are known, see for example US2002/0155816, which discloses systems and methods for measuring data quality in a wireless communication network. One such system comprises a means for obtaining information related to a location of the mobile wireless device within the wireless data network, a means for establishing a data call between the mobile wireless device and the wireless data network via an electromagnetic signal, a means for obtaining information related to the quality of data service associated with the data call, and a means for correlating the information related to the quality of data service with the information related to the location of the mobile wireless device.
A drawback of these existing data quality measurements systems is that these systems do not directly correlate the measurements on the lower layers to the perception of the user of the wireless data services.
A major drawback of existing systems for measuring data quality in wireless communication networks is that these systems do not correlate the lower layer measurements, to the perception of the performance of the wireless data service by the user. Examples of these lower layer measurements for e.g. GPRS networks are “number of used timeslots”, “bit/frame error rates”, “carrier/interference ratio “(C/I)”, “coding scheme usage”, “cell reselections”.
The aim of the present invention is to provide a method and system to assess the end-to-end quality as perceived by the user of a wireless datacommunication system by correlating lower layer measurements to the perception of the user.
It is another aim of the present invention to provide a system for identifying underlying causes for a low quality as perceived by the user.
According to the present invention, a method is provided for determining a perceived quality indicator for end-to-end data transfer in a data network, e.g. the download and/or response time or network throughput in a GPRS network, comprising measuring at least one system performance indicator during transfer of a predefined data type specimen, and calculating the perceived quality indicator for said predefined data transfer type and at least one other data transfer type from said measurement.The at least one system performance indicator may be a measured lower network layer indicator which is indirectly or directly linked to the quality indicator to be estimated.
The at least one system performance indicator may in a first embodiment comprise an indirect quality indicator, i.e. an indicator which may be measured, but is not directly related to a perceived quality. The system performance indicator may e.g. be at least one lower network layer performance indicator as normally measured for assessing network performance. The method further comprises mapping the at least one lower network layer performance parameter to the quality indicator. The quality indicator may be an estimated session time for the at least one other data transfer type (or directly related thereto, the throughput speed for the at least one other data transfer type).
The lower network layer performance indicators may e.g. be the percentage of time in which a specific coding scheme is used in the downlink, or the number of used timeslots during the download. In the case of FTP downloads, these performance parameters show a high correlation with actual session times, and can thus be used to derive estimated session times from data gathered from a single large file FTP download. For browsing, it has been found that the measured C/I value (Carrier/Interference ratio) may advantageously be used as low level performance parameter, e.g. in combination with the Number of used time slot parameter.
The mapping of the measured indicators using the lower network layerperformance indicators may be a linear mapping (using a single parameter) or a multi-dimensional mapping, e.g. using two parameters (two-dimensional mapping). For FTP downloads, correlations between actual session times and estimated session times have been found, with the combination of the two previously mentioned lower network layer indicators providing the best result. For browsing, good results have been achieved using the indicators number of used time slots and averaged measured C/I value.
In a further embodiment, the measured lower network layer performance indicators are modified to model more accurately the estimation of the session time. Integer values of the at least one lower network layer Iperformance indicator may e.g. be mapped to real values. For example, for FTP downloads, the number of used timeslots (an integer value) may be mapped to real values (integer value 1 is mapped as real value x1; integer value 2 is mapped as real value x2), and the percentage of time that an indicator has values in a predefined range may be clipped (e.g. all values between 0% and 5are mapped to 5% For browsing, good results have been achieved when the number of used time slots are mapped differently, i.e. different values for x1 and x2.
In a further group of embodiments, which may advantageously be used for determining estimated session times as quality indicators for FTP downloads, the at least one lower network layer performance indicator is a direct perceived quality indicator. In this case, the at least one lower network layer performance indicator is the measured throughput speed of a download of the predefined data transfer type specimen (e.g. a 512 kByte file), and the quality indicator is derived from the measured throughput speed using a moving window averaging estimation, in which the size of the moving window corresponds to the at least one other data transfer type. The at least one other data transfer type may e.g. be an FTP download of a smaller size file.
A final perceived quality indicator is calculated in a further embodiment from the percentage increase in the perceived quality indicator (such as the estimated session time) for the at least one other data transfer type. This allows to obtain a final quality figure, e.g. on a scale from 1-5, which allows easier interpretation and comparison.
Underlying causes in the system may be determined from further analysis of the contribution of each lower network layer performance indicator to the estimated session time. To allow for analyzing the underlying causes for bad performance, direct and/or statistical back mapping of the at least one lower network layer performance indicator may be used. In this manner, it is e.g. possible to determine whether voice congestion or cell reselection is the major source of bad performance, by looking at the contributions of various lower network layer performance indicatorss to the derived perceived quality indicator.
The predefined data transfer type specimen as used for measuring the at least one lower network layer performance indicator (indirect or direct) in all of the above described embodiments may e.g. be a FTP download of a large size file (e.g. 512 kByte). The quality indicator for data transfer types may then be derived from this at least one lower network layer performance indicator. Other data transfer types may be FTP downloads of different size of files (e.g. 4, 32 and 128 kByte), a browsing session, audio streaming data, video streaming data, chat sessions, etc.
The present method allows to derive a number of quality indicators for different types of data transfers using only a single field measurement (the large file size download). The quality indicator is e.g. a session time, such as the sum of the response time and download time in an FTP session.
In a further aspect, the present invention relates to a measurement system for determining a quality indicator for end-to-end data transfer in a wireless data network, comprising a data network analysis system connected to the wireless data transfer network for measuring at least one system performance indicator using a predefined data transfer type specimen, in which the measurement system is further equipped with processing means which are arranged for deriving the quality indicator for at least one other data transfer type from the at least one performance indicator. Furthermore, the processing means may be arranged to execute the method according to one of the present methods. This measurement system provides advantages analogue to the advantages described above in relation to the present method.
In an even further aspect, the present invention relates to a computer program comprising computer readable software instructions, which when loaded on a computer system, provides the computer system with the functionality of the method according to one of the present methods. Also, the present invention relates to a computer program product, such as a floppy disc or CDROM, comprising a computer program according to the present invention.
The present invention will now be explained in further detail using a number of exemplary embodiments, with reference to the accompanying drawings.
In
The measurement system 10 may be connected to the GPRS-network 1 using a proper wireless terminal 7 and a dedicated interface 6. The measurement system 10 comprises a processing system 2 (which may comprise one or more processors or other processing means), connected to input/output elements 5 (such as a keyboard, mouse, trackball, etc.), a display 4 (such as a monitor or LCD display), and memory 3, e.g. in the form of a hard disk, semiconductor memory (ROM, RAM, etc.) or removable memory means (floppy disc, CDROM, etc.). The processor 2 is arranged to execute a software program (e.g. stored in one of the memories 3) which comprises program code which allows the processor 2 to execute the present method.
Since the aim of the present method is to correlate quality measurements on the lower layers of the GPRS network to the quality as perceived by the user, it is important to establish which factors determine the perception of the user. For data services (FTP file downloads, browsing sessions, streaming audio/video data, chat sessions, etc.) it is known that the determining factors are:
For normal data services the TCP layer guarantees the integrity, so this factor is not taken into account within the present invention.
It has been found that the combined effect of response and download time is the most significant factor that determines the end-to-end perceived quality.
With FTP there is in general no distinction made between download time and response time and “FTP download time” (or session time) is in this application used to indicate the time between start and stop of the FTP session. With browsing there is in general a distinction made between download time and response time and in this application the sum of the two is referred to as “browsing time” (or session time).
Since the aim of the present method is to correlate quality measurements on the lower layers of the GPRS network to the quality as perceived by the user, it is important to establish the main factors influencing the performance within the GPRS network 1 and the lower layer measurements which indicate these factors. These factors are known to be:
The influence of congestion is pre-dominantly seen in the time dependent variances of the quality, at a certain point in time the measured quality differs from the same measurement at another point in time, because the number of active users varies. Congestion within a GPRS network has to forms, voice congestion and data congestion. Voice congestion can be detected by looking at the measurement of “the number of used time slots”. Data congestion is more difficult to detect.
The influence of radio conditions and cell reselections is pre-dominantly seen in the location dependent variances of the quality, the quality differs depending on the location where the wireless terminal 7 is used. Radio conditions can be detected from “C/I”, “bit/frame error rates” and “coding scheme usage” measurements. Cell reselections can be detected by looking at the “number of used time slots” measurements, which drops to zero when a cell reselection occurs. With voice congestion this drop in “number of used time slots” rarely goes to zero.
Server/Core network performance is mostly out of the control of a service provider and will especially dominate performance for high speed links. For low speed links, server performance should generally not be the limiting factor but it is clear that in assessing GPRS quality over the Internet the server and the public domain part of the connection that is used may have a significant influence. Therefore measurements that use public domain servers are not suited for field measurements. Two solutions are possible to overcome this problem when using the present method:
In the following adescription will be given for the exemplary determination of perceived quality indicators for “FTP file download” using a GPRS network 1.
FTP Download Perceived Quality Indicator
To keep the amount of final data manageable the following file sizes can be used for FTP download testing:
As described above with reference to
Although this direct approach allows to quantify the quality it does not allow to find the underlying causes for bad performance. Therefore the next step is to analyse the data with the second assessment approach (block 16).
The second assessment approach analyses the download results for 4, 32,128 and 512 kByte files over a wide range of network conditions and tries to find lower layer indicators that best correlate with the download time. For each of the four file sizes mappings are searched for, preferably using the same combination of lower layer indicators for all four file sizes but with a different mapping. These lower layer indicators are less influenced by the transport/application layer behaviour for large file downloads and will thus also allow correct quality predictions on the 4 kByte time scale. Furthermore the mapping should allow to analyse the underlying cause for bad performance. In a first analysis, to determine the mapping to be used in the present invention, lower layer indicators were correlated with a single linear mapping between download time and indicator, for all four different files sizes (4, 32, 128 and 512 kByte). The best indicators were:
When searching for the best two-dimensional mapping the afore mentioned indicator set gave the best correlation of all possible pairs.
A first detailed analysis showed that data congestion and voice congestion are the main effects that need to be modelled accurately in order to get higher correlations.
The indicator “number of used timeslots” is influenced by both cell-reselections and voice congestion. With cell-reselections the “number of used timeslots” is zero during the reselection process while for voice congestion it is decreased, e.g. from 3 down to 1. It turned out that the impact of a lower average “number of used timeslots” was far less in the case of voice congestion than in the case of cell reselections.
An effective way of modelling can be carried out in mapping the use of 1 and 2 time slots, as occurs with voice congestion, to the real values x1 and x2 respectively (i.e. all occurrences of 1 time slot are substituted by x1 and 2 by x2, effectively decreasing the impact of the use of 1 and 2 timeslots). This modified indicator (modified low level performance indicator) is denoted by MOD(UsedTimeslotsDL).
A second detailed analysis showed that the indicator for percentage of time a specific coding scheme is used, had little influence in the range between 0 and 5%. This is most probably caused by the fact that signalling information is always coded with the specific coding scheme so that the coding scheme usage below a certain percentage gives no information about the quality of the channel. This can be taken into account by mapping all values between 0 and 5% to 5% (clipping). This modified indicator is denoted byMOD(CodingSchemeDL).
The final two-dimensional mapping that is used in the mapping tool (block 14) gives a better correlation between model prediction and measured download time over all download times using the same indicator set for all four file sizes (4, 32, 128, 512 kByte). This correlation is up to a level that can be used in a tool, especially when it is combined with the analysis of the first assessment approach (see below). An example of the relation between measured and predicted download time is given in
Based on the two assessment approaches an FTP download quality assessment tool is implemented as a post processing on the analysis tool output that uses the following strategy:
From the above mentioned modified lower layer indicators the download time for each of the three file sizes (4, 32 and 128 kByte) is predicted by (second assessment approach) using the following mappings:
for 4, 32 and 128 kByte files respectively, in which a1,a2, a3, b1, b2, b3, c1, c2, c3 are predetermined mapping coefficients.
The tool allows to further analyse the underlying cause of bad performance by:
Other Applications
The present invention can also be applied for other data transfer types than the FTP file download as discussed in the example above. Further possible applications in which end-to-end perceived quality can be determined using the present invention include, but are not limited to, browsing, streaming audio, streaming video, chatting etc. For each of the different types of applications, other (combinations of) performance indicators (direct performance indicator(s) such as throughput speed, or indirect performance indicator(s) such as number of used cells, etc (see above)) may be used.
Perception Based Assessment Procedure
This section deals with the problem of how to interpret the time variant behaviour for each of the three file sizes (4, 32, 128 kByte, coinciding with the time scales of 4, 12 and 40 seconds). For the largest file (512 kByte) quality is measured directly from the download time of the file and time variant behaviour has to be assessed on the basis of multiple downloads (e.g. 10 downloads as recommended).
The basic idea in the data interpretation is that the download time has to vary over more than 25% in order to have a perceptual effect and has to vary by a factor of two for perceiving a significant degradation. The idea of quality categorization can be used for both FTP download and browsing. This leads to the following quality classes using the ITU-T 5 point rating scale.
Using this categorization one can quantify the time variant behaviour of the network, over the duration of the 512 kByte download, for each of the three file sizes for FTP download and for the browsing session. If a complete quality analysis is required one should thus calculate the percentages in each class (i.e. % excellent, % good, etc.) and use those five numbers to quantify the end-to-end quality for each of the four file sizes and the browsing session separately. For quantifying the time variant behaviour on the largest time scale (>140 s) one has to perform multiple FTP downloads of the 512 kByte file.
A first global quality analysis in the final post processing tool is carried out by calculating the percentage poor or bad as is commonly used in speech transmission quality assessment. This percentage is calculated from the lower layer indicators as well as from the application throughput (see above). This allows for an easy to interpret small set of quality numbers. For each of the three FTP download time scales and the single browsing time scale, two quality numbers in the form of a percentage poor or bad, are thus given. When they are about the same the degradation is not caused by data congestion, when they are significantly different data congestions plays an important role.
By a further analysis of the lower layer indicators one can find the cause for bad performance. When the average MOD(CodingSchemeDL) is high and C/I is low, then bad radio performance dominates the quality. When the average MOD(UsedTimeslotsDL) is low then voice congestion or cell reselection dominates the quality. By checking the distribution of the indicator number of used timeslots one can distinguish voice congestion from cell reselections. When many zero's are present cell reselection dominates the quality, when the values 1 and 2 dominate the distribution, voice congestion dominates the quality.
For the person skilled in the art, it will be clear that the above exemplary embodiments are only described for a better understanding. The exemplary embodiments are not meant to limit the scope of this application, which is defined by the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 03076894.9 | Jun 2003 | EP | regional |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/EP04/06538 | 6/16/2004 | WO | 11/30/2005 |
