Patent Application
20140337871
The present invention generally relates to a method to measure quality of experience of a video service, said video service being over-the-top video streaming provided to a user application under request by means of a network, and more particularly to a method that comprises calculating a Key Performance Indicator, or KPI, from measurable network parameters of said network for each video provided by said video service, assigning a Key Quality Indicator, or KQI, to each KPI by means of analytical models and calculating a global KQI function of a set of KQIs.
Typically, the way of measuring the Quality of Experience of the video services has been through the performance of subjective tests or difference analysis between the source and the received streams. The insights of some of the methods that are used are described next.
MOS (Mean Opinion Score)
The Mean Opinion Score (MOS) test has been used for decades in telephony networks to obtain the user's QoE of this voice service. Modern telecommunication networks provide a wide array of services using many transmission systems. In particular, the rapid deployment of digital technologies has led to an increased need for evaluating the transmission characteristics of new transmission equipment. In many circumstances, it is necessary to determine the subjective effects of some new transmission equipment or modification to the transmission characteristics of a telephone network.
In multimedia (audio, voice telephony, or video) especially when codecs are used to compress the bandwidth requirement (for example, of a digitized voice connection from the standard 64 kilobit/second PCM modulation), the mean opinion score (MOS) provides a numerical indication of the perceived quality from the users' perspective of received media after compression and/or transmission.
MOS tests are specified by ITU-T recommendation P.800 [1]. This Recommendation describes methods for obtaining subjective evaluations of transmission systems and components.
The MOS provides a numerical indication of the perceived quality from the users' perspective of received media after compression and/or transmission. It is expressed as a single number in the range 1 to 5, where 1 is lowest perceived quality, and 5 is the highest perceived.
PEVQ (Perceptual Evaluation of Video Quality)
PEVQ (Perceptual Evaluation of Video Quality) is a standardized E2E measurement algorithm to score the picture quality of a video presentation by means of a 5-point mean opinion score (MOS). The measurement algorithm can be applied to analyse visible artefacts caused by a digital video encoding/decoding (or transcoding) process, RF- or IP-based transmission networks and end-user devices. Application scenarios address next generation networking and mobile services and include IPTV (Standard-definition television and HDTV), streaming video, Mobile TV, video telephony, video conferencing and video messaging.
PEVQ is based on modelling the behaviour of the human visual tract and besides an overall quality MOS score (as a figure of merit) abnormalities in the video signal are quantified by a variety of KPIs, including PSNR, distortion indicators and lip-sync delay.
PEVQ is a full-reference algorithm and analyses the picture pixel-by-pixel after a temporal alignment (also referred to as ‘temporal registration’) of corresponding frames of reference and test signal. PEVQ MOS results range from 1 (bad) to 5 (excellent).
DiversifEye (Shenick Network Systems)
DiversifEye's is a measurement system that makes a per flow analysis enabling QoE performance testing on each and every flow. Their analysis is based on a ‘per flow’ basis as they claim that it is the only mechanism to guarantee performance at the user's granularity. Its architecture is used to emulate and assess subscriber Quality of Experience, network and device performance limitations, along with application performance under varying loads and conditions. The key strength to ‘Per flow’ testing is the microscopic view gained on live tests. This is representative of the in-depth performance analysis of each and every individual end point's quality of experience, on a per application basis.
A core benefit of per flow analysis is the ability to view the impact that various traffic flows and network devices (settings and conditions) have on a variety of traffic types. In diversifEye the Per flow assessment is enabled on all application flows including Secure VPNs, LTE—GTP tunnels, OTT—adaptive streams, IP-TV, VoD, VoIP, Telepresence, Data (web, email, P2P), etc. Per flow provides the necessary performance details on all delay sensitive applications and is critical in defining traffic management settings, along with guaranteeing a high subscriber Quality of Experience (QoE).
Additional benefits of diversifEye's per flow architecture is the ability to actively monitor live test runs. Users of diversifEye can add performance monitoring algorithms to each and every emulated application flow. During live test runs the algorithm actively monitors the performance on the associated flow, when the flow's performance crosses the assigned threshold level an event notification is generated.
Its purpose is to run tests rather than real network measurements as it is used to emulate both client and server side applications. The result is a set of real time performance statistics for each and every individual application flow request or activity.
Problems with existing solutions
The current solutions for video QoE measurement have several drawbacks:
Difference analysis: The measurement paradigm is to assess degradations of a decoded video sequence output from the network in comparison to the original reference picture. With this kind of analysis, at least two measuring points are required.
In MOS methodology, a single test condition (the video sequence) is presented to the viewers once only. They should then give the quality according to the predefined scale. This methodology obviously lacks of precision and objectivity.
Finally, subjective video quality tests are quite expensive in terms of time (preparation and running) and human resources.
It is necessary to offer an alternative to the state of the art which covers the gaps found therein, particularly related to the lack of proposals which really provide techniques to obtain quality of experience measurements in video services in an efficient and objective way.
To that end, the present invention provides a method to measure quality of experience of a video service, said video service being over-the-top video streaming provided to a user application under request by means of a network.
On contrary to the known proposals, the method of the invention, in a characteristic manner, comprises calculating a Key Performance Indicator, or KPI, from measurable network parameters of said network for each video provided by said video service, assigning a Key Quality Indicator, or KQI, to each KPI by means of analytical models and calculating a global KQI function of a set of KQIs.
Other embodiments of the method of the invention are described according to appended claims 2 to 11, and in a subsequent section related to the detailed description of several embodiments.
The previous and other advantages and features will be more fully understood from the following detailed description of embodiments, with reference to the attached drawings which must be considered in an illustrative and non-limiting manner, in which:
The basic concept of the invention relies in the use of a method that collects network measurements in just one point and for a very particular application (OTT Video Streaming). From these measurements, the method follows an analytical model in order to calculate some Key Performance Indicators (KPIs) required to obtain the user's QoE. As a function of these KPIs, the QoE is expressed in terms of Key Quality Indicators (KQIs) following another analytical model. The information flow of the method was depicted in
The OTT Video streaming traffic is generated by the platforms that host videos and stream them to the users after their request. These videos are delivered to the user via HTTP/TCP connections following the client-server model. Among the most popular portals that generate this type of traffic are YouTube, Metacafe and Megavideo.
By its nature, this type of traffic is elastic, although specific characteristics from inelastic applications are necessary. It is required that the connection has enough throughput to allow the application to play the content in a smooth and seamless manner from the beginning so that the user perceives that the application works properly and the application does not require a great queuing capacity. To achieve this behaviour, it should be required that the application input (data throughput) is at least equal to that required for viewing (the bitrate of the video). Additionally, it is required that the waiting time for reproduction start is low, although this parameter is less relevant for the purposes of perceived quality.
Therefore, the qualitative parameters of quality for the video streaming are the smooth playback (without interruption) and a low initial waiting time.
Instead of using a global KPI, specific KPIs have been used for each video playback. Precisely, the KPI that has been chosen to reflect the smooth video playback is the interruption rate (number of interruptions or breaks during the video playback time). An interruption occurs when the instantaneous throughput received is exceeded by the bitrate of the video.
Next, the process of determining when the interruptions occur and the calculation of the particular KPI of each video are detailed. After that, it is described how to calculate the associated particular KQI and a possible method to calculate a global KQI.
KPIs Obtainment Model from Network Parameters
The model focuses on how to identify an interruption in the user's application. To do this, it is required to know how the user's application works (for instance, an embedded Flash player in a web page) and which are the conditions that must appear before an interruption occurs.
There is an initial buffering time (T0) before the start of the video playback that allows the application to protect against possible variations in the received throughput. This initial time T0 may vary among the different streaming sites, but it generally corresponds to the time required to acquire a volume of bytes B0 that allows the view of a concrete video playback time τ0, different for each streaming site. The time τ0 is independent of the video bitrate, but it determines the initial volume of bytes to be acquired B0. Specifically, the initial volume of bytes B0 is given by the following equation:
B
0
=r·τ
0
where:
After this time, the video player starts the playback and must receive enough bytes (enough throughput) to remain above the video encoding bitrate, so that there are no interruptions. Specifically, the condition of not interrupted playback is given by the following equation:
B(t)≧r·(t−T0)
where:
If the volume of bytes received at any given time is below the theoretical line of video playback, there will be an interruption.
At time Ti in which the interruption occurs, the condition of the equation is not accomplished, i.e., the particular condition of the scenario is the following:
B(Ti)<r·(Ti−T0)
After the first interruption, it will take a time T1 until it buffers again a volume of B0 bytes in order to allow the playback of a video time τ0. After this time, the non-interruption condition changes and it becomes as follows:
B(t)≧r·(t−T0−T1)
Generally, after n interruptions:
Following the previous steps, it is possible to determine when there has been an interruption. Obviously, it is necessary to know the average video bitrate. Although knowing this rate may seem complicated, an evolved DPI could do it because the information travels inside the data packets. In fact, it is possible to obtain the codification bitrate of the videos from popular streaming sites such as YouTube and Megavideo (totalling more than 50% of video streaming traffic).
For each displayed video, it is possible to calculate the particular KPI (interruptions rate during the display time) using the following equation:
During a sufficiently large period of time, the particular KPI of each video playback will be obtained and these KPIs will be used to calculate the individual KQIs and, ultimately, the global KQI.
KQIs Obtainment Model from Network Parameters
The particular KQI must capture the impact of its associated KPI. Prior to the establishment of a mathematical function that relates KQIs and KPIs, the scale of KQIs must be always defined and decide whether the variation between the extremes of the scale must be linear, concave or convex.
In this case, the scale of KQI ranges from 1 to 5, where score 1 is the worst quality (dreadful) and the value 5 corresponds to the highest quality (ideal). A video is considered ideal if the number of interruptions detected during playback is zero. Meanwhile, if the number of interruptions is equal to or greater than 4 interruptions per minute, the quality is considered dreadful.
Regarding the concavity, the chosen function must be concave upward, so that the KQI responds more sharply to low values of the KPI. For interruption rates greater than 2 interruptions per minute, the degradation of the quality is lower as the quality is low by itself.
With these features, the function chosen to calculate the KQI is the following:
After obtaining the particular KQIs of all the videos during a significant period, a global KQI representing the overall quality of the video streaming service could be obtained. There are several options for calculating this global KQI:
The average of the particular KQIs.
The median or the 50th percentile of the particular KQIs distribution function.
The x-th percentile of the of the particular KQIs cumulative distribution function, i.e. the value of KQI for which x% of the videos have a quality lower than that KQI.
To calculate the global KQI, the videos whose playback time d is less than a certain threshold (few seconds) should be discarded as not being reliable enough.
As an example,
Implementation of the Invention
The invention has been deployed in a general purpose hardware as a prototype to evaluate the perceived QoE of the users from the YouTube video streaming service. This method performs the functions described before to obtain the KQI from some measurable network parameters.
In the analysis performance, real network measurements from an evolved DPI have been used. In
In this study, around 30,000 video playbacks were analysed, obtaining their particular KPI. More than 92% of the video playbacks had zero interruptions. As a function of these KPIs, the method is able to obtain the particular KQIs of each video playback. In
As it has been previously explained, there are many possibilities of obtaining a global KQI. From
In this particular case, it seems reasonable to choose the average of the particular KQIs as the way of obtaining the global KQI as its value is the most representative of the QoE perceived by the users of the service.
Advantages of the Invention
The present invention has the following advantages:
With this method, only one point of measurement is required in the network. Unlike the systems based on difference analysis, this Video QoE measurement method just needs to measure the parameters in one point of the network (for instance, one link) and with the information gathered from those measurements, it is able to evaluate the QoE perceived by the users of a particular service.
Due to the automation of the QoE evaluation, the results of the analysis are fully objective. Through the application of the models of the invention, the method is able to evaluate the user's perspective (subjective by definition) automatically and continuously without requiring human opinion or presence.
The Video QoE measurement method saves the time and human resources costs that are intrinsic in the subjective video quality tests.
A person skilled in the art could introduce changes and modifications in the embodiments described without departing from the scope of the invention as it is defined in the attached claims.
Acronyms
[1] ITU-T Recommendation P.800: Methods for subjective determination of transmission quality: http://www.itu.int/rec/T-REC-P.800-199608-l/en
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/EP2011/070897 | 11/24/2011 | WO | 00 | 6/16/2014 |
| Number | Date | Country | |
|---|---|---|---|
| 61540281 | Sep 2011 | US |
