QUALITY ASSESSMENT OF DIGITAL MEDIA

Abstract

A method for quality assessment of digital media. The method includes dividing the digital media to a plurality of frame subsets, generating a plurality of frame-level feature sets by generating an mth frame-level feature set, and extracting a score distribution for a plurality of scores from the plurality of frame-level feature sets. Each of the plurality of frame subsets includes a plurality of frames. The mth frame-level feature set is associated with an mth frame subset of the plurality of frame subsets where m∈[1, M] and M is a number of the plurality of frame subsets. The mth frame-level feature set includes a plurality of feature sets. The score distribution is associated with the digital media.

Description

TECHNICAL FIELD

The present disclosure generally relates to media quality assessment, and particularly, to media quality assessment based on neural networks.

BACKGROUND

Quality assessment of digital media (i.e., image, video, or audio) maps digital media, regarded as distorted views of real-world scenes, to corresponding quality scores on a numerical scale. Methods for media quality assessment may be evaluated based on a performance for benchmark media quality databases that are labeled by subjective quality scores obtained from human expert viewers. Predicting a quality score measure may facilitate a content provider to guarantee a desired percentage of satisfied users. One quality score measure is mean opinion score (MOS) that represents an overall quality of digital media and is given as a decimal number. However, image, video, or audio signals with approximately same MOS may have different quality score distributions. These distributions may include information about how much consensus or divergence exists among individual ratings.

Conventional quality assessment methods may extract predefined features from digital media. These features may then be used to predict MOS. These methods may be useful for MOS prediction of synthetically distorted signals. However, they may provide low quality predictions for real-world data since real-world data may suffer from various combinations of defects, such as optical distortions imposed by cameras at recording times, distortions caused by data compression, and data loss within communication channels. There is, therefore, a need for a quality assessment method for digital media that may provide a high quality prediction for a quality score distribution of a real-world image. There is also a need for a video quality assessment method that may provide a high quality prediction for a quality score distribution of a real-world audio/video.

SUMMARY

This summary is intended to provide an overview of the subject matter of the present disclosure, and is not intended to identify essential elements or key elements of the subject matter, nor is it intended to be used to determine the scope of the claimed implementations. The proper scope of the present disclosure may be ascertained from the claims set forth below in view of the detailed description below and the drawings.

In one general aspect, the present disclosure describes an exemplary method for quality assessment of digital media. An exemplary method may include dividing the digital media into a plurality of frame subsets, generating a plurality of frame-level feature sets by generating an m^th frame-level feature set, and extracting a score distribution for a plurality of scores from the plurality of frame-level feature sets. In an exemplary embodiment, each of the plurality of frame subsets may include a plurality of frames. An exemplary m^th frame-level feature set may be associated with an m^th frame subset of the plurality of frame subsets where m∈[1, M] and M is a number of the plurality of frame subsets. An exemplary m^th frame-level feature set may include a plurality of feature sets. An exemplary score distribution may be associated with the digital media.

In an exemplary embodiment, dividing the digital media into the plurality of frame subsets may include dividing an audio signal into a plurality of audio segments. An exemplary audio signal may be associated with the digital media. In an exemplary embodiment, dividing the audio signal into the plurality of audio segments may include obtaining each of the plurality of audio segments by applying a window function to a plurality of overlapping portions of the audio signal. Each successive pair of the plurality of overlapping portions may include an overlap between 0.2 and 0.6 of a length of the window function.

In an exemplary embodiment, generating the m^th frame-level feature set may include feeding a first plurality of feature maps to a first three dimensional (3D) convolutional layer of a plurality of 3D convolutional layers, generating an (i+1)^th plurality of feature maps by applying an i^th 3D convolutional layer of the plurality of 3D convolutional layers on an i^th plurality of feature maps where 1≤i≤N and N is a number of the plurality of 3D convolutional layers, and obtaining each respective feature set of the plurality of feature sets from a respective plurality of feature maps. In an exemplary embodiment, the plurality of 3D convolutional layers may be associated with a neural network. In an exemplary embodiment, the first plurality of feature maps may include the m^th frame subset.

In an exemplary embodiment, generating the (i+1)^th plurality of feature maps may include generating an (i+1)^th plurality of filtered feature maps by applying an i^th 3D convolution filter on the i^th plurality of feature maps, generating an (i+1)^th plurality of normalized feature maps by applying a batch normalization process on the (i+1)^th plurality of filtered feature maps, generating an (i+1)^th plurality of non-linear feature maps by applying an i^th non-linear activation function on each of the (i+1)^th plurality of normalized feature maps, and obtaining the (i+1)^th plurality of feature maps by applying a max pooling on each of the (i+1)^th plurality of non-linear feature maps. An exemplary i^th 3D convolution filter may be associated with an i^th 3D convolutional layer. An exemplary batch normalization process may be associated with the i^th 3D convolutional layer. Each exemplary normalized feature map of the (i+1)^th plurality of normalized feature maps may be associated with a respective filtered feature map of the (i+1)^th plurality of filtered feature maps. An exemplary i^th non-linear activation function may be associated with the i^th 3D convolutional layer. An exemplary max pooling may be associated with the i^th 3D convolutional layer.

In an exemplary embodiment, obtaining each respective feature set of the plurality of feature sets may include applying a global average pooling on a respective plurality of non-linear feature maps. In an exemplary embodiment, the global average pooling may include obtaining an average of each non-linear feature map of the respective plurality of non-linear feature maps.

In an exemplary embodiment, extracting the score distribution from the plurality of frame-level feature sets may include generating a recurrent output by feeding the plurality of frame-level feature sets to a recurrent neural network and generating the score distribution from the recurrent output.

In an exemplary embodiment, feeding the plurality of frame-level feature sets to the recurrent neural network may include feeding the plurality of frame-level feature sets to a long short-term memory (LSTM) network. In an exemplary embodiment, the LSTM network may include a plurality of LSTM layers.

In an exemplary embodiment, extracting the score distribution may include generating a first fully connected output of a plurality of fully connected outputs and generating a O^h fully connected output of the plurality of fully connected outputs. In an exemplary embodiment, generating the first fully connected output may include applying the recurrent output to the first fully connected layer. In an exemplary embodiment, generating the O^h fully connected output may include applying a (k−1)^th fully connected output of the plurality of fully connected outputs to a k^th fully connected layer of the plurality of fully connected layers where 1≤k≤L and L is a number of the plurality of fully connected layers. In an exemplary embodiment, applying the (k−1)^th fully connected output to the k^th fully connected layer may include applying an (N+k)^th non-linear activation function on the (k−1)^t fully connected output. In an exemplary embodiment, an (N+L)^th non-linear activation function may include a softmax function. In an exemplary embodiment, a zeroth fully connected output may include the recurrent output. In an exemplary embodiment, an L^th fully connected output of the plurality of fully connected outputs may include the score distribution.

In an exemplary embodiment, extracting the score distribution from the plurality of frame-level feature sets may include extracting a plurality of statistical features from the plurality of frame-level feature sets and obtaining the score distribution by applying a regression method on the plurality of statistical features through feeding the plurality of statistical features to a regression layer.

Other exemplary systems, methods, features and advantages of the implementations will be, or will become, apparent to one of ordinary skill in the art upon examination of the following figures and detailed description. It is intended that all such additional systems, methods, features and advantages be included within this description and this summary, be within the scope of the implementations, and be protected by the claims herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawing figures depict one or more implementations in accord with the present teachings, by way of example only, not by way of limitation. In the figures, like reference numerals refer to the same or similar elements.

FIG. 1A shows a flowchart of a method for quality assessment of an image, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 1B shows a flowchart for generating a plurality of feature maps, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 1C shows a flowchart for training a convolutional neural network, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 1D shows a flowchart of an iterative process, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 1E shows a flowchart for generating a plurality of updated weights, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 1F shows a flowchart for generating a plurality of training feature maps, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 1G shows a flowchart for extracting a plurality of feature maps, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 1H shows a flowchart for extracting a score distribution, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 2A shows a schematic of a convolutional neural network, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 2B shows a schematic of a convolutional layer, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 2C shows a schematic of a convolutional neural network in a training phase, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3A shows a flowchart of a method for quality assessment of a video, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3B shows a flowchart of training a neural network, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3C shows a flowchart of a second iterative process, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3D shows a flowchart of generating a plurality of updated weights, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3E shows a flowchart of an iteration of the third iterative process, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3F shows a flowchart of an iteration of a first iterative process, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3G shows a flowchart of generating a frame-level feature set, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3H shows a flowchart of generating a plurality of feature maps, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3I shows a flowchart of a method for generating a first feature set, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3J shows a flowchart of a method for generating a second feature set, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3K shows a flowchart of a method for generating a third feature set, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3L shows a flowchart of a method for generating a fourth feature set, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3M shows a flowchart of a method for generating a fifth feature set, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3N shows a flowchart of a method for generating a sixth feature set, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3O shows a flowchart of a method for generating a seventh feature set, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 3P shows a flowchart of for extracting a score distribution, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 4A shows a schematic of a neural network, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 4B shows a schematic of a neural network in a training phase, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 4C shows a schematic of a plurality of convolutional layers, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 4D shows a schematic of a convolutional layer, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 4E shows a schematic of a recurrent neural network, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 5A shows a flowchart of a method for quality assessment of digital media, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 5B shows a flowchart of a method for generating a frame-level feature set from a frame subset, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 5C shows a flowchart of generating a plurality of feature maps by a three-dimensional (3D) convolutional layer, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 5D shows a flowchart for extracting a score distribution from a plurality of frame-level feature sets, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 6A shows a schematic of a system for quality assessment of digital media, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 6B shows an audio signal, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 6C shows a block diagram of a feature extraction block, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 6D shows a schematic of a 3D convolutional layer, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 7 shows a high-level functional block diagram of a computer system, consistent with one or more exemplary embodiments of the present disclosure.

FIG. 8 shows a ground truth distribution and a score distribution of an image, consistent with exemplary embodiments of the present disclosure.

FIG. 9 shows a ground truth distribution and a score distribution of a video, consistent with exemplary embodiments of the present disclosure.

FIG. 10 shows a ground truth distribution and a score distribution of digital media, consistent with exemplary embodiments of the present disclosure.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent that the present teachings may be practiced without such details. In other instances, well known methods, procedures, components, and/or circuitry have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

The following detailed description is presented to enable a person skilled in the art to make and use the methods and devices disclosed in exemplary embodiments of the present disclosure. For purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that these specific details are not required to practice the disclosed exemplary embodiments. Descriptions of specific exemplary embodiments are provided only as representative examples. Various modifications to the exemplary implementations will be readily apparent to one skilled in the art, and the general principles defined herein may be applied to other implementations and applications without departing from the scope of the present disclosure. The present disclosure is not intended to be limited to the implementations shown, but is to be accorded the widest possible scope consistent with the principles and features disclosed herein.

Herein is disclosed an exemplary method and system for quality assessment of an image. An exemplary method may provide a prediction of a score distribution for an exemplary image utilizing a convolutional neural network. In doing so, a number of feature maps may be generated by a number of cascaded convolutional layers. Each convolution layer may generate an exemplary set of feature maps from its input and may pass the exemplary set of feature maps to a next convolutional layer. Each convolutional layer may apply filtering, normalization, and non-linear operations on an exemplary input provided by a previous convolutional layer. To utilize different combinations of feature maps, each set of feature maps generated by each respective convolutional layer may be directly utilized for prediction of the score distribution. For a size of feature maps to be independent of a size of the image, a predefined number of features may be extracted from each respective feature map by applying a global averaging process on each feature map prior to prediction of the score distribution. As a result, a fixed number of inputs may be provided to be applied to a number of fully connected layers that are utilized for score distribution prediction. Each of the fully connected layers may apply a non-linear activation function on its input and may pass a result to a next fully connected layer. Finally, the score distribution may be achieved at an output of the last fully connected layer.

Herein is further disclosed an exemplary method and system for quality assessment of digital media (i.e., audio or video signals). An exemplary method may provide a prediction of a score distribution for exemplary digital media. In doing so, a number of feature sets may be extracted from an exemplary set of audio/video frames by a feature extraction block. An exemplary feature extraction block may generate feature sets by a number of hand-crafted features. An exemplary feature extraction block may alternatively generate feature sets by a number of cascaded two-dimensional (2D) or three-dimensional (3D) convolutional layers. Each convolutional layer may apply filtering, normalization, and non-linear operations on an exemplary input provided by a previous convolutional layer. Afterwards, feature sets may be analyzed to extract score distribution.

In an exemplary embodiment, feature sets may be fed to an exemplary recurrent neural network. An exemplary recurrent neural network may extract spatiotemporal (i.e., related to time and space) information of the video/audio. When feature sets of all frames of the video/audio are fed to the exemplary recurrent neural network, an exemplary recurrent neural network output may be fed to a number of cascaded fully connected layers that are utilized for score distribution prediction. Each of the fully connected layers may apply a non-linear activation function on its input and may pass a result to a respective next fully connected layer. Finally, the score distribution may be achieved at an output of the last fully connected layer.

FIG. 1A shows a flowchart of a method for quality assessment of an image, consistent with one or more exemplary embodiments of the present disclosure. An exemplary method 100 may include generating an (i+1)^th plurality of feature maps from an image (step 102), obtaining an (i+1)^th feature set of a plurality of feature sets from the (i+1)^th plurality of feature maps (step 104), and extracting a score distribution for a plurality of scores from the plurality of feature sets (step 106).

FIG. 2A shows a schematic of a convolutional neural network, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, different steps of method 100 may be implemented utilizing an exemplary convolutional neural network 200A. In an exemplary embodiment, convolutional neural network 200A may include a plurality of convolutional layers 202, a plurality of fully connected layers 204, and a global average pooling layer 205. In an exemplary embodiment, plurality of convolutional layers 202 may include a first (1^st ) convolutional layer 206 and an i^th convolutional layer 208, where 1≤i≤N and N is a number of plurality of convolutional layers 202. In an exemplary embodiment, first (1^st) convolutional layer 206 may be configured to receive an image 210. In an exemplary embodiment, convolutional neural network 200A may be implemented utilizing a processor. In an exemplary embodiment, first (1^st) convolutional layer 206 may receive image 210 from the processor. In an exemplary embodiment, the processor may receive image 210 from an imaging device (such as a camera) or a memory. In an exemplary embodiment, plurality of fully connected layers 204 may include a first (1^st) fully connected layer 212.

FIG. 2B shows a schematic of a convolutional layer, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, i^th convolutional layer 208 may be configured to extract an (i+1)^th plurality of feature maps 214 from an i^th plurality of feature maps 216. For example, first (1^st) convolutional layer 206 may be configured to extract a second (2^nd) plurality of feature maps from a first (1^st) plurality of feature maps.

For further detail with respect to step 102, FIG. 1B shows a flowchart for generating a plurality of feature maps, consistent with one or more exemplary embodiments of the present disclosure. Referring to FIGS. 1B, 2A, and 2B, in an exemplary embodiment, generating (i+1)^th plurality of feature maps 214 may include training convolutional neural network 200A (step 108), feeding image 210 to first (1^st) convolutional layer 206 of plurality of convolutional layers 202 (step 110), and extracting (i+1)^th plurality of feature maps 214 (step 112).

In further detail with regards to step 108, FIG. 1C shows a flowchart for training a convolutional neural network, consistent with one or more exemplary embodiments of the present disclosure. FIG. 2C shows a schematic of a convolutional neural network in a training phase, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, training convolutional neural network 200A may include initializing convolutional neural network 200A by a plurality of initial weights (step 114) and repeating an iterative process until a termination condition is satisfied (step 116).

For further detail regarding step 114, in an exemplary embodiment, initializing convolutional neural network 200A may include generating the plurality of initial weights. In an exemplary embodiment, a respective number of initial weights of the plurality of initial weights may be multiplied by an output of each respective convolutional layer of plurality of convolutional layers 202. In an exemplary embodiment, a respective multiplication result may be fed to a subsequent convolutional layer of plurality of convolutional layers 202. In an exemplary embodiment, a respective number of initial weights of the plurality of initial weights may be multiplied by an output of each respective fully connected layer of plurality of fully connected layers 204. In an exemplary embodiment, a respective multiplication result may be fed to a subsequent fully connected layer of plurality of fully connected layers 204.

In an exemplary embodiment, generating the plurality of initial weights may include generating a plurality of random variables from a predefined probability distribution. In an exemplary embodiment, the predefined probability distribution may be determined by a designer of convolutional neural network 200A according to a required range of each initial weight. In an exemplary embodiment, the predefined probability distribution may be selected from Gaussian or uniform probability distributions.

In further detail with respect to step 116, FIG. 1D shows a flowchart of an iterative process, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, the iterative process may include generating a plurality of updated weights (step 118) and replacing the plurality of initial weights with the plurality of updated weights (step 120). In an exemplary embodiment, the plurality of updated weights may be generated by feeding a training image 218 of a plurality of training images to convolutional neural network 200A.

In further detail with respect to step 118, FIG. 1E shows a flowchart for generating a plurality of updated weights, consistent with one or more exemplary embodiments of the present disclosure. Referring to FIGS. 1E and 2C, in an exemplary embodiment, generating the plurality of updated weights may include generating an (i+1)^th plurality of training feature maps 220 from training image 218 (step 122), obtaining an (i+1)^th training feature set 222 of a plurality of training feature sets 224 from (i+1)^th plurality of training feature maps 220 (step 124), extracting a training distribution 226 from plurality of training feature sets 224 (step 126), generating a plurality of adjustment values by minimizing a loss function (step 128), and obtaining the plurality of updated weights (step 130). In an exemplary embodiment, the iterative process may be repeated until a value of the loss function becomes lower than a predefined threshold. In an exemplary embodiment, the predefined threshold may be set by a designer of convolutional neural network 200A according to a required performance of convolutional neural network 200A. In an exemplary embodiment, to increase the performance of convolutional neural network 200A, a value of the predefined threshold may be decreased. In an exemplary embodiment, the loss function may be defined as a function of different components (e.g., mean, variance, etc.) of training distribution 226 and a ground truth distribution associated with the training image.

In an exemplary embodiment, the loss function may be defined by the following:

$L\left(p,q\right)=\sum _{m=1}^M{p}_m\log \left(\frac{p_m}{q_m}\right)$

where L(⋅,⋅) is the loss function, p is training distribution 226, q is a ground truth distribution associated with the training image, p_m is an m^th component of the training distribution, q_m is an m^th component of the ground truth distribution, and M is a number of the plurality of scores. An exemplary ground truth distribution may be obtained from a plurality of viewers. In doing so, each viewer of the plurality of viewers may assign a score from the plurality of scores to training image 218. Next, in an exemplary embodiment, a histogram of scores for training image 218 may be obtained and regarded as the ground truth distribution associated with training image 218. In an exemplary embodiment, the loss function may represent a measure of distance between the ground truth distribution and training distribution 226. Other exemplary types of loss functions may also be defined in step 116, e.g., earth mover's distance, mean square error, or mean absolute error.

For further detail with respect to step 122, FIG. 1F shows a flowchart for generating a plurality of training feature maps, consistent with one or more exemplary embodiments of the present disclosure. Referring to FIGS. 1F and 2C, in an exemplary embodiment, generating (i+1)^th plurality of training feature maps 220 may include feeding training image 218 to first (1^st) convolutional layer 206 of plurality of convolutional layers 202 (step 132) and extracting (i+1)^th plurality of training feature maps 220 from an output 228 of i^th convolutional layer 208 (step 134).

For further detail with respect to step 132, in an exemplary embodiment, training image 218 may include a first (1^st) plurality of training feature maps. Therefore, feeding training image 218 to first (1^st) convolutional layer 206 may include feeding each training feature map of first (1^st) plurality of training feature maps to first (1^st) convolutional layer 206. In an exemplary embodiment, each feature map of the first (1^st) plurality of training feature maps may be associated with a channel of training image 218, e.g., a red, a green, or a blue channel.

In further detail regarding step 134, in an exemplary embodiment, convolutional neural network 200A may be a deep neural network. Therefore, a performance of training convolutional neural network 200A may be negatively impacted by overfitting. In an exemplary embodiment, when a deep neural network gets impacted by overfitting, the deep neural network may provide very low values of error defined as a difference between ground truth and training outputs. However, the performance of the deep neural network for test data may be drastically degraded. To resolve this issue, in an exemplary embodiment, extracting (i+1)^th plurality of training feature maps 220 may include applying a first dropout process on the output of i^th convolutional layer 208. The first dropout process may eliminate an element of each of (i+1)^th plurality of training feature maps 220 with a predefined probability that may be set by a designer of convolutional neural network 200A. Therefore, extracting each of (i+1)^th plurality of training feature maps 220 associated with training image 218 may be subjected to a specific first dropout process that may eliminate a specific set of elements of (i+1)^th plurality of training feature maps 220. In an exemplary embodiment, the predefined probability may be adjusted such that a negative impact of overfitting is suppressed, i.e., a value of the loss function in a testing phase becomes lower than a required threshold.

In training convolutional neural network 200A, a value of each element in (i+1)^th plurality of feature maps 214 may vary in a large interval. Consequently, in an exemplary embodiment, a training speed of convolutional neural network 200A may be low, since convolutional neural network 200A may need to be trained for a very wide interval of values at each layer. To compensate this issue, in an exemplary embodiment, each of plurality of convolutional layers 202 may include a batch normalization process. In an exemplary embodiment, the batch normalization process may normalize (i+1)^th plurality of feature maps 214 utilizing an average and a standard deviation of a set of (i+1)^th plurality of feature maps 214 associated with a batch of training images. In doing so, the plurality of training images may be partitioned into a plurality of training image batches. Next, an exemplary set of (i+1)^th plurality of feature maps 214 associated with each of training image batches may be obtained. Afterwards, in an exemplary embodiments, an average and a standard deviation of the set of (i+1)^th plurality of feature maps 214 may be obtained and all elements of the set of (i+1)^th plurality of feature maps 214 may be normalized in accordance to the average and the standard deviation. Next, in an exemplary embodiments, all elements of the set of (i+1)^th plurality of feature maps 214 may be scaled and shifted by a scale and a shift variable which may be learned during training process. Therefore, in an exemplary embodiments, all elements of (i+1)^th plurality of feature maps 214 may follow a normal distribution which may considerably reduce a required time for training convolutional neural network 200A. In an exemplary embodiment, extracting (i+1)^th plurality of training feature maps 220 may be similar to extracting (i+1)^th plurality of feature maps 214. Moreover, in an exemplary embodiment, obtaining an (i+1)^th training feature set 222 may be similar to obtaining (i+1)^th feature set 243.

Referring again to FIGS. 1E and 2C, step 126 may include extracting a training distribution 226 from plurality of training feature sets 224. In further detail with respect to step 126, in an exemplary embodiment, extracting training distribution 226 may include feeding plurality of training feature sets 224 to first (1^st) fully connected layer 212 of plurality of fully connected layers 204. In an exemplary embodiment, extracting training distribution 226 may include applying a second dropout process on a k^th fully connected output 252. In an exemplary embodiment, the second dropout process may include eliminating each element of k^th fully connected output 252 with a predefined probability for training image 218. Therefore, extracting each training distribution associated with each of the plurality of training images may be subjected to a specific second dropout process that may eliminate a specific set of output elements of k^th fully connected output 252.

For further detail with regards to step 128, in an exemplary embodiment, each of the plurality of adjustment values may be associated with a respective initial weight of the plurality of initial weights. In an exemplary embodiment, minimizing the loss function may be performed utilizing a gradient descent method. In an exemplary gradient descent method, each adjustment value of the plurality of adjustment values may be proportional to a gradient of the loss function associated with each initial weight of the plurality of initial weights. Exemplary adjustment values associated with each convolutional layer of plurality of convolutional layers 202 and each fully connected layer of plurality of fully connected layers 204 may be obtained utilizing a back propagation algorithm. For further detail with respect to step 130, in an exemplary embodiment, the plurality of updated weights may be obtained by adding each of the plurality of adjustment values to a respective initial weight of the plurality of initial weights.

Referring again to FIG. 1B, in an exemplary embodiment, step 110 may include feeding image 210 to first (1^st) convolutional layer 206. In further detail regarding step 110, in an exemplary embodiment, image 210 may include a first (1^st) plurality of feature maps. In an exemplary embodiment, each feature map of the first (1^st) plurality of feature maps may be associated with a channel of image 210. For example, each feature map of the first (1^st) plurality of feature maps may be associated with one of a red, a green, or a blue channel of image 210. In an exemplary embodiment, feeding image 210 to first (1^st) convolutional layer 206 may include feeding each of the first (1^st) plurality of feature maps to first (1^st) convolutional layer 206.

For further detail with respect to step 112, FIG. 1G shows a flowchart for extracting an (i+1)^th plurality of feature maps, consistent with one or more exemplary embodiments of the present disclosure. Referring to FIG. 2B, in an exemplary embodiment, i^th convolutional layer 208 may include an i^th plurality of filters 230, a batch normalization layer 232, an i^th non-linear activation layer 234, and a pooling layer 236. In an exemplary embodiment, extracting (i+1)^th plurality of feature maps 214 may include generating an (i+1)^th plurality of filtered feature maps 238 (step 136), generating an (i+1)^th plurality of normalized feature maps 240 (step 138), generating an (i+1)^th plurality of non-linear feature maps 242 (step 140), and generating (i+1)^th plurality of feature maps 214 (step 142).

In further detail with regards to step 136, in an exemplary embodiment, generating (i+1)^th plurality of filtered feature maps 238 may include applying i^th plurality of filters 230 on i^th plurality of feature maps 216. In an exemplary embodiment, each filter of i^th plurality of filters 230 may include N_i number of two dimensional filters, where N_i is a number of i^th plurality of feature maps 216. An exemplary two dimensional filter may include a matrix which may perform a convolution operation on a window of a respective feature map of i^th plurality of feature maps 216. In other words, an exemplary filter of i^th plurality of filters 230 may include a three dimensional filter which may be generated by stacking N_i two dimensional filters. In an exemplary embodiment, applying i^th plurality of filters 230 on i^th plurality of feature maps 216 may include applying each of two dimensional filters on a respective feature map of i^th plurality of feature maps 216 and obtaining a respective result. In an exemplary embodiment, the respective result may correspond to an output of an exemplary two dimensional filter. An element of each of (i+1)^th plurality of filtered feature maps 238 may be obtained by a summation over respective results associated with each of two dimensional filters. Next, in an exemplary embodiment, a respective filtered feature map of (i+1)^th plurality of filtered feature maps 238 may be obtained by sweeping each two dimensional filter on the respective feature map. In an exemplary embodiment, applying i^th plurality of filters 230 on i^th plurality of feature maps 216 may include applying a separable convolution on i^th plurality of feature maps 216. An exemplary separable convolution may break down N_i two dimensional convolutions into two simpler types of convolutions, i.e., a depth wise convolution and a point wise convolution. This may reduce a number of all weights of a neural network and may result in computational efficiency.

For further detail with respect to step 138, in an exemplary embodiment, generating (i+1)^th plurality of normalized feature maps 240 may include applying a batch normalization process on (i+1)^th plurality of filtered feature maps 238. In an exemplary embodiment, the batch normalization process may be associated with batch normalization layer 232. In an exemplary embodiment, each normalized feature map of (i+1)^th plurality of normalized feature maps 240 may be associated with a respective filtered feature map of (i+1)^th plurality of filtered feature maps 238. The batch normalization process may be similar to that of step 134. However, there may be no training image batches in generating (i+1)^th plurality of normalized feature maps 240. Therefore, in an exemplary embodiment, an average and a standard deviation used for batch normalization may be obtained from a set of averages and a set of standard deviations associated with all batches of training image batches.

In further detail regarding step 140, in an exemplary embodiment, generating (i+1)^th plurality of non-linear feature maps 242 may include implementing an i^th non-linear activation function on each of (i+1)^th plurality of normalized feature maps 240. In an exemplary embodiment, the i^th non-linear activation function may be associated to the i^th non-linear activation layer 234. In an exemplary embodiment, implementing the i^th non-linear activation function may include implementing one of a rectified linear unit (ReLU) function or an exponential linear unit (ELU) function. In an exemplary embodiment, implementing each of the i^th non-linear activation function may include implementing other types of activation functions such as leaky ReLU, scaled ELU, parametric ReLU, etc.

For further detail with regards to step 142, in an exemplary embodiment, generating (i+1)^th plurality of feature maps 214 may include applying a max pooling on each of (i+1)^th plurality of non-linear feature maps 242. In an exemplary embodiment, the max pooling may be performed utilizing pooling layer 236. In an exemplary embodiment, generating (i+1)^th plurality of feature maps 214 may further include applying an average pooling on each of (i+1)^th plurality of non-linear feature maps 242. In an exemplary embodiment, the average pooling may be associated with pooling layer 236. An exemplary pooling layer may apply a window of a predefined size on each of (i+1)^th plurality of non-linear feature maps 242 and obtain a maximum or average of windowed version of each of (i+1)^th plurality of non-linear feature maps 242. Therefore, in an exemplary embodiment, each of (i+1)^th plurality of feature maps 214 may be generated by sweeping the window on each of (i+1)^th plurality of non-linear feature maps 242. As a result, a size of each of (i+1)^th plurality of non-linear feature maps 242 may be smaller than a size of a respective non-linear feature map of (i+1)^th plurality of non-linear feature maps 242. Consequently, a number of required operations may proportionally be decreased.

Referring again to FIGS. 1A, 2A, and 2B, in an exemplary embodiment, step 104 may include obtaining (i+1)^th feature set 243 of a plurality of feature sets 246 from (i+1)^th plurality of feature maps 214. In further detail regarding step 104, in an exemplary embodiment, obtaining an (i+1)^th feature set 243 may include applying a global average pooling on (i+1)^th plurality of non-linear feature maps 242. In an exemplary embodiment, global average pooling layer 205 may be configured to perform the global average pooling. In an exemplary embodiment, the global average pooling may include obtaining an average of each non-linear feature map of (i+1)^th plurality of non-linear feature maps 242. In an exemplary embodiment, global average pooling layer 205 may be configured to obtain a set of statistical features from each of (i+1)^th plurality of non-linear feature maps 242. For example, global average pooling layer 205 may be configured to obtain a standard deviation, or a moment of an arbitrary order. In an exemplary embodiment, a size of an output of a convolutional layer may be proportional to a size of an input of the convolutional layer. In an exemplary embodiment, when a size of the input of the convolutional layer is H×W and a size of a two dimensional filter corresponding to the convolutional layer is M×N, a size of an output of the convolutional layer may become (H−M 301)×(W−N+1). Obtaining each feature set from each of (i+1)^th plurality of non-linear feature maps 242 using global statistics, i.e., using all elements of each of (i+1)^th plurality of non-linear feature maps 242, may result in feature sets of a size that may be independent of a size of each of i^th plurality of feature maps 216. As a result, in an exemplary embodiment, a score distribution 244 with a predefined number of plurality of scores for image 210 may be obtained from convolutional neural network 200A where image 210 may be of an arbitrary size.

In an exemplary embodiment, obtaining the plurality of feature sets from each convolutional layer of plurality of convolutional layers 202 may lead to different combinations of feature maps. In an exemplary embodiment, feature sets obtained from both low convolutional layers (e.g., first (1^st) convolutional layer 206) and high convolutional layers (e.g., an N^th convolutional layer, where N is a number of plurality of convolutional layers 202) may provide more information to plurality of fully connected layers 204 compared with obtaining features sets solely from the N^th convolutional layer. On the other hand, plurality of fully connected layers 204 may be connected to low convolutional layers through feature sets obtained from feature maps associated low convolutional layers. Therefore, a value of gradient may be considerable with respect to weights in low convolutional layers, which may prevent a vanishing gradient phenomenon.

For further detail with respect to step 106, FIG. 1H shows a flowchart of for extracting a score distribution, consistent with one or more exemplary embodiments of the present disclosure. Referring to FIG. 2A, in an exemplary embodiment, the plurality of scores may be associated with image 210. In an exemplary embodiment, score distribution 244 may be extracted by feeding plurality of feature sets 246 to first (1^st) fully connected layer 212. In an exemplary embodiment, extracting score distribution 244 may include generating a first (1^st) fully connected output 248 of a plurality of fully connected outputs 250 (step 144 ) and generating k^th fully connected output 252 of the plurality of fully connected outputs (step 146 ).

In further detail with regards to step 144, in an exemplary embodiment, generating first (1^st) fully connected output 248 may include applying plurality of feature sets 246 to first (1^st) fully connected layer 212 of plurality of fully connected layers 204. In an exemplary embodiment, applying plurality of feature sets 246 to first (1^st) fully connected layer 212 may include arranging plurality of feature sets 246 into a feature vector and applying the feature vector to first (1^st) fully connected layer 212.

For further detail regarding step 146, in an exemplary embodiment, generating k^th fully connected output 252 may include applying a (k−1)^th fully connected output 254 of plurality of fully connected outputs 250 to a k^th fully connected layer 256 of plurality of fully connected layers 204, where 1≤k≤L, and L is a number of plurality of fully connected layers 204. In an exemplary embodiment, applying the (k−1)^th fully connected output to k^th fully connected layer 256 may include implementing an (N+k)^th non-linear activation function on (k−1)^th fully connected output 254. In an exemplary embodiment, implementing the (N+k)^th non-linear activation function may include implementing one of a rectified linear unit (ReLU) function or an exponential linear unit (ELU) function. In an exemplary embodiment, implementing the (N+k)^th non-linear activation function may include implementing other types of activation functions such as leaky ReLU, scaled ELU, parametric ReLU, etc. In an exemplary embodiment, an (N+L)^th non-linear activation function may include a softmax function. In an exemplary embodiment, a 0^th fully connected output may include plurality of feature sets 246. In an exemplary embodiment, an L^th fully connected output 258 of plurality of fully connected outputs 250 may include score distribution 244.

FIG. 3A shows a flowchart of a method for quality assessment of a video, consistent with one or more exemplary embodiments of the present disclosure. An exemplary method 300 may include training a neural network (step 301), repeating a first iterative process M times (step 302), and extracting a score distribution for a plurality of scores (step 304).

FIG. 4A shows a schematic of a neural network, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, different steps of method 300 may be implemented utilizing a neural network 400.

In further detail regarding step 301, FIG. 3B shows a flowchart of training a neural network, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, training neural network 400 may include initializing neural network 400 by a plurality of initial weights (step 306) and repeating a second iterative process until a termination condition is satisfied (step 308).

FIG. 4B shows a schematic of a neural network in a training phase, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, neural network 400 may include a feature extraction unit 401, a recurrent neural network 402, and a plurality of fully connected layers 404.

In further detail with respect to step 306, in an exemplary embodiment, initializing neural network 400 may include generating the plurality of initial weights. In an exemplary embodiment, neural network 400 may include a plurality of neurons. In an exemplary embodiment, each neuron of the plurality of neurons may include a respective neuron output of a plurality of neuron outputs. In an exemplary embodiment, a respective number of initial weights of the plurality of initial weights may be multiplied by each respective neuron output of the plurality of neuron outputs. In an exemplary embodiment, a respective multiplication result may be fed to a subsequent neuron of the plurality of neurons. In an exemplary embodiment, generating the plurality of initial weights may include generating a plurality of random variables from a predefined probability distribution. In an exemplary embodiment, the predefined probability distribution may be determined by a designer of neural network 400 according to a required range of each initial weight based on a saturation region of a plurality of non-linear activation functions applied on the plurality of neurons. In an exemplary embodiment, the predefined probability distribution may be selected from Gaussian or uniform probability distributions.

For further detail with regard to step 308, in an exemplary embodiment, the plurality of updated weights may be generated by feeding a training video of a plurality of training videos to neural network 400. In an exemplary embodiment, the training video may be fed to neural network 400 by the processor. In an exemplary embodiment, the processor may receive the training video from a video capturing device (such as a camera) or a memory.

In an exemplary embodiment, the second iterative process may be repeated until a value of the loss function becomes lower than a predefined threshold. In an exemplary embodiment, the predefined threshold may be set by a designer of neural network 400 according to a required performance of neural network 400. In an exemplary embodiment, to increase the performance of neural network 400, i.e., to achieve lower values of the loss function, a value of the predefined threshold may be decreased. In an exemplary embodiment, the loss function may be defined as a function of different components (e.g., mean, variance, etc.) of a training score distribution 405 and a ground truth distribution.

In an exemplary embodiment, the loss function may be defined by the following:

$\begin{array}{cc}L\left(p,q\right)=\{\begin{array}{c}{\sum }_{m=1}^M{p}_m\log \left(\frac{p_m}{q_m}\right),\mathrm{or}\\ {\sum }_{m=1}^Mp_m-q_m\end{array}& \mathrm{Equation}\left(1\right)\end{array}$

where L(⋅,⋅) is the loss function, p is training score distribution 405, q is the ground truth distribution, p_m is an m^th component of training score distribution 405, q_m is an m^th component of the ground truth distribution, and M is a number of the plurality of scores. An exemplary ground truth distribution may be obtained from a plurality of viewers. In doing so, each viewer of the plurality of viewers may assign a score from the plurality of scores to the training video. Next, in an exemplary embodiment, a histogram of scores for the training video may be obtained and regarded as the ground truth distribution. In an exemplary embodiment, the loss function may represent a measure of distance between the ground truth distribution and training score distribution 405. Other exemplary types of loss functions may also be defined in step 308, e.g., earth mover's distance, mean square error, or mean absolute error.

FIG. 3C shows a flowchart of a second iterative process, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, the second iterative process may include generating a plurality of updated weights (step 310) and replacing the plurality of initial weights with the plurality of updated weights (step 312).

In further detail with respect to step 310, FIG. 3D shows a flowchart of generating a plurality of updated weights, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, generating the plurality of updated weights may include repeating a third iterative process M times (step 314), extracting training score distribution 405 for the plurality of scores (step 316), generating a plurality of adjustment values by minimizing a loss function (step 318), and obtaining the plurality of updated weights (step 320).

For further detail regarding step 314, FIG. 3E shows a flowchart of an m^th iteration of the third iterative process, consistent with one or more exemplary embodiments of the present disclosure. Referring to FIGS. 3E and 4B, in an exemplary embodiment, an m^th iteration of the third iterative process, where m∈[1, M] and M is a number of a plurality of video frames 419, may include generating an m^th training frame-level feature set 406 (step 322), generating a first training recurrent output based on a zeroth training recurrent output (step 324), and generating an m^th training recurrent output 408 based on an (m−1)^th training recurrent output (step 326 ).

For further detail with respect to step 322, in an exemplary embodiment, m^th training frame-level feature set 406 may be extracted from an m^th training video frame 410 of a plurality of training video frames 412. In an exemplary embodiment, the training video may include plurality of training video frames 412. In an exemplary embodiment, m^th training frame-level feature set 406 may be generated utilizing feature extraction unit 401. In an exemplary embodiment, neural network 400 may be a deep neural network. An exemplary deep neural network may include a plurality of layers between an input layer of the deep neural network and an output layer of the deep neural network. Therefore, a performance of training neural network 400 may be negatively impacted by overfitting. In an exemplary embodiment, when a deep neural network gets impacted by overfitting, the deep neural network may provide very low values of error defined as a difference between ground truth and training outputs. However, the performance of the deep neural network for test data may be drastically degraded. To resolve this issue, in an exemplary embodiment, generating m^th training frame-level feature set 406 may include applying a first dropout process on feature extraction unit 401.

In further detail regarding step 324, in an exemplary embodiment, the first training recurrent output may be generated by feeding a first training frame-level feature set to recurrent neural network 402. In an exemplary embodiment, the zeroth training recurrent output may be equal to a zero vector. In an exemplary embodiment, the zero vector may include zero elements. For further detail with respect to step 326, in an exemplary embodiment, m^th training recurrent output 408 may be generated by feeding m^th training frame-level feature set 406 to recurrent neural network 402.

Referring again to FIG. 3D, in an exemplary embodiment, step 316 may include extracting training score distribution 405. In further detail with regard to step 316, in an exemplary embodiment, training score distribution 405 may be extracted from an M^th training recurrent output. Referring again to FIG. 4A, in an exemplary embodiment, training score distribution 405 may be extracted from the training video. In an exemplary embodiment, training score distribution 405 may be extracted by feeding the M^th training recurrent output to a first fully connected layer 416 of plurality of fully connected layers 404. In an exemplary embodiment, extracting training score distribution 405 may include a second dropout process on plurality of fully connected layers 404.

For further detail regarding step 318, in an exemplary embodiment, minimizing the loss function may be performed utilizing a gradient descent method. In an exemplary gradient descent method, each adjustment value of the plurality of adjustment values may be proportional to a gradient of the loss function with respect to a respective initial weight of the plurality of initial weights. In an exemplary embodiment, the plurality of adjustment values may be obtained utilizing a back propagation algorithm. For further detail with respect to step 320, in an exemplary embodiment, the plurality of updated weights may be obtained by adding each of the plurality of adjustment values to a respective initial weight of the plurality of initial weights.

Referring again to FIG. 3A, in an exemplary embodiment, step 302 may include repeating a first iterative process M times. In further detail regarding step 302, FIG. 3F shows a flowchart of an m^th iteration of a first iterative process, consistent with one or more exemplary embodiments of the present disclosure. Referring again to FIGS. 3F and 4A, In an exemplary embodiment, an m^th iteration of the first iterative process may include generating an m^th frame- level feature set 420 (step 328), generating a first recurrent output of a plurality of recurrent outputs based on a zeroth recurrent output of the plurality of recurrent outputs (step 330), and generating an m^th recurrent output of the plurality of recurrent outputs based on an (m−1)^th recurrent output of the plurality of recurrent outputs (step 332).

In an exemplary embodiment, the first iterative process may be repeated utilizing one or more processors. In an exemplary embodiment, the video may include plurality of video frames 419. In an exemplary embodiment, m^th frame-level feature set 420 may be generated utilizing feature extraction unit 401. In an exemplary embodiment, feature extraction unit 401 may be implemented utilizing one or more processors. An exemplary m^th frame-level feature set 420 may be extracted from an m^th video frame 424 of plurality of video frames 419. An exemplary m^th frame-level feature set 420 may include a plurality of feature sets 426.

For further details with respect to step 328, FIG. 3G shows a flowchart of a method for generating an m^th frame-level feature set, consistent with one or more exemplary embodiments of the present disclosure. An exemplary method 328A may include a first implementation of step 328. In an exemplary embodiment, generating m^th frame-level feature set 420 may include feeding m^th video frame 424 to a first convolutional layer of a plurality of convolutional layers (step 334), generating a second plurality of feature maps (step 336), generating an (i+1)^th plurality of feature maps (step 338), and obtaining an (i+1)^th feature set of plurality of feature sets 426 from (i+1)^th plurality of feature maps (step 340).

FIG. 4C shows a schematic of a plurality of convolutional layers, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, neural network 400 may further include a plurality of convolutional layers 430. In an exemplary embodiment, plurality of convolutional layers 430 may include a first convolutional layer 428 and an i^th convolutional layer 438. An exemplary first convolutional layer 428 may be configured to generate a second plurality of feature maps 432. In an exemplary embodiment, i^th convolutional layer 438 may be configured to generate an (i+1)^th plurality of feature maps 434.

In an exemplary embodiment, plurality of convolutional layers 430 may include a first implementation of feature extraction unit 401. In an exemplary embodiment, second plurality of feature maps 432 may be generated by applying m^th video frame 424 to first convolutional layer 428. In an exemplary embodiment, (i+1)^th plurality of feature maps 434 may be generated by applying an i^th plurality of feature maps 436 to i^th convolutional layer 438 of plurality of convolutional layers 430, where 1≤i≤N and N is a number of plurality of convolutional layers 430.

In further detail with regard to step 334, in an exemplary embodiment, m^th video frame 424 may be fed to first convolutional layer 428 by the processor. In an exemplary embodiment, the processor may receive m^th video frame 424 from a video capturing device (such as a camera) or a memory. For further detail regarding step 336, in an exemplary embodiment, second plurality of feature maps 432 may be generated by applying m^th video frame 424 to first convolutional layer 428. In an exemplary embodiment, generating the second plurality of feature maps may be similar to generating (i+1)^th plurality of feature maps 434.

For further detail with respect to step 338, FIG. 311 shows a flowchart of generating an (i+1)^th plurality of feature maps, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, generating (i+1)^th plurality of feature maps 434 may include generating an (i+1)^th plurality of filtered feature maps (step 342), generating an (i+1)^th plurality of normalized feature maps (step 344), generating an (i+1)^th plurality of non-linear feature maps (step 346), and obtaining (i+1)^th plurality of feature maps 434 (step 348).

FIG. 4D shows a schematic of a convolutional layer, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, i^th convolutional layer 438 may include an i^th plurality of filters 446, a batch normalization layer 448, an i^th activation layer 450, and a pooling layer 452.

In further detail with respect to step 342, in an exemplary embodiment, generating an may include applying i^th plurality of filters 446 on i^th plurality of feature maps 436. In an exemplary embodiment, each filter of i^th plurality of filters 446 may include N_i number of two dimensional filters, where N_i is a number of i^th plurality of feature maps 436. An exemplary two dimensional filter may include a matrix which may perform a convolution operation on a window of a respective feature map of i^th plurality of feature maps 436. In other words, an exemplary filter of i^th plurality of filters 446 may include a three dimensional filter which may be generated by stacking N_i two dimensional filters. In an exemplary embodiment, applying i^th plurality of filters 446 on i^th plurality of feature maps 436 may include applying each of two dimensional filters on a respective feature map of i^th plurality of feature maps 436 and obtaining a respective result. In an exemplary embodiment, the respective result may correspond to an output of an exemplary two dimensional filter. An element of each of may be obtained by a summation over respective results obtained from each of two dimensional filters. Next, in an exemplary embodiment, a respective filtered feature map of may be obtained by sweeping each two dimensional filter on the respective feature map. In an exemplary embodiment, applying i^th plurality of filters 446 on i^th plurality of feature maps 436 may include applying a separable convolution on i^th plurality of feature maps 436. An exemplary separable convolution may break down N_i two dimensional convolutions into two simpler types of convolutions, i.e., a depth wise convolution and a point wise convolution. This may reduce a number of all weights of a neural network and may result in computational efficiency.

For further detail regarding step 344, in an exemplary embodiment, generating an (i+1)^th plurality of normalized feature maps 442 may include applying a batch normalization process on . In an exemplary embodiment, batch normalization layer 448 may be configured to perform the batch normalization process. In an exemplary embodiment, each normalized feature map of (i+1)^th plurality of normalized feature maps 442 may be obtained from a respective filtered feature map of.

In further detail with regard to step 346, in an exemplary embodiment, generating an (i+1)^th plurality of non-linear feature maps 444 may include implementing an i^th non-linear activation function on each of (i+1)^th plurality of normalized feature maps 442. In an exemplary embodiment, i^th activation layer 450 may be configured to implement the i^th non-linear activation function.

For further detail with respect to step 348, in an exemplary embodiment, generating (i+1)^th plurality of feature maps 434 may include applying a max pooling on each of (i+1)^th plurality of non-linear feature maps 444. In an exemplary embodiment, the max pooling may be applied utilizing a pooling layer 452. In an exemplary embodiment, generating (i+1)^th plurality of feature maps 434 may further include applying an average pooling on each of (i+1)^th plurality of non-linear feature maps 444. An exemplary pooling layer may apply a window of a predefined size on each of (i+1)^th plurality of non-linear feature maps 444 and obtain a maximum or average of windowed version of each of (i+1)^th plurality of non-linear feature maps 444. Therefore, in an exemplary embodiment, each of (i+1)^th plurality of feature maps 434 may be generated by sweeping the window on each of (i+1)^th plurality of non-linear feature maps 444. As a result, a size of each of (i+1)^th plurality of feature maps 434 may be smaller than a size of a respective non-linear feature map of (i+1)^th plurality of non-linear feature maps 444. Consequently, a number of required operations may proportionally be decreased.

Referring again to FIG. 3G, in an exemplary embodiment, step 340 may include obtaining an (i+1)^th feature set 436 of plurality of feature sets 426 from (i+1)^th plurality of feature maps 434. In further detail regarding step 340, in an exemplary embodiment, obtaining (i+1)^th feature set 436 may include applying a global average pooling on (i+1)^th plurality of non-linear feature maps 444. In an exemplary embodiment, neural network 400 may further include a global average pooling layer 454. In an exemplary embodiment, the global average pooling may be applied utilizing global average pooling layer 454. In an exemplary embodiment, the global average pooling may include obtaining an average of each non-linear feature map of (i+1)^th plurality of non-linear feature maps 444. In an exemplary embodiment, the global average pooling may include obtaining a set of statistical features from each of (i+1)^th plurality of non-linear feature maps 444. For example, global average pooling layer 205 may include obtaining a standard deviation or a moment of an arbitrary order. In an exemplary embodiment, a size of an output of convolutional layer may be proportional to a size of an input of the convolutional layer. In an exemplary embodiment, when a size of the input of the convolutional layer is H×W and a size of a two dimensional filter corresponding to the convolutional layer is M×N, a size of an output of the convolutional layer may become (H−M+1)×(W−N+1). Obtaining each feature set from each of (i+1)^th plurality of non-linear feature maps 444 using global statistics, i.e., using all elements of each of (i+1)^th plurality of non-linear feature maps 444, may result in feature sets of a size that may be independent of a size of each of i^th plurality of feature maps 436. As a result, in an exemplary embodiment, a score distribution 445 with a predefined number of plurality of scores for the video may be obtained from neural network 400 where the video may be of an arbitrary size.

In an exemplary embodiment, obtaining plurality of feature sets 426 from each convolutional layer of plurality of convolutional layers 430 may lead to different combinations of feature maps. In an exemplary embodiment, feature sets obtained from both low convolutional layers (e.g., first convolutional layer 428) and high convolutional layers (e.g., an N^th convolutional layer, where N is a number of plurality of convolutional layers 430) may provide more information to recurrent neural network 402 compared with obtaining features sets solely from the N^th convolutional layer. On the other hand, recurrent neural network 402 may be connected to low convolutional layers through feature sets obtained from feature maps extracted from low convolutional layers. Therefore, a value of gradient may be considerable with respect to weights in low convolutional layers, which may prevent a vanishing gradient phenomenon.

Referring again to FIG. 3F, in an exemplary embodiment, step 328 may include generating the m^th frame-level feature set 420. In an exemplary embodiment, generating the m^th frame-level feature set 420 may include generating the plurality of feature sets 426. FIG. 31 shows a flowchart of a method for generating a first feature set, consistent with one or more exemplary embodiments of the present disclosure. An exemplary method 328B may include a second implementation of step 328. An exemplary method 328B may include generating a first feature set of plurality of feature sets 426. In an exemplary embodiment, method 328 B may include generating a horizontal gradient feature map (step 349), generating a vertical gradient feature map (step 350), obtaining a mean horizontal gradient feature (step 351), obtaining a standard deviation horizontal gradient feature (step 352), obtaining a mean vertical gradient feature (step 353), and obtaining a standard deviation vertical gradient feature (step 354).

In an exemplary embodiment, step 349 may include generating the horizontal gradient feature map. In an exemplary embodiment, the horizontal gradient feature map may be generated by applying a horizontal gradient filter of a size r₁×p₁ on an m^th luminance channel of m^th video frame 424, where r₁ and p₁ are positive integers.

In an exemplary embodiment, step 350 may include generating the vertical gradient feature map. In an exemplary embodiment, the vertical gradient feature map may be generated by applying a vertical gradient filter of a size r₁×p₁ on the m^th luminance channel.

In an exemplary embodiment, step 351 may include obtaining the mean horizontal gradient feature map. An exemplary mean horizontal gradient feature may be obtained by averaging the horizontal gradient feature map. In an exemplary embodiment, the first feature set may include the mean horizontal gradient feature.

In an exemplary embodiment, step 352 may include obtaining a standard deviation horizontal gradient feature. An exemplary standard deviation horizontal gradient feature may be obtained by obtaining a standard deviation of the horizontal gradient feature map. In an exemplary embodiment, the first feature set may further include the standard deviation horizontal gradient feature.

In an exemplary embodiment, step 353 may include obtaining a mean vertical gradient feature. An exemplary mean vertical gradient feature may be obtained by averaging the vertical gradient feature map. In an exemplary embodiment, the first feature set may further include the mean vertical gradient.

In an exemplary embodiment, step 354 may include obtaining the standard deviation vertical gradient feature. An exemplary standard deviation vertical gradient feature may be obtained by obtaining a standard deviation of the vertical gradient feature map. In an exemplary embodiment, the first feature set may further include the standard deviation vertical gradient feature.

In an exemplary embodiment, the horizontal gradient filter together with the vertical gradient filter may provide an approximation of a gradient of the m^th luminance channel. In an exemplary embodiment, the horizontal gradient filter together with the vertical gradient filter may be utilized for extracting edge information from m^th

video frame 424.

FIG. 3J shows a flowchart of a method for generating a second feature set, consistent with one or more exemplary embodiments of the present disclosure. An exemplary method 328C may include a third implementation of step 328. An exemplary method 328C may include generating a second feature set of plurality of feature sets 426. In an exemplary embodiment, method 328C may include generating a Laplacian feature map (step 355), generating a mean Laplacian feature (step 356), and generating a standard deviation Laplacian feature (step 357).

In an exemplary embodiment, step 355 may include generating the Laplacian feature map. An exemplary Laplacian feature map may be generated by applying a Laplacian filter of size r₂ ×p₂ on the m^th luminance channel. In an exemplary embodiment, step 356 may include generating the mean Laplacian feature. An exemplary mean Laplacian feature may be generated by averaging the Laplacian feature map. In an exemplary embodiment, the second feature set may include the mean Laplacian feature.

In an exemplary embodiment, step 357 may include generating the standard deviation Laplacian feature. An exemplary standard deviation Laplacian feature may be generated by obtaining a standard deviation of the Laplacian feature map. In an exemplary embodiment, the second feature set may further include the standard deviation Laplacian feature. An exemplary Laplacian filter may be utilized to highlight regions in m^th video frame 424 with high intensity change. Therefore, Laplacian feature map may include edge information of m^th video frame 424.

FIG. 3K shows a flowchart of a method for generating a third feature set, consistent with one or more exemplary embodiments of the present disclosure. An exemplary method 328D may include a fourth implementation of step 328. An exemplary method 328D may include generating a third feature set of plurality of feature sets 426. In an exemplary embodiment, method 328D may include generating a plurality of mean subtracted contrast normalized (MSCN) values (step 358), generating a first generalized Gaussian distribution (GGD) feature and a second GGD feature (step 359), generating a plurality of pair-wise product sets (step 360), generating a plurality of first asymmetric GGD (AGGD) features, a plurality of second AGGD features, and a plurality of third AGGD features (step 361), generating a plurality of mean AGGD features (step 362), generating an m^th down-sampled luminance channel (step 363), generating a plurality of down-sampled MSCN values (step 364), generating a first down-sampled GGD feature and a second down-sampled GGD feature (step 365), generating a plurality of down-sampled pair-wise product sets (step 366), generating a plurality of first down-sampled AGGD features, a plurality of second down-sampled AGGD features, and a plurality of third down-sampled AGGD features (step 367), and generating a plurality of mean down-sampled AGGD features (step 368).

In an exemplary embodiment, step 358 may include generating the plurality of MSCN values. In an exemplary embodiment, plurality of MSCN values may be generated by generating an (r, p)^th MSCN value of the plurality of MSCN values. In an exemplary embodiment, the (r, p)^th MSCN value may be generated according to an operation defined by the following:

$\begin{array}{cc}{\hat{F}}_1\left(r,p\right)=\frac{F_1\left(r,p\right)-{\mu }_1\left(r,p\right)}{{\sigma }_1\left(r,p\right)+C_1},& \mathrm{Equation}\left(2\right)\end{array}$

where {circumflex over (F)}₁(r, p) is the (r, p)^th MSCN value where r∈[1, H], p∈[1, W], H is a height of m^th video frame 424, W is a width of m^th video frame 424, and C₁ is a first constant number. In an exemplary embodiment, F₁(r, p) may include an (r, p)^th luminance value of a plurality of luminance values. In an exemplary embodiment, the m^th luminance channel may include the (r, p)^th luminance value. In an exemplary embodiment, μ₁(r, p) may include an (r, p)^th first weighted local mean of a plurality of first weighted local means. In an exemplary embodiment, the (r, p)^th first weighted local mean may be defined by the following:

μ₁(r, p)=Σ_k=−K1^KiΣ_l=−L1^L1w_k,lF₁(r+k, p+l),   Equation (3)

where K₁ is a height of a first Gaussian filter, L₁ is a width of the first Gaussian filter, and w_k,l is a (k, l)^th weight of the first Gaussian filter. In an exemplary embodiment, the (k, l)^th weight of the first Gaussian filter may be defined by the following:

$\begin{array}{cc}{w}_{k,l}=\frac{e^{-a_1\left(k^2+l^2\right)}}{{\sum }_{k=-K_1}^{K_1}{\sum }_{l=-L_1}^{L_1}{e}^{-a_1\left(k^2+l^2\right)}},& \mathrm{Equation}\left(4\right)\end{array}$

where α₁ is a second constant number. In an exemplary embodiment, σ₁(r, p) may include an (r, p)^th first weighted local standard deviation of a plurality of first weighted local standard deviations. In an exemplary embodiment, the (r, p)^th first weighted local standard deviation may be defined by the following:

$\begin{array}{cc}{\sigma }_1\left(r,p\right)=\sqrt{{\sum }_{k=-K_1}^{K_1}{\sum }_{l=-L_1}^{L_1}{w_{k,l}\left[F_1\left(r+k,p+l\right)-{\mu }_1\left(r,p\right)\right]}^2},& \mathrm{Equation}\left(5\right)\end{array}$

In an exemplary embodiment, step 359 may include generating the first GGD feature and the second GGD feature. In an exemplary embodiment, the first GGD feature and the second GGD feature may be generated by applying a moment-matching method on the plurality of MSCN values. In an exemplary embodiment, the third feature set may include the first GGD feature and the second GGD feature. In an exemplary embodiment, the moment-matching method may include a GGD defined by the following:

$\begin{array}{cc}{f}_1\left(x;{\alpha }_1,{\beta }_1\right)=\frac{{\alpha }_1}{2{\beta }_1\Gamma \left(\frac{1}{{\alpha }_1}\right)}{e}^{-\frac{{x}^{{\alpha }_1}}{{\beta }_1^{{\alpha }_1}}},& \mathrm{Equation}\left(6\right)\end{array}$

where f₁(.;.,.) is the GGD, Γ(.) is a gamma function, α₁ is the first GDD feature, and β₁ is the second GDD feature.

In an exemplary embodiment, step 360 may include generating the plurality of pair-wise product sets. In an exemplary embodiment, the plurality of pair-wise product sets may be generated by generating a plurality of pair-wise product values. In an exemplary embodiment, a pair-wise product set of the plurality of pair-wise product sets may include a respective plurality of pair-wise product values. In an exemplary embodiment, generating the plurality of pair-wise product values may include generating an (r, p, {tilde over (r)}, {tilde over (p)})^th pair-wise product value of the plurality of pair-wise product values. In an exemplary embodiment, the (r, p, {tilde over (r)}, {tilde over (p)})^th pair-wise product value may be generated according to an operation defined by the following:

G ₁(r, p, {tilde over (r)}, {tilde over (p)})={circumflex over (F)}₁(r, p)×{circumflex over (F)}₁({tilde over (r)}, {tilde over (p)})   Equation (7)

where G₁(r, p, {tilde over (r)}, {tilde over (p)}) is the (r, p, {tilde over (r)}, {tilde over (p)})^th pair-wise product value, |r−{tilde over (r)}|+|p−{tilde over (p)}|=1, {tilde over (r)}∈[1, H], and {tilde over (p)}∈[1, W].

In an exemplary embodiment, step 361 may include generating the plurality of first AGGD features, the plurality of second AGGD features, and the plurality of third AGGD features. In an exemplary embodiment, the plurality of first AGGD features, the plurality of second AGGD features, and the plurality of third AGGD features may be generated by applying the moment-matching method on the plurality of pair-wise product sets. In an exemplary embodiment, each of the plurality of pair-wise product sets may include a respective plurality of first AGGD features, a respective plurality of second AGGD features, and a respective plurality of third AGGD features. In an exemplary embodiment, the third feature set may further include the plurality of first AGGD features, the plurality of second AGGD features, and the plurality of third AGGD features. In an exemplary embodiment, the moment-matching method may include an AGGD defined by the following:

$\begin{array}{cc}{f}_2\left(x;{\gamma }_1,{\beta }_{l,1},{\beta }_{r,1}\right)=\{\begin{array}{cc}\frac{{\gamma }_1}{\left({\beta }_{l,1},{\beta }_{r,1}\right)\Gamma \left(\frac{1}{{\gamma }_1}\right)}{e}^{-\frac{{x}^{{\gamma }_1}}{{\beta }_l^{{\gamma }_1}}},& \forall x\le 0\\ \frac{{\gamma }_1}{\left({\beta }_{l,1},{\beta }_{r,1}\right)\Gamma \left(\frac{1}{{\gamma }_1}\right)}{e}^{-\frac{{x}^{{\gamma }_1}}{{\beta }_{r,1}^{{\gamma }_1}}},& \forall x>0\end{array}& \mathrm{Equation}\left(8\right)\end{array}$

where f₂(.;.,.,.) is the AGGD, γ₁ is a first AGGD feature of the plurality of first AGGD features, is a second AGGD feature of the plurality of second AGGD features, and β_r,1 is a third AGGD feature of the plurality of third AGGD features.

In an exemplary embodiment, step 362 may include generating the plurality of mean AGGD features. In an exemplary embodiment, the plurality of mean AGGD features may be generated by generating each of the plurality of mean AGGD features according to a set of operations defined by the following:

$\begin{array}{cc}{\eta }_1=\left({\beta }_{r,1}-{\beta }_{l,1}\right)\frac{\Gamma \left(\frac{2}{{\gamma }_1}\right)}{\Gamma \left(\frac{1}{{\gamma }_1}\right)},& \mathrm{Equation}\left(9\right)\end{array}$

where η₁ is a mean AGGD feature of the plurality of mean AGGD features. In an exemplary embodiment, the plurality of mean AGGD features may be obtained from a respective pair-wise product set. In an exemplary embodiment, the third feature set may further include the plurality of mean AGGD features.

In an exemplary embodiment, step 363 may include generating the m^th down-sampled luminance channel. In an exemplary embodiment, the m^th down-sampled luminance channel may be generated by down sampling the m^th luminance channel with a factor of 2.

In an exemplary embodiment, step 364 may include generating the plurality of down-sampled MSCN values. In an exemplary embodiment, the plurality of down-sampled MSCN values may be generated by generating a (t, s)^th down-sampled MSCN value of the plurality of down-sampled MSCN values. In an exemplary embodiment, the (t, s)^th down-sampled MSCN value may be generated according to an operation defined by the following:

$\begin{array}{cc}{\hat{F}}_2\left(t,s\right)=\frac{F_2\left(t,s\right)-{\mu }_2\left(t,s\right)}{{\sigma }_2\left(t,s\right)+C_2},& \mathrm{Equation}\left(10\right)\end{array}$

where {circumflex over (F)}₂(t, s) is the (t, s)^th down-sampled MSCN value,

$t\in \left[1,\lfloor \frac{H}{2}\rfloor \right],s\in \left[1,\lfloor \frac{W}{2}\rfloor \right],$

and C₂ is a third constant number. In an exemplary embodiment, F₂(t, s) may include a (t, s)^th down-sampled luminance value of a plurality of down-sampled luminance values. In an exemplary embodiment, the (t, s)^th down-sampled luminance value may be obtained from the m^th down-sampled luminance channel. In an exemplary embodiment, μ₂(t, s) may include a (t, s)^th second weighted local mean of a plurality of second weighted local means. In an exemplary embodiment, the (t, s)^th second weighted local mean may be defined by the following:

μ₂(t, s)=Σ_k=−K2^K2Σ_l=−L2^L2g_k,lF₂(t+k, s+l),   Equation (11)

where K₂ is a height of a second Gaussian filter, L₁ is a width of the second Gaussian filter, and g_k,l is a (k, l)^th weight of the second Gaussian filter. In an exemplary embodiment, the (k, l)^th weight of the second Gaussian filter may be defined by the following:

$\begin{array}{cc}{g}_{k,l}=\frac{e^{-a_2\left(k^2+l^2\right)}}{{\sum }_{k=-K}^K{\sum }_{l=-L}^L{e}^{-a_2\left(k^2+l^2\right)}},& \mathrm{Equation}\left(12\right)\end{array}$

where α₁ is a fourth constant number. In an exemplary embodiment, σ₁(r, p) may include a (t, s)^th second weighted local standard deviation of a plurality of second weighted local standard deviations. In an exemplary embodiment, the (t, s)^th second weighted local standard deviation may be defined by the following:

σ₂(t, s)=√{square root over (Σ_k=−K^KΣ_l=−L^Lg_k,l[F₂(t+k, s+l)−μ₂(t, s)]²)},   Equation (13)

In an exemplary embodiment, step 365 may include generating the first down-sampled GGD feature and the second down-sampled GGD feature. In an exemplary embodiment, the first down-sampled GGD feature and the second down-sampled GGD feature may be generated by applying the moment-matching method on the plurality of down-sampled MSCN values. In an exemplary embodiment, the third feature set may further include the first down-sampled GGD feature and the second down-sampled GGD feature. In an exemplary embodiment, the moment-matching method may include the GGD defined by the following:

$\begin{array}{cc}{f}_1\left(x;{\alpha }_2,{\beta }_2\right)=\frac{{\alpha }_2}{2{\beta }_2\Gamma \left(\frac{1}{{\alpha }_2}\right)}{e}^{-\frac{{x}^{{\alpha }_2}}{{\beta }_2^{{\alpha }_2}}},& \mathrm{Equation}\left(14\right)\end{array}$

where α₂ is the first down-sampled GDD feature, and β₂ is the second down-sampled GDD feature.

In an exemplary embodiment, step 366 may include generating the plurality of down-sampled pair-wise product sets. In an exemplary embodiment, the plurality of down-sampled pair-wise product sets may be generated by generating a plurality of down-sampled pair-wise product values. In an exemplary embodiment, each of the plurality of down-sampled pair-wise product sets may be obtained from the plurality of down-sampled pair-wise product values. In an exemplary embodiment, generating the plurality of down-sampled pair-wise product values may include generating a (t, s, {tilde over (t)}, {tilde over (s)})^th down-sampled pair-wise product value of the plurality of down-sampled pair-wise product values. In an exemplary embodiment, the(t, s, {tilde over (t)}, {tilde over (s)})^th down-sampled pair-wise product value may be generated according to an operation defined by the following:

G ₂(t, s, {tilde over (t)}, {tilde over (s)})={circumflex over (F)}₂(t, s)×{circumflex over (F)}₂({tilde over (t)}, {tilde over (s)}),   Equation (15)

where G₂(t, s, {tilde over (t)}, {tilde over (s)}) is the (t, s, {tilde over (t)}, {tilde over (s)}) down-sampled pair-wise product value,

$t-\tilde{t}+s-\tilde{s}=1,\tilde{t}\in \left[1,\lfloor \frac{H}{2}\rfloor \right],\mathrm{and}$ $s\in \left[1,\lfloor \frac{W}{2}\rfloor \right].$

In an exemplary embodiment, step 367 may include generating the plurality of first down-sampled AGGD features, the plurality of second down-sampled AGGD features, and the plurality of third down-sampled AGGD features. In an exemplary embodiment, the plurality of first down-sampled AGGD features, the plurality of second down-sampled AGGD features, and the plurality of third down-sampled AGGD features may be generated by applying the moment-matching method on the plurality of down-sampled pair-wise product sets. In an exemplary embodiment, each of the plurality of first down-sampled AGGD features, the plurality of second down-sampled AGGD features, and the plurality of third down-sampled AGGD features may be obtained from a respective down-sampled pair-wise product set of the plurality of down-sampled pair-wise product sets. In an exemplary embodiment, the third feature set may further include the plurality of first down-sampled AGGD features, the plurality of second down-sampled AGGD features, and the plurality of third down-sampled AGGD features. In an exemplary embodiment, the moment-matching method may include the AGGD defined by the following:

$\begin{array}{cc}{f}_2\left(x;{\gamma }_2,{\beta }_{l,2},{\beta }_{r,2}\right)=\{\begin{array}{cc}\frac{{\gamma }_2}{\left({\beta }_{l,2},{\beta }_{r,2}\right)\Gamma \left(\frac{1}{{\gamma }_2}\right)}{e}^{-\frac{{x}^{{\gamma }_2}}{{\beta }_{l,2}^{{\gamma }_2}}},& \forall x\le 0\\ \frac{{\gamma }_2}{\left({\beta }_{l,2},{\beta }_{r,2}\right)\Gamma \left(\frac{1}{{\gamma }_2}\right)}{e}^{-\frac{{x}^{{\gamma }_2}}{{\beta }_{r,2}^{{\gamma }_2}}},& \forall x>0\end{array}& \mathrm{Equation}\left(16\right)\end{array}$

where γ₂ is a first down-sampled AGGD feature of the plurality of first down-sampled AGGD features, β_t,2 is a second down-sampled AGGD feature of the plurality of second down-sampled AGGD features, and β_r,2 is a third down-sampled AGGD feature of the plurality of third down-sampled AGGD features.

In an exemplary embodiment, step 368 may include generating the plurality of mean down-sampled AGGD features. In an exemplary embodiment, the plurality of mean down-sampled AGGD features may be generated by generating each of the plurality of mean down-sampled AGGD features according to an operation defined by the following:

$\begin{array}{cc}{\eta }_2=\left({\beta }_{r,2}-{\beta }_{l,2}\right)\frac{\Gamma \left(\frac{2}{\gamma ,2}\right)}{\Gamma \left(\frac{1}{\gamma ,2}\right)},& \mathrm{Equation}\left(17\right)\end{array}$

where η₂ is a mean down-sampled AGGD feature of the plurality of mean down-sampled AGGD features. In an exemplary embodiment, each of mean down-sampled AGGD features may be obtained from a respective down-sampled pair-wise product set. In an exemplary embodiment, third feature set may further include the plurality of mean down-sampled AGGD features.

FIG. 3L shows a flowchart of a method for generating a fourth feature set, consistent with one or more exemplary embodiments of the present disclosure. An exemplary method 328E may include a fifth implementation of step 328. An exemplary method 328 E may include generating a fourth feature set of plurality of feature sets 426. In an exemplary embodiment, method 328E may include obtaining a mean luminance feature (step 369), and obtaining a standard deviation luminance feature (step 370).

In an exemplary embodiment, step 369 may include obtaining the mean luminance feature. An exemplary mean luminance feature may be obtained by averaging the m^th luminance channel. In an exemplary embodiment, fourth feature set may include the mean luminance feature.

In an exemplary embodiment, step 370 may include obtaining the standard deviation luminance feature. An exemplary standard deviation luminance feature may be obtained by obtaining a standard deviation of the m^th luminance channel. In an exemplary embodiment, the fourth feature set may further include the standard deviation luminance feature.

FIG. 3M shows a flowchart of a method for generating a fifth feature set, consistent with one or more exemplary embodiments of the present disclosure. An exemplary method 328F may include a sixth implementation of step 328. An exemplary method 328F may include generating a fifth feature set of plurality of feature sets 426. In an exemplary embodiment, method 328F may include obtaining a first mean chrominance feature (step 371), obtaining a first standard deviation chrominance feature (step 372), obtaining a second mean chrominance feature (step 373), and obtaining a second standard deviation chrominance feature (step 374).

In an exemplary embodiment, step 371 may include obtaining the first mean chrominance feature. An exemplary first mean chrominance feature may be obtained by averaging a first chrominance channel of m^th video frame 424. In an exemplary embodiment, the fifth feature set may include the first mean chrominance feature.

In an exemplary embodiment, step 372 may include obtaining the first standard deviation chrominance feature. In an exemplary embodiment, the first standard deviation chrominance feature may be obtained by obtaining a standard deviation of the first chrominance channel. In an exemplary embodiment, the fifth feature set may further include the first standard deviation chrominance channel.

In an exemplary embodiment, step 373 may include obtaining the second mean chrominance feature. An exemplary second mean chrominance feature may be obtained by averaging a second chrominance channel of m^th video frame 424. In an exemplary embodiment, the fifth feature set may further include the second mean chrominance feature. In an exemplary embodiment, step 374 may include obtaining the second standard deviation chrominance feature. An exemplary second standard deviation chrominance feature may be obtained by obtaining a standard deviation of the second chrominance channel. In an exemplary embodiment, the fifth feature set may further include the second standard deviation chrominance channel.

FIG. 3N shows a flowchart of a method for generating a sixth feature set, consistent with one or more exemplary embodiments of the present disclosure. An exemplary method 328 G may include a seventh implementation of step 328. An exemplary method 328 G may include generating a sixth feature set of plurality of feature sets 426. In an exemplary embodiment, method 328 G may include obtaining a first colorfulness feature map (step 375), obtaining a second colorfulness feature map (step 376), obtaining a first mean colorfulness feature (step 377), obtaining a first standard deviation colorfulness feature (step 378), obtaining a second mean colorfulness feature (step 379), obtaining a second standard deviation colorfulness feature (step 380), obtaining a third mean colorfulness feature (step 381), obtaining a third standard deviation colorfulness feature (step 382), and obtaining an overall colorfulness feature (step 383).

In an exemplary embodiment, step 375 may include obtaining the first colorfulness feature map. An exemplary first colorfulness feature map may be obtained according to an operation defined by the following:

rg=R−G,   Equation (18)

where rg is the first colorfulness feature map, R is a red channel of m^th video frame 424, and G is a green channel of m^th video frame 424.

In an exemplary embodiment, step 376 may include obtaining the second colorfulness feature map. An exemplary second colorfulness feature map may be obtained according to a set of operations defined by the following:

yb=0.5(R+G)−B,   Equation (19)

where yb is the second colorfulness feature map and B is a blue channel of m^th video frame 424.

In an exemplary embodiment, step 377 may include obtaining the first mean colorfulness feature. An exemplary first mean colorfulness feature may be obtained by averaging the first colorfulness feature map. In an exemplary embodiment, the sixth feature set may include the first mean colorfulness feature.

In an exemplary embodiment, step 378 may include obtaining the first standard deviation colorfulness feature. An exemplary first standard deviation colorfulness feature may be obtained by obtaining a standard deviation of the first colorfulness feature map. In an exemplary embodiment, the sixth feature set may further include the first standard deviation colorfulness feature.

In an exemplary embodiment, step 379 may include obtaining the second mean colorfulness feature. An exemplary second mean colorfulness feature may be obtained by averaging the second colorfulness feature map. In an exemplary embodiment, the sixth feature set may further include the second mean colorfulness feature.

In an exemplary embodiment, step 380 may include obtaining the second standard deviation colorfulness feature. An exemplary second standard deviation colorfulness feature may be obtained by obtaining a standard deviation of the second colorfulness feature map. In an exemplary embodiment, the sixth feature set may further include the second standard deviation colorfulness feature.

In an exemplary embodiment, step 381 may include obtaining the third mean colorfulness feature. An exemplary third mean colorfulness feature may be obtained according to an operation defined by the following:

μ_yrgb=√{square root over (μ_rg²+μ_yb²)},   Equation (20)

where μ_yrgb is the third mean colorfulness feature, μ_rg is the first mean colorfulness feature, and μ_yb is the second mean colorfulness feature. In an exemplary embodiment, the sixth feature set may further include third mean colorfulness feature.

In an exemplary embodiment, step 382 may include obtaining the third standard deviation colorfulness feature. An exemplary third standard deviation colorfulness feature may be obtained according to an operation defined by the following:

σ_yrgb=√{square root over (σ_rg²+σ_yb²)},   Equation (21)

where σ_yrgb is the third standard deviation colorfulness feature, σ_rg is the first standard deviation colorfulness feature, and σ_yb is the second standard deviation colorfulness feature. In an exemplary embodiment, the sixth feature set may further include third standard deviation colorfulness feature.

In an exemplary embodiment, step 383 may include obtaining the overall colorfulness feature. An exemplary overall colorfulness feature may be obtained according to an operation defined by:

M _yrgb=σ_yrgb+0.3μ_yrgb,   Equation (22)

where M_yrgb is the overall colorfulness feature. In an exemplary embodiment, the sixth feature set may further include the overall colorfulness feature.

FIG. 30 shows a flowchart of a method for generating a seventh feature set, consistent with one or more exemplary embodiments of the present disclosure. An exemplary method 32811 may include an eighth implementation of step 328. An exemplary method 32811 may include generating a seventh feature set of plurality of feature sets 426. In an exemplary embodiment, method 32811 may include obtaining a difference luminance channel (step 384), obtaining a mean temporal feature (step 385), and obtaining a standard deviation temporal feature (step 386).

In an exemplary embodiment, step 384 may include obtaining the difference luminance channel. Referring to FIG. 4A, an exemplary difference luminance channel may be obtained by subtracting an (m−1)^th luminance channel of an (m−1)^th video frame 455 from the m^th luminance channel.

In an exemplary embodiment, step 385 may include obtaining the mean temporal feature. An exemplary mean temporal feature may be obtained by averaging the difference luminance channel. In an exemplary embodiment, the seventh feature set may include the mean temporal feature.

In an exemplary embodiment, step 386 may include obtaining the standard deviation temporal feature. An exemplary standard deviation temporal feature may be obtained by obtaining a standard deviation of the difference luminance channel. In an exemplary embodiment, the seventh feature set may further include the standard deviation temporal feature.

Referring again to FIG. 3F, in an exemplary embodiment, step 330 may include generating a first recurrent output. In further detail regarding step 330, in an exemplary embodiment, the first recurrent output may be generated utilizing the processor. In an exemplary embodiment, the first recurrent output may be generated by feeding a first frame-level feature set to recurrent neural network 402.

FIG. 4E shows a schematic of a recurrent neural network, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, the zeroth recurrent output may be equal to the zero vector. Referring to FIGS. 4A and 4E, in an exemplary embodiment, feeding the first frame-level feature set to recurrent neural network 402 may include feeding the first frame-level feature set to a long short-term memory (LSTM) network 402A. In an exemplary embodiment, LSTM network 402A may include a plurality of LSTM layers 425. In an exemplary embodiment, LSTM network 402A may include a plurality of feedback connections. In an exemplary embodiment, the plurality of feedback connections may feed a preceding output of LSTM network 402A to a present input of the LSTM network 402A. In an exemplary embodiment, LSTM network 402A may include a plurality of cell states. In an exemplary embodiment, the plurality of cell states may be updated according to the present input. As a result, temporal information of the video may be extracted from the plurality of cell states.

For further detail with regard to step 332, in an exemplary embodiment, an m^th recurrent output 457 may be generated utilizing the processor. In an exemplary embodiment, recurrent neural network 402 may be configured to generated an m^th recurrent output 457 based on an (m−1)^th recurrent output 459.

In an exemplary embodiment, an m^th recurrent output 457 may be generated by feeding m^th frame-level feature set 420 to recurrent neural network 402. In an exemplary embodiment, feeding m^th frame-level feature set 420 to recurrent neural network 402 may include feeding m^th frame-level feature set 420 to the LSTM network.

Referring again to FIG. 3A, in an exemplary embodiment, step 304 may include extracting score distribution 445. In further detail with respect to step 304, FIG. 3P shows a flowchart of for extracting a score distribution, consistent with one or more exemplary embodiments of the present disclosure. Referring again to FIGS. 3P and 4A, in an exemplary embodiment, extracting score distribution 445 may include generating a first fully connected output 456 of a plurality of fully connected outputs 458 (step 388) and generating a k^th fully connected output 460 of plurality of fully connected outputs 458 (step 390). In an exemplary embodiment, score distribution 445 may be extracted utilizing the processor.

For further detail regarding step 388, in an exemplary embodiment, generating first fully connected output 456 may include applying an M^th recurrent output 427 of the plurality of recurrent outputs to first fully connected layer 416. For further detail regarding step 390, in an exemplary embodiment, generating k^th fully connected output 460 may include applying a (k−1)^th fully connected output 462 of plurality of fully connected outputs 458 to a k^th fully connected layer 464 of plurality of fully connected layers 404, where 1≤k≤L, and L is a number of plurality of fully connected layers 404. In an exemplary embodiment, applying (k−1)^th fully connected output 462 to k^th fully connected layer 464 may include implementing an (N+k)^th non-linear activation function on (k−1)^th fully connected output 462. In an exemplary embodiment, implementing the (N+k)^th non-linear activation function may include implementing one of a rectified linear unit (ReLU) function or an exponential linear unit (ELU) function. In an exemplary embodiment, implementing the (N+k)^th non-linear activation function may include implementing other types of activation functions such as leaky ReLU, scaled ELU, parametric ReLU, etc. In an exemplary embodiment, an (N+L)^th non-linear activation function may include a softmax function. In an exemplary embodiment, a zeroth fully connected output may include M^th recurrent output 427. In an exemplary embodiment, an L^th fully connected output 466 of plurality of fully connected outputs 458 may include score distribution 445.

FIG. 5A shows a flowchart of a method for quality assessment of digital media, consistent with one or more exemplary embodiments of the present disclosure. An exemplary method 500 may include dividing the digital media into a plurality of frame subsets (step 501), generating a plurality of frame-level feature sets from the plurality of frame subsets by generating an m^th frame-level feature set (step 502), and extracting a score distribution for a plurality of scores from the plurality of frame-level feature sets (step 504). In an exemplary embodiment, each of the plurality of frame subsets may include a plurality of frames. An exemplary m^th frame-level feature set may be associated with an m^th frame subset of the plurality of frame subsets where m∈[1, M] and M is a number of the plurality of frame subsets. In an exemplary embodiment, the m^th frame subset may be extracted from the m^th frame-level feature set, as described below. An exemplary m^th frame-level feature set may include a plurality of feature sets. An exemplary score distribution may be associated with the digital media. In an exemplary embodiment, the score distribution may include a distribution of scores that may represent the quality of the digital media. An exemplary score distribution may include information about how much consensus or divergence exists for qualifying exemplary digital media among individual ratings.

FIG. 6A shows a schematic of a system for quality assessment of digital media, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, different steps of method 500 may be implemented utilizing a system 600. In an exemplary embodiment, system 600 may include a plurality of feature extraction blocks 601 and a score extraction block 602.

Referring to FIGS. 5A and 6A, in an exemplary embodiment, step 501 may include dividing the digital media into a plurality of frame subsets 604. In an exemplary embodiment, each of plurality of frame subsets 604 (for example, an m^th frame subset 606) may include a plurality of frames. In an exemplary embodiment, the digital media may be divided into plurality of frame subsets 604 utilizing one or more processors, as described below.

Exemplary digital media may include a video that may be recorded at a given frame rate. An exemplary frame rate may refer to a frequency (i.e., rate) of capturing consecutive video frames of the video. In an exemplary embodiment, the frame rate may be determined by a recording device (such as a camera) that may be utilized for recording the video. Therefore, in an exemplary embodiment, dividing the digital media into plurality of frame subsets 604 in step 501 may include arranging a given number of successive video frames of the video in a corresponding frame subset. As a result, each exemplary video frame may be placed in one of plurality of frame subsets 604. In an exemplary embodiment, a number of video frames that may be placed in a frame subset may be set to a value between 0.2×FR and 0.8×FR where FR is the frame rate of the video. In an exemplary embodiment, the above range may be obtained empirically to achieve a more accurate score distribution, as described below.

FIG. 6B shows an audio signal, consistent with one or more exemplary embodiments of the present disclosure. Exemplary digital media may include an audio signal 608. In an exemplary embodiment, dividing the digital media to plurality of frame subsets 604 may include dividing audio signal 608 into a plurality of audio segments 610. In an exemplary embodiment, dividing the audio signal into plurality of audio segments 610 may include obtaining each of plurality of audio segments 610 by applying a window function to a plurality of overlapping portions (for example, portions 608A and 608B) the audio signal 608. An exemplary window function may include a function (for example, a rectangular function) that has non-zero values inside a time interval of a corresponding portion (for example, interval 612 corresponding to portion 608A) and is zero outside said time interval. An exemplary window function may be applied to a corresponding portion of audio signal 608 by multiplying a value of the window function by a value of the corresponding portion at each respective moment inside a time interval of the portion to obtain a corresponding audio segment. For example, an audio segment 610A may be obtained by multiplying values of portion 608 A by values of a corresponding window function at each time instance within interval 612. Each successive pair (for example, portions 608A and 608B) of the plurality of overlapping portions may include an overlap (for example, overlap 614). In an exemplary embodiment, a length of overlap 614 may be set to a value between 0.2 and 0.6 of a length of the window function (i.e., a length of interval 612). In an exemplary embodiment, the above range may be obtained empirically to achieve a more accurate score distribution, as described below.

Referring again to FIGS. 5A and 6B, in an exemplary embodiment, step 502 may include generating a plurality of frame-level feature sets 616 from plurality of frame subsets 604 utilizing plurality of feature extraction blocks 601. Each exemplary frame-level feature set may be generated by a corresponding feature extraction block of plurality of feature extraction blocks 601. For example, an m^th frame-level feature set 616A of plurality of frame-level feature sets 616 may be extracted from m^th frame subset 606 utilizing a feature extraction block 601 A of plurality of feature extraction blocks 601 feature extraction unit 401. In an exemplary embodiment, plurality of feature extraction blocks 601 may be implemented utilizing one or more processors.

Referring again to FIGS. 5A, 6A, and 6B, in an exemplary embodiment, generating m^th frame-level feature set 616A in step 502 may include extracting the plurality of feature sets from an m^th audio segment 610B of plurality of audio segments 610. Exemplary plurality of feature sets may include a fundamental frequency of m^th audio segment 610B. An exemplary fundamental frequency may be calculated by finding a lowest frequency of the frequency spectrum of m^th audio segment 610B in a frequency domain.

Exemplary plurality of feature sets may further include a bandwidth of m^th audio segment 610B. An exemplary bandwidth may be obtained by calculating a difference between a highest frequency of the frequency spectrum of m^th audio segment 610B and a lowest frequency of the frequency spectrum of m^th audio segment 610B in a frequency domain. Exemplary plurality of feature sets may further include a mean of amplitude variations of m^th audio segment 610B. An exemplary mean of amplitude variations may be calculated by averaging amplitude values of m^th audio segment 610B over time. Exemplary plurality of feature sets may further include a standard deviation of the amplitude variations of m^th audio segment 610B. An exemplary standard deviation of the amplitude variations may refer to an amount of deviation of the amplitude variations from an expected value of the amplitude of m^th audio segment 610B. Exemplary plurality of feature sets may further include a zero crossing rate of m^th audio segment 610B. An exemplary zero crossing rate may be calculated by counting a number of sign changes (for example, from positive to negative) of a mathematical function representing audio segment 610B (i.e., the number of times that audio segment 610B crosses the time axis).

Exemplary plurality of feature sets may further include a histogram of wavelet coefficients of m^th audio segment 610B. To obtain an exemplary histogram, wavelet coefficients of a wavelet transform of m^th audio segment 610B may be calculated. Exemplary wavelet coefficients may be obtained by calculating a convolution of audio segment 610B and an exemplary wavelet basis function. An exemplary histogram may then be obtained by acquiring a distribution of different ranges of wavelet coefficient values. Exemplary plurality of feature sets may further include a histogram of fast Fourier transform (FFT) coefficients of m^th audio segment 610B. To obtain an exemplary histogram, FFT coefficients of m^th audio segment 610B may be calculated. Exemplary FFT coefficients may be obtained by applying ann FFT algorithm (such as the Cooley-Tukey algorithm) to m^th audio segment 610B. An exemplary histogram may then be obtained by acquiring a distribution of different ranges of FFT coefficient values. Exemplary plurality of feature sets may further include an audio spectrum centroid of m^th audio segment 610B. An exemplary audio spectrum centroid may be obtained by calculating a first moment of m^th audio segment 610B in a frequency domain. In an exemplary embodiment, the first moment of m^th audio segment 610B in the frequency domain may be calculated by averaging values of the frequency spectrum of m^th audio segment 610B.

For further details with respect to step 502, FIG. 5B shows a flowchart of a method for generating a frame-level feature set from a frame subset, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, generating m^th frame-level feature set 616A in step 502 may include feeding a first plurality of feature maps to a first three dimensional (3D) convolutional layer of a plurality of 3D convolutional layers (step 506), generating an (i+1)^th plurality of feature maps by applying an i^th 3D convolutional layer of the plurality of 3D convolutional layers on an i^th plurality of feature maps where 1≤i≤N and N is a number of the plurality of 3D convolutional layers (step 508), and obtaining each respective feature set of the plurality of feature sets from a respective plurality of feature maps (step 510). An exemplary first plurality of feature maps may include an exemplary m^th frame subset.

FIG. 6C shows a block diagram of a feature extraction block, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, feature extraction block 601A may include a plurality of 3D convolutional layers 618. In an exemplary embodiment, plurality of 3D convolutional layers 618 may include a first 3D convolutional layer 618 A and/or an i^th 3D convolutional layer 618B where 1≤i≤N and N is a number of plurality of 3D convolutional layers 618. In an exemplary embodiment, i^th 3D convolutional layer 618B may generate an (i+1)^th plurality of feature maps 620 from an i^th plurality of feature maps 622 by applying a 3D convolution to i^th plurality of feature maps 622. In an exemplary embodiment, the first (i.e., i=1) plurality of feature maps may include m^th frame subset 606.

In further detail with regard to step 506, in an exemplary embodiment, m^th frame subset 606 may be fed to first 3D convolutional layer 618A by one or more processors. Exemplary one or more processors may receive m^th frame subset 606 from a video capturing device (such as a camera) or a memory. In an exemplary embodiment, first 3D convolutional layer 618A may apply a 3D convolution on m^th frame subset 606, as described below in more detail under step 508.

For further detail with respect to step 508, FIG. 5C shows a flowchart of generating a plurality of feature maps by a 3D convolutional layer, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, generating (i+1)^th plurality of feature maps 620 may include generating an (i+1)^th plurality of filtered feature maps from i^th plurality of feature maps 622 (step S12), generating an (i+1)^th plurality of normalized feature maps from the (i+1)^th plurality of filtered feature maps (step 514), generating an (i+1)^th plurality of non-linear feature maps from the (i+1)^th plurality of normalized feature maps (step 516), and obtaining (i+1)^th plurality of feature maps 620 from the (i+1)^th plurality of non-linear feature maps (step 518), as described in further detail below.

FIG. 6D shows a schematic of a 3D convolutional layer, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, different steps of the flowchart of FIG. 5C may be implemented utilizing i^th 3D convolutional layer 618B. In an exemplary embodiment, i^th 3D convolutional layer 618B may include an i^th 3D convolution filter 624, a batch normalization layer 626, an i^th activation layer 628, and a max pooling layer 630. In an exemplary embodiment, first 3D convolutional layer 618A may have a similar structure and function to i^th 3D convolutional layer 618B, as described below.

In further detail with respect to step 512, in an exemplary embodiment, generating an (i+1)^th plurality of filtered feature maps 632 may include applying i^th 3D convolution filter 624 on i^th plurality of feature maps 622. In an exemplary embodiment, i^th 3D convolution filter 624 may perform a three-dimensional convolution operation on a 3D cube of i^th plurality of feature maps 622. An exemplary three-dimensional convolution operation may be applied on the 3D cube in x, y, and t dimensions where x and y represent horizontal and vertical dimensions of each feature map of i^th plurality of feature maps 622, respectively, and t represents the time dimension. Referring to FIGS. 6C and 6D, in an exemplary embodiment, i^th plurality of feature maps 622 may be generated from m^th frame subset 606 through one or more 3D convolutional layers (such as first 3D convolutional layer 618A). Since m^th frame subset 606 may include a plurality of two-dimensional (2D) video frames that may be sequentially captured through time, each exemplary feature map of i^th plurality of feature maps 622 may include a 2D map in x and y directions that may correspond to a 2D video frame captured at a respective moment. As a result, in an exemplary embodiment, successive feature maps of i^th plurality of feature maps 622 may form a 3D cube by adding third dimension t to x and y dimensions of each 2D map. As a result, by applying i^th 3D convolution filter 624 on an exemplary 3D portion of i^th plurality of feature maps 622, a corresponding portion of (i+1)^th plurality of filtered feature maps 632 may be generated. In an exemplary embodiment, i^th 3D convolution filter 624 may move through i^th plurality of feature maps 622 in three dimensions until 3D convolution is performed on all portions of i^th plurality of feature maps 622, thereby generating all portions of (i+1)^th plurality of filtered feature maps 632.

Referring again to FIGS. 5C and 6C, in an exemplary embodiment, step 514 may include generating an (i+1)^th plurality of normalized feature maps 634 by applying a batch normalization process on (i+1)^th plurality of filtered feature maps 632. An exemplary batch normalization process may normalize (i+1)^th plurality of filtered feature maps 632 utilizing an average and a standard deviation of a set of (i+1)^th plurality of filtered feature maps 632. In an exemplary embodiment, batch normalization layer 626 may be configured to perform the batch normalization process. In an exemplary embodiment, each normalized feature map of (i+1)^th plurality of normalized feature maps 634 may be obtained from a respective filtered feature map of (i+1)^th plurality of filtered feature maps 632. Further detail of an exemplary batch normalization process may be similar to batch normalization in plurality of convolutional layers 202 described above under step 134.

In further detail with regard to step 516, in an exemplary embodiment, generating an (i+1)^th plurality of non-linear feature maps 636 may include implementing an i^th non-linear activation function on each of (i+1)^th plurality of normalized feature maps 634. An exemplary i^th non-linear activation function may refer to a function that takes each of plurality of normalized feature maps 634 as input data and generates each of (i+1)^th plurality of non-linear feature maps 636 (i+1)^th by applying a non-linear transformation on the input data. In an exemplary embodiment, i^th activation layer 628 may be configured to implement the i^th non-linear activation function.

For further detail with respect to step 518, in an exemplary embodiment, generating (i+1)^th plurality of feature maps 620 may include applying a max pooling on each of (i+1)^th plurality of non-linear feature maps 636. An exemplary max pooling may refer to a pooling operation that calculates a maximum value for patches of each of (i+1)^th plurality of non-linear feature maps 636 to generate a respective downsampled (i.e., pooled) feature map. As a result, exemplary max pooling may prevent over-fitting by providing an abstracted form of data representation. In an exemplary embodiment, the max pooling may be applied utilizing max pooling layer 630. In an exemplary embodiment, generating (i+1)^th plurality of feature maps 620 may further include applying an average pooling on each of (i+1)^th plurality of non-linear feature maps 636. An exemplary max pooling layer may apply a window of a predefined size on each of (i+1)^th plurality of non-linear feature maps 636 and may obtain a maximum or average of windowed version of each of (i+1)^th plurality of non-linear feature maps 636. Therefore, in an exemplary embodiment, each of (i+1)^th plurality of feature maps 620 may be generated by sweeping the window on each of (i+1)^th plurality of non-linear feature maps 636. As a result, a size of each of (i+1)^th plurality of feature maps 620 may be smaller than a size of a respective non-linear feature map of (i+1)^th plurality of non-linear feature maps 636. Consequently, a number of required operations may proportionally be decreased.

Referring again to FIGS. 5B, 6C, and 6D, in an exemplary embodiment, m^th frame-level feature set 616 A may include a plurality of feature sets 638. In an exemplary embodiment, step 510 may include obtaining each respective feature set (for example, an (i+1)^th feature set 638A of plurality of feature sets 638 from a respective plurality of non-linear feature maps (for example, (i+1)^th plurality of feature maps 620).

In further detail regarding step 510, in an exemplary embodiment, obtaining (i+1)^th feature set 638 A may include applying a global average pooling on (i+1)^th plurality of non-linear feature maps 636. In an exemplary embodiment, the global average pooling may refer to obtaining an average of each non-linear feature map of (i+1)^th plurality of non-linear feature maps 636. An exemplary global average pooling may include obtaining a set of statistical features from each of (i+1)^th plurality of non-linear feature maps 636. In an exemplary embodiment, system 600 may further include a global average pooling layer 640. An exemplary global average pooling may be applied utilizing global average pooling layer 640. For example, global average pooling layer 640 may obtain a standard deviation or a moment of an arbitrary order. An exemplary size of an output of each 3D convolutional layer may be proportional to a size of an input of the 3D convolutional layer. Therefore, obtaining each feature set from each of (i+1)^th plurality of non-linear feature maps 636 using global statistics, i.e., using all elements of each of (i+1)^th plurality of non-linear feature maps 636, may result in feature sets of a size that may be independent of a size of each of i^th plurality of feature maps 622. As a result, an exemplary score distribution 642 with a predefined number of plurality of scores for the digital media may be obtained from system 600 where the digital media may be of an arbitrary size.

Referring again to FIGS. 5A, 6A, and 6C, in an exemplary embodiment, step 502 may include generating plurality of frame-level feature sets 616. In an exemplary embodiment, generating plurality of frame-level feature sets 616 may include generating m^th frame-level feature set 616A. In an exemplary embodiment, generating m^th frame-level feature set 616A may include generating each of plurality of feature sets 638.

In an exemplary embodiment, any of plurality of feature sets 638 (for example, (i+1)^th feature set 638A) may include a gradient feature set. In an exemplary embodiment, the gradient feature set may refer to a set of features that may include a gradient (i.e., derivative) of different frames in m^th frame subset 606 in different dimensions (for example, horizontal direction x, vertical direction y, or temporal direction t). An exemplary gradient feature set may include a first mean horizontal gradient feature. An exemplary first mean horizontal gradient feature may be obtained by averaging a first horizontal gradient feature map. In an exemplary embodiment, the first horizontal gradient feature map may be generated by applying a horizontal gradient filter (i.e., a 2D filter that may generate derivatives at each pixel in a 2D slice in horizontal direction x and temporal direction t) of a size r₁×p₁ on a horizontal spatiotemporal slice of an m^th luminance channel of m^th frame subset 606 where r₁ and p₁ are positive integers. An exemplary horizontal spatiotemporal slice may include a 2D slice of m^th frame subset 606 in horizontal direction x and temporal direction t. Applying an exemplary horizontal gradient filter to the horizontal spatiotemporal slice may generate a derivative of each pixel in the horizontal spatiotemporal slice in horizontal direction x and temporal direction t.

An exemplary gradient feature set may further include a second mean horizontal gradient feature. In an exemplary embodiment, the second mean horizontal gradient feature may be obtained by averaging a second horizontal gradient feature map. An exemplary second horizontal gradient feature map may be generated by applying the horizontal gradient filter on a vertical spatiotemporal slice of the m^th luminance channel. In an exemplary embodiment, the vertical spatiotemporal slice may include a 2D slice of m^th frame subset 606 in vertical direction y and temporal direction t. Applying an exemplary horizontal gradient filter to the vertical spatiotemporal slice may generate a derivative of each pixel in the vertical spatiotemporal slice in horizontal direction x and temporal direction t.

An exemplary gradient feature set may further include a first standard deviation horizontal gradient feature. In an exemplary embodiment, the first standard deviation horizontal gradient feature may be obtained by calculating a standard deviation of the first horizontal gradient feature map.

An exemplary gradient feature set may further include a second standard deviation horizontal gradient feature. In an exemplary embodiment, the second standard deviation horizontal gradient feature may be obtained by calculating a standard deviation of the second horizontal gradient feature map.

An exemplary gradient feature set may further include a first mean vertical gradient feature. In an exemplary embodiment, the first mean vertical gradient feature may be obtained by averaging a first vertical gradient feature map. In an exemplary embodiment, the first vertical gradient feature map may be generated by applying a vertical gradient filter of a size r₁×p₁ (i.e., a 2D filter that may generate derivatives at each pixel in a 2D slice in vertical direction y and temporal direction t and may have a same size as the horizontal gradient filter) on the horizontal spatiotemporal slice. Applying an exemplary vertical gradient filter to the horizontal spatiotemporal slice may generate a derivative of each pixel in the horizontal spatiotemporal slice in vertical direction y and temporal direction t.

An exemplary gradient feature set may further include a second mean vertical gradient feature. In an exemplary embodiment, the second mean vertical gradient feature may be obtained by averaging a second vertical gradient feature map. An exemplary second vertical gradient feature map may be generated by applying the vertical gradient filter on the vertical spatiotemporal slice. Applying an exemplary vertical gradient filter to the vertical spatiotemporal slice may generate a derivative of each pixel in the vertical spatiotemporal slice in vertical direction y and temporal direction t.

An exemplary gradient feature set may further include a first standard deviation vertical gradient feature. In an exemplary embodiment, the first standard deviation vertical gradient feature may be obtained by calculating a standard deviation of the first vertical gradient feature map.

An exemplary gradient feature set may further include a second standard deviation vertical gradient feature. In an exemplary embodiment, the second standard deviation vertical gradient feature may be obtained by calculating a standard deviation of the second vertical gradient feature map.

In an exemplary embodiment, any of plurality of feature sets 638 (for example, (i+1)^th feature set 638A) may include a Laplacian feature set. In an exemplary embodiment, the Laplacian feature set may refer to a set of features that may include a sum of second order partial derivatives at each pixel of different spatiotemporal slices in two different dimensions (for example, horizontal direction x and temporal direction t or vertical direction y and temporal direction t). An exemplary Laplacian feature set may include a first mean Laplacian feature. In an exemplary embodiment, the first mean Laplacian feature may be generated by averaging a first Laplacian feature map. An exemplary first Laplacian feature map may be generated by applying a Laplacian filter (i.e., a 2D filter that may generate a sum of 2D derivatives at each pixel in a 2D slice in two different dimensions) of a size r₂×p₂ on the horizontal spatiotemporal slice where r₂ and p₂ are positive integers. Applying an exemplary Laplacian filter to the horizontal spatiotemporal slice may generate a sum of 2D derivatives at each pixel in the horizontal spatiotemporal slice in horizontal direction x and temporal direction t.

An exemplary Laplacian feature set may further include a second mean Laplacian feature. In an exemplary embodiment, the second mean Laplacian feature may be generated by averaging a second Laplacian feature map. An exemplary second Laplacian feature map may be generated by applying the Laplacian filter on the vertical spatiotemporal slice. Applying an exemplary Laplacian filter to the vertical spatiotemporal slice may generate a sum of 2D derivatives at each pixel in the vertical spatiotemporal slice in vertical direction y and temporal direction t.

An exemplary Laplacian feature set may further include a first standard deviation Laplacian feature. In an exemplary embodiment, the first standard deviation Laplacian feature may be generated by calculating a standard deviation of the first Laplacian feature map. An exemplary Laplacian feature set may further include a second standard deviation Laplacian feature. In an exemplary embodiment, the second standard deviation Laplacian feature may be generated by calculating a standard deviation of the second Laplacian feature map.

In an exemplary embodiment, any of plurality of feature sets 638 (for example, (i+1)^th feature set 638A) may include a frame-rate feature set. An exemplary a frame-rate feature set may include a frame rate of m^th frame subset 606. An exemplary frame rate may be equal to N/T where N is a number of frames in m^th frame subset 606 and T is a temporal duration of obtaining m^th frame subset 606.

In an exemplary embodiment, step 504 may include extracting score distribution 642 from plurality of frame-level feature sets 616. In an exemplary embodiment, score extraction block 602 may be utilized for extracting score distribution 642 from plurality of frame-level feature sets 616. In an exemplary embodiment, plurality of frame-level feature sets 616 may include different type of statistical features (such as exemplary feature sets described above). In an exemplary embodiment, score extraction block 602 may include a regression layer that may be configured to obtain score distribution 642 by applying a regression method on statistical features of plurality of frame-level feature sets 616. An exemplary regression method may estimate score distribution 642 based on a statistical analysis on plurality of frame-level feature sets 616. Different regression methods may be employed by an exemplary regression layer for estimating score distribution 642, such as support vector regression (SVR), random forest, decision tree, extra trees, Gaussian process, k-nearest neighbors, multilayer perceptron (MLP) regression, ensemble learning methods including bagging, stacking and boosting methods (such as gradient boosting, XGBoost, CatBoost, and LightBoost), etc.

For further detail regarding step 504, FIG. 5D shows a flowchart for extracting a score distribution from a plurality of frame-level feature sets, consistent with one or more exemplary embodiments of the present disclosure. In an exemplary embodiment, extracting score distribution 642 from plurality of frame-level feature sets 616 may include generating a recurrent output by feeding plurality of frame-level feature sets 616 to a recurrent neural network (step 520) and generating score distribution 642 from the recurrent output (step 522).

In further detail with respect to step 520, in an exemplary embodiment, sore distribution extractor 602 may include a recurrent neural network similar to recurrent neural network 402 of FIG. 4B described above. In an exemplary embodiment, plurality of frame-level feature sets 616 may be fed to the recurrent neural network to generate an exemplary recurrent output. In an exemplary embodiment, feeding plurality of frame-level feature sets 616 to the recurrent neural network may include feeding plurality of frame-level feature sets 616 to a LSTM network similar to LSTM network 402A of FIG. 4E that may include a plurality of LSTM layers 425. Further details regarding an exemplary LSTM network may be similar to details of LSTM network 402A, as discussed above.

Referring again to FIG. 5A and 6A, in an exemplary embodiment, step 504 may include extracting score distribution 642. In an exemplary embodiment, further details of step 504 may be similar to step 304 of FIG. 3P. In an exemplary embodiment, sore distribution extractor 602 may further include a plurality of fully connected layers similar to plurality of fully connected layers 404 of FIG. 4A. An exemplary recurrent output may be applied to a first fully connected layer (similar to first fully connected layer 416 of FIG. 4A) of the plurality of fully connected layers to extract score distribution 642. Further detail regarding extracting score distribution 642 utilizing an exemplary plurality of fully connected layers may be similar to that of plurality of fully connected layers 404, as discussed above.

FIG. 7 shows an example computer system 700 in which an embodiment of the present invention, or portions thereof, may be implemented as computer-readable code, consistent with exemplary embodiments of the present disclosure. For example, different steps of method 100, method 300, and method 500 may be implemented in computer system 700 using hardware, software, firmware, tangible computer readable media having instructions stored thereon, or a combination thereof and may be implemented in one or more computer systems or other processing systems. Hardware, software, or any combination of such may embody any of the modules and components in FIGS. 1A-6D.

If programmable logic is used, such logic may execute on a commercially available processing platform or a special purpose device. One ordinary skill in the art may appreciate that an embodiment of the disclosed subject matter can be practiced with various computer system configurations, including multi-core multiprocessor systems, minicomputers, mainframe computers, computers linked or clustered with distributed functions, as well as pervasive or miniature computers that may be embedded into virtually any device.

For instance, a computing device having at least one processor device and a memory may be used to implement the above-described embodiments. A processor device may be a single processor, a plurality of processors, or combinations thereof. Processor devices may have one or more processor “cores.”

An embodiment of the invention is described in terms of this example computer system 500. After reading this description, it will become apparent to a person skilled in the relevant art how to implement the invention using other computer systems and/or computer architectures. Although operations may be described as a sequential process, some of the operations may in fact be performed in parallel, concurrently, and/or in a distributed environment, and with program code stored locally or remotely for access by single or multi-processor machines. In addition, in some embodiments the order of operations may be rearranged without departing from the spirit of the disclosed subject matter.

Processor device 704 may be a special purpose (e.g., a graphical processing unit) or a general-purpose processor device. As will be appreciated by persons skilled in the relevant art, processor device 704 may also be a single processor in a multi-core/multiprocessor system, such system operating alone, or in a cluster of computing devices operating in a cluster or server farm. Processor device 704 may be connected to a communication infrastructure 706, for example, a bus, message queue, network, or multi-core message-passing scheme.

In an exemplary embodiment, computer system 700 may include a display interface 702, for example a video connector, to transfer data to a display unit 730, for example, a monitor. Computer system 700 may also include a main memory 708, for example, random access memory (RAM), and may also include a secondary memory 710. Secondary memory 710 may include, for example, a hard disk drive 712, and a removable storage drive 714. Removable storage drive 714 may include a floppy disk drive, a magnetic tape drive, an optical disk drive, a flash memory, or the like. Removable storage drive 714 may read from and/or write to a removable storage unit 718 in a well-known manner. Removable storage unit 718 may include a floppy disk, a magnetic tape, an optical disk, etc., which may be read by and written to by removable storage drive 714. As will be appreciated by persons skilled in the relevant art, removable storage unit 718 may include a computer usable storage medium having stored therein computer software and/or data.

In alternative implementations, secondary memory 710 may include other similar means for allowing computer programs or other instructions to be loaded into computer system 700. Such means may include, for example, a removable storage unit 722 and an interface 720. Examples of such means may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM, or PROM) and associated socket, and other removable storage units 722 and interfaces 720 which allow software and data to be transferred from removable storage unit 722 to computer system 700.

Computer system 700 may also include a communications interface 724. Communications interface 724 allows software and data to be transferred between computer system 700 and external devices. Communications interface 724 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. Software and data transferred via communications interface 724 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals capable of being received by communications interface 724. These signals may be provided to communications interface 724 via a communications path 726. Communications path 726 carries signals and may be implemented using wire or cable, fiber optics, a phone line, a cellular phone link, an RF link or other communications channels.

In this document, the terms “computer program medium” and “computer usable medium” are used to generally refer to media such as removable storage unit 718, removable storage unit 722, and a hard disk installed in hard disk drive 712. Computer program medium and computer usable medium may also refer to memories, such as main memory 708 and secondary memory 710, which may be memory semiconductors (e.g. DRAMs, etc.).

Computer programs (also called computer control logic) are stored in main memory 708 and/or secondary memory 710. Computer programs may also be received via communications interface 724. Such computer programs, when executed, enable computer system 700 to implement different embodiments of the present disclosure as discussed herein. In particular, the computer programs, when executed, enable processor device 704 to implement the processes of the present disclosure, such as the operations in methods 100, 300, and 500 illustrated respectively by flowchart 100 of FIG. 1A, flowchart 300 of FIG. 3A, and flowchart 500 of FIG. 5A discussed above. Accordingly, such computer programs represent controllers of computer system 700. Where exemplary embodiments of methods 100, 300, and 500 are implemented using software, the software may be stored in a computer program product and loaded into computer system 700 using removable storage drive 714, interface 720, and hard disk drive 712, or communications interface 724.

Embodiments of the present disclosure also may be directed to computer program products including software stored on any computer useable medium. Such software, when executed in one or more data processing device, causes a data processing device to operate as described herein. An embodiment of the present disclosure may employ any computer useable or readable medium. Examples of computer useable mediums include, but are not limited to, primary storage devices (e.g., any type of random access memory), secondary storage devices (e.g., hard drives, floppy disks, CD ROMS, ZIP disks, tapes, magnetic storage devices, and optical storage devices, MEMS, nanotechnological storage device, etc.).

The embodiments have been described above with the aid of functional building blocks illustrating the implementation of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.

EXAMPLE 1

In this example, a performance of a method (analogous to method 100) for quality assessment of an image (analogous to image 210) is demonstrated. Different steps of the method are implemented utilizing a convolutional neural network (analogous to convolutional neural network 200A). The image and a plurality of training images include three channels associated with a red, a green, and a blue channel. The convolutional neural network includes four convolutional layers (each analogous to i^th convolutional layer 208) including 32, 64, 128, and 256 number of two dimensional filters (analogous to i^th plurality of filters 230). All of two dimensional filters are of a 5×5 size. Each convolutional layer includes a batch normalization layer, an ELU non-linear activation function layer, a dropout layer, and a max pooling layer. The dropout layer is configured to apply a first dropout process in which dropout probabilities of convolutional layers are set to about 0.2, 0.2, 0.3, and 0.4 from first (1^st) convolutional layer (analogous to first (1^st) convolutional layer 206) to 4^th convolutional layer, respectively. The pooling layer of each convolutional layer includes a max pooling layer with a window of 3×3 size. A global average layer (analogous to global average pooling layer 205) obtains an average of each of a plurality of feature maps (analogous to (i+1)^th plurality of feature maps 214). The convolutional neural network further includes three fully connected layers (analogous to plurality of fully connected layers 204), including 512, 512, and 5 neurons. A second dropout process is applied to two first fully connected layers with a dropout probability of 0.2. FIG. 8 shows a ground truth distribution and a score distribution of an image, consistent with exemplary embodiments of the present disclosure. As FIG. 8 shows, a score distribution 802 (analogous to score distribution 244) obtained by the convolutional neural network matches a ground truth distribution 804 for the image.

EXAMPLE 2

In this example, a performance of a method (analogous to method 300) for quality assessment of a video is demonstrated. Different steps of the method are implemented utilizing a neural network (analogous to neural network 400). An m^th frame-level feature set (analogous to m^th frame-level feature set 420) includes a plurality of feature sets (analogous to plurality of feature sets 426). The plurality of feature sets include a plurality of hand-crafted feature sets (analogous to the first feature set to the seventh feature set). The plurality of hand-crafted feature sets include 51 features.

The neural network further includes a recurrent neural network (recurrent neural network 402 ). The recurrent neural network includes three LSTM layers (analogous to plurality of LSTM layers 425) including 204, 204, and 102 neurons. The neural network further includes three fully connected layers (analogous to plurality of fully connected layers 404), including 102, 102, and 5 neurons. A second dropout process is applied to two first fully connected layers with a dropout probability of 0.5. FIG. 9 shows a ground truth distribution and a score distribution of a video, consistent with exemplary embodiments of the present disclosure. As FIG. 9 shows, a score distribution 902 (analogous to score distribution 445) obtained by the neural network matches a ground truth distribution 904 for the video.

EXAMPLE 3

In this example, a performance of a method (analogous to method 500) for quality assessment of digital media (analogous to 604) is demonstrated. Different steps of an exemplary method are implemented utilizing a system similar to system 600. Exemplary digital media includes a plurality of frame subsets analogous to plurality of frame subsets 604. Each frame in an exemplary frame subset of the plurality of frame subsets includes three channels associated with a red, a green, and a blue channel. An exemplary system includes four 3D convolutional layers (each analogous to i^th 3D convolutional layer 618B) including 32, 64, 128, and 256 3D convolution filters (each analogous to i^th 3D convolution filter 624). All of 3D convolution filters are of a 5×5×5 size. Each convolutional layer includes a batch normalization layer, an ELU non-linear activation function layer, a dropout layer, and a max pooling layer. The dropout layer is configured to apply a first dropout process in which dropout probabilities of 3D convolutional layers are set to about 0.2, 0.2, 0.3, and 0.3 from a first (1^st) 3D convolutional layer (analogous to first 3D convolutional layer 618A) to a 4^th 3D convolutional layer, respectively. The pooling layer of each 3D convolutional layer includes a max pooling layer with a window of 3×3 size. A global average layer (analogous to global average pooling layer 640) obtains an average of each of a plurality of feature maps (analogous to (i+1)^th plurality of feature maps 620).

An exemplary system further includes a recurrent neural network (analogous to recurrent neural network 402). An exemplary recurrent neural network includes three LSTM layers (analogous to plurality of LSTM layers 425) including 204, 204, and 102 neurons. An exemplary system further includes three fully connected layers (analogous to plurality of fully connected layers 404), including 102, 102, and 5 neurons. A second dropout process is applied to two first fully connected layers with a dropout probability of 0.2. FIG. 10 shows a ground truth distribution and a score distribution of digital media, consistent with exemplary embodiments of the present disclosure. As FIG. 10 shows, a score distribution 1002 (analogous to score distribution 642) obtained by an exemplary matches a ground truth distribution 1004 for the media.

While the foregoing has described what may be considered to be the best mode and/or other examples, it is understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples, and that the teachings may be applied in numerous applications, only some of which have been described herein. It is intended by the following claims to claim any and all applications, modifications and variations that fall within the true scope of the present teachings.

Unless otherwise stated, all measurements, values, ratings, positions, magnitudes, sizes, and other specifications that are set forth in this specification, including in the claims that follow, are approximate, not exact. They are intended to have a reasonable range that is consistent with the functions to which they relate and with what is customary in the art to which they pertain.

The scope of protection is limited solely by the claims that now follow. That scope is intended and should be interpreted to be as broad as is consistent with the ordinary meaning of the language that is used in the claims when interpreted in light of this specification and the prosecution history that follows and to encompass all structural and functional equivalents. Notwithstanding, none of the claims are intended to embrace subject matter that fails to satisfy the requirement of Sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in such a way. Any unintended embracement of such subject matter is hereby disclaimed.

Except as stated immediately above, nothing that has been stated or illustrated is intended or should be interpreted to cause a dedication of any component, step, feature, object, benefit, advantage, or equivalent to the public, regardless of whether it is or is not recited in the claims.

It will be understood that the terms and expressions used herein have the ordinary meaning as is accorded to such terms and expressions with respect to their corresponding respective areas of inquiry and study except where specific meanings have otherwise been set forth herein. Relational terms such as first and second and the like may be used solely to distinguish one entity or action from another without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by “a” or “an” does not, without further constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various implementations. This is for purposes of streamlining the disclosure, and is not to be interpreted as reflecting an intention that the claimed implementations require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed implementation. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

While various implementations have been described, the description is intended to be exemplary, rather than limiting and it will be apparent to those of ordinary skill in the art that many more implementations and implementations are possible that are within the scope of the implementations. Although many possible combinations of features are shown in the accompanying figures and discussed in this detailed description, many other combinations of the disclosed features are possible. Any feature of any implementation may be used in combination with or substituted for any other feature or element in any other implementation unless specifically restricted. Therefore, it will be understood that any of the features shown and/or discussed in the present disclosure may be implemented together in any suitable combination. Accordingly, the implementations are not to be restricted except in light of the attached claims and their equivalents. Also, various modifications and changes may be made within the scope of the attached claims.

Claims

1. A method for quality assessment of digital media, the method comprising: dividing, utilizing one or more processors, the digital media into a plurality of frame subsets, each of the plurality of frame subsets comprising a plurality of frames;generating, utilizing the one or more processors, a plurality of frame-level feature sets by generating an mth frame-level feature set associated with an mth frame subset of the plurality of frame subsets where m∈[1, M] and M is a number of the plurality of frame subsets, the mth frame-level feature set comprising a plurality of feature sets; andextracting, utilizing the one or more processors, a score distribution for a plurality of scores associated with the digital media from the plurality of frame-level feature sets.

2. The method of claim 1, wherein generating the mth frame-level feature set comprises: feeding a first plurality of feature maps to a first three dimensional (3D) convolutional layer of a plurality of 3D convolutional layers associated with a neural network, the first plurality of feature maps comprising the mth frame subset;generating an (i+1)th plurality of feature maps by applying an ith 3D convolutional layer of the plurality of 3D convolutional layers on an ith plurality of feature maps where 1≤i≤N and N is a number of the plurality of 3D convolutional layers; andobtaining each respective feature set of the plurality of feature sets from a respective plurality of feature maps.

3. The method of claim 2, wherein generating the (i+1)th plurality of feature maps comprises: generating an (i+1)th plurality of filtered feature maps by applying an ith 3D convolution filter on the ith plurality of feature maps, the ith 3D convolution filter associated with the ith 3D convolutional layer;generating an (i+1)th plurality of normalized feature maps by applying a batch normalization process associated with the ith 3D convolutional layer on the (i+1)th plurality of filtered feature maps, each normalized feature map of the (i+1)th plurality of normalized feature maps associated with a respective filtered feature map of the (i+1)th plurality of filtered feature maps;generating an (i+1)th plurality of non-linear feature maps by applying an ith non-linear activation function associated with the ith 3D convolutional layer on each of the (i+1)th plurality of normalized feature maps; andobtaining the (i+1)th plurality of feature maps by applying a max pooling associated with the ith 3D convolutional layer on each of the (i+1)th plurality of non-linear feature maps.

4. The method of claim 3, wherein obtaining each respective feature set of the plurality of feature sets comprises applying a global average pooling on a respective plurality of non-linear feature maps, the global average pooling comprising obtaining an average of each non-linear feature map of the respective plurality of non-linear feature maps.

5. The method of claim 1, wherein generating the mth frame-level feature set comprises generating the plurality of feature sets by: generating a gradient feature set of the plurality of feature sets, comprising: generating a first horizontal gradient feature map by applying a horizontal gradient filter of a size r1×r1 on a horizontal spatiotemporal slice of an mth luminance channel of the mth frame subset where r1 and p1 are positive integers;generating a second horizontal gradient feature map by applying the horizontal gradient filter on a vertical spatiotemporal slice of the mth luminance channel;generating a first vertical gradient feature map by applying a vertical gradient filter of a size r1×p1 on the horizontal spatiotemporal slice;generating a second vertical gradient feature map by applying the vertical gradient filter on the vertical spatiotemporal slice;obtaining a first mean horizontal gradient feature by averaging the first horizontal gradient feature map, the first mean horizontal gradient feature associated with the first feature set;obtaining a second mean horizontal gradient feature by averaging the second horizontal gradient feature map, the second mean horizontal gradient feature associated with the gradient feature set;obtaining a first standard deviation horizontal gradient feature by calculating a standard deviation of the first horizontal gradient feature map, the first standard deviation horizontal gradient feature associated with the gradient feature set;obtaining a second standard deviation horizontal gradient feature by calculating a standard deviation of the second horizontal gradient feature map, the second standard deviation horizontal gradient feature associated with the gradient feature set;obtaining a first mean vertical gradient feature by averaging the first vertical gradient feature map, the first mean vertical gradient feature associated with the gradient feature set;obtaining a second mean vertical gradient feature by averaging the second vertical gradient feature map, the second mean vertical gradient feature associated with the gradient feature set;obtaining a first standard deviation vertical gradient feature by calculating a standard deviation of the first vertical gradient feature map, the first standard deviation vertical gradient feature associated with the gradient feature set; andobtaining a second standard deviation vertical gradient feature by calculating a standard deviation of the second vertical gradient feature map, the second standard deviation vertical gradient feature associated with the gradient feature set;generating a Laplacian feature set of the plurality of feature sets, comprising: generating a first Laplacian feature map by applying a Laplacian filter of a size r2×p2 on the horizontal spatiotemporal slice where r2 and p2 are positive integers;generating a second Laplacian feature map by applying the Laplacian filter on the vertical spatiotemporal slice;generating a first mean Laplacian feature by averaging the first Laplacian feature map, the first mean Laplacian feature associated with the Laplacian feature set;generating a second mean Laplacian feature by averaging the second Laplacian feature map, the second mean Laplacian feature associated with the Laplacian feature set;generating a first standard deviation Laplacian feature by calculating a standard deviation of the first Laplacian feature map, the first standard deviation Laplacian feature associated with the Laplacian feature set; andgenerating a second standard deviation Laplacian feature by calculating a standard deviation of the second Laplacian feature map, the second standard deviation Laplacian feature associated with the Laplacian feature set; andgenerating a frame-rate feature set of the plurality of feature sets by obtaining a frame rate of the mth frame subset according to N/T where N is a number of frames in the mth frame subset and T is a temporal duration of obtaining the mth frame subset.

6. The method of claim 5, wherein dividing the digital media to the plurality of frame subsets comprises setting a number of the plurality of frames to a value between 0.2×FR and 0.8×FR where FR is the frame rate.

7. The method of claim 1, wherein extracting the score distribution from the plurality of frame-level feature sets comprises: generating a recurrent output by feeding the plurality of frame-level feature sets to a recurrent neural network; andgenerating the score distribution from the recurrent output.

8. The method of claim 7, wherein feeding the plurality of frame-level feature sets to the recurrent neural network comprises feeding the plurality of frame-level feature set to a long short-term memory (LSTM) network, the LSTM network comprising a plurality of LSTM layers.

9. The method of claim 7, wherein extracting the score distribution from the recurrent output comprises: generating a first fully connected output of a plurality of fully connected outputs by applying the recurrent output to a first fully connected layer of a plurality of fully connected layers; andgenerating a kth fully connected output of the plurality of fully connected outputs by applying a (k−1)th fully connected output of a plurality of fully connected outputs to a kth fully connected layer of the plurality of fully connected layers, wherein applying the (k−1)th fully connected output to the kth fully connected layer comprises applying an (N+k)th non-linear activation function on the (k−1)th fully connected output, an (N+L)th non-linear activation function comprising a softmax function where 1≤k≤L and L is a number of the plurality of fully connected layers, a zeroth fully connected output comprises the recurrent output and an Lth fully connected output of the plurality of fully connected outputs comprises the score distribution.

10. The method of claim 1, wherein extracting the score distribution from the plurality of frame-level feature sets comprises: extracting a plurality of statistical features from the plurality of frame-level feature sets; andobtaining the score distribution by applying a regression method on the plurality of statistical features by feeding the plurality of statistical features to a regression layer.

11. The method of claim 1, wherein dividing the digital media into the plurality of frame subsets comprises dividing an audio signal associated with the digital media into a plurality of audio segments.

12. The method of claim 11, wherein dividing the audio signal to the plurality of audio segments comprises obtaining each of the plurality of audio segments by applying a window function to a plurality of overlapping portions of the audio signal, each successive pair of the plurality of overlapping portions comprising an overlap between 0.2 and 0.6 of a length of the window function.

13. The method of claim 11, wherein generating the mth frame-level feature set comprises extracting the plurality of feature sets from an mth audio segment of the plurality of audio segments by: calculating a fundamental frequency of the mth audio segment;calculating a bandwidth of the mth audio segment;calculating a mean of amplitude variations of the mth audio segment;calculating a standard deviation of the amplitude variations of the mth audio segment;calculating a zero crossing rate of the mth audio segment;obtaining a histogram of wavelet coefficients of the mth audio segment;obtaining a histogram of fast Fourier transform (FFT) coefficients of the mth audio segment; andobtaining an audio spectrum centroid of the mth audio segment by calculating a first moment of the mth audio segment in a frequency domain.

14. A system for quality assessment of digital media, the system comprising: a memory having processor-readable instructions stored therein; andone or more processors configured to access the memory and execute the processor-readable instructions, which, when executed by the one or more processors configures the one or more processors to perform a method, the method comprising: dividing the digital media into a plurality of frame subsets, each of the plurality of frame subsets comprising a plurality of frames;generating a plurality of frame-level feature sets by generating an mth frame-level feature set associated with an mth frame subset of the plurality of frame subsets where m∈[1, M] and M is a number of the plurality of frame subsets, the mth frame-level feature set comprising a plurality of feature sets; andextracting a score distribution for a plurality of scores associated with the digital media from the plurality of frame-level feature sets.

15. The system of claim 14, wherein generating the mth frame-level feature set comprises: feeding a first plurality of feature maps to a first three dimensional (3D) convolutional layer of a plurality of 3D convolutional layers associated with a neural network, the first plurality of feature maps comprising the mth frame subset;generating an (i+1)th plurality of feature maps by applying an ith 3D convolutional layer of the plurality of 3D convolutional layers on an ith plurality of feature maps where 1≤i≤N and N is a number of the plurality of 3D convolutional layers; andobtaining each respective feature set of the plurality of feature sets from a respective plurality of feature maps.

16. The system of claim 15, wherein generating the (i+1)th plurality of feature maps comprises: generating an (i+1)th plurality of filtered feature maps by applying an ith 3D convolution filter on the ith plurality of feature maps, the ith 3D convolution filter associated with the ith 3D convolutional layer;generating an (i+1)th plurality of normalized feature maps by applying a batch normalization process associated with the ith 3D convolutional layer on the (i+1)th plurality of filtered feature maps, each normalized feature map of the (i+1)th plurality of normalized feature maps associated with a respective filtered feature map of the (i+1)th plurality of filtered feature maps;generating an (i+1)th plurality of non-linear feature maps by applying an ith non-linear activation function associated with the ith 3D convolutional layer on each of the (i+1)th plurality of normalized feature maps; andobtaining the (i+1)th plurality of feature maps by applying a max pooling associated with the ith 3D convolutional layer on each of the (i+1)th plurality of non-linear feature maps.

17. The system of claim 16, wherein obtaining each respective feature set of the plurality of feature sets comprises applying a global average pooling on a respective plurality of non-linear feature maps, the global average pooling comprising obtaining an average of each non-linear feature map of the respective plurality of non-linear feature maps.

18. The system of claim 14, wherein generating the mth frame-level feature set comprises generating the plurality of feature sets by: generating a gradient feature set of the plurality of feature sets, comprising: generating a first horizontal gradient feature map by applying a horizontal gradient filter of a size r1×p1 on a horizontal spatiotemporal slice of an mth luminance channel of the mth frame subset where r1 and p1 are positive integers;generating a second horizontal gradient feature map by applying the horizontal gradient filter on a vertical spatiotemporal slice of the mth luminance channel;generating a first vertical gradient feature map by applying a vertical gradient filter of a size r1×p1 on the horizontal spatiotemporal slice;generating a second vertical gradient feature map by applying the vertical gradient filter on the vertical spatiotemporal slice;obtaining a first mean horizontal gradient feature by averaging the first horizontal gradient feature map, the first mean horizontal gradient feature associated with the first feature set;obtaining a second mean horizontal gradient feature by averaging the second horizontal gradient feature map, the second mean horizontal gradient feature associated with the gradient feature set;obtaining a first standard deviation horizontal gradient feature by calculating a standard deviation of the first horizontal gradient feature map, the first standard deviation horizontal gradient feature associated with the gradient feature set;obtaining a second standard deviation horizontal gradient feature by calculating a standard deviation of the second horizontal gradient feature map, the second standard deviation horizontal gradient feature associated with the gradient feature set;obtaining a first mean vertical gradient feature by averaging the first vertical gradient feature map, the first mean vertical gradient feature associated with the gradient feature set;obtaining a second mean vertical gradient feature by averaging the second vertical gradient feature map, the second mean vertical gradient feature associated with the gradient feature set;obtaining a first standard deviation vertical gradient feature by calculating a standard deviation of the first vertical gradient feature map, the first standard deviation vertical gradient feature associated with the gradient feature set; andobtaining a second standard deviation vertical gradient feature by calculating a standard deviation of the second vertical gradient feature map, the second standard deviation vertical gradient feature associated with the gradient feature set;generating a Laplacian feature set of the plurality of feature sets, comprising: generating a first Laplacian feature map by applying a Laplacian filter of a size r2×p2 on the horizontal spatiotemporal slice where r2 and p2 are positive integers;generating a second Laplacian feature map by applying the Laplacian filter on the vertical spatiotemporal slice;generating a first mean Laplacian feature by averaging the first Laplacian feature map, the first mean Laplacian feature associated with the Laplacian feature set;generating a second mean Laplacian feature by averaging the second Laplacian feature map, the second mean Laplacian feature associated with the Laplacian feature set;generating a first standard deviation Laplacian feature by calculating a standard deviation of the first Laplacian feature map, the first standard deviation Laplacian feature associated with the Laplacian feature set; andgenerating a second standard deviation Laplacian feature by calculating a standard deviation of the second Laplacian feature map, the second standard deviation Laplacian feature associated with the Laplacian feature set; andgenerating a frame-rate feature set of the plurality of feature sets by obtaining a frame rate of the mth frame subset according to N/T where N is a number of frames in the mth frame subset and T is a temporal duration of obtaining the mth frame subset.

19. The system of claim 14, wherein extracting the score distribution from the plurality of frame-level feature sets comprises: generating a recurrent output by feeding the plurality of frame-level feature sets to a recurrent neural network; andgenerating the score distribution from the recurrent output.

20. The system of claim 19, wherein extracting the score distribution from the recurrent output comprises: generating a first fully connected output of a plurality of fully connected outputs by applying the recurrent output to a first fully connected layer of a plurality of fully connected layers; andgenerating a kth fully connected output of the plurality of fully connected outputs by applying a (k−1)th fully connected output of a plurality of fully connected outputs to a kth fully connected layer of the plurality of fully connected layers,wherein applying the (k−1)th fully connected output to the kth fully connected layer comprises applying an (N+k)th non-linear activation function on the (k−1)th fully connected output, an (N+L)th non-linear activation function comprising a softmax function where 1≤k≤L and L is a number of the plurality of fully connected layers, a zeroth fully connected output comprises the recurrent output and an Lth fully connected output of the plurality of fully connected outputs comprises the score distribution.