Patent Application
20190043222
One or more aspects of embodiments according to the present disclosure relate to displays, and more particularly to a system and method for testing the operation of a display and the quality of an image being displayed.
Electronic devices such as cellular phones, wearable devices, tablet computers, laptops, monitors, and televisions all utilize displays in various forms and formats. Displays are generally composed of patterned arrays of pixels configured to emit light in various colors. These pixels are generally composed of organic light emitting diodes (OLEDs), liquid crystals used in combination with a backlight (e.g., LCDs), micro light emitting diodes (microLEDs), etc. Each pixel generally includes multiple subpixels that are each configured to emit specific wavelengths of light. For example, subpixels typically include red, blue, and green subpixels. These subpixels and pixels may be arranged and driven in a number of ways, with the goal of accurately reproducing an image to the human visual system. A common configuration of pixels is a RGB stripe configuration which has red, blue, and green subpixels in every pixel. In some cases, the subpixels may be subsampled, for example, an RGBG display has a green subpixel for each pixel, but only one red and one blue subpixel for two pixels.
For a variety of reasons, various compression techniques may be applied to an image that is transmitted to the display. In some cases, the compression technique utilized may result in what is termed as a “visually lossless” image being displayed by the display meaning that a typical person would not perceive any artifacts introduced by the compression technique. Some techniques, however, result in perceivable artifacts being noticeable to a user observing a display. The artifacts may include incorrect colors, blur, flickering, aliasing, blockiness, etc.
Both the display itself and any compression used may therefore introduce artifacts into an image being displayed. A number of systems have been created to test both displays and the images being provided to the displays for visual artifacts. For example, subjective test scoring using the ISO/IEC 29170-2:2017 standard procedure using human test subjects and a calibrated monitor is considered to be a highly effective way to evaluate the coding of image sequences. These subjective tests often involve a person viewing tens or hundreds of images and providing an evaluation for each one. Conducting these tests can be time consuming for both the test subjects and the test administrators. Thus, objective testing systems have been utilized to assist in evaluation.
Common objective testing systems utilize a variety of metrics to attempt to quantify visual quality, but they have not proven to be as effective as subjective test scoring. Thus common objective testing systems that use peak signal to noise ratios (PSNRs) and pattern-color sensitivity are typically used for performing preliminary testing before utilizing human test subjects. One of the issues with the objective measures is that they do not incorporate display spatial characteristics (e.g. subpixel configuration) or human perception characteristics. Thus, an improved objective testing system that takes into account display and human characteristics is needed.
The above information is only for enhancement of understanding of the background of embodiments of the present disclosure, and therefore may contain information that does not form the prior art
Some embodiments of the present disclosure provide a system and method for image testing allowing a user to compare images to determine the perceivable differences. The system and method applies at least one display property to a test image to generate a display modified test image and applies the at least one display property to a reference image to generate a display modified reference image. A human eye model is applied to the display modified test image to generate an eye modified test image and the human eye model is applied to the display modified reference image to generate an eye modified reference image. The eye modified test image may then be compared with the eye modified reference image to determine human perceivable differences between the test image and the reference image.
In various embodiments, the at least one display property includes at least one of a subpixel layout or a spectral emission.
In various embodiments, applying at least one display property to the test image to generate the display modified test image includes: generating a subpixel matrix for a display based on the subpixel layout; applying the subpixel matrix to the test image; converting the test image to the spectral domain from 3-color space; and applying the spectral emission to the test image to generate the display modified test image.
In various embodiments, applying at least one display property to the reference image to generate the display modified reference image includes: applying the subpixel matrix to the reference image; converting the reference image to the spectral domain from 3-color space; and applying the spectral emission to the reference image to generate the display modified test image.
In various embodiments, applying human eye model to the display modified test image to generate an eye modified test image comprises applying an optical transfer function to the display modified test image; and applying human eye model to the display modified reference image to generate an eye modified reference image comprises applying the optical transfer function to the display modified reference image.
In various embodiments, the system and method further include: converting the eye modified test image from the spectral domain to 3-color space; converting the eye modified reference image from the spectral domain to 3-color space; applying a cone absorption mosaic to the eye modified test image for a plurality of eye positions to generate a plurality of cone absorption test images; applying a cone absorption mosaic to the eye modified reference image for the plurality of eye positions to generate a plurality of cone absorption reference images; averaging the plurality of cone absorption test images to generate the eye modified test image; and averaging the plurality of cone absorption reference images to generate the eye modified reference image.
In various embodiments, comparing the eye modified test image with the eye modified reference image includes: transforming the eye modified reference image and eye modified test image into CIELAB color space; calculating the Euclidean distance between each pixel in the eye modified reference image and the corresponding pixel in the eye modified reference image; and applying a pooling function to the Euclidean distances.
In various embodiments, the pooling function includes at least one of applying a mean to the Euclidean distances or selecting a value according to a percentile rank.
In various embodiments, the test image may include an image resulting from a lossy compression algorithm being applied to the reference image.
In various embodiments, an image testing system for comparing human perceivable differences between two images, includes: a processor and a memory storing instructions causing the processor to: apply, using a display model module, at least one display property to a test image to generate a display modified test image; apply, using the display model module, the at least one display property to a reference image to generate a display modified reference image; apply, using an eye model module, a human eye model to the display modified test image to generate an eye modified test image; apply, using the eye model module, a human eye model to the display modified reference image to generate an eye modified reference image; and compare, using a comparison module, the eye modified test image with the eye modified reference image to determine human perceivable differences between the test image and the reference image.
In various embodiments, the at least one display property includes at least one of a subpixel layout or a spectral emission.
In various embodiments, the display module is configured to: generate a subpixel matrix for a display based on the subpixel layout; apply the subpixel matrix to the test image; convert the test image to the spectral domain from 3-color space; and apply the spectral emission to the test image to generate the display modified test image.
In various embodiments, the display module is further configured to: apply the subpixel matrix to the reference image; convert the reference image to the spectral domain from 3-color space; and apply the spectral emission to the reference image to generate the display modified test image.
In various embodiments, the eye model module is configured to: apply an optical transfer function to the display modified test image; and apply the optical transfer function to the display modified reference image.
In various embodiments, the eye model module is further configured to: convert the eye modified test image from the spectral domain to 3-color space; convert the eye modified reference image from the spectral domain to 3-color space; apply a cone absorption mosaic to the eye modified test image for a plurality of eye positions to generate a plurality of cone absorption test images; apply a cone absorption mosaic to the eye modified reference image for the plurality of eye positions to generate a plurality of cone absorption reference images; average the plurality of cone absorption test images to generate the eye modified test image; and average the plurality of cone absorption reference images to generate the eye modified reference image.
In various embodiments, the comparison module is configured to: transform the eye modified reference image and eye modified test image into CIELAB color space; calculate the Euclidean distance between each pixel in the eye modified reference image and the corresponding pixel in the eye modified reference image; and applying a pooling function to the Euclidean distances.
In various embodiments, the pooling function includes at least one of applying a mean to the Euclidean distances or selecting a value according to a percentile rank.
In various embodiments, the test image includes an image resulting from a lossy compression algorithm being applied to the reference image.
In various embodiments, a method for comparing images for image testing includes: applying a display model to an input image to generate a display modified input image, wherein the applying the display model includes: generating a subpixel matrix for a display based on a subpixel layout; applying the subpixel matrix to an input image; converting the input image to the spectral domain from 3-color space; applying a spectral emission to the input image to generate the display modified input image; applying an optical transfer function to the display modified input image to generate a retinal modified input image; and comparing the retinal modified input image with a reference image to determine human perceivable differences between the input image and the reference image.
In various embodiments, comparing the retinal modified input image with the reference image includes: transforming the eye modified input image and reference image into CIELAB color space; and calculating the difference between each pixel in the eye modified input image and the corresponding pixel in the reference image.
Some embodiments can be understood in more detail from the following description taken in conjunction with the accompanying drawings, in which:
Features of the inventive concept and methods of accomplishing the same may be understood more readily by reference to the following detailed description of embodiments and the accompanying drawings. Hereinafter, embodiments will be described in more detail with reference to the accompanying drawings, in which like reference numbers refer to like elements throughout. The present disclosure, however, may be embodied in various different forms, and should not be construed as being limited to only the illustrated embodiments herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the aspects and features of the present disclosure to those skilled in the art. Accordingly, processes, elements, and techniques that are not necessary to those having ordinary skill in the art for a complete understanding of the aspects and features of the present disclosure may not be described. Unless otherwise noted, like reference numerals denote like elements throughout the attached drawings and the written description, and thus, descriptions thereof will not be repeated. In the drawings, the relative sizes of elements, layers, and regions may be exaggerated for clarity.
Embodiments of the present disclosure include a system and method for image testing. The system allows for the objective evaluation of image quality being displayed on a given display system. The system allows for the inclusion of information about the display that will be displaying the image such as subpixel arrangement and spectral emission. The system also incorporates the physics of the human eye. A reference and a test image may be supplied to the system which in turns modifies the images according to the display and eye information. The modified images may then be compared to determine the human perceivable visual quality of the test image.
Lossless compression algorithms by definition allow for data to be compressed and perfectly reconstructed into its original form. Lossy compression algorithms are generally capable of further reducing the size of data (e.g., an image or video) at the expense of losing some of the original data. In the field of image and video, a compression algorithm is said to be “visually lossless” when the differences between the original image and the compressed image are imperceptible to a human when displayed on a display. Being able to predict when a compression algorithm will result in a “visually lossless” picture is important for determining the compression parameters.
As discussed above, current methods of evaluating image quality use either subjective testing (e.g., with human evaluations) or objective calculations. Subjective tests include human test subjects reviewing large numbers of images and may include substantial analysis by the test administrator. Conventional objective tests are capable of measuring difference between two images, however, they lack the ability to incorporate how a human would perceive those differences on a real world display. Thus, these conventional methods are either too time consuming (e.g., with subjective testing) or are generally ineffective (e.g., with objective testing). In various embodiments, the present system improves on the previous systems by incorporating display data and human physiological data into the objective evaluation. For example, prior objective systems relied upon the differences between two images.
Embodiments of the present system improve on the deficiencies of past systems by providing an increased accuracy objective testing system. In various embodiments, the testing system employs a computational observer (e.g., a computational human observer) to process the images to find the perceivable differences. For example, in various embodiments, the system is configured to transform the image being tested to account for the specific display being used to show the image. For example, many displays may not exactly reproduce the source image due to their configuration and characteristics. Furthermore, the system may then take into account the perceptional abilities of the human eye. Thus, when the modified images are compared, the comparison shows an estimate of what an actual human would perceive differently when viewing the images on the display. Embodiments of the present disclosure provide for a specific combination of techniques that provide the speed and ease of use of an objective testing system with the accuracy of subjective testing (e.g., an improved correlation).
In various embodiments, the present system provides high levels of versatility to the user. For example, the system may be used with any display and images of any size. Furthermore, a user may apply various human physiological models according to their preferences and current research. The system for image testing utilizes physics-based models, and therefore can be easily modified based on application. Thus, the system employees a set of physics and physiological-based rules to provide improvements on prior systems and methods.
Referring to
In various embodiments, the testing system 100 includes a display model module 110, an eye model module 120, and a comparison module 130. In various embodiments, the test image includes any image that is a modified version of the reference image. For example, the test image may include an image that results from the use of a lossy compression algorithm used on the reference image. In other embodiments, the test image may be an image of the reference image being shown on a display (e.g., to compare the differences between displays).
In various embodiments, the display model module receives the test image and the reference image (S210). In various embodiments, the display model module 110 is configured to apply the physical properties of a display to each of the reference image and the test image (e.g., to modify the image as if it were being displayed by the display). For example, a given display may have a particular subpixel arrangement and a spectral emission.
Referring to
In various embodiments, the display model module 110 also incorporates the spectral emission of a display. The spectral emission of the display, may, for example, be known or may be physically measured using a spectral radiometer. For example, the first display 310 may have a first spectral emission 314, the second display 320 may have a second spectral emission 324, and the third display may have a third spectral emission 334. Once the spectral emission of the display is known, the user may provide the spectral emission to the display model module 110. The spectral emission of the display and the subpixel arrangement may then be used by the display model module 110 to modify an input image (e.g., the reference image and the test image) with the display properties and provide a display modified image to the eye model module (S220). Thus, the reference and test images may be modified based on an actual display that the images could be shown on.
In various embodiments, the eye model module 120 is configured to receive display module modified input images (e.g., the reference image and the test image) and apply a human eye model to the images (S230). In various embodiments, the human eye model may include an optical transfer function modeled on the perception of a human.
Referring to
In various embodiments, the display and eye modified reference and test images are received by the comparison module 130. The comparison module 130 may then determine the differences between the transformed reference and test images (S240). For example, in various embodiments the comparison module 130 may transform the reference and test images into CIELAB color space (e.g., CIE L*a*b*). The transformed images may then be differenced and an output showing the perceivable differences may be generated. Thus, the user may objectively evaluate the quality of the test image.
Referring to
In various embodiments, a computational observer 505 may be configured to receive an input image (S610) and process it based on the display and human physiology. For example, the computational observer 505 may be configured to receive an RGB image (e.g., 3-color space image), apply a display's properties to the image to convert it to a spectral image, apply human physiological properties to the image, and convert the image back to RGB for further processing. In various embodiments, the computational observer 505 may utilize Image System Engineering Toolbox for BIOlogy (ISETBIO) for performing operations. In various embodiments, the computational observer 505 includes a display model module 510 and an eye model module 520.
In various embodiments, the display model module 510 receives the input image and a display subpixel arrangement. The display model module 512 includes a spatial arrangement module for modeling the displays subpixel layout and a spectral properties module 514 for modeling the displays spectral emission. Using the subpixel arrangement, the spatial arrangement module 512 may create a virtual subpixel matrix for the display (S620). The image may then be applied to the virtual subpixel matrix to simulate the subpixel array displaying the input image (S630). In various embodiments, the spectral properties module 514 may receive the simulated subpixel array and convert the image from an RGB image to a spectrum image according to the spectral properties of the display (S640). For example, as described above, a user may measure the actual spectral properties of the display and supply the spectral properties to the spectral properties module 514. In various embodiments, the display modified spectral image may be supplied to the eye model module 520 which applies a human retinal model (S650).
In various embodiments, the eye model module 520 may include a retinal model module 522 for simulating the retinal response to the supplied image and a cone absorption module 524 for simulating the absorption of image on simulated cones. In various embodiments, the retinal model module 522 is configured to apply an optical transfer function to the supplied spectral image (e.g., as discussed above). In various embodiments, the retinal modified image may be converted back to 3-color space from the spectral space (S660). In various embodiments, cone absorption module 524 is configured to simulate the absorption of various colors based on sample arrangements of an eye's cones. The cone absorption module 524 applies cone absorption to the image (S670). The processed image may then be passed from the computational observer module 505 to the comparison module 530 which performs a comparison between the test image and the reference image (S680).
In various embodiments, describing a single pixel in the spectral domain may include 31 values (e.g., 31 8-bit values) instead of the 3 values (e.g., 3 8-bit or 10-bit values) used in the RGB domain. An input image may for example include a relatively high resolution. For example, an input image may include a FullHD resolution (1920×1080 pixels), a 4 k resolution (3840×2160 pixels), an 8 k resolution (7680×4320 pixels), or any other resolution. Thus, converting an image from 3-color space to a spectral image may result in a relatively large size (e.g. a 10× increase in size). In various embodiments, the testing system 500 may be configured to split an input image supplied to the system into a plurality of portions and the test system 500 and method 600 may run multiple times until each full image has been processed. In various embodiments, the each portion of each image may be processed and the comparison module 530 may receive the complete processed test and reference images for comparison. In other embodiments, corresponding portions of the test and reference images may be processed either consecutively or concurrently and the comparison module may compare each corresponding portion.
Referring to
In various embodiments, the cone mosaics may be applied for a plurality of eye positions (e.g., for twenty eye positions). For example, a person's eyes may make a variety of movements due to tremors, drift, and microcascades. Each of the resulting images may then be averaged (S750) and the resulting image may be output (S760).
Referring to
Referring to
In the preceding description, for the purposes of explanation, numerous specific details are set forth to provide a thorough understanding of various embodiments. It is apparent, however, that various embodiments may be practiced without these specific details or with one or more equivalent arrangements. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring various embodiments.
It will be understood that when an element, layer, region, or component is referred to as being “on,” “connected to,” or “coupled to” another element, layer, region, or component, it can be directly on, connected to, or coupled to the other element, layer, region, or component, or one or more intervening elements, layers, regions, or components may be present. However, “directly connected/directly coupled” refers to one component directly connecting or coupling another component without an intermediate component. Meanwhile, other expressions describing relationships between components such as “between,” “immediately between” or “adjacent to” and “directly adjacent to” may be construed similarly. In addition, it will also be understood that when an element or layer is referred to as being “between” two elements or layers, it can be the only element or layer between the two elements or layers, or one or more intervening elements or layers may also be present.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the singular forms “a” and “an” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “have,” “having,” “includes,” and “including,” when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.
As used herein, the term “substantially,” “about,” “approximately,” and similar terms are used as terms of approximation and not as terms of degree, and are intended to account for the inherent deviations in measured or calculated values that would be recognized by those of ordinary skill in the art. “About” or “approximately,” as used herein, is inclusive of the stated value and means within an acceptable range of deviation for the particular value as determined by one of ordinary skill in the art, considering the measurement in question and the error associated with measurement of the particular quantity (i.e., the limitations of the measurement system). For example, “about” may mean within one or more standard deviations, or within ±30%, 20%, 10%, 5% of the stated value. Further, the use of “may” when describing embodiments of the present disclosure refers to “one or more embodiments of the present disclosure.” As used herein, the terms “use,” “using,” and “used” may be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Also, the term “exemplary” is intended to refer to an example or illustration.
When a certain embodiment may be implemented differently, a specific process order may be performed differently from the described order. For example, two consecutively described processes may be performed substantially at the same time or performed in an order opposite to the described order.
Various embodiments are described herein with reference to sectional illustrations that are schematic illustrations of embodiments and/or intermediate structures. As such, variations from the shapes of the illustrations as a result, for example, of manufacturing techniques and/or tolerances, are to be expected. Further, specific structural or functional descriptions disclosed herein are merely illustrative for the purpose of describing embodiments according to the concept of the present disclosure. Thus, embodiments disclosed herein should not be construed as limited to the particular illustrated shapes of regions, but are to include deviations in shapes that result from, for instance, manufacturing. For example, an implanted region illustrated as a rectangle will, typically, have rounded or curved features and/or a gradient of implant concentration at its edges rather than a binary change from implanted to non-implanted region. Likewise, a buried region formed by implantation may result in some implantation in the region between the buried region and the surface through which the implantation takes place. Thus, the regions illustrated in the drawings are schematic in nature and their shapes are not intended to illustrate the actual shape of a region of a device and are not intended to be limiting.
The electronic or electric devices and/or any other relevant devices or components according to embodiments of the present disclosure described herein may be implemented utilizing any suitable hardware, firmware (e.g. an application-specific integrated circuit), software, or a combination of software, firmware, and hardware. For example, the various components of these devices may be formed on one integrated circuit (IC) chip or on separate IC chips. Further, the various components of these devices may be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on one substrate. Further, the various components of these devices may be a process or thread, running on one or more processors, in one or more computing devices, executing computer program instructions and interacting with other system components for performing the various functionalities described herein. The computer program instructions are stored in a memory which may be implemented in a computing device using a standard memory device, such as, for example, a random access memory (RAM). The computer program instructions may also be stored in other non-transitory computer readable media such as, for example, a CD-ROM, flash drive, or the like. Also, a person of skill in the art should recognize that the functionality of various computing devices may be combined or integrated into a single computing device, or the functionality of a particular computing device may be distributed across one or more other computing devices without departing from the spirit and scope of the exemplary embodiments of the present disclosure.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present disclosure belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and/or the present specification, and should not be interpreted in an idealized or overly formal sense, unless expressly so defined herein.
The foregoing is illustrative of example embodiments, and is not to be construed as limiting thereof. Although a few example embodiments have been described, those skilled in the art will readily appreciate that many modifications are possible in the example embodiments without materially departing from the novel teachings and advantages of example embodiments. Accordingly, all such modifications are intended to be included within the scope of example embodiments as defined in the claims. In the claims, means-plus-function clauses are intended to cover the structures described herein as performing the recited function and not only structural equivalents but also equivalent structures. Therefore, it is to be understood that the foregoing is illustrative of example embodiments and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed example embodiments, as well as other example embodiments, are intended to be included within the scope of the appended claims. The inventive concept is defined by the following claims, with equivalents of the claims to be included therein.
The present application claims priority to and the benefit of U.S. Provisional Patent Application Ser. No. 62/542,245, filed Aug. 7, 2017 and entitled “MEASURES FOR VISUALLY LOSSLESS TESTING,” the entire content of which is hereby expressly incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62542245 | Aug 2017 | US |
