Patent Grant
10839499
The present invention relates an image processing apparatus and a superimposed image generation method that are configured to display two or more pieces of image data in a superimposed manner.
Various kinds of technologies have been developed in order to enhance the image quality in video display in television broadcasting and distribute moving images. These days, in addition to the technologies of enhancing resolutions and color gamut, technologies of processing signals of HDR (High Dynamic Range) with luminance range extended is gaining popularity. As compared with conventional SDR (Standard Dynamic Range), HDR has approximately 100 times greater allowable range of luminance, so that an object that is glaring in a real world such as the reflection beam of solar light can be represented more really also on an image.
In not only television broadcasting and distributed moving images but also the world of computer graphics such as game images, realizing the image representation of HDR allows a virtual world more realistic. On the other hand, especially in the case of content that requires the presentation of various kinds of information such as with games, it is desired to display, according to circumstances, various kinds of information such as dialog boxes, help screens, control panels, and indicators as superimposed on a main image. In this case, display technologies are required that a world view pursuing the reality expressed by the main image is not hampered by the hiding and, at the same time, the necessary information can be visually recognized at a glance.
Therefore, the present invention addresses the above-identified and other problems associated with related-art methods and apparatuses and solves the addressed problems by providing a technology that realizes a superimposition display screen well balanced in two or more images regardless of the difference between luminance ranges.
In carrying out the present invention and according to one aspect thereof, there is provided an image processing apparatus. The image processing apparatus has a first image acquisition block configured to acquire data of a first image, a second image acquisition block configured to acquire data of a second image to be displayed on the above-mentioned first image in a superimposed manner, a luminance range adjustment block configured to adjust a luminance range of at least one of the above-mentioned first image and the above-mentioned second image in accordance with a predetermined rule, and an output value computation block configured to execute alpha blending processing by use of data of the image adjusted in the luminance range so as to generate data of a superimposed image, thereby outputting the generated data.
In carrying out the present invention and according to another aspect thereof, there is provided a superimposed image generation method. The superimposed image generation method has a step of acquiring data of a first image, a step of acquiring data of a second image to be displayed on the above-mentioned first image in a superimposed manner, a step of adjusting a luminance range of at least one of the above-mentioned first image and the above-mentioned second image in accordance with a predetermined rule stored in a memory, and a step of executing alpha blending processing by use of data of the image adjusted in the luminance range so as to generate data of a superimposed image, thereby outputting the generated data to a display apparatus.
It should be noted that any combinations of the components mentioned above and those obtained by converting the expression of the present invention between a method, an apparatus, a system, a computer program, and a recording medium recording a computer program are also valid as embodiments of the present invention.
According to the present invention, two or more images can be displayed in a superimposed manner with good balance regardless of the difference between luminance ranges.
Now, referring to
The display apparatus 16 is realized by a liquid crystal display, a plasma display, or an organic EL (Electroluminescence) display that is configured to display images. Further, the display apparatus 16 may have a speaker through which audio data is outputted. The input apparatus 14 and the display apparatus 16 may be connected to the information processing apparatus 10 with a cable or in a wireless manner by use of LAN (Local Area Network) or the like. It should be noted that the external shapes of the input apparatus 14, the display apparatus 16, and the information processing apparatus 10 are not limited to those illustrated; for example, two or three thereof may be formed in an integral manner.
The information processing apparatus 10 receives a signal related with a user manipulation from the input apparatus 14, executes the corresponding processing on the signal so as to generate the data of a display image, and outputs the generated data to the display apparatus 16. The information processing apparatus 10 may be any one of a game machine, a personal computer, a tablet terminal, a mobile terminal, and a mobile telephone. In accordance with a form of the information processing apparatus 10 as described above and an application selected by a user, the contents of the processing to be executed by the information processing apparatus 10 may be various.
For example, the information processing apparatus 10 makes an electronic game specified by the user go on in accordance with a user manipulation and generates the data of a game screen thereof with a predetermined frame rate, thereby outputting the generated data. Alternatively, the data of moving images may be acquired from a server via the network 8 that is sequentially decoded for output. Thus, the purpose of use of the information processing apparatus 10 may be various and the contents of the information processing to be executed accordingly are different from each other, so that detailed description is skipped. The following mainly describes a technique of displaying such two or more images in a superimposed manner as images generated as a result of the above-mentioned information processing or images representative of the information to be presented.
Referring to
Making the image 200a be seen through via the image 200b in a region as wide as possible allows the displaying of necessary information without interrupting a world view of the image 200a such as a game or a moving image. In addition, temporally changing the transparency of the image 200b can provide the performance of making the dialog box gradually appear or disappear.
Other various examples than that of the case in which two or more images are displayed in a superimposed manner may be understood by those skilled in the art. In the case of a racing game for example, it may be conceptualized that an image indicative of a bird's eye view of an entire course is additionally displayed to a main image indicative of driver's field of view. In the case of displaying a movie, it may be conceptualized that an image indicative of bibliographic information such as a plot and performers and manipulation panels of such as playback, pause, and fast are additionally displayed. In accordance with situations and display contents, the position, area, and period with regard to the superimposition may be various.
In what follows, an image, as with the image 200a, that is positioned in the background at the time of superimposition is referred to as a first image; an image, as with the image 200b, that is positioned in the foreground at the time of superimposition is referred to as a second image. However, the principles of operation to be described below holds true if three or more images are displayed in a superimposed manner. In order to display images in a superimposed manner by providing transparency as described above, luminance Fout of each pixel of the display image 202 is determined by alpha blending basically expressed by the following equation.
Fout=(1−α)Fb1+αFb2 (Equation 1)
where, Fb1 and Fb2 are luminance of the pixel concerned in the first image and the second image, respectively, and α is a general α value set to the pixel concerned of the second image, namely, a value indicative of transparency, from 0 to 1.0. It should be noted that, in the case of an RGB (Red, Green, Blue) image, luminance Fb1 and Fb2 are set to each of the three channels concerned; here, the luminance of the three channels are generically referred to as luminance Fb1 and Fb2. For example, varying the α value in an entire image from 0 to 1, the color of the second image gradually darkens from the state in which only the first image is displayed, finally displaying the second image in opaqueness. Setting the α value to an intermediate value that is larger than 0 and smaller than 1 makes the second image translucent, thereby making the first image be seen transparently.
Hence, as illustrated in the display image 204, even if the second image is set to a level near opaqueness, a sun 206 having high luminance in the first image, for example, may possibly be seen dominant. As a result, it is possible that the second image is hard to see or displayed letters become scarcely readable. In addition, making the second image be nearer as possible for easier visual recognition denotes going away from the transparency desired as image representation.
Further, a slight change in α value causes a problem of a large change in the luminance of a display image. For example, as with described above, if the intense white (Fb1=10000 nit) of the first image is superimposed with the black (Fb2=0 nit) of the second image, even a change as slight as 0.01 of α value results in a change of 100 nit in luminance Fout of the display image.
Hence, also in the case where gradation is represented by gradually changing α value on an image plane or an image is cross-faded by a temporal change of α value, a smooth change may not be realized. Especially, to a change of α value from 0.99 to 1.0, a drastic change from white to black or the like will take place on the display image.
It should be noted that, even if the second image is generated with a luminance range of HDR and if the luminance range of a region in which letters and diagrams to be shown more clearly are represented is as small as SDR, then the same situation as described above takes place. Therefore, in the present embodiment, in displaying two or more images in a superimposed manner by alpha blending, the luminance range of the images are adjusted in accordance with α value so as to realize the originally intended image representation.
Referring to
The CPU 23 controls the entire information processing apparatus 10 by executing the operating system stored in the storage block 34. In addition, the CPU 23 executes various programs read from a removable recording medium and loaded into the main memory 26 or downloaded via the communication block 32. The communication block 32 may establish communication with an external apparatus such as a server via the network 8, thereby acquiring the data of electronic content such as moving images and send the data generated inside the information processing apparatus 10.
The GPU 24 has a geometry engine function and a rendering processor function and executes drawing processing in accordance with drawing commands from the CPU 23, thereby storing the data of a display image into a frame buffer not depicted. Then, the display image stored in the frame buffer is converted into a video signal which is outputted to the output block 36. The main memory 26, made up of a RAM (Random Access Memory), stores programs and data that are necessary for the execution of processing.
The information processing apparatus 10 has an input information acquisition block 50 for acquiring input information from the input apparatus 14, an information processing block 52 for executing information processing in accordance with a user manipulation in a game or the like, an image generation block 54 for generating data of an image to be displayed, an image data storage block 53 for storing data necessary for the generation of an image, a superimposition processing block 56 for superimposing two or more images with each other, and an image output block 58 for outputting the data of a display image to the display apparatus 16.
The input information acquisition block 50, realized by the input block 38 and the CPU 23, acquires the data indicative of the contents of a user manipulation from the input apparatus 14. Here, a user manipulation may be such manipulations by a user that are executed on a general-purpose information processing apparatus as the selection of an application to be executed or the content to be outputted, the starting or ending of processing, and the inputting of commands. If an image taking apparatus or any one of various sensors is introduced for the input apparatus 14, the input information acquisition block 50 may acquire the data of a taken image or detection results. In addition, the input information acquisition block 50 may acquire the data of electronic content such as a moving image from a server via the network 8. The input information acquisition block 50 supplies the acquired data to the information processing block 52 and the image generation block 54 from time to time.
The information processing block 52, realized by the CPU 23 and the main memory 26, executes information processing of a game and the like on the basis of the data supplied from the input information acquisition block 50. As described above, the processing to be executed by the information processing block 52 is not especially limited in the contents as far as the displaying of an image is involved. The image generation block 54, realized by the GPU 24 and the main memory 26, generates an image to be displayed in accordance with a request from the information processing block 52. For example, if the information processing block 52 executes a game, the image generation block 54 draws a game image in accordance with a user manipulation with a predetermined frame rate by use of the data stored in the image data storage block 53.
Alternatively, the data of a moving image specified by the information processing block 52 may be decoded and expanded. The moving image may be one acquired from an image taking apparatus that is the input apparatus 14, one distributed from a server via the network 8, or one stored in the image data storage block 53. Subsequently, a game image or a moving image generated as described above is referred to as a first image. Further, the image generation block 54 generates a second image as required. The second image may be one that is displayed from time to time on demand by a user or one that displayed with a timing determined by the information processing block 52 in accordance with the progress of a game or a moving image. Alternatively, the second image may be one that is normally displayed in a corner for example of a screen.
For generating an image concerned, the data stored in the image data storage block 53 may be used or the data acquired by the input information acquisition block 50 from an external apparatus such as an image taking apparatus or a server may be used. Incidentally, the second image is an image that is displayed on the first image in a superimposed manner, in which the degree of display transparency is set by α value in advance. The α value may be set for each pixel or region or fixed for the entire image. Further, the setting concerned may be temporally changed.
During a period in which it is unnecessary for the second image to be superimposed, the image generation block 54 outputs the data of the generated first image to the image output block 58. During a period in which it is necessary for the second image to be superimposed, the image generation block 54 outputs the data of the first image and the second image to the superimposition processing block 56.
The superimposition processing block 56, realized by the CPU 23, the GPU 24, and the main memory 26, has a first image acquisition block 60 for acquiring the data of the first image, the second image acquisition block 62 for acquiring the data of the second image, the luminance range adjustment block 64 for adjusting luminance range, an adjustment rule storage block 66 for storing the adjustment rule of luminance range, and an output value computation block 68 for computing a pixel value of a display image resulted from the superimposition of the second image on the first image.
The first image acquisition block 60 and the second image acquisition block 62 acquire the data of the first image and the data of the second image respectively from the image generation block 54. As described above, the data of the second image includes the setting data of α value. On the basis of the setting of the α value concerned, the luminance range adjustment block 64 adjusts the luminance range of the first image, the luminance range of the second image, or the luminance ranges of both as required. Qualitatively, the luminance that is excessive in the entire or a part of the region is reduced so as to make the balance between the luminance levels of both be put in an intended state upon superimposition, allowing the visual recognition of required information at a glance. Here, the degree of this reduction is changed with the α value set to the second image.
For example, assume the case in which the luminance range of the first image is large and therefore may be a subject of adjustment. In the setting where α value is near 0, the center of gravity is possibly placed on the first image rather than the second image, so that the luminance range of the first image may not be reduced much. On the other hand, in the setting where α value is near 1, the center of gravity is possibly placed on the second image rather than the first image, so that the degree of the reduction in the luminance range of the first image is set large, thereby preventing the interference to the second image. Thus, estimating an intended balance through the setting of α value and accordingly adjusting the luminance range allow the necessary information to be surely presented without unnecessarily suppressing the high contrast realized on an image.
In the case where the first image and the second image have the luminance ranges approximately equal to each other or there is no need for suppressing the luminance in terms of expression, the adjustment processing may be skipped. Whether or not the adjustment is to be executed may be set for each image, each region, or each pixel. Such setting information is supplied by the image generation block 54 along with the data of the second image. The luminance range adjustment block 64 references the adjustment rule stored in the adjustment rule storage block 66 in advance so as to execute the adjustment. If two or more adjustment rules are stored, the image generation block 54 specifies any one of the rules or sends the notification of necessary parameters. A specific example of adjustment rules will be described later.
The output value computation block 68 executes alpha blending by use of the pixel values of the first image and the second image with the luminance ranges adjusted as required, thereby computing the pixel values of a display image. In the case of an RGB image, the computation is executed for each channel. The output value computation block 68 outputs the results of the computation to the image output block 58. The image output block 58, realized by the GPU 24, the main memory 26, and the output block 36, sequentially outputs, to the display apparatus 16, the data of the first image supplied from the image generation block 54 or the data of the image superimposed with the second image supplied from the superimposition processing block 56.
Consequently, an image superimposed with another image as required is displayed on a main image such as a game image or a moving image with an intended balance regardless of the original luminance range. It should be noted that the superimposition processing block 56 may be realized as an image processing apparatus that inputs the data of two types of images from the outside and externally outputs the data of the image obtained by superimposing these two types of images with each other. Alternatively, separate from an apparatus that processes a game or the like so as to generate the first image, an apparatus having the function of generating the second image and the function of the superimposition processing block 56 may be arranged as an image processing apparatus.
The following describes a technique of luminance range adjustment that is executed by the luminance range adjustment block 64. Referring to
The horizontal axis of the graph is indicative of the α value to be set to the second image, allowing the setting from 0 to 1.0. It should be noted that the maximum luminance FH and FS may be the maximum values of the luminance definable with each of HDR and SDR or the maximum values of the luminance actually given as a pixel value on each image. In the former, FH takes a value of 10000 nit, 1000 nit, or 2000 nit, and FS takes a value of 100 nit, 400 nit, or 280 nit. In description below, the same holds true.
As depicted in the above-mentioned graph, since the adjustment of luminance range is basically unnecessary if α=0, the maximum luminance becomes FH. Then, as the setting value of α gets larger, the maximum luminance gets smaller, so that the amount of adjustment is controlled such that the maximum luminance reaches FS when α=1.0. This setup allows the optimization of a luminance range such that the luminance range corresponds to the intended balance in visual recognition estimated from the setting of α value. In addition, in a mode where α value is gradually varied on an image plane or along the time axis, the smooth variation in transparency can be realized also with respect to displaying.
As described above, a conversion rule for uniquely determine the luminance after adjustment from the original luminance is determined in advance so as to vary the maximum luminance with α value. It should be noted that the illustrated example is illustrative only, being qualitatively indicative of the variation in the amount of adjustment relative to α value; namely, the variation in the actual maximum luminance is not limited. That is, in the above-mentioned graph, the maximum luminance is linearly decreased relative to α value; however, the maximum luminance may be a function of quadratic or higher of α or a discontinuous variation. For example, if it is found beforehand that α value is set to only 0.5 and 1.0 as required by the configuration of screen, only the conversion rule for the α value concerned may be set. Further, the conversion rules may be switched between the finite ranges of α value.
The adjustment of the luminance range by the luminance range adjustment block 64 and the alpha blending by the output value computation block 68 are collectively expressed by the following equation:
Fout=(1−α)·f(α,Fb1)+αFb2 (Equation 2)
Where, Fb1 and Fb2 are the luminance of the original pixels of the first image and the second image as with equation 1, f being indicative of a function for converting the luminance of the first image into the luminance after the adjustment. As described above, function f depends on the α value of the second image. For example, when α=0, namely, the second image is transparent, f is the original value, namely,
f(0,Fb1)=Fb1.
When α=1, namely, the second image is opaque, then the f is a function that maximizes the amount of adjustment.
If both are HDR, it is possible that the first image and the second image are to be adjusted together. In this case, extending equation 2 allows the expression as follows:
Fout=(1−α)·f1(α,Fb1)+α·f2(α,Fb2) (Equation 3)
where, f1 is a function for converting the luminance of the first image and f2 is a function for converting the luminance of the second image. According to equation 3, if there is an object having remarkably high luminance in either the first image or the second image, a situation in which another object having relatively low luminance cannot be seen as intended due to the interference of the object of remarkably high luminance at the time of superimposition can be avoided.
It should be noted that, in the present embodiment, the α value of the second image positioned foreground at the time of superimposition is focused, so that function f1 is set such that the luminance range is narrowed relative to the increase in the α value as depicted in
Fout=(1−α)Fb1+α·f(α,Fb2) (Equation 4)
Referring to
That is, maximum value Vp(α) of function f has a variation as illustrated in
If Vp(α) is a linear function as illustrated in
It should be noted that, if the second image is to be adjusted, rewriting α to 1−α for example can linearly decrease an output value as the α value increases. In
It should be noted that, in the description referencing
In place of equation 5, a technology of tone mapping for converting the original image data into the luminance suitable for displaying may be applied. For example, one of the following equations is used for a tone map:
Example 1: N·Fb/(Fb+M)
Example 2: (N·Fb)/((M·Fb/V)+1)
where N an M may be constants or functions dependent on α. For example, if N=1, M(α)=α, then function f can be expressed as follows:
Alternatively, when α=0, a tone map is used to provide a value obtained by interpolating the luminance after adjustment in accordance with the α value when α=0 and α=1.0 in the range of 0<α<1. That is, in 0<α<1, the following function f may be provided:
f(α,Fb)=α·f(1.0,Fb)+(1−α)·f(0,Fb).
It should be note that, in any case, in use of a tone map, not only luminance Fb of a subject pixel but also the value after adjustment may be obtained by use of the luminance values of surrounding pixels. Also, an appropriate combination of a known tone map and the adjustment according to α value may be used. Further, the α value described so far may be replaced by a function of α value. For the function concerned, function A(α) to be described later may be introduced.
If, in equation 6, V=Vp(0) and luminance Fb1 of the first image that is a subject pixel is 10000 nit and luminance Fb2 of the second image is 50 nit, then the luminance of the first image after adjustment is 10000 nit for α=0, 196.1 nit for α=0.5, and 99.0 nit for α=1.0. As a result, luminance Fout after alpha blending is 10000 nit for α=0, 123 nit for α=0.5, and 50 nit for α=1.0 according to equation 2.
Further, if example 2 is applied in a range (α>0.1 for example) of the α values to be used and f(0, Fb)=Fb in α=0, then, in the pixel composition described above, 10000 nit is obtained when α=0, 99.0 nit when α=0.5, and 99.0 nit when α=1.0. As a result, luminance Fout after alpha blending is 10000 nit when α=0, 75 nit when α=0.5, and 50 nit when α=1.0 according to equation 2.
Referring to
In the example illustrated in
Thus, for luminance Fb of the range of such a degree of SDR as not to hamper the visibility after superimposition, the adjustment more than necessary can be avoided by keeping the original value. It should be noted that the number of divisions of the luminance range is not limited to two. Further, in addition to the switching between the execution and non-execution of adjustment for each luminance block, the functions f for specifying conversion rules may be switched. For example, with the function f determined by another method to be described later being a candidate of introduction, an optimum function may be selected for each luminance block.
It should be noted that, especially when combining the functions f determined by different methods, a luminance block for linear or curve interpolation may be arranged so as to continuously vary function f relative to luminance Fb. This arrangement can prevent the generation of a large difference between the luminance after adjustment in the vicinity of the boundary between the luminance blocks.
Referring to
Referring to
f(α,Fb)=min(Vc(α),Fb) (Equation 7)
where, Vc(α) is the upper limit of the luminance after adjustment, being determined in accordance with the α value. In the diagram, the upper limit at the time of α=0, 0.5, and 1.0 is expressed as Vc(0), Vc(0.5), and Vc(1.0).
For example, if the luminance range of an HDR image is reduced to the luminance range of SDR, Vc(0)=FH and Vc(1.0)=FS are set. However, if restraint conditions are considered that, when α=0, the luminance range is not adjusted and, when α=1.0, the luminance is contained within the SDR luminance range, the definition may be made by the following conditions:
Vc(0)≥FH
Vc(1.0)≤FS
Then, Vc(α) is set so as to satisfy the conditions mentioned above within the range of 0≤α≤1. For example, Vc(α) is defined as follows:
Vc(α)=Vc(0)−A(α)·(Vc(0)−Vc(1.0)) (Equation 8)
where, A(α) is a monotonically increasing function that satisfies A(0)=0 and A(1.0)=1.0. That is, a value obtained by executing weighted average on Vc(0) and Vc(1.0) by the weight depending on α according to equation 7 becomes Vc(α). By this definition, the upper limit Vc(α) in accordance with α value is determined and therefore the clamping of luminance with a degree in accordance with α value can be provided in accordance with equation 7.
Function A(α) is appropriately determined in accordance with the contents of an actual image or the purpose of the composition thereof. That is, since function A(α) is an element for determining how much the pixel of high luminance should be clamped in an intermediate value of α, it is desired to optimize the function from the viewpoint whether a desired effect can be achieved by gradually varying the α value in addition to the ease of visual recognition of an image to be actually displayed. The following definitions can be considered for function A(α):
Example 1: A(α)=α;
Example 2: A(α)=α2; and
Example 3: A(α)=1−(1−α)2.
Alternatively, α may be replaced by function B(α) of α in the examples 1 to 3 mentioned above. However, function B(α) is a monotonically increasing function that satisfies B(0)=0 and B(1.0)=1.0. For example, use of function B(α) in place of example 2 mentioned above, equation 8 is expressed as follows:
Vc(α)=Vc(0)−B(α)2·(Vc(0)−Vc(1.0)).
Function B(α) may be defined as follows:
Example 4: when 0≤α≤a1,B(α)=0
a monotonically increasing function from 0 to 1.0 when a1<α≤1.0.
Example 5: when 0≤α≤a2, a monotonically increasing function from 0 to 1.0.
when a2<α≤1.0, B(α)=1.0.
According to example 4 mentioned above, the adjustment results in which the upper limit of luminance does not vary due to the variation of α when 0<α≤a1. According to example 5 mentioned above, the adjustment results in which the upper limit of luminance does not vary due to the variation of α when a2<α≤1.0.
Referring to
Example 6: when 0≤α≤a1,B(α)=0.
when a1≤α≤a2, a monotonically increasing function from 0 to 1.0.
when a2<α≤1.0, B(α)=1.0.
For example, if a1=0.2 and a2=0.5, then B(α)=0 in α≤0.2, so that A(α)=0, resulting in Vc(α)=Vc(0); namely, no clamp occurs. In α>0.5, B(α)=1.0, so that A(α)=1, resulting in Vc(α)=Vc(1.0); namely, the luminance is clamped to Vc(1.0). It should be noted that function B(α) when a1<α≤a2 is not limited to a linear function. In addition, this setting may be made on a discrete α value as described above.
Referring to
Example 6′: when 0≤α≤a,B(α)=0.
when a<α≤1.0, B(α)=1.0.
Thus function B(α) itself may be discontinuously varied.
In this case, according to equation 8,
when 0≤α≤a, Vc(α)=Vc(0).
when a<α≤1.0, Vc(α)=Vc(1.0).
Only when the α value is larger than a, a mode is realized in which the luminance is clamped to Vc(1.0).
In equation 8, various combinations of Vc(0), Vc(1.0), A(α), and B(α) from the candidates as described above can optimize the upper limit of luminance in accordance with situations. For example, when example 1 mentioned above is employed as Vc(0)=FH=10000 nit, Vc(1.0)=FS=100 nit, and function A(α), the upper limit Vc(0.5) of the luminance after adjustment when α=0.5 is 5050 nit from equation 7. As a result, the luminance after adjustment relative to luminance Fb of the original image can be determined by the following function f:
f=min(5050,Fb)
When example 3 is employed in place of example 1 mentioned above as function A(α), the upper limit Vc(0.5) of the luminance after adjustment when α=0.5 is 2575 from equation 8. As a result, the luminance after adjustment relative to luminance Fb of the original image can be determined by function f.
f=min(2575,Fb)
It should be noted that the upper limit Vc(α) of the luminance after adjustment obtained from equation 8 by use of functions A(α) and B(α) illustrated so far can also be applied to the maximum value Vp(α) of each function illustrated in
As described above, the adjustment of a luminance range by the present embodiment is applicable to the entire first image or the second image or only a partial region thereof.
In displaying this image 300 in a translucent manner, it may be considered that the region 302 is subject to the adjustment of luminance range, while the region 304 is out of the adjustment. At this time, the image generation block 54 generates a mask image 310 with the necessity or unnecessity of adjustment expressed by a pixel value as illustrated in the lower of this diagram, for example, thereby supplying the generated mask image to the superimposition processing block 56 along with the data of the second image. The luminance range adjustment block 64 of the superimposition processing block 56 executes the luminance range adjustment only on the region specified by the control entity 310.
Consequently, only the necessary parts can be adjusted in accordance with the configuration of an image. As a result, an intention in the image representation that the second image is superimposingly displayed in a translucent manner and the ease of visual recognition of information can go in a balanced manner. In addition, a luminance range can be adjusted also on a region that does not require detail understanding, thereby avoiding the damage of the world view expressed by the first image such as a game more than necessary. It should be noted that the mask image illustrated in
According to the present embodiment described so far, in the image representation for simultaneously displaying two or more images in a superimposed manner, at least one luminance range is adjusted so as to avoid the occurrence of a remarkable difference in the luminance of superimposed pixels. This setup provides the balanced composition by using the related-art alpha blending mechanism without change also when displaying an image having an extremely wide luminance range such as HDR.
For example, if an image for which it is general to be displayed in a comparatively low luminance level, such as a character string, is composed in a translucent manner on an image desired to be expressed in a real manner with a high contrast such as a taken image or a virtual world based on computer graphics, the latter image tends to be dominant. Therefore, in order to make a character string or the like easy to be visibly recognizable, it may be considered to lower transparency, which damages the originally intended effects of translucence. According to the present embodiment, the desired information can be visibly recognized at a glance, while the impression of an image based on translucence is kept.
Further, by varying the degree of adjustment in accordance with the α value for use in setting transparency or switching between adjustment and non-adjustment on an image plane or between adjustment techniques, the suppression of luminance more than necessary can be avoided, thereby keeping the world view of an image represented with a high contrast as long as possible. In the adjustment of luminance, by properly selecting a function with which the luminance after adjustment is uniquely determined relative to the original luminance, the optimum adjustment in accordance with the contents and configuration of the image and the purpose of composition thereof can be executed on a pixel basis. It is especially effective to control the adjustment with such a fine granularity as with a moving image in which, whether or not a figure of high luminance appears and, if found appearing, the position thereof changes according to situations.
While the embodiment of the present invention has been described using specific terms, such description is for illustrative purpose only, and it is to be understood by those skilled in the art that changes and variations may be made without departing from the spirit or scope of the claims of the present invention.
For example, in the present embodiment, an example of adjusting the range of luminance as the pixel values of three channels of RGB has been mainly described but the subject of range adjustment is not limited thereto; namely, any luminance component expressed by an image can be adjusted in range. Hence, the luminance range adjustment block 64 may execute at least one of the processing of converting a color space for range adjustment on an image to be adjusted and the processing of converting a color space for alpha blending after adjustment. For example, it is also practicable that the data of RGB indicative of the original image is converted into the data of luminance and color difference (YUV), the range of luminance (U) is adjusted to be returned to the RGB data, and then the processing of alpha blending is executed. This technique also achieves the same effects as those provided by the present embodiment.
8 . . . Network, 10 . . . Information processing apparatus, 14 . . . Input apparatus, 16 . . . Display apparatus, 23 . . . CPU, 24 . . . GPU, 26 . . . Main memory, 50 . . . Input information acquisition block, 52 . . . Information processing block, 53 . . . Image data storage block, 54 . . . Image generation block, 56 . . . Superimposition processing block, 58 . . . Image output block, 60 . . . First image acquisition block, 62 . . . Second image acquisition block, 64 . . . Luminance range adjustment block, 66 . . . Adjustment rule storage block, 68 . . . Output value computation block
As described above, the present invention is applicable to various kinds of information processing apparatuses such as a game apparatus, an image processing apparatus, an image reproduction apparatus, and a personal computer, and an information processing system that includes any thereof.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2016-120299 | Jun 2016 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2017/005717 | 2/16/2017 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2017/217011 | 12/21/2017 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 9672603 | Koike | Jun 2017 | B2 |
| 20040095358 | Takagi | May 2004 | A1 |
| 20060274206 | Jang | Dec 2006 | A1 |
| 20100201709 | Yang | Aug 2010 | A1 |
| 20110063325 | Saunders | Mar 2011 | A1 |
| 20120301050 | Wakazono | Nov 2012 | A1 |
| 20150213586 | Koike | Jul 2015 | A1 |
| Number | Date | Country |
|---|---|---|
| 2230662 | Sep 2010 | EP |
| 2012247873 | Dec 2012 | JP |
| 2013044959 | Mar 2013 | JP |
| 2015141333 | Aug 2015 | JP |
| 2016058889 | Apr 2016 | JP |
| Entry |
|---|
| International Search Report for corresponding PCT Application No. PCT/JP2017/005717, 4 pages, dated May 16, 2017. |
| International Preliminary Report on Patentability and Written Opinion for corresponding PCT Application No. PCT/JP2017/005717, 14 pages, dated Oct. 18, 2018. |
| Extended European Search Report for corresponding EP Application No. 17812920, 8 pages, dated Nov. 4, 2019. |
| Number | Date | Country | |
|---|---|---|---|
| 20190355107 A1 | Nov 2019 | US |
