Patent Application
20100002008
1. Field of the Invention
The present invention relates to an image input/output device having an image display function and an image pickup function and a method of inputting images by using the image input/output device, and a method of correcting a photo-reception level in such an image input/output device.
2. Description of the Related Art
In image display devices, there are some display devices including touch panels. In the touch panels, in addition to a resistance type touch panel using a change in electric resistance and a capacitance type touch panel using a change in capacitance, there is an optical type touch panel optically detecting a finger or the like. In the optical type touch panel, for example, light from a backlight is modulated with a liquid crystal element, and an image is displayed on a display plane. Meanwhile, the light emitted from the display plane and reflected by an adjacent object such as a finger is received by photo-reception elements aligned in the display plane. Thereby a position or the like of the adjacent object is detected. Japanese Unexamined Patent Publication No. 2004-127272 discloses such an image display device. The display device described in Japanese Unexamined Patent Publication No. 2004-127272 includes a display section which has a display section displaying an image and an image pickup section picking up an image of an object.
In the above-described image display device, the light emitted from the backlight is reflected by the external adjacent object, and the returned light is detected. However, light intensity of the backlight is not uniform in a display plane in many cases (for example, a central part of the display plane has light intensity higher than that of a surrounding part). Thus, the light intensity of the light reflected by the adjacent object is varied in the central part and the surrounding part. Moreover, although the photo-reception elements receiving the reflected light are aligned along the display plane, photo-reception sensitivity is generally varied in each photo-reception element. Therefore, an image pickup signal obtained from the photo-reception elements is strongly influenced by a light distribution in the display plane of the light emitted from the backlight, and variation of the photo-reception sensitivity among the photo-reception elements.
Under this situation, when an image pickup signal is binarized to determine a position of the adjacent object, binarization may be inappropriately performed due to error of the image pickup signal which is caused by ununiformity in the light intensity of the backlight and variation of the photo-reception sensitivity of each photo-reception element. As a result, there is a risk that the adjacent object is not accurately detected, and position detection error occurs. That is, in the image display device including the optical type touch panel of the related art, it is difficult to detect a finger or the like with high accuracy.
In view of the foregoing, it is desirable to provide an image input/output device capable of detecting an adjacent object such as a finger with high accuracy and a method of inputting images by using the image input/output device, and a method of correcting a photo-reception level in such an image input/output device.
According to an embodiment of the present invention, there is provided an image input/output device including: an input/output panel including a plurality of display elements displaying an image on a display plane based on an image signal, and including a plurality of photo-reception elements arranged along the display plane to receive lights which are emitted from the display plane and then reflected from an external adjacent object; a correction section correcting photo-reception signals from the photo-reception elements with use of an in-plane correction table where correction coefficients are associated with positions on the display plane, respectively, a value distribution profile in the display plane of the correction coefficients being defined based on both of a light intensity distribution profile in the display plane of the light emitted from the display plane and a photo-reception sensitivity distribution profile in the display plane of the photo-reception elements; and an image processing section obtaining information concerning one or more of position, shape and size of the external adjacent object based on the photo-reception signals corrected by the correction section.
In the image input/output device according to the embodiment of the present invention, the input/output panel is configured with a so-called display plane with a touch panel capable of displaying an image with the plurality of display elements and detecting the external adjacent object with the plurality of photo-reception elements. Here, the external adjacent object is, for example, a finger, and reflects the light emitted from the display plane. This reflected light is received by the photo-reception elements, and converted to a photo-reception signal. This photo-reception signal is corrected by the correction section using the in-plane correction table. The in-plane correction table is configured in such a manner that correction coefficients are associated with positions on the display plane, respectively, a value distribution profile in the display plane of the correction coefficients being defined based on both of a light intensity distribution profile in the display plane of the light emitted from the display plane and a photo-reception sensitivity distribution profile in the display plane of the photo-reception elements. Thereby, correction corresponding to the light intensity distribution profile and the photo-reception sensitivity distribution profile in the display plane is performed to the photo-reception signal.
In particular, in the case where the value distribution profile of the correction coefficient is defined as an inversion of a synthesized distribution profile which is a synthesis of the light intensity distribution profile and the photo-reception sensitivity distribution profile, and the correction section corrects the photo-reception signals from the photo-reception elements through multiplying the photo-reception signals by the correction coefficients, respectively, it is possible to obtain the photo-reception signal with a uniform level in which influence from the light intensity distribution profile in the display plane of the emitted light and the photo-reception sensitivity distribution profile in the display plane of the photo-reception elements is canceled.
According to the embodiment of the present invention, there is provided a method of inputting images including: receiving lights by a plurality of photo-reception elements arranged along the display plane, the lights being emitted from a display plane and then reflected from an external adjacent object; correcting photo-reception signals from the photo-reception elements with use of correction coefficients having a value distribution profile in the display plane, the value distribution profile being defined as an inversion of a synthesized distribution profile which is a synthesis of a light intensity distribution profile in the display plane of the lights emitted from the display plane and a photo-reception sensitivity distribution profile in the display plane of the photo-reception elements; and obtaining information concerning one or more of position, shape and size of the external adjacent object based on the photo-reception signals corrected by the correction section.
In the method of inputting the images according to the embodiment of the present invention, correction is performed to the photo-reception signal from each of the photo-reception elements with use of correction coefficients having a value distribution profile in the display plane, the value distribution profile being defined as an inversion of a synthesized distribution profile which is a synthesis of a light intensity distribution profile in the display plane of the lights emitted from the display plane and a photo-reception sensitivity distribution profile in the display plane of the photo-reception elements. As a result, the influence from the light intensity distribution profile in the display plane of the emitted light and the photo-reception sensitivity distribution profile of the photo-reception elements is eliminated from the photo-reception signal.
In the image input/output device according to the embodiment of the present invention, photo-reception signals from the photo-reception elements are corrected with use of an in-plane correction table where correction coefficients are associated with positions on the display plane, respectively, a value distribution profile in the display plane of the correction coefficients being defined based on both of a light intensity distribution profile in the display plane of the light emitted from the display plane and a photo-reception sensitivity distribution profile in the display plane of the photo-reception elements, and information concerning one or more of position, shape and size of the external adjacent object is obtained based on the photo-reception signals corrected by the correction section. Thus, by appropriately setting the correction coefficient, it is possible to perform a suitable correction to the photo-reception signal corresponding to the light intensity distribution profile and the photo-reception sensitivity distribution profile in the display plane. In particular, in the case where the value distribution profile of the correction coefficient is defined as an inversion of a synthesized distribution profile which is a synthesis of the light intensity distribution profile and the photo-reception sensitivity distribution profile, and the correction section corrects the photo-reception signals from the photo-reception elements through multiplying the photo-reception signals by the correction coefficients, respectively, it is possible to obtain the photo-reception signal with a uniform level in which influence from the light intensity distribution profile in the display plane of the emitted light and the photo-reception sensitivity distribution profile in the display plane of the photo-reception elements is canceled. Therefore, it is possible to detect the adjacent object such as a finger with high accuracy.
In the method of inputting the images according to the embodiment of the present invention, correction is performed to the photo-reception signal from each of the photo-reception elements with use of correction coefficients having a value distribution profile in the display plane, the value distribution profile being defined as an inversion of a synthesized distribution profile which is a synthesis of a light intensity distribution profile in the display plane of the lights emitted from the display plane and a photo-reception sensitivity distribution profile in the display plane of the photo-reception elements. As a result, it is possible to obtain the photo-reception signal with a uniform level in which influence from the light intensity distribution profile in the display plane of the emitted light and the photo-reception sensitivity distribution profile in the display plane of the photo-reception elements is canceled. Therefore, it is possible to detect the adjacent object such as a finger with high accuracy.
Other and further objects, features and advantages of the invention will appear more fully from the following description.
A preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.
As illustrated in
As illustrated in
The display signal processing section 12 illustrated in
As illustrated in
The photo-emission side scanner 41 has a function to select a photo-emission cell CW to be driven in response to the photo-emission timing control signal output from the display signal hold control section 40. Specifically, the photo-emission side scanner 41 supplies a photo-emission selection signal to each pixel 16 through a gate line for photo-emission connected to each pixel 16 in the input/output panel 11, and controls the photo-emission device selection switch. That is, when voltage is applied and the photo-emission element selection switch of a certain pixel 16 is turned on with the photo-emission selection signal, photo-emission with luminance corresponding to the voltage supplied from the display signal driver 42 is performed in that pixel 16.
The display signal driver 42 has a function to supply display data to a photo-emission cell CW to be driven in response to the display signal of one horizontal line output from the display signal hold control section 40. Specifically, the display signal driver 42 supplies voltage corresponding to the display data, to the pixel 16 selected by the above-described photo-emission side scanner 41 through a data supply line connected to each pixel 16 in the input/output panel 11. The photo-emission side scanner 41 and the display signal driver 42 line-sequentially operate in conjunction with each other. Thereby, an image corresponding to arbitrary display data is displayed on the input/output panel 11.
The photo-reception side scanner 43 has a function to select a photo-reception cell CR to be driven in response to the photo-reception timing control signal output from the display signal hold control section 40. Specifically, the photo-reception side scanner 43 supplies a photo-reception selection signal to each pixel 16 through a gate line for photo-reception connected to each pixel 16 in the input/output panel 11, and controls a photo-reception element selection switch. That is, similarly to the above-described operation of the photo-emission side scanner 41, when voltage is applied and the photo-reception element selection switch of a certain pixel 16 is turned on with the photo-reception selection signal, the photo-reception signal detected from that pixel 16 is output to a photo-reception signal receiver 45. Thereby, for example, it is possible that the photo-reception cell CR receives and detects the light reflected by the external adjacent object which is contact with or close to the display plane on the basis of the light emitted from a certain photo-emission cell CW. The photo-reception side scanner 43 outputs a photo-reception block control signal to the photo-reception signal receiver 45 and a photo-reception signal hold section 46, and controls these blocks contributing to the photo-reception operation. In the image display device according to the embodiment, the gate line for photo-emission and the gate line for photo-reception described above are separately connected to each photo-emission photo-reception cell CWR. Thereby, the photo-emission side scanner 41 and the photo-reception side scanner 43 operate independently from each other.
The photo-reception signal processing section 13 (correction section) illustrated in
As illustrated in
The photo-reception signal hold section 46 has a function to restructure the photo-reception signal output from the photo-reception receiver 45 into a photo-reception signal for each screen (each display of one field) in response to the photo-reception block control signal output from the photo-reception side scanner 43, and to store and hold the photo-reception signal in, for example, a field memory of SRAM. Data of the photo-reception signal stored in the photo-reception signal hold section 46 is output to a position detection section 47 in the image processing section 14 (
The image processing section 14 (
The electric device main body 20 (
As illustrated in
Next, a method of forming the in-plane correction table 13a will be described.
A reference reflecting plate (not illustrated in the figure) with a surface having a uniform reflectance is located to face the input/output panel 11 so that the whole surface of the input/output panel 11 is covered (S10). Under this situation, all of the photo-emission cells CW (liquid crystal cells) as the display elements 11a are set in a white display state (that is, the highest gray-scale state) with a reference image display signal from the display signal generating section (
Next, a photo-reception signal output from each photo-reception element 11b is read (S12). The photo-reception element 11b used in the embodiment is arranged in each pixel 16. Therefore, to reduce capacity of the memory which stores the in-plane correction table 13a, the plurality of photo-reception elements 11b arranged immediately adjacent to one another constitute one group as described above, and a correction coefficient is obtained for each group. As an example, as illustrated in
Next, average intensity of the photo-reception signals output from the photo-reception elements 11b which constitute each group is calculated and obtained. That is, the average value of the photo-reception intensity of the reflected light is obtained for each group, and the obtained value is regarded as a group average value. Further, the maximum value among the plurality of group average values obtained from the calculation is regarded as a maximum group average value (S14).
Each group average value obtained in S14 is divided by the maximum group average value to obtain a normalized value, and an inverse of the normalized value is obtained by an inverse calculation (S16). The result is regarded as a correction coefficient. In this case, since the above normalized value may be always 1.0 or less, the correction coefficient as the inverse of the normalized value may be always 1.0 or more. For this reason, the capacity of the memory necessary for storing is small in a comparison with the case where the correction coefficient is in a range of 1.0 or less. Moreover, since it is generally considered that the variation of the light intensity distribution profile in the display plane of the backlight and the variation of the photo-reception sensitivity of each photo-reception element 11b are not remarkably large, each group average value is approximately slightly lower than the maximum group average value. Naturally, the value of the correction coefficient as the result of the inverse calculation is within a relatively-narrow range which is approximately slightly above 1.0. Thus, as will be described later, the capacity of the memory necessary for storing is small also from this viewpoint. In this manner, the correction coefficients of all the groups are obtained by performing the calculation of S16 for each group. Thereby, the in-plane correction table 13a, for example, as illustrated in
The in-plane correction table 13a in
Next, operations of the image input/output device 1 according to the embodiment will be described.
The display data output from the electric device main body 20 is input to the display signal processing section 12. The display signal processing section 12 drives the input/output panel 11 so that an image is displayed on the input/output panel 11 on the basis of the display data.
The input/output panel 11 displays the image on the display element 11a by using the light emitted from the backlight. Meanwhile, the input/output panel 11 drives the photo-reception element 11b. When the external adjacent object such as a finger in contact with or close to the display element 11a, the image displayed on the display element 11a is reflected by the external adjacent object, and this reflected light is detected in the photo-reception element 11b. The photo-reception signal is output from the photo-reception element 11b by this detection (S20). At this time, the ununiform state in the light intensity in the plane of the backlight is as indicated with curve A in
The photo-reception signal processing section 13 inputs the photo-reception signal and performs process such as amplification, and processes the photo-reception signal by using the in-plane correction table 13a read from the above-described storage unit (S22). Specifically, the photo-reception signal processing section 13 calculates (multiply) the input photo-reception signal by a value in the in-plane correction table 13a in
Next, the image processing section 14 inputs the in-plane corrected picked-up image, and performs binarization to the picked-up image (S24). That is, the image processing section 14 stores a threshold value which is previously set. For example, the image processing section 14 compares and determines whether the signal intensity of the picked-up image data is smaller, or equal to or larger than the threshold value, and performs binarization for setting “0” or “1”. Thereby, the section receiving light which is reflected by the external adjacent object is set to “1” and another section is set to “0”
The image processing section 14 removes an isolated point from the binarized picked-up image (S26). That is, in the case of the above-described binarization, the image processing section 14 removes noise by removing the section of “1” isolated from the section corresponding to the external adjacent object.
After this, the image processing section 14 performs labeling (S28). That is, in the case of the above-described binarization, the image processing section 14 performs labeling to the section set to “1”. Then, the image processing section 14 detects the region set to “1” as the region of the external adjacent object, and obtains a center of gravity or a central coordinate of this region. Such data is output to the control section 21 as the point information.
Next, the control section 21 uses the point information input from the image processing section 14, and performs necessary processing such as changing the display image. For example, in the case where a certain operation menu is displayed on the screen, the control section 21 detects which button in the operation menu is selected by a user's finger, and executes the command corresponding to the selected button.
In this manner, according to the embodiment, the in-plane correction table 13a is used in compensating the variation of the light intensity distribution profile in the plane of the backlight for displaying the image and the variation of the photo-reception sensitivity of each photo-reception element 11b, and the photo-reception signal from the photo-reception element which receives the light emitted from the backlight and reflected by the external adjacent object is corrected. Thereby, the image input/output device 1 performs the image processing with high accuracy on the basis of the corrected photo-reception signal. As a result, the external adjacent object is accurately detected.
According to the embodiment, since the correction coefficient may be always 1.0 or more, the capacity of the memory necessary for storing is small in comparison with the case where the correction coefficient is within the range of 1.0 or less. Moreover, since the correction coefficient is within the relatively-narrow range which is approximately slightly above 1.0, the number of bits for expressing the correction coefficient is reduced, and this also contributes to the reduction in the capacity of the memory necessary for storing.
According to the embodiment, in the plurality of pixels 16, the pixels 16 immediately adjacent to one another constitute one group, and the correction coefficient is obtained for each group by using the photo-reception elements 11b arranged in each group. Thereby, the in-plane correction table 13a is obtained. Therefore, the amount of data in the in-plane correction table 13a is reduced, and the memory capacity is small in comparison with the case where the in-plane correction table 13a is obtained by obtaining the correction coefficient for each pixel 16.
For example, the correction coefficient in the in-plane correction table 13a may be obtained to approximately three digits after the decimal point, and stored. The correction coefficient may be obtained with rougher accuracy (for example, to one digit below the decimal point). Specifically, the in-plane correction table 13a is typically set by 6 bits. However, when the in-plane correction table 13a is set by 4 bits with rough accuracy, the amount of data in the in-plane correction table 13a is reduced, and the memory capacity of the above-described storage unit is reduced.
Moreover, according to the embodiment, in the case where the in-plane correction table 13a is obtained and recorded in the memory before delivering the image input/output device 1 to users, labor hour when the users form the in-plane correction table 13a is saved. However, in the case where formation of the in-plane correction table 13a is optional for users, even if the input/output panel 11 changes with the passage of time, it is possible to appropriately form the in-plane correction table 13a corresponding to the change with time, and the appropriately-corrected picked-up image is obtained all the time with high accuracy even after a long time duration in use.
In the embodiment, when the in-plane correction table 13a is formed, as indicated in S14 and S16 in
In the case where the number of the correction coefficients in the in-plane correction table 13a is set small (set with roughness), and, in the in-plane correction table 13a, there is no correction coefficient corresponding to a photo-reception element group, data interpolation is performed on the basis of an existing correction coefficient of another group, and the correction coefficient may be obtained for the photo-reception element group with no corresponding correction coefficient in the in-plane correction table 13a. In this manner, correction may be performed by using the correction coefficient obtained by this interpolation. For example, the correction coefficient of the group may be interpolated with the correction coefficients of immediately adjacent groups. Thereby, it is prevented that the correction coefficient rapidly changes in the groups immediately adjacent to one another, and a gradual change in the in-plane correction table 13a is possible. Moreover, the memory capacity necessary for storing the in-plane correction table 13a is reduced.
In the embodiment, in the photo-reception elements 11b arranged in matrix, the plurality of photo-reception elements 11b immediately adjacent to one another constitute one group, and the correction coefficient is obtained for each group. Thereby, the in-plane correction table 13a is obtained. However, it is also possible that the correction coefficient is obtained for each photo-reception element 11b arranged in each pixel 16, and the plurality of correction coefficients are brought together. Thereby, the in-plane correction table 13a is obtained. In this case, the correction coefficient is finely obtained in the plane in comparison with the case where the correction coefficient is obtained for each group. Therefore, the in-plane corrected image with higher accuracy is obtained.
Modifications described below are also possible.
An image input/output device according to Modification 1 includes a display signal processing section 12, an input/output panel 11, a photo-reception signal processing section 13 and an image processing section 14, which are arranged in a display 10, and a control section 21 arranged in an electric device main body 20 using the display 10.
In the above-described embodiment, the reference image with a uniform luminance is used when forming the in-plane correction table 13a in the image input/output device according to the embodiment. However, in Modification 1, an arbitrary reference image is displayed on an input/output panel 11, and an in-plane correction table 13a is formed by using the arbitrary reference image. This reference image is arbitrary to have patterns with a plurality of luminance levels in one frame.
In the configuration of such an image input/output device, the display signal processing section 12 is connected to a previous stage of the input/output panel 11 and a previous stage of the photo-reception signal processing section 13. The display signal processing section 12 outputs, to the input/output panel 11, a drive signal to display the reference image on the basis of the display data. The display signal processing section 12 outputs, to the photo-reception signal processing section 13, a luminance data of the reference image to be displayed on the input/output panel 11.
In the input/output panel 11, when the reference image is reflected by a reference reflecting plate, a photo-reception element 11b receives the reflected light of the reference image, and outputs a photo-reception signal to the photo-reception signal processing section 13. The photo-reception signal processing section 13 is also connected to a subsequent stage of the input/output panel 11. When forming the in-plane correction table 13a, the photo-reception signal is supplied from the input/output panel 11 to the photo-reception signal processing section 13. The photo-reception signal processing section 13 multiplies a coefficient in an inverted light intensity table by the photo-reception signal, where the inverted light intensity table is obtained from the luminance data of the reference image corresponding to the photo-reception signal obtained from the input/output panel 11. Thereby, the influence of the reference image which has the ununiform luminance in the plane is eliminated from the photo-reception signal.
After this, the photo-reception signal processing section 13 performs processing indicated with S14 to S18 in
Operations of the image input/output device according to Modification 1 are the same as those of the image input/output device described in the embodiment. Therefore, the descriptions are omitted.
When forming the in-plane correction table 13a, in the image input/output device, the coefficient in the inverted light intensity table which is obtained from the luminance data of the reference image corresponding to the photo-reception signal obtained from the input/output panel 11 is used, the reference image being output from the display signal processing section 12 to the photo-reception signal processing section 13. Then, correction is performed so that the influence of the reference image with the ununiform luminance in the plane is eliminated from the photo-reception signal obtained from the input/output panel 11.
For this reason, the image input/output device obtains the in-plane correction table 13a without using the reference image with the ununiform luminance described in the embodiment. Therefore, it is unnecessary to store the reference image with the uniform luminance, the reference image being necessary only when forming the in-plane correction table 13a.
In the case where the reference image displayed on the input/output panel 11 continuously changes, the display signal processing section 12 obtains an inverse table of the image data for each frame, and outputs the inverse table to the photo-reception signal processing section 13. The photo-reception signal processing section 13 performs correction by using the photo-reception signal input from the input/output panel 11, and the inverse table obtained from the image data when the photo-reception signal is obtained in the input/output panel 11. After that, the processing is performed as indicated with S14 to 18 in
In the image input/output devices 1 and 2 described in the embodiment, and Modification 1 and Modification 2, the configuration where a liquid crystal display panel is used as the input/output panel 11 is described. However, the image input/output device according to the present invention may be configured with an organic electroluminescence (EL) panel or the like as the input/output panel. When bias voltage in a forward direction is applied, the organic EL element emits light. When bias voltage in a reverse direction is applied, the organic EL element receives light and generates a current. For this reason, the organic EL element includes the display element 11a and the photo-reception element 11b. In the configuration of the input/output panel 11, the organic EL element is arranged in each pixel 16. By applying bias voltage in the forward direction to each organic EL element in response to the display data, the organic EL element emits light and displays the image. By applying bias voltage in the reverse direction to other organic EL elements, the organic EL element receives the reflected light. At this time, when the in-plane correction method described in the embodiment is performed to the light intensity distribution profile in the display plane when the organic EL element performs photo-emission by applying the bias voltage in the forward direction, and the photo-reception sensitivity distribution profile in the plane when the organic EL element performs photo-reception by applying the bias voltage in the reverse direction, it is possible to correct these distributions in the plane.
Hereinbefore, the present invention is described with the embodiment and the modifications. However, the present invention it not limited to those, and various modifications may be made. For example, the case where one photo-reception cell is provided corresponding to one photo-emission cell as illustrated in
The present application contains subject matter related to that disclosed in Japanese Priority Patent Application JP 2008-176685 filed in the Japan Patent Office on Jul. 7, 2008, the entire content of which is hereby incorporated by reference.
It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2008-176685 | Jul 2008 | JP | national |
