Patent Grant
8368794
The present invention relates to a display apparatus which is worn by a user as part of an eyeglass-like or head-mount wearable unit and which has display means positioned in front of the user's eyes for image display purposes, as well as to a display apparatus for use with that display apparatus.
A variety of wearable display apparatuses have so far been proposed, as described illustratively in Japanese Patent Laid-open Nos. Hei 8-126031, Hei 9-27970 and Hei 9-185009. Patent Documents disclose display apparatuses each worn by a user as part of an eyeglass-like or head-mount wearable unit with a display unit located immediately before the user's eyes for image display.
However, traditional display apparatuses such as those cited above have yet to relieve the user of the need to operate their keys or other controls. These apparatuses have yet to offer the kind of display that would suitably reflect the user's intentions and biometric status.
The present invention has been made in view of the above circumstances and provides an apparatus and a method whereby display operations are performed in a manner precisely reflecting the user's status, i.e., his or her intentions and physical conditions.
In carrying out the present invention and according to one embodiment thereof, there is provided a display apparatus including: display means for displaying images as it is positioned in front of the eyes of a user; user information acquisition means for acquiring information about either behavior or physical status of the user; and control means for determining either intention or status of the user in accordance with the information acquired by the user information acquisition means, the control means further controlling display operations of the display means in a manner reflecting the result of the determination.
Preferably, the display apparatus according to an embodiment of the present invention may further include image pickup means for picking up images. In this structure, the control means may preferably control the display means to display image data picked up by the image pickup means in accordance with the information acquired by the user information acquisition means.
Preferably, the display apparatus of the present invention may further include reproduction means for reproducing data from a recording medium. In this structure, the control means may preferably control the display means to display the data reproduced by the reproduction means in accordance with the information acquired by the user information acquisition means. The control means may preferably control reproduction operations of the reproduction means in accordance with the information acquired by the user information acquisition means while the data reproduced by the reproduction means is being displayed by the display unit.
Preferably, the display apparatus according to an embodiment of the present invention may further include reception means for receiving data through communication with an external apparatus. In this structure, the control means may preferably control the display means to display the data received by the reception means in accordance with the information acquired by the user information acquisition means.
Preferably, the display means may be capable of switching between a see-through state and a display state, the see-through state being a state in which the display means remains either transparent or translucent, the display state being a state in which the display means displays supplied data.
The user information acquisition means may preferably be a sensor configured to detect acceleration, angular velocity, or vibration.
The user information acquisition means may preferably be a sensor configured to detect movements of the head, arms, hands, legs, or whole body of the user.
The user information acquisition means may preferably be a sensor configured to detect a stationary state, a walking state, and a running state of the user.
The user information acquisition means may preferably be a visual sensor configured to detect visual information about the user.
The user information acquisition means may preferably be a sensor configured to detect, as visual information about the user, the line of sight, focal distance, pupil dilation, retinal pattern, or blinks of the user.
The user information acquisition means may preferably be a biometric sensor configured to detect biometric information about the user.
The user information acquisition means may preferably be a sensor configured to detect, as biometric information about the user, the heart rate, pulse rate, perspiration, brain wave, galvanic skin reflex, blood pressure, body temperature, or breathing rate of the user.
The user information acquisition means may preferably be a biometric sensor configured to detect information about a tense state or an excited state of the user.
The control means may preferably control the display means to start and end the display operations thereof.
The control means may preferably control the display means to switch between the see-through state and the display state mentioned above.
The control means may preferably control switching between sources which supply data to the display means for display.
The control means may preferably control the display means to scale up and down a display image.
The control means may preferably control the display means to split an onscreen display.
The control means may preferably control the display means to control display brightness.
The control means may preferably control signal processing of an image signal to be displayed by the display means.
The control means may preferably control the display means to scroll images or to feed pages.
The control means may preferably determine an input operation regarding the image displayed by the display means in accordance with the information acquired by the user information acquisition means.
According to another embodiment of the present invention, there is provided a display method for use with a display apparatus having display means for displaying images as it is positioned in front of the eyes of a user, and user information acquisition means for acquiring information about either behavior or physical status of the user, the display method including the steps of: acquiring information about the behavior or physical status of the user; and determining either intention or status of the user in accordance with the information acquired in the information acquiring step, the determining step further controlling display operations of the display means in a manner reflecting the result of the determination.
Where the present invention embodied as outlined above is in use, the inventive display apparatus is worn by the user as part of an eyeglass-like or head-mount wearable unit having display means positioned in front of the user's eyes for visual recognition. Wearing the display apparatus allows the user to view what is being displayed by the display means such as images picked up by image pickup means, images constituted by the data reproduced by reproduction means, or images composed of the data received by reception means of the apparatus.
It is preferred that the display operations be controlled in a manner suitably reflecting the user's intention and status. More specifically, the display operations may involve causing the display means to switch between the display state and the see-through state; selecting sources from which to acquire the image data to be displayed; adjusting diverse display settings such as splitting of the onscreen display, scale-up and scale-down of display images, control of display brightness and other display quality-related attributes, scrolling of images and feeding of pages; inputting data to the display means for display; and causing the display means to display reproduced images. The present invention proposes that these operations be carried out not by the user manipulating switches and controls but by the display apparatus itself acquiring information about the user's behavior or physical status, determining the user's intention or status based on the acquired information, and executing suitable controls in accordance with the result of the determination.
According to an embodiment of the present invention, as outlined above, the display apparatus determines the user's intention or status based on the information about the user's behavior or physical status and controls the display operations accordingly while the display means is displaying images immediately before the user's eyes. This structure permits appropriate display performance reflecting the user's intention or status without bothering the user to manipulate controls or switches. The inventive display apparatus thus provides excellent ease of use for the user while offering varieties of visual experiences.
Since the display means can be placed in the transparent or translucent state as desired, the user can lead normal daily life while continuously wearing the display apparatus. The advantages of the inventive apparatus can thus be exploited by the user while maintaining his or her usual lifestyle.
The display apparatus and display method practiced as preferred embodiments of the present invention will now be described under the following headings:
[1. Appearance of the Display Apparatus and its Relations to External Apparatuses]
[2. Typical Structure the Display Apparatus]
[3. Typical Displays]
[4. Determination of User Status]
[5. Various Operations]
[6. Effects, Variations and Extensions of the Embodiments]
[1. Appearance of the Display Apparatus and its Relations to External Apparatuses]
Worn by the user as shown in
With the display unit 2 worn by the user, an image pickup lens 3a is oriented forward so as to pick up images of objects in the user's line of sight.
A lighting element 4a is positioned in a manner giving illumination in the direction in which the image pickup lens 3a picks up the images. The lighting element 4a is typically formed by a light-emitting diode (LED).
A pair of earphone-like speakers 5a are furnished to be inserted into the user's ears (only the left-hand side is shown in
A pair of microphones 6a and 6b for picking up external sounds are positioned at a right-hand edge of the display unit 2 for the right eye and at a left-hand edge of the display unit 2 for the left eye.
What is shown in
As another alternative, the pair of earphone-like speakers 5a for stereo sound effects may be replaced by a single earphone for one-ear use. Likewise, the pair of microphones 6a and 6b may be replaced by a single microphone.
The structure of
Alternatively, the structure may dispense with the image pickup function. As another alternative, the display apparatus 1 may be free of microphones and/or earphone-like speakers. As a further alternative, the display apparatus 1 may dispense with the lighting element 4a.
The internal structure of the display apparatus 1, which will be discussed later in detail, may include a reproduction capability (storage unit 25 in
Where these capabilities are implemented, the sources from which to derive data as the images to be displayed by the display unit 2 may include an image pickup capability block, a reproduction capability block, and a communication capability block.
(a) of
(b) of
The external image pickup apparatus 70 may be of diverse types. It may be an image pickup apparatus in the possession of the user utilizing the display apparatus 1 or an image pickup apparatus owned by an acquaintance of the user of the display apparatus 1. Alternatively, the external image pickup apparatus 70 may be an apparatus set up as part of the facility of an image-providing public service or a piece of equipment established by a private service-offering company, both capable of communicating with the display apparatus 1.
(c) of
Typically, the content source apparatus 71 may be an AV (audio visual) apparatus such as a video unit, a TV tuner or a home server; or an information processing apparatus such as a PC (personal computer), PDA (personal digital assistant), or a mobile phone. As in the case above, the content source apparatus 71 may be an apparatus in the possession of the user handling the display apparatus 1, an apparatus owned by an acquaintance of the user, an apparatus set up as part of the facility of a content-providing public service, or a server established by a private content-offering company.
The data sent from the content source apparatus 71 to the display apparatus 1 for display purposes may also be of diverse kinds: moving image contents such as movies and video clips; still image contents picked up by digital still cameras or the like and stored on the recording medium; electronic book data; computer use data prepared by the user on his or her PC and recorded to the recording medium, including image data, text data, and spreadsheet data; video game images derived from video game programs recorded on the recording medium; and any other data that may be available in suitable format for display use.
(d) of
[2. Typical Structure the Display Apparatus]
A system controller 10 is typically composed of a microcomputer including a CPU (central processing unit), a ROM (read only memory), a RAM (random access memory), a nonvolatile memory, and an interface. In operation, the system controller 10 controls the display apparatus 1 as a whole.
The system controller 10 controls the components of the display apparatus 1 in accordance with the user's status. That is, the system controller 10 runs on a program that detects and determines the user's status and controls the internal components in their operation in a manner reflecting what is detected and determined. In functional terms, the system controller 10 thus has two capabilities: a user status determination capability 10a to determine the user's status, and an operation control capability 10b to give control instructions to the components in accordance with the result of the determination by the user status determination capability 10a, as shown in
The display apparatus 1 has an image pickup unit 3, an image pickup control unit 11, and an image pickup signal processing unit 15 for picking up images of the scene in front of the user.
The image pickup unit 3 has a lens block, a lens drive block, and a solid state imaging device array. The lens block is made up of the image pickup lens 3a (
The image pickup signal processing unit 15 includes a video A/D converter, and a sample-and-hold/AGC (automatic gain control) circuit that performs gain control and waveform shaping on the signal acquired by the solid state imaging devices of the image pickup unit 3. Working in this manner, the image pickup signal processing unit 15 obtains an image pickup signal in digital data form. The image pickup signal processing unit 15 also carries out white balance control, brightness control, white signal adjustment, and camera shake correction on the image pickup signal.
Under control of the system controller 10, the image pickup control unit 11 controls the image pickup unit 3 and image pickup signal processing unit 15 in operation. Illustratively, the image pickup control unit 11 controls activation and deactivation of the image pickup unit 3 and image pickup signal processing unit 15. The image pickup control unit 11 also effects motor control on the image pickup unit 3 to regulate auto focusing, automatic exposure, aperture opening, and zoom functions.
The image pickup control unit 11 has a timing generator. Timing signals generated by the timing generator are used to control the solid state imaging devices, as well as the sample-and-hold/AGC circuit and the video A/D converter in the image pickup signal processing unit 11 in their signal processing. Image pickup frame rate can also be controlled as part of the timing control scheme.
Furthermore, the image pickup control unit 11 controls the solid state imaging devices and the image pickup signal processing unit 15 in terms of imaging sensitivity and signal processing. Illustratively, as part of the imaging sensitivity control scheme, gain control is effected on the signal read from the solid state imaging devices. Controls are likewise performed on the black level setting, on coefficients used in image pickup signal processing in the digital data stage, and on the amount of correction in camera shake correction processing. In terms of imaging sensitivity, it is possible to carry out overall sensitivity adjustments regardless of wavebands and to perform sensitivity adjustments taking specific wavebands into consideration such as infrared and ultraviolet ranges. Sensitivity adjustments dependent on wavelengths can be brought about by the introduction of a wavelength filter in the image pickup lens block and by the execution of wavelength filter calculations on the image pickup signal. More specifically, the image pickup control unit 11 can effect sensitivity control by suitably adjusting the wavelength filter being inserted and by designating appropriate filter calculation coefficients.
The image pickup signal (acquired image data) picked up by the image pickup unit 3 and processed by the image pickup signal processing unit 15 is supplied to an image input/output control unit 27.
Under control of the system controller 10, the image input/output control unit 27 controls the transfer of image data, specifically between the image pickup block (image pickup signal processing unit 15), the display block (display image processing unit 12), a storage unit 25, and a communication unit 26.
Illustratively, the image input/output control unit 27 supplies the display image processing unit 12, storage unit 25, and communication unit 26 with the image data constituting the image pickup signal processed by the image pickup signal processing unit 15.
The image input/output control unit 27 also supplies the display image processing unit 12 and communication unit 26 with the image data reproduced illustratively from the storage unit 25.
Furthermore, the image input/output control unit 27 supplies the display image processing unit 12 and storage unit 25 with the image data received illustratively by the communication unit 26.
The display apparatus 1 has the display unit 2, display image processing unit 12, a display drive unit 13, and a display control unit 14 making up the structure for offering displays to the user.
Illustratively, the image data picked up by the image pickup unit 3 and processed by the image signal processing unit 15 as the image pickup signal may be sent to the display image processing unit 12 through the image input/output control unit 27. The display image processing unit 12 is what is generally referred to as a video processor that carries out various display-related processes on the supplied image data. These processes include brightness level adjustment, color correction, contrast adjustment, and sharpness control (edge enhancement) effected on the image data. The display image processing unit 12 is further capable of generation of a partially scaled-up or scaled-down image from the supplied image data, splitting and superimposing of display images based on the supplied image data, generation of characters and graphic images, and superimposing of a generated image onto the supplied image data. That is, the supplied image data can be processed in diverse fashion by the display image processing unit 12.
The display drive unit 13 is constituted by pixel drive circuits that enable the display unit 2 (e.g., LCD) to display the image data supplied from the display image processing unit 12. More specifically, the drive circuits furnish the pixels arranged in matrix fashion in the display unit 2 with drive signals based on video signals occurring at predetermined horizontal and vertical drive timings, whereby the image of the supplied data is displayed. The display drive unit 13 may place the display unit 2 in the see-through state by suitably controlling the transmittance of each of the pixels in the unit 2.
The display control unit 14 controls the display image processing unit 12 in processing and the display drive unit 13 in operation under control of the system controller 10. More specifically, the display control unit 14 causes the display image processing unit 12 to carry out the above-described processes and enables the display drive unit 13 to switch between the see-through state and the image display state.
The image data reproduced from the storage unit 25 or the image data received by the communication unit 26 may be supplied to the display image processing unit 12 via the image input/output control unit 27. In such cases, the reproduced image or the received image is output by the display unit 2 thanks to the above-described workings of the display image processing unit 12 and display drive unit 13.
The display apparatus 1 also includes an audio input unit 6, an audio signal processing unit 16, and an audio output unit 5.
The audio input unit 6 is made up of the microphones 6a and 6b shown in
The audio data acquired by the audio input unit 6 is supplied to the audio input/output control unit 28.
Under control of the system controller 10, the audio input/output control unit 28 controls transfer of the supplied audio data, specifically the transfer of audio signals between the audio input unit 6, audio signal processing unit 16, storage device 25, and communication unit 26.
For example, the audio input/output control unit 28 supplies the audio data obtained by the audio input unit 6 to the audio signal processing unit 16, to the storage unit 25, or to the communication unit 26.
The audio input/output control unit 28 further supplies the audio data reproduced illustratively from the storage unit 25 to the audio signal processing unit or to the communication unit 26.
The audio input/output control unit 28 also forwards the audio data received illustratively by the communication unit 26 to the audio signal processing unit 16 or to the storage unit 25.
The audio signal processing unit 16 is typically constituted by a digital signal processor and a D/A converter. The audio data obtained by the audio input unit 6 or the audio data coming from the storage unit 25 or from the communication unit 26 is supplied to the audio signal processing unit 16 through the audio input/output control unit 28. Under control of the system controller 10, the audio signal processing unit 16 performs such processes as sound level control, sound quality control, and acoustic effect adjustments on the supplied audio data. The audio data thus processed is converted to an analog signal that is sent to the audio output unit 5. The audio signal processing unit 16 is not limited to the structure for digital signal processing; the unit 16 may also be implemented in a structure including analog amplifiers and analog filters for analog signal processing.
The audio output unit 5 is made up of the above-described pair of earphone-like speakers 5a in
The audio input unit 6, audio signal processing unit 16, and audio output unit 5 combine to let the user listen to external sounds, to the sounds reproduced from the storage unit 25, or to the sounds received by the communication unit 26. The audio output unit 5 may be structured alternatively as so-called bone conduction speakers.
The storage unit 25 allows data to be written to and read from a suitable recording medium. Typically, the storage unit 25 is implemented as a hard disk drive (HDD). Obviously, the recording medium may be of diverse types including a solid state memory such as a flash memory, a memory card containing a solid state memory, an optical disk, a magneto-optical disk, and a hologram memory. Any of such recording media may be adopted as long as the storage unit 25 can properly write and read data to and from the selected medium.
The image data picked up by the image pickup unit 3 and processed by the image pickup signal processing unit 15 as the image pickup signal, or the image data received by the communication unit 26 can be sent to the storage unit 25 via the image input/output control unit 27. The audio data obtained by the audio input unit 6 or the image data received by the communication unit 26 may be forwarded to the storage unit 25 through the audio input/output control unit 28.
Under control of the system controller 10, the storage unit 25 encodes the supplied image data or audio data preparatory to recording to the recording medium and then writes the encoded data to the recording medium.
The storage unit 25 also reproduces image data or audio data from the recording medium under control of the system controller 10. The reproduced image data is output to the image input/output control unit 27; the reproduced audio data is sent to the audio input/output control unit 28.
The communication unit 26 sends and receives data to and from external apparatuses. Diverse kinds of external apparatuses may communicate with the communication unit 26, including the image pickup apparatus 70 and content source apparatus 71 shown in
The communication unit 26 may communicate with the external apparatus over a network such as a wireless LAN or a Bluetooth setup with short-distance wireless links to network access points. Alternatively, the communication unit 26 may engage in direct wireless communication with an external apparatus having a suitable communication capability.
The image data picked up by the image pickup unit 3 and processed by the image pickup signal processing unit 15 as the image pickup signal, or the image data reproduced from the storage unit 25 can be sent to the communication unit 26 through the image input/output control unit 27. The audio data acquired by the audio input unit 6 or reproduced from the storage unit 25 can be supplied to the communication unit 26 via the audio input/output control unit 28.
Under control of the system controller 10, the communication unit 26 encodes and modulates the supplied image data or audio data preparatory to transmission, and transmits the data thus processed to the external apparatus.
The communication unit 26 further receives data from the external apparatus. The image data received and demodulated by the communication unit 26 is output to the image input/output control unit 27; the audio data received and demodulated likewise is forwarded to the audio input/output control unit 28.
The display apparatus 1 also includes a lighting unit 4 and a lighting control unit 18. The lighting unit 4 is made up of the lighting element 4a shown in
With the lighting element 4a of the lighting unit 4 positioned in a manner giving illumination in the forward direction, the lighting unit 4 provides illumination in the user's line of sight.
The display apparatus 1 further includes a visual sensor 19, an acceleration sensor 20, a gyro 21, and biometric sensors 22 constituting the structure for acquiring user information.
The visual sensor 19 detects information about the user's vision. Illustratively, the visual sensor 19 is capable of detecting such sight-related information as the line of sight, focal distance, pupil dilation, retinal pattern, and blinks of the user.
The acceleration sensor 20 and gyro 21 output signals reflecting the user's movements. Illustratively, these sensors detect the movements of the user's head, neck, whole body, arms, and legs.
The biometric sensors 22 detect biometric information about the user. Illustratively, the biometric sensors 22 detect the heart rate, pulse rate, perspiration, brain wave, galvanic skin reflex (GSR), blood pressure, body temperature, and breathing rate of the user. The signals detected by the biometric sensors constitute information by which to determine the user's diverse states: a tense state, an excited state, a calm state, a sleepy state, or a comfortable state, or an uncomfortable state.
The visual sensor 19, acceleration sensor 20, gyro 21, and biometric sensors 22 combine to acquire information about the user wearing the display apparatus 1 in connection with the user's behavior or physical status. The user information thus acquired is supplied to the system controller 10.
By operating its user status determination capability 10a, the system controller 10 determines the user's intention or status reflecting the acquired user information. With the user's intention or status thus determined, the system controller 10 controls display-related operations through the processing of its operation control capability 10b. More specifically, the system controller 10 instructs the display control unit to control the display image processing unit 12 and display drive unit 13 in operation, selects the source from which to obtain the data to be displayed, or controls the storage unit 25 in reproduction or the communication unit 26 in communication.
Although the visual sensor 19, acceleration sensor 20, gyro 21 and biometric sensors 22 were shown to constitute the structure for acquiring user information, the installation of all these components is not requisite. Other sensors may be added as well, including a voice sensor for detecting the user's voice and a lip-reading sensor for reading the user's lip movements.
The structure in
In the structure of
In other words, the setup of
Although not shown, other structures than those in
[3. Typical Displays]
The system controller 10 controls display-related operations in response to the user's intention or status, selecting the source that supplies the data to be displayed and carrying out processes on display images. This allows the user to view diverse forms and contents of displays on the display unit 2.
(a) of
(b) of
(c) of
Although not shown, if the system controller 10 causes the image pickup control unit 11 to get the image pickup unit 3 picking up a wide-angle shot image, then that wide-angle shot will be displayed by the display unit 2. Switchover between telephoto and wide-angle shot settings may be effected either by controlling the zoom lens drive in the image pickup unit 3 or by getting the image pickup signal processing unit 15 suitably processing signals.
(b) and (c) of
More specifically, (a) of
(b) of
(c) of
In each of the cases of (a), (b) and (c) of
More specifically, (a) of
(b) of
(c) of
More specifically, (a) of
(b) of
(c) of
In the preceding cases, the content source apparatus 71 is typically one of AV apparatuses including video players or an information processing apparatus such as a personal computer. The display apparatus 1 receives and displays the image data sent from any of these content source apparatuses 71. The user wearing the display apparatus 1 can then verify the displayed images and perform various operations accordingly.
(a) of
(b) of
(c) of
Obviously, it is also possible for each of a plurality of areas making up the screen to display an image from each of different sources.
(a) of
If the system controller 10 gives an image scale-up instruction to the display image processing unit 12 through the display control unit 14, then the display unit 2 may display a scaled-up image such as one shown in (b) of
(a) of
This is an image at a low brightness level and it may not be easy for the user to watch it comfortably.
In such a case, the system controller 10 may instruct the display control unit 14 (display image processing unit 12, display drive unit 13) to increase brightness, adjust contrast, and/or enhance sharpness, thereby causing the display unit 2 to display a brighter, clearer image such as one shown in (b) of
The foregoing displays are merely examples and are not limitative of the present invention. Many other forms of displays can be implemented by the embodiment of the invention. Specifically, such displays are brought about by selecting the source out of the image pickup capability block, reproduction capability block, and reception capability block as desired; by suitably controlling the image pickup capability block or reproduction capability block in operation; or by properly controlling the display image processing unit 12 or display drive unit 13 in terms of processing or in operation.
Illustratively, if the image pickup capability block is selected as the display image source, then it is possible to implement the following displays: telephoto view display, wide-angle view display, zoom-in and zoom-out displays shifting from telephoto to wide-angle settings or vice versa, scale-up display, scale-down display, display at variable frame rates (e.g., images picked up at high or low frame rates), high intensity display, low intensity display, variable contrast display, variable sharpness display, display at enhanced imaging sensitivity, display at enhanced infrared imaging sensitivity, display at enhanced ultraviolet imaging sensitivity, and display of images with their specific wavebands suppressed.
If the reproduction capability block or reception capability block is selected as the display image source, then it is possible to implement displays of images reproduced at varied speeds such as high-speed reproduction, slow reproduction, frame-by-frame advance reproduction; as well as changing of pages being displayed or scrolling of images as is carried out on a personal computer or the like.
Where the display image processing unit 12 is suitably controlled in terms of processing, it is conceivable to implement displays with diverse onscreen effects, such as images with pixelated mosaic effect, images in reverse video, soft-focus images, partially highlighted images, and images in varied ambient colors.
[4. Determination of User Status]
As described above, the display apparatus 1 embodying the present invention has the visual sensor 19, acceleration sensor 20, gyro 21, and biometric sensors 22 constituting the structure for acquiring user information.
The visual sensor 19 is used to detect information about the user's sight. For example, the visual sensor 19 may be formed by an image pickup arrangement located close to the display unit 2 to take pictures of the user's eyes. Images of the user's eyes are picked up by that image pickup arrangement and acquired by the system controller 10 wherein the user status determination capability 10a analyses the obtained images. The analysis provides information about the line of sight, focal distance, pupil dilation, retinal pattern, and blinks of the user. The information thus acquired is used as the basis for determining the user's status or intentions.
Alternatively, the visual sensor 19 may be formed by a light-emitting arrangement and a light-receiving arrangement, the light-emitting arrangement being located close to the display unit 2 and emitting light to the user's eyes, the light-receiving arrangement receiving reflected light from the user's eyes. Illustratively, the thickness of the user's crystalline lens is detected from a received-light signal. The detected lens thickness in turn serves as the basis for detecting the focal distance of the user's eye.
By detecting the user's line of sight, the system controller 10 can determine which part of the image displayed illustratively by the display unit 2 is attracting the user's attention.
It is also possible for the system controller 10 to recognize the user's line of sight as an input operation. For example, the user's line of sight being moved rightward or leftward may be interpreted as instructions input to the display apparatus 1.
By detecting the user's focal distance, it is possible to determine whether the scene currently attracting the user's attention is nearby or far away. This in turn permits execution of zoom control and scale-up/scale-down control. Illustratively, the user's act of looking at a distant object may be interpreted as a cue to bring about a telephoto view display.
By detecting the degree of the user's pupil dilation, it is possible to determine how the user is reacting to glare of the ambient brightness if the see-through state is in effect or to glare of the onscreen image brightness if the monitor display state is in use. According to the user's reaction thus determined, the system controller 10 may control brightness and imaging sensitivity.
The user's retinal pattern may be detected illustration for use in authenticating the user's identity. Since every person has his or her unique retinal pattern, the user wearing the display apparatus is identified so that the apparatus may be controlled in a manner suitable for the detected identity. Alternatively, the display apparatus may enable its monitor to be operated only for a specifically authenticated user.
By detecting the user's blinks, it is possible to determine the degree of the user's eyestrain due to glare or other causes. It is also possible to interpret the user's blinks as input operations deliberately executed by the user. For example, three consecutive blinks made by the user may be interpreted as a cue to bring about a particular operation.
The acceleration sensor 20 and gyro 21 output signals corresponding to the user's movements. Illustratively, the acceleration sensor 20 is suitable for detecting linear movements while the gyro 21 is suited to sense rotary or vibratory motions.
When appropriately positioned on the user's body, the acceleration sensor 20 and gyro 21 can detect movements of the user's whole body or body parts.
For example, the acceleration sensor 20 and gyro 21 may be mounted inside the eyeglass-like display apparatus 1 shown in
The above sensor setup primarily detects the user's behavior in terms of head or neck movements. Illustratively, the sensors can detect whether the user is looking upward or downward. The user's head directed downward may be interpreted to mean that the user is looking at a nearby object like a book as in reading; the user's head oriented upward may be interpreted to signify that the user is looking at a faraway object.
Upon detecting the user shaking his or her head, the system controller 10 can interpret the motion as the user's deliberate action. For example, if the user is found to shake his head twice leftward, the action may be recognized as a particular input operation.
The acceleration sensor 20 and gyro 21 may be suitably arranged to detect whether the user is at rest (stationary), is walking, or is running. It is also possible for these sensors to determine whether the user has sat down from his or her standing position or has now stood up from the sitting position.
As another alternative, the acceleration sensor 20 and gyro 21 may be attached to the user's arms or legs apart from the wearable unit mounted on the user's head. This sensor setup makes it possible to detect movements of only the arms or the legs of the user.
The biometric sensors 22 are designed to detect information about the heart rate, pulse rate, perspiration, brain wave (e.g., alpha wave, beta wave, theta wave, gamma wave), galvanic skin reflex, body temperature, blood pressure, and breathing (e.g., rate and depth of breathing, amount of ventilation) of the user. These items of information, when acquired, allow the system controller 10 to detect whether the user is in a tense state, an excited state, a calm state, a comfortable state, or an uncomfortable state.
Whether or not the display apparatus 1 is currently worn by the user may be determined based on biometric information, and this information may be used for operational purposes. Illustratively, if the display apparatus 1 is found unmounted by the user, then the system controller 10 may place the apparatus 1 in standby mode in which checks are made only for biometric information. As soon as the display apparatus 1 is found worn by the user, the system controller 10 may activate the apparatus 1. When the user is found to have unmounted the display apparatus 1, the system controller 10 may place the apparatus 1 back into standby mode.
The information detected by the biometric sensors 22 may also be utilized for authentication of the user's identity (so that the apparatus should be worn by the properly identified person).
The biometric sensors 22 may be located inside the head mount frame of the eyeglass-like display apparatus 1. Illustratively, the information described above may be derived from the temples and back of the user's head. Alternatively, the biometric sensors 22 may be positioned elsewhere on the user's body, independent of the head mount frame of the display apparatus 1.
[5. Various Operations]
As described above, the display apparatus 1 embodying the present invention has its system controller control display-related operations in accordance with the user information detected by the visual sensor 19, acceleration sensor 20, gyro 21, and biometric sensors 22. In this manner, the system controller 10 allows the display unit 2 to display images that reflect the user's intention and status and are thus fit for his or her preferences.
Such operations carried out under control of the system controller 10 will now be described below.
In step F101, the system controller 10 controls the display control unit 14 to place the display unit 2 in the see-through state. Illustratively, when the display apparatus 1 is initially switched on, the system controller 10 goes to step F101 and puts the display unit 2 in the see-through state.
With the display unit 2 in the see-through state, the system controller 10 goes to step F102 and checks to determine whether a display start trigger is detected. The display start trigger signifies an event that cues the system controller 10 in causing the display unit 2 to start its display operation in view of the user's intention or status determined by the user status determination capability 10a. The system controller 10 recognizes a specific display start trigger in the form of the user's explicit operation, user's deliberate act (i.e., action interpreted as an operation), or the user's unconscious movement or status (including the user's perception). Some examples of the trigger are described below.
If a display start trigger is detected, then the system controller 10 goes to step F103 and controls the start of display. More specifically, the system controller 10 instructs the display control unit 14 to control the display image processing unit 12 and display drive unit 13 in order to have the display unit 2 display supplied data as a normal pickup image.
The display image processing unit 12 is supplied with image data through the image input/output control unit 27. If a plurality of image data sources exist as shown in
If the default source is the storage unit 25, then the system controller 10 in step F103 may start up the storage unit 25 and get the display unit 2 to display reproduced contents or a menu screen for selection of contents coming from the storage unit 25. If the default source is the communication unit 26, then the system controller 10 in step F103 may start up the communication unit 26 and get the display unit 2 to display a communication-ready screen or images of data received from an external apparatus.
Obviously, if there is one source, the image data from that source need be supplied to the display image processing unit 12.
As another alternative, when step F103 is reached in which to control the start of display, the display unit 2 may be arranged to display a menu screen or a source selection screen as its initial screen, with no image data supplied from image sources.
Although the process of
While the image coming from a given source is being displayed by the display unit 2, the system controller 10 goes to step F104 and checks to determine whether a display control trigger has occurred. If no display control trigger is detected, then the system controller 10 goes to step F105 and checks to see if a source switch trigger has occurred. If no source switch trigger is detected, then the system controller 10 goes to step F106 and determines if a display end trigger has occurred.
The display control trigger signifies an event that cues the system controller 10 in making changes to how images are to be displayed or how the display image data is to be processed, in view of the user's intention or status determined by the user status determination capability 10a.
The source switch trigger means an event that cues the system controller 10 in switching sources of display image data in view of the user's intention or status determined by the user status determination capability 10a, where the display apparatus 1 has a plurality of sources as shown in
The display end trigger refers to an event that cues the system controller 10 in causing the display unit 2 to end its display operation and enter the see-through state in view of the user's intention or status determined by the user status determination capability 10a.
The triggers above are detected by the system controller 10 as representative of the user's deliberate operation (any action recognized as an operation) or the user's unconscious operation or status (i.e., user's physical or mental state). How such triggers are recognized and what will be controlled as a result of the detected trigger will be discussed below in more detail.
If a display control trigger is detected, the system controller 10 goes from step F104 to step F107 and controls image display operation. Specifically, the system controller 10 instructs the display control unit 14 to have the display unit 2 give a display reflecting the user's intention or status at that point. Depending on the currently selected source, the system controller may control the image pickup capability block, the storage unit 25 in operation, or the communication unit 26 in operation.
Following the display operation control in step F107, the system controller 10 continues to check for triggers in steps F104, F105 and F106.
If a source switch trigger is detected, the system controller 10 goes from step F105 to step F108 and controls source switching operation. Specifically, the system controller 10 instructs the image input/output control unit 27 and/or the audio input/output control unit 28 to control the newly switched source in operation and to have the display image processing unit 12 and/or audio signal processing unit 16 supplied with the image data and/or audio data coming from the new source.
The source switch control in step F108 illustratively switches the display unit 2 from the state in which the images picked up by the image pickup unit 3 have been displayed, to the state in which the images reproduced from the storage unit 25 are displayed.
Following the source switch control in step F108, the system controller 10 continues to check for triggers in steps F104, F105 and F106.
If a display end trigger is detected, then the system controller 10 goes from step F106 back to step F101 and instructs the display control unit 14 to put the display unit 2 in the see-through state. The currently selected image data source is instructed to end its image feeding operation.
While the display apparatus 1 remains active and is worn by the user, the operation control capability 10b of the system controller 10 carries out the process outlined in
During the process, four kinds of controls are effected: display start control following detection of the display start trigger, display operation control following detection of the display control trigger, source switch control following detection of the source switch trigger, and control to put the display unit 2 in the see-through state following detection of the display end trigger. Further details of the checks for triggers and specific controls to be executed following trigger detection will be discussed below in reference to
(a) and (b) of
In step F200 of (a) of
It is assumed that the user's specific movement, such as moving down his head twice, shaking his head right and left in one stroke, or rotating his head in a circular movement, is predetermined as representative of the user's decision to start the display operation. If the detection information from the acceleration sensor 20 and/or the gyro 21 is found to indicate the user's specific moment designating the start of display, then the system controller 10 goes from step F201 to step F202 and recognizes a display start trigger.
With the display start trigger detected in step F202, control of the process in
The user's specific behavior to be detected by the acceleration sensor 20 or gyro 21 as representative of the user's decision to start monitor display is not limited to the example cited above. Alternatively, the user may jump, wave his hands, swing his arms, swing his legs, or perform any other bodily motion to gain the same result.
(b) of
In step F210, the system controller 10 analyzes the information coming from the visual sensor 19. Illustratively, if the visual sensor 19 includes an image pickup device to pick up images of the user's retinal pattern, then the system controller 10 analyzes the image thus acquired by the visual sensor 19.
If the user's three consecutive blinks are predetermined as representative of the user's decision to start the display operation, then the system controller 10 checks for that particular behavior through image analysis. Upon detecting three consecutive blinks of the user, the system controller 10 goes from step F211 to F212 and recognizes a display start trigger.
With the display start trigger detected in step F212, control of the process in
In step F103, the system controller 10 controls the display unit 2 to start its image display.
The user's specific behavior to be detected by the visual sensor 19 as representative of the user's decision to start monitor display is not limited to the example cited above. Alternatively, the user may roll the eyes in a circular motion, move the eyes right and left or up and down in two strokes, or do any other suitable eye movement to gain the same result.
(c) of
In step F230, the system controller 10 checks for such information as brain wave, heart rate, and galvanic skin reflex from the biometric sensors 22.
In step F231, the system controller 10 checks the information from the biometric sensors 22 to determine whether the user is wearing the display apparatus 1. The fact that biometric information starts getting acquired by the biometric sensors 22 is interpreted as the user starting to wear the display apparatus 1.
Upon detecting the user wearing the display apparatus 1, the system controller 10 goes from step F231 to step F232 and recognizes a display start trigger. With the display start trigger detected in step F232, the system controller 10 reaches step F103 in
Whether or not the user is wearing the display apparatus 1 is determined as described above on the basis of the vital reaction information from the biometric sensors 22.
As soon any of such findings as the pulse rate, brain wave, and galvanic skin reflex starts getting detected, the display start trigger is recognized. The user's act of wearing the display apparatus 1 thus prompts the system controller 10 to start the display operation.
It is also possible to start display control when the display apparatus 1 is worn not by any user but by a particular user. The user's retinal pattern detected by the visual sensor 19 or the detection signal from the biometric sensors 22 may be used to identify individual users. Where the retinal patterns or specific biometric information about expected users is registered in advance, the system controller 10 may determine which user is currently wearing the display apparatus 1 in accordance with the incoming information.
Specifically, when a certain user wears the display apparatus 1, the system controller 10 authenticates the user's identity. If a particular user's identity is recognized, then the system controller 10 detects the display start trigger and performs display start control accordingly.
It is also possible to detect the display start trigger by interpreting the user's diverse unconscious behavior or physical status.
For example, a sudden shift in the user's line of sight in the information coming from the visual sensor 19 may be interpreted as a display start trigger. The display start trigger may be recognized alternatively by use of other information from the visual sensor 19 or on the basis of input sounds from the audio input unit 6.
When an image is displayed by the display unit 2 in response to the display start trigger as described above, a see-through state area AR1 may be left intact on the screen of the display unit 2 while another area AR2 is arranged to display the image being picked up, as shown in (b) of
Although not shown in
Conversely, the system controller 10 may switch off the display apparatus 1 when detecting the apparatus 1 getting unmounted by the user.
Described below in reference to
In step F300 of
If the user's line of sight is found moving leftward, then the system controller 10 goes from step F301 to step F303 and recognizes an image-back trigger.
If the user's line of sight is found moving rightward, then the system controller 10 goes from step to step F304 and recognizes an image-forward trigger.
In any of the cases above, control of the process in
Although the example above was shown to utilize detection of the user's line of sight moving right or left, this is not limitative of the present invention. Alternatively, the user's line of sight moving up or down may be detected and interpreted as a screen scroll trigger, whereby the display screen may be scrolled as desired by the user.
In step F400 of
In step F401, the system controller 10 checks biometric information such as the brain wave, heart rate, perspiration, and blood pressure coming from the biometric sensors 22.
Given such information from the visual sensor 19 and biometric sensors 22, the system controller 10 checks to determine whether the user is in a comfortable state with regard to the image being displayed by the display unit 2.
If the user's state of visual perception is found to be uncomfortable, then the system controller 10 goes from step F402 to step F403 and recognizes an image adjustment trigger. The system controller 10 then goes to step F404 and calculates modified settings deemed to bring about comfort to the user in his or her situation, such as adjusted settings of display brightness, contrast, and sharpness.
Following steps F403 and F404, control of the process in
For example, the user's state of visual perception may deteriorate by eyestrain or as a result of aberrant ambient brightness against which the image picked up by the image pickup capability block is being displayed. In such a case, the above processing corrects the situation and allows the user to restore a pleasant state of visual perception. Illustratively, a dimmed unclear image such as one shown in (a) of
In step F500, the system controller 10 checks for information detected by the acceleration sensor 20 and/or gyro 21 (acceleration signal, angular velocity signal). In step F501, the system controller 10 determines the movement of the user's head. For example, checks are made to see if the user's head has moved backward twice, forward twice, or leftward twice.
If the user's head is found to have moved backward twice, then the system controller 10 goes from step F502 to step F505 and recognizes an image scale-up trigger.
In this case, control of the process in
If the user's head is found to have moved forward twice, then the system controller 10 goes from step F503 to step F506 and recognizes an image scale-down trigger. In this case, control of the process in
If the user's head is found to have moved leftward twice, then the system controller 10 goes from step F504 to step F507 and recognizes an image scale reset (default) trigger. In this case, too, control of the process in
In step F600 of
Upon detecting a movement of the user's line of light, the system controller 10 goes from step F601 to step F603 and recognizes a trigger for moving the cursor (i.e., onscreen pointer) on the display screen. In step F604, the system controller 10 calculates the direction and amount of the cursor movement over the screen based on the direction and mount of the detected line-of-sight movement.
In this case, the system controller 10 reaches step F107 in
Upon detecting the user's two consecutive blinks, the system controller 10 goes from step F602 to step F605 and recognizes a trigger for a click operation. In this case, the system controller 10 reaches step F107 in
The processing above allows the user's eye movements to be used as triggers commensurate with the operations of a mouse attached to a typical computer system. When a video game image such as one in (b) of
The processes in
A display control trigger can be recognized whenever predetermined conditions are found to be met following detection of the user's behavior and physical status from diverse information coming from the visual sensor 19, acceleration sensor 20, gyro 21, and biometric sensors 22.
When a display control trigger is recognized under diverse conditions, the display image processing unit 12 may be operated to execute scale-up/scale-down control; brightness, contrast and sharpness adjustment; control on image effects including pixelated mosaic effect, images in reverse video, and soft-focus images; and control on split-screen display and strobe-induced image display, in accordance with the user's behavior and physical status in effect.
If the image pickup capability block is selected as the source, with picked-up images displayed by the display unit 2, then the image pickup unit 3 and image pickup signal processing unit 15 may be operated to execute zoom control between telephoto and wide-angle shot settings, imaging sensitivity control, switchover between imaging frame rates, and changes to infrared or ultraviolet imaging sensitivity settings.
If the reproduced image from the storage unit 25 or the received image from the communication unit 26 is displayed by the display unit 2, then diverse display control triggers may be recognized and the storage unit 25 or communication unit 26 may be operated accordingly. That is, the operations may deal with variable speed reproduction controls such as fast-forward/fast-rewind, instant access to desired positions, frame-by-frame feed, slow reproduction, and pause; page feed, page scroll, shift of a highlighted portion within a list display, cursor movement, decision-finalizing operation, and video game actions.
In other words, the conditions under which to detect display control triggers may be associated with the corresponding control operations in a large number of combinations.
When onscreen displays on the display unit 2 are to be changed in response to the display control trigger as described above, the wider area AR1 may be left unchanged in its see-through state or normal display sate while the smaller area AR2 is arranged to display images in a different form, as shown in (b) of
The foregoing description given in reference to
That is, the user's deliberate actions as well as his unconscious behavior and physical status may be detected from the information supplied by the visual sensor 19, acceleration sensor 20, gyro 21, and biometric sensors 22. When a particular condition is found to be met, the presence of a source switch trigger may be detected.
With the source switch trigger thus recognized, the system controller 10 may go to step F108 in
Described below in reference to
(a) of
In step F800 of (a) of
Upon detecting the user's particular behavior designating the desire to end monitor display, the system controller 10 goes from step F801 to step F802 and recognizes a display end trigger.
After step F802, control of the process in
(b) of
In step F810 of (b) of
Upon detecting three consecutive blinks on the user's part, the system controller goes from step F811 to step F812 and recognizes a display end trigger.
After step F812, control of the process in
Where the user's desire to bring about the see-through state is expressed by behavior as outlined in (a) and (b) of
(a) of
In step F900 of (a) of
Upon detecting the user starting to walk or run, the system control goes from step F901 to step F902 and recognizes that a display end trigger has occurred.
After step F902, control of the process in
Interpreting the user's walking or running state as the cue to bring back the see-through state is desirable in terms of ensuring the user's safety.
However, the switch back to the see-through state on cue is not limitative of the invention. Alternatively, upon detection of the user's walking or running state, monitor display may be switched to the normal pickup image equivalent to the see-through state as shown in (b) of
(b) of
In step F910 of (b) of
In step F911, the system controller 10 analyses information from the visual sensor 19 to check the user's pupil dilation and blinks (number of blinks per unit time).
On the basis of the information coming from the biometric sensors 22 and visual sensor 19, the system controller 10 checks to determine whether the user's eyes are unduly strained.
Upon detecting the user's eyestrain, the system controller 10 goes from step F912 to step F913 and recognizes that a display end trigger has occurred.
After step F913, control of the process in
When the display operation of the display unit 2 is brought to an end in response to the user's physical status such as eyestrain, the display apparatus 1 can be used in a manner minimizing the burdens on the user's physical condition.
[6. Effects, Variations and Extensions of the Embodiments]
The preferred embodiment of the present invention has been described above. The description has focused, among others, on the display unit 2 which is part of an eyeglass-like or head mount-type wearable unit of the display apparatus 1 and which displays images immediately before the user's eyes. With this setup, information about the user's movements and physical status is used as the basis for determining the user's intentions and biometric status. The result of what has been determined is in turn used to control display-related operations, whereby a suitable display is implemented corresponding to the user's intentions and status with little operational effort on the user's part. The inventive display apparatus 1 thus provides excellent usability and offers the user varieties of visual experiences.
With the display unit 2 placed in the transparent or translucent see-through state through control of its transmittance, the display apparatus 1 when worn by the user does not bother him in his daily pursuits. The benefits of the inventive display apparatus 1 can be enjoyed by the user while leading ordinary life.
In the foregoing description, emphasis was placed on how to control display operations. Alternatively, the user's behavior and biometric status may be used as the basis for controlling the switchover between power-on, power-off, and standby state of the apparatus and for adjusting the volume and quality of sounds being output by the audio output unit 5. Illustratively, sound level control may be effected by checking the information from the biometric sensors 22 to see how comfortable the user is at present.
The display apparatus 1 may incorporate a character recognition block that recognizes characters in images, and a speech synthesis block that performs speech synthesizing processes. With such modifications in place, the character recognition block recognizes any characters that may be found included in the images that have been picked up, reproduced, or received. Given the recognized characters, the speech synthesis block generates read-aloud voice signals causing the audio output unit 5 to read the characters aloud.
The appearance and structures of the display apparatus 1 are not limited to those shown in
Although the display apparatus 1 was described as mountable on the eyeglass-like or head mount-type wearable unit, this is not limitative of the invention. The display apparatus 1 according to the invention need only be structured to give display immediately before the user's eyes. Illustratively, the inventive apparatus may be of a headphone type, a neck band type, a ear-hook type, or any other wearable type. As other alternatives, the display apparatus may be worn by the user in a manner attached to the user's eyeglasses, visor, headphones, or other daily-worn accessories with a clip or like fittings.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2006-244686 | Sep 2006 | JP | national |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/JP2007/065083 | 8/1/2007 | WO | 00 | 2/25/2009 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2008/029570 | 3/13/2008 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 5905525 | Ishibashi et al. | May 1999 | A |
| 5978015 | Ishibashi et al. | Nov 1999 | A |
| 6091546 | Spitzer | Jul 2000 | A |
| 6388638 | Fukushima et al. | May 2002 | B2 |
| 6549231 | Matsui | Apr 2003 | B1 |
| 6558050 | Ishibashi | May 2003 | B1 |
| 7855743 | Sako et al. | Dec 2010 | B2 |
| 20010038360 | Fukushima et al. | Nov 2001 | A1 |
| 20030174211 | Imaoka et al. | Sep 2003 | A1 |
| 20030234885 | Pilu | Dec 2003 | A1 |
| 20050190989 | Kondo et al. | Sep 2005 | A1 |
| 20050256675 | Kurata | Nov 2005 | A1 |
| 20060061544 | Min et al. | Mar 2006 | A1 |
| 20070195012 | Ichikawa et al. | Aug 2007 | A1 |
| 20080062291 | Sako et al. | Mar 2008 | A1 |
| 20080062297 | Sako et al. | Mar 2008 | A1 |
| 20080211921 | Sako et al. | Sep 2008 | A1 |
| Number | Date | Country |
|---|---|---|
| 0 709 816 | May 1996 | EP |
| 0 827 337 | Mar 1998 | EP |
| 1 541 966 | Jun 2005 | EP |
| 8 5954 | Jan 1996 | JP |
| 8 123979 | May 1996 | JP |
| 8 126031 | May 1996 | JP |
| 9 27970 | Jan 1997 | JP |
| 9 185009 | Jul 1997 | JP |
| 11-146057 | May 1999 | JP |
| 2000 284214 | Oct 2000 | JP |
| 2000-307715 | Nov 2000 | JP |
| 2001 13941 | Jan 2001 | JP |
| 2003 76353 | Mar 2003 | JP |
| 2004 180208 | Jun 2004 | JP |
| 2005 172851 | Jun 2005 | JP |
| 1462543 | Dec 2003 | TW |
| 2002 019728 | Mar 2002 | WO |
| 2005 122128 | Dec 2005 | WO |
| Entry |
|---|
| Examination and Search Report issued Nov. 23, 2010 in Singaporean Application No. 200901147-9. |
| Supplementary European Search Report issued May 20, 2011, in European Patent Application No. 07791763.1. |
| European Office Action issued Feb. 14, 2012, in European Patent Application No. 07 791 763.1. |
| Japanese Office Action issued Mar. 21, 2012, in Japanese Patent Application No. 2006-244686, filed Sep. 8, 2006. |
| Office Action issued Mar. 12, 2012 to Chinese patent application No. 200780031798.4, with English translation. |
| Office Action issued Jan. 10, 2012 to Taiwanese patent application No. 10120023100. |
| Office Action issued Aug. 21, 2012 in Japanese Patent Application No. 2006-244686. |
| Number | Date | Country | |
|---|---|---|---|
| 20100013739 A1 | Jan 2010 | US |
