The present invention is related generally to gaze tracking systems and methods. More particularly but not exclusively, the present invention relates to apparatus and methods for enhancing the performance and response of imaging sensors used for gaze tracking applications by combining pixel specific filtering with sensor elements to facilitate image processing.
In typical imaging applications, an imaging device (also denoted herein as an imager) is used to capture digital images based on light focused on or incident on a photosensitive element of the device. Digital imaging devices utilize photoelectronic imaging sensors consisting of arrays of pixels. Photoelectronic sensors used in many applications are based on semiconductor technologies such as Charge-Coupled Device (CCDs) and Complementary Metal-Oxide-Semiconductor (CMOS). While standard implementations of these imaging sensors are suitable for many applications, the pixel arrays associated with standard imaging devices are typically homogeneous, having the same imaging and photosensitivity characteristic throughout the sensor.
In some applications, it may be desirable to have additional control over pixel-specific characteristics of the imaging sensor and/or over associated pixel-specific processing. Accordingly, there is a need in the art for imaging devices that provide more pixel-specific configurations and controls.
The present invention is related generally to gaze tracking systems and methods.
In one aspect, the present invention is directed to a filtering assembly for an imaging apparatus comprising a filter array including a plurality of filter elements, said plurality of filter elements including a first filter element configured to filter light according to a first range of wavelengths and a second filter element configured to filter light according to a second range of wavelengths and a filter map, said filter map including a set of data corresponding to characteristics of ones of the plurality of filter elements.
In another aspect, the present invention is directed to an imaging apparatus comprising an imaging sensor having a plurality of pixel elements disposed in an array, said pixel elements configured for sensing light, a filter array optically coupled to the pixel array, said filter array including a plurality of filter elements matched to ones of a corresponding plurality of the pixel elements and a filter map, said filter map including a set of data corresponding to ones of the plurality of filter elements.
In another aspect, the present invention is directed to a method of processing images for gaze tracking applications comprising receiving a first set of data representing sensor data provided by ones of a plurality of sensor elements of a pixel array, receiving a filter map, said filter map including data associated with characteristics of ones of a plurality of filter elements associated with corresponding ones of the plurality of sensor elements and generating a first processed image, said processed image generated at least in part by adjusting the first set of data based on the filter map.
Additional aspects of the present invention are further described and illustrated herein with respect to the following detailed description and appended drawings.
For a better understanding of the nature of the features of the invention, reference should be made to the following detailed description taken in conjunction with the accompanying drawings, in which:
a illustrates details of an embodiment of an imager, in accordance with aspects of the present invention.
b illustrates details of an embodiment of an image sensor, in accordance with aspects of the present invention.
a illustrates details of embodiments of image sensor filtering element configurations, in accordance with aspects of the present invention.
b illustrates details of an enhanced image sensor including a pixel array sensor and a filter array, in accordance with aspects of the present invention.
c illustrates details of embodiments of a filter array in accordance with aspects of the present invention.
a illustrates details of an embodiment of a process for sub-image enhancement, in accordance with aspects of the present invention.
b illustrates details of embodiments of sub-images and sub-image enhancement, in accordance with aspects of the present invention.
This application is related to U.S. Provisional Patent Application Ser. No. 60/955,639, entitled APPLICATIONS BASED ON GAZE TRACKING INTEGRATED WITH OTHER SENSORS, ACTUATORS AND ACTIVE ELEMENTS, to U.S. Provisional Patent Application Ser. No. 60/957,164, entitled SYNCHRONIZATION OF IMAGE SENSOR ELEMENT EXPOSURE AND ILLUMINATION FOR GAZE TRACKING APPLICATIONS, to U.S. Provisional Patent Application Ser. No. 61/021,945, entitled APPARATUS AND METHODS FOR SPATIAL REGISTRATION OF USER FEATURES IN GAZE TRACKING APPLICATIONS, to U.S. Provisional Patent Application Ser. No. 61/040,709, entitled APPARATUS AND METHODS FOR GLINT SIGNAL OPTIMIZATION AND SPATIAL REGISTRATION, to U.S. Utility patent application Ser. No. 12/139,369, entitled PLATFORM AND METHOD FOR CLOSED-LOOP CONTROL OF ILLUMINATION FOR GAZE TRACKING APPLICATION, and to U.S. Utility patent application Ser. No. 12/025,716, entitled GAZE TRACKING USING MULTIPLE IMAGES. The content of each of these applications is hereby incorporated by reference herein in its entirety for all purposes. These applications may be denoted collectively herein as the “related applications” for purposes of brevity.
The present invention is related generally to gaze tracking systems and methods. More particularly but not exclusively, the present invention relates to apparatus and methods for enhancing the performance and response of imaging sensors used for gaze tracking applications.
Embodiments of various aspects of the present invention are further described below with respect to the appended drawings. It is noted that the embodiments described herein are provided for purposes of illustration, not limitation, and other embodiments including fewer components or stages, more components or stages and/or different components or stages are fully contemplated within the spirit and scope of the present invention.
Various embodiments of the present invention are described in detail below with reference to the figures, wherein like elements are referenced with like numerals throughout unless noted otherwise.
Gaze tracking systems are used to measure and track the relative position of a user's attention when viewing a reference component, such as a computer display screen or other point of interest. The relative position of the user is typically determined with respect to a particular frame of reference, which then allows for tracking of the user's gaze and/or other user related parameters, including those described herein and in the related applications. For example, in a gaze tracking application for use on a computer system, the most relevant frame of reference would typically be the computer's display or monitor, and the user's gazing direction may be determined by generating images of the user, and in particular user features such as the eyes and reflections from the eyes (i.e., glints), and then determining gaze from those images. A core component of such a system are imaging devices, which are components for receiving and capturing images of the user. The present invention is directed to apparatus and methods for enhancing the configuration and performance of imaging devices to increase overall system performance in applications such as gaze tracking, as well as other applications.
Attention is now directed to
Light source 60 may include fixed or dynamically adjustable elements for generating and controlling light illumination, typically at IR wavelengths, but also, in some embodiments, at visible or other wavelengths. The output light from source 60 may be modulated in amplitude, may be time varying, such as by turning light output on and off, may be adjusted by wavelength, and/or may be adjusted by position or rotation. In some embodiments two or more light sources 60 may be combined in a single component source or module to provide multiple light output functionality.
In a typical gaze tracking application, output light 13a is generated by light source 60 and reflected from features of the eye 10, with the reflected light as well as any ambient or other light (incoming sensor light 13b) received at imager module 80. Imager module 80 includes one or more imaging sensor elements configured to capture incoming light and generate one or more images for further processing in processor module 40. Imager module 80 may also include optical elements such as lenses and associated mechanical assemblies, filters, mirrors, electronics, processors, embedded software/firmware and/or memory, as well as housings and/or other related electronic or mechanical components.
Processor module 40 is configured to receive one or more images from imager module 80 to generate user tracking data as well as provide data to light control module 50 to adjust the output of light source(s) 60 to optimize tracking or feature recognition performance. Processor module 40 may also be connected to display 70 to provide on-display images from the target object, such as cursors or other indications of the user's point of regard and/or other displays or information. It is noted that the processing and control functionality illustrated in
Attention is now directed to
Additional details of sensor 220 are illustrated in
In accordance with one aspect of the present invention, a set of filter elements 332 may be applied to the sensor pixels of a sensor array 320 in combination with a substrate 310, as shown in
Particular characteristics of the pixels 322 of sensor array 320 may be determined and mapped into a pixel map 320b, with characteristics or parameters associated with one or more pixels 322 (typically all pixels 322) of sensor array 320 stored in the pixel map 320b as shown in
In addition, a filter array 330, matched to the sensor array 320, may be included in the sensor element. The filter array 330 may also be denoted herein as a Gaze Tracker Filter Array, abbreviated as a GTFA. As shown in
For example, as shown in
A variety of other filter array pixel configurations may also be used. For example,
In addition, in some embodiments the characteristics of the filter array may be dynamically alterable based on particular image, spatial, temporal and/or other characteristics of the image received from the target object and/or from information provided by a processor such as processor 40, via a filter control signal (not shown), or by another processor or other component of imager 80 or system 100. For example, in one embodiment the filter array may include LCD elements (or other elements known or developed in the art) configured to allow dynamic adjustment of filter characteristics such as intensity, polarization and/or passed or attenuated wavelengths based on the provided control signal. Data associated with this dynamically adjustable information may then be provided, typically simultaneously, to an associated filter map 330b as further described below.
GTFA 330 also includes a filter map 330b as shown in
As noted previously, In typical embodiments, GTFA 330 comprises a one dimensional or multi-dimensional mosaic pattern of filter elements 332, where the filter elements 332 modify the spectral response of corresponding pixel elements of the sensor array 320. In some embodiments, GTFA 330 may be constructed in a filter-on-window configuration, which is a manufacturing method allowing placement of filter elements onto the window of a sensor, such as sensor 320. This may be done with CCD or CMOS sensors, as well as with other sensor elements. Alternately, in some embodiments, GTFA 330 may be constructed using a filter-on-die configuration, which is a manufacturing method wherein the filtering elements are placed directly onto the silicon surface of the sensor (such as the CCD or CMOS sensor).
In some embodiments, GTFA 330 may be a separate component, such as in a filter-on-window implementation, or may be integral with the sensor 320, such as in a filter-on-die implementation. As a separate component, GTFA 330 is aligned and mated to the sensor 320, such as through mechanical alignment and mounting techniques as are know or developed in the art. In some embodiments, GTFA 330 may be constructed of passive, discrete optical filter elements. Each passive filter element may have different optical absorptive properties. Alternately, GTFA 330 may be constructed with one or more active elements, which may be addressable and programmable, such as in conjunction with digital electronics element 250 of
As an integral component of sensor 320 (i.e., in a filter-on-die configuration), GTFA 330 may have a filtering pattern construction based on known fabrication technologies for manufacturing filter arrays. For example, a Bayer Color Filter Array (BCFA) implementation may be used, where the BCFA is a mosaic pattern consisting of a single wavelength of filter elements (such as red, green and blue), which is commonly used for capturing and reconstructing color images. In addition, the GTFA 330 filter elements may be constructed by controlling and/or modifying the inherent optical reflectivity and transmissive properties of silicon during pixel sensor 320 manufacturing. The QE of an imager's pixel cavity at wavelengths of interest may be controlled accordingly.
GTFA 330 elements may also be constructed by controlling the placement of optical dead structures and/or modifying the absorption losses within an imager's pixel cavity. The QE of an imager's pixel cavity at wavelengths of interest may be controlled accordingly. The GTFA 330 elements may also be constructed by doping the corresponding imager's pixel cavity (such as, for example, by using ion implantation techniques) to create different optical absorptive properties.
b illustrates a composite sensor 340 including sensor array 320 combined with filter array 330 and a substrate 310. Composite sensor 340 may be used in applications as sensor 220 as shown in
Images obtained from a filtered sensor, such as sensor 340, may then be processed as illustrated in processing embodiment 400 of
Process 400 begins with a start acquisition stage 410, where image acquisition may be triggered by the processor 40 in conjunction with light source 60. For example, processor 40, in conjunction with control module 50, may direct light source 60 to provide IR light (and/or visible or other wavelengths of light) to the user's eye 10 as shown in
For example, in some embodiments specific processing is dependent on the particular sensor and filter array 330 and filter map data 330b. In one embodiment, a pattern composed of blue, green, red (for color imaging) and IR filters may be used in a 2×2 matrix, with the green signal value doubled to allow chromatic reconstruction of the scene in a standard implementation. Alternately, if alternate rows are comprised of IR filters, one row may be subtracted from the adjacent row to obtain the IR response. In addition, it is noted that the above described processing may be implemented in a fashion that is different from that used in conventional imaging applications where chromatic and spatial reconstruction are desired. In many embodiments of the present invention, the acquired images and associated processing are not ultimately intended for direct display to an end user, as is the case with a conventional imaging system, but rather is typically used to provide information such as gazing direction data and associated motion or tracking data.
It is noted that the processing described with respect to
Process 500 begins with a start acquisition stage 510, where image acquisition may be triggered by the processor 40 in conjunction with light source 60. For example, processor 40, in conjunction with control module 50, may direct light source 60 to provide IR light (and/or visible or other wavelengths of light) to the user's eye 10 as shown in
Once a particular sub-image region of interest is determined (or alternately, if the entire acquired image is used), two (or more) sub-images may be extracted from the received image as shown in
Because certain characteristics of the eye provide greater reflection to IR illumination (such as glints 556a), the images can be processed to separate the IR specific features as shown in image 562. For example, in a typical embodiment, the visible light only image 552b can be subtracted from the visible+IR image 552a to generate image 562, which illustrates the enhanced glint 556c. In addition to subtraction, other processing may be performed at stage 560, such as by thresholding the subtracted images (i.e., applying a threshold filter to assign pixel values above a threshold to while and pixel values below a threshold to black). Any other desired additional processing may be done at stage 570, with the processed data then stored in a memory of the sensor element and/or output at stage 580. It is noted that the processing described with respect to
It is noted that in various embodiments the present invention may relate to processes or methods such as are described or illustrated herein and/or in the related applications. These processes are typically implemented in one or more modules comprising systems as described herein and/or in the related applications, and such modules may include computer software stored on a computer readable medium including instructions configured to be executed by one or more processors. It is further noted that, while the processes described and illustrated herein and/or in the related applications may include particular stages, it is apparent that other processes including fewer, more, or different stages than those described and shown are also within the spirit and scope of the present invention. Accordingly, the processes shown herein and in the related applications are provided for purposes of illustration, not limitation.
As noted, some embodiments of the present invention may include computer software and/or computer hardware/software combinations configured to implement one or more processes or functions associated with the present invention such as those described above and/or in the related applications. These embodiments may be in the form of modules implementing functionality in software and/or hardware software combinations. Embodiments may also take the form of a computer storage product with a computer-readable medium having computer code thereon for performing various computer-implemented operations, such as operations related to functionality as describe herein. The media and computer code may be those specially designed and constructed for the purposes of the present invention, or they may be of the kind well known and available to those having skill in the computer software arts, or they may be a combination of both.
Examples of computer-readable media within the spirit and scope of the present invention include, but are not limited to: magnetic media such as hard disks; optical media such as CD-ROMs, DVDs and holographic devices; magneto-optical media; and hardware devices that are specially configured to store and execute program code, such as programmable microcontrollers, application-specific integrated circuits (“ASICs”), programmable logic devices (“PLDs”) and ROM and RAM devices. Examples of computer code may include machine code, such as produced by a compiler, and files containing higher-level code that are executed by a computer using an interpreter. Computer code may be comprised of one or more modules executing a particular process or processes to provide useful results, and the modules may communicate with one another via means known in the art. For example, some embodiments of the invention may be implemented using assembly language, Java, C, C#, C++, or other programming languages and software development tools as are known in the art. Other embodiments of the invention may be implemented in hardwired circuitry in place of, or in combination with, machine-executable software instructions.
The description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that specific details are not required in order to practice the invention. Thus, the foregoing descriptions of specific embodiments of the invention are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed; obviously, many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, they thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the following claims and their equivalents define the scope of the invention.
This application claims priority under 35 U.S.C. §119(e) to co-pending U.S. Provisional Patent Application Ser. No. 60/953,679, entitled OPTIMIZATION OF IMAGES SENSORS FOR USE IN GAZE TRACKING APPLICATIONS, filed on Aug. 2, 2007. This application is related to U.S. Provisional Patent Application Ser. No. 60/955,639, entitled APPLICATIONS BASED ON GAZE TRACKING INTEGRATED WITH OTHER SENSORS, ACTUATORS AND ACTIVE ELEMENTS, filed on Aug. 14, 2007, to U.S. Provisional Patent Application Ser. No. 60/957,164, entitled SYNCHRONIZATION OF IMAGE SENSOR ELEMENT EXPOSURE AND ILLUMINATION FOR GAZE TRACKING APPLICATIONS, filed on Aug. 21, 2007, to U.S. Provisional Patent Application Ser. No. 61/021,945, entitled APPARATUS AND METHODS FOR SPATIAL REGISTRATION OF USER FEATURES IN GAZE TRACKING APPLICATIONS, filed Jan. 18, 2008, to U.S. Provisional Patent Application Ser. No. 61/040,709, entitled APPARATUS AND METHODS FOR GLINT SIGNAL OPTIMIZATION AND SPATIAL REGISTRATION, filed on Mar. 30, 2008, to. U.S. Utility patent application Ser. No. 12/139,369, entitled PLATFORM AND METHOD FOR CLOSED-LOOP CONTROL OF ILLUMINATION FOR GAZE TRACKING APPLICATION, filed on Jun. 13, 2008, and to U.S. Utility patent application Ser. No. 12/025,716, entitled GAZE TRACKING USING MULTIPLE IMAGES, filed on Feb. 4, 2008. The content of each of these applications is hereby incorporated by reference herein in its entirety for all purposes.
Number | Date | Country | |
---|---|---|---|
60953679 | Aug 2007 | US | |
60955639 | Aug 2007 | US | |
60957164 | Aug 2007 | US | |
61021945 | Jan 2008 | US | |
61040709 | Mar 2008 | US |