Patent Grant
10327308
The present application is related to a mobile device, and more specifically to methods and systems that contain a display operable as an infrared emitter, and a light sensor operable as an infrared receiver.
Current face recognition and detection systems utilizing infrared (IR) imaging, such as Google's Tango, Apple's FaceID, require an IR light source. The IR light source is a stand-alone IR emitter. Additionally, these IR systems also use a stand-alone IR receiver to receive reflected light from the IR emitter. Placing additional sensors inside a mobile device, increases the cost of manufacturing of the mobile device, and decreasing the reliability of the mobile device because foreign substances such as dirt and water can get inside the cavities drilled into the mobile device chassis. Further, the images recorded using the stand-alone IR receiver tend to be low resolution due to limited resolution of the stand-alone IR receiver.
Presented here is a system to record high-resolution infrared images without the need to include additional stand-alone sensors into the mobile device. According to one embodiment, an organic light emitting diode (OLED) display is modified to emit IR and near-IR light in a large field. The modified display allows for depth sensing and infrared imaging without a stand-alone emitter. Additionally, an IR shutter filter can be applied to the existing front facing red, green, blue (RGB) camera that would only be in place when the display is in IR emission mode. The combination of these two technologies allows a facial recognition system using existing hardware, and not require additional sensors or emitters to achieve face recognition.
Mobile Device
Presented here is a system to record high-resolution infrared images without the need to include additional stand-alone sensors into the mobile device. According to one embodiment, an organic light emitting diode (OLED) display is modified to emit IR and near-IR light in a large field. IR spectrum spans 700 nm-1 mm wavelengths, while near-IR spectrum spans 700 nm-2500 nm wavelengths. Visible spectrum spans 380 nm-700 nm. The modified display allows for depth sensing and infrared imaging without a stand-alone emitter. Additionally, an IR shutter filter can be applied to the existing front facing red, green, blue (RGB) camera that would only be in place when the display is in IR emission mode. The combination of these two technologies allows a facial recognition system using existing hardware, and not require additional sensors or emitters to achieve face recognition.
In one embodiment the regions 125, 135 can be dual-mode regions, meaning that regions 125, 135 can change modes between being transparent and being opaque. When the region 125, 135 are opaque, they can display part of an image shown on the display screen 110. In another embodiment the regions 125, 135 can be permanently transparent, and not configured to act as a part of the display 110. Regions 125, 135 can be the same size as, or slightly larger than the light sensor 120, and the IR emitter 130, respectively. When the region 125, 135 are dual-mode regions, the dual-mode regions 125, 135 can be selectively addressable region as described in the U.S. patent application Ser. No. 15/336,380, filed Mar. Oct. 27, 2016, and incorporated by reference herein in its entirety.
The optional IR emitter 130 can emit a flash infrared light just prior to the light sensor 120 sensing the infrared light. The processor of the mobile device 100 can send an instruction to the IR emitter 130 to flash and nearly simultaneously send an instruction to the light sensor 120 to record the IR image. The IR emitter 130 can emit the flash of IR light through an opaque region 135, or the processor can send an instruction to the region 135 to turn transparent as the IR emitter 130 flashes the infrared light. The IR emitter 130 can occupy a small region of the display 110, as shown in
The display 110 can be a dual-mode display transmitting both visible and IR light. The display screen 110 can include multiple display members, such as OLEDs and/or LCDs. A first mode of operation of the dual-mode display screen 110 includes multiple display members transmitting visible light and forming an image of the display screen for a viewer to see. A second mode of operation of the dual-mode display screen 110 includes multiple display members transmitting infrared (IR) light into an environment surrounding the display screen. Enabling the display 110 to transmit both visible and IR light renders the IR emitter 130 not necessary. Consequently, the IR emitter 130 can be excluded from the mobile device 100, thus minimizing the number of ports drilled into the mobile device 100, and minimizing the number of failure points of the mobile device 100.
The light sensor 120 can record a picture through the region 125 of the display 110. The light sensor 120 can record an image when the region 125 is transparent and/or when the region 125 is displaying an image. The light sensor 120 can record an infrared (IR) image and/or the light sensor 120 can record a visible light spectrum image.
The light sensor 120 can be a dual-mode light sensor, e.g. a dual-mode camera, including multiple imaging members such as a photodiode that can detect only visible light, only IR light, or both visible light and IR light. The photodiodes can be arranged to form a CMOS or a CCD sensor. A first mode of operation of the dual-mode light sensor 120 can include receiving visible light and forming an image of the visible light. A second mode of operation of the dual-mode light sensor 120 can include receiving infrared light and forming an image of the infrared light.
A controller, i.e. a processor, can synchronize the functioning of the dual-mode display screen 110 and the dual-mode light sensor 120 by sending a first instruction to the dual-mode display to transmit infrared light, and sending a second instruction to the dual-mode camera to receive infrared light. As a result, the mobile device 100 can form the IR image.
Infrared Display Screen
The dual-mode display member 200, 210, 220, 230 can emit both IR light and a specific part of the visible light spectrum. For example, the dual-mode display member 210 can transmit both red light and IR light, the dual-mode display member 200, 220 can transmit both green light and IR light, while the dual-mode display member 230 can transmit both blue light and IR light. In other examples, instead of transmitting red, green, and blue light, in addition to IR light, the dual-mode display member 200, 210, 220, 230 can transmit cyan, yellow, magenta in addition to IR, cyan yellow magenta, and white in addition to IR, etc.
By default, the dual-mode display member 200, 210, 220, 230 can transmit only the visible light, or can transmit both visible light and IR light. When the default for dual-mode display member 200, 210, 220, 230 is to transmit only the visible light, a controller can send an instruction to the dual-mode display member 200, 210, 220, 230 to transmit IR light.
The dual-mode display member 200, 210, 220, 230 can be arranged in various patterns. In
The dual-mode display member 200, 210, 220, 230 can include OLEDs, and/or LCDs. For example, the dual-mode display member 200, 210, 220, 230 can include OLEDs emitting green and infrared light, red and infrared light, green and infrared light, and blue and infrared light respectively. In another example, the dual-mode display member 200, 210, 220, 230 can include an LCD display stack. The LCD display stack includes: a visible light backlight to emit visible light; an infrared backlight to emit infrared light; a polarizer to adjust a polarization of the infrared light and the visible light; and a liquid crystal to selectively transmit the polarized infrared light and the polarized visible light. Further, the LCD display stack needs a filtering member such as a color filter that selectively allows red, green, blue and IR light to pass through the color filter.
The display screen 110 includes a display member 240, 250, 260, 270 (only four marked for brevity). The display member 240 can transmit IR light, display member 250 can transmit red light, display member 260 can transmit green light, and display member 270 can transmit blue light. When a controller sends an instruction to the dual-mode display screen 110 to transmit IR light, the display member 240 activates and transmits IR light.
The display members 240, 250, 260, 270 can be laid out in a regular pattern as shown in
The density of the display member 240 the transmit IR light can be the same as the density of display members 250, 260270 transmitting visible light, as shown in
Two filtering members 420, 430 can be placed over the display member 400, 410. Filtering member 420 allows visible light to pass substantially blocking all other light, while filtering member 430 allows infrared light to pass substantially blocking all other light. Spectral filters 440, 450 of the filtering member 420 can be placed above the display members 400, 410 respectively. The spectral filters 440, 450 allow visible light emitted by the display members 400, 410 to pass. For example, spectral filter 440 allows green light to pass, while spectral filter 450 allows red light to pass. The spectral filters 460, 470 of the filtering member 430 allow infrared light to pass, while substantially blocking all other wavelengths.
When the controller determines that display screen 110 should emit infrared light, the controller sends an instruction to the one or more switching members 480, 490, 405 to replace the filtering member 420 with the filtering member 430. When the filtering member 430 is placed on top of the display members 400, 410, infrared light passes through the display screen 110, while the rest of the wavelengths are substantially blocked, including the visible light transmitted by the display member 400, 410.
The switching members 480, 490, 405 can be a micro electromechanical devices (MEMS). The switching member 480, 490, 405 can include a piezo electric material which expands when power is applied to it. As shown in
When the controller determines that the display screen 110 should return to emitting visible light, the controller can send an instruction to the one or more switching members 480, 490, 405 to retract, thus replacing filtering member 430 with the filtering member 420. In some embodiments, one or more optional switching members 415, 425 can be placed on the opposite side of the filter is shown in
In another embodiment, instead of waiting for the controller to send the instruction to place the filtering member 420 above the display member 400410, the switching members 480, 490, 405, 415, 425, 435 can automatically place the filtering member 420 above the display member 400, 410, when recording of the IR image is completed.
The display screen can include multiple dual-mode display members repetitively arranged, such as shown in
In addition, a dual-mode power source can be connected to the dual-mode display member. A first mode of operation of the dual-mode power source can be to provide a first amount of power and to cause the dual-mode display member to transmit visible light. A second mode of operation of dual-mode power source can be to provide a second amount of power and to cause the dual-mode display member to transmit IR light. The dual-mode power source can provide varying voltage. In some embodiments, the second amount of power can be lower than the first amount of power. A controller can send an instruction to the dual-mode power source to provide first, or second amount of power.
Further, a filtering member including multiple spectral filters can be provided, as shown in
Infrared Light Sensor
The dual-mode imaging members 710, 720 detect electromagnetic waves in a visible light spectrum and form a visible light image, as well as detecting electromagnetic waves in an IR spectrum and forming an IR image. The dual-mode imaging members 710, 720 can be wide dynamic range photodiodes, which can convert light in the range 300 nm-2500 nm or 300 nm-1 mm to electricity.
The filtering member 750 is placed between the dual-mode imaging member 710, 720 and incoming electromagnetic waves 755. The filtering member 750 can consist of a single filter, or can be divided into two or more rows of filters. The filtering member 750 includes multiple spectral filters, which include a visible light filter 760, 770, 780, 790 (only four labeled for brevity) and an IR filter 730 (only one drawn for brevity). The visible light filter 760, 770, 780, 790 substantially blocks IR light from reaching the dual-mode imaging member 710, 720, while the IR filter 730 substantially blocks visible light from reaching the dual-mode imaging member 710, 720. In
A controller can be coupled to the dual-mode imaging member 710, 720. Based on the user input, the controller can determine whether to form the visible light image or the IR image. The user input can be voice input, textual input, and/or gestural input. For example, when a phone is held at a predefined angle, such as within 45° away from vertical, the phone is locked, and the light sensor 700 detects a face, the controller can send an instruction to the display to emit IR light, and to the light sensor 700 to record IR light. Consequently, authentication of the user's face can be performed to unlock the phone. In another example, the user can give a voice command to the mobile device to record an IR image.
One or more switching member 705 move the filtering member 750 and positions the IR filter 730 between the dual-mode imaging member 710, 720 and incoming electromagnetic waves 755 when the controller determines to form the IR image. The switching member 705 can also be coupled to the controller, and receive instructions, such as an instruction to position the IR filter 730 on top of the dual-mode imaging member 710, 720. Once the IR image is recorded, the switching member 705 can automatically position the visible light filter between the dual-mode imaging member 710, 720 and incoming electromagnetic waves 755. In another example, the switching member 705 can wait to receive an instruction from the controller to position the visible light filter above the dual-mode imaging member 710, 720.
The switching member 705 can include a micro-electromechanical mechanical (MEMS) device, a nano-electromechanical device, a piezo actuator, etc. For example, the switching member 705 can include a piezo electric material which expands when power is applied to the piezo electric material. The expansion of the piezo electric material moves the filtering member 750 to position the IR filter 730 between the dual-mode imaging member 710, 720 and incoming electromagnetic waves 755 and removes the visible light filter from the dual-mode imaging member 710, 720.
The filtering member 750 can move along a guide in response to the switching member 705 moving the filtering member. The guide can be a rail, a groove in a substrate 715 on which the filtering member 750 is placed, etc.
The filtering member 800 can be moved left and right using one or more devices 860, 870, placed on either side of the filtering member 800, as described in this application. Devices 860, 870 can be MEMS devices, Piezo actuators, and/or springs, as described in this application. In one embodiment, the filtering member 800 can be moved to the right or left by one column, aligning the columns of infrared filters 800, 810, 820 above the dual-mode imaging member 710, 720, or aligning columns of visible light filters 830, 840, 850, respectively, as shown in
The light sensor array 910 includes multiple sensors. Beneath each IR filter is an IR sensor, such as IR sensor 950. Beneath each visible light filter, such as red, green, blue, is a visible light sensor. Each sensor records a scalar value representing intensity of light passing through the filter associated with the color filter array 900.
The multiple color filters associated with the color filter array 900 can include a color set such as: red, green, blue, IR; red, green, blue, emerald, IR; cyan, yellow, magenta, IR; cyan, yellow, green, magenta, IR; red, green, blue, white, IR; etc. The colors can be distributed within the color filter array 900 in a regular pattern, or an irregular pattern. The color filter array 900 shown in
When the light sensor 700 in
In step 1010, a filtering member including a visible light filter and an IR filter selectively filters electromagnetic waves into the visible light spectrum and the IR spectrum. The visible light filter substantially blocks IR light from reaching the dual-mode imaging member, while the IR filter substantially blocks visible light from reaching the dual-mode imaging member.
In step 1020, the controller determines whether to form the visible light image or the IR image. This determination can be based on a user input such as textual, gestural, or voice input. For example, the user can give a voice instruction to record an infrared image, or the user can press a button on a display instructing the controller to record the infrared image. In another example, the controller can determine based on the positioning of the device 100 in
Before recording the infrared image, a switching member can move the filtering member and position the IR filter between the dual-mode imaging member and incoming electromagnetic waves when the controller determines to form the IR image. After the IR image is recorded, the switching member can automatically position the visible light filter between the dual-mode imaging member and the incoming electromagnetic waves, or the switching member can wait for an instruction from the controller to position the visible light filter above the dual-mode imaging member.
The filtering member can move along a guide in response to the switching member moving the filtering member. The guide can be MEMS rail, or a groove in a substrate on which the filtering member rests.
The switching member can be a MEMS device, a MEMS spring, a Piezo actuator, etc. In one embodiment, when power is provided to the switching member, the switching member expands and moves the filtering member to position the IR filter between the dual-mode imaging member and incoming electromagnetic waves. As a result, the visible light filter is positioned away from the dual-mode imaging member. In another embodiment, when power is provided to the switching member, a MEMS blocker keeping the MEMS spring compressed is removed, and the MEMS spring expands, thus moving and positioning the filtering member, as described above.
The light sensor can receive both the visible image and the IR image, and produce a grayscale image for the IR light, and for visible light, as described in this application. For example, the grayscale images produced can be the red image, the green image, the blue image, and the infrared image. To record the infrared image, a processor discards the red, green, and blue images, and only analyzes the infrared image. The controller can discard the IR image when the controller determines to form the visible light image, and discard the visible light image when the controller determines to form the IR image.
Computer
In the example of
The video display of the computer system 1100 can be the display 110 in
This disclosure contemplates the computer system 1100 taking any suitable physical form. As example and not by way of limitation, computer system 1100 may be an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC) (such as, for example, a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile telephone, a personal digital assistant (PDA), a server, or a combination of two or more of these. Where appropriate, computer system 1100 may include one or more computer systems 1100; be unitary or distributed; span multiple locations; span multiple machines; or reside in a cloud, which may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 1100 may perform without substantial spatial or temporal limitation one or more steps of one or more methods described or illustrated herein. As an example and not by way of limitation, one or more computer systems 1100 may perform in real time or in batch mode one or more steps of one or more methods described or illustrated herein. One or more computer systems 1100 may perform at different times or at different locations one or more steps of one or more methods described or illustrated herein, where appropriate.
The processor may be, for example, a conventional microprocessor such as an Intel Pentium microprocessor or Motorola power PC microprocessor. One of skill in the relevant art will recognize that the terms “machine-readable (storage) medium” or “computer-readable (storage) medium” include any type of device that is accessible by the processor.
The memory is coupled to the processor by, for example, a bus. The memory can include, by way of example but not limitation, random access memory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM). The memory can be local, remote, or distributed.
The bus also couples the processor to the non-volatile memory and drive unit. The non-volatile memory is often a magnetic floppy or hard disk, a magnetic-optical disk, an optical disk, a read-only memory (ROM), such as a CD-ROM, EPROM, or EEPROM, a magnetic or optical card, or another form of storage for large amounts of data. Some of this data is often written, by a direct memory access process, into memory during execution of software in the computer 1100. The non-volatile storage can be local, remote, or distributed. The non-volatile memory is optional because systems can be created with all applicable data available in memory. A typical computer system will usually include at least a processor, memory, and a device (e.g., a bus) coupling the memory to the processor.
Software is typically stored in the non-volatile memory and/or the drive unit. Indeed, storing and entire large program in memory may not even be possible. Nevertheless, it should be understood that for software to run, if necessary, it is moved to a computer readable location appropriate for processing, and for illustrative purposes, that location is referred to as the memory in this paper. Even when software is moved to the memory for execution, the processor will typically make use of hardware registers to store values associated with the software, and local cache that, ideally, serves to speed up execution. As used herein, a software program is assumed to be stored at any known or convenient location (from non-volatile storage to hardware registers) when the software program is referred to as “implemented in a computer-readable medium.” A processor is considered to be “configured to execute a program” when at least one value associated with the program is stored in a register readable by the processor.
The bus also couples the processor to the network interface device. The interface can include one or more of a modem or network interface. It will be appreciated that a modem or network interface can be considered to be part of the computer system 1100. The interface can include an analog modem, isdn modem, cable modem, token ring interface, satellite transmission interface (e.g. “direct PC”), or other interfaces for coupling a computer system to other computer systems. The interface can include one or more input and/or output devices. The I/O devices can include, by way of example but not limitation, a keyboard, a mouse or other pointing device, disk drives, printers, a scanner, and other input and/or output devices, including a display device. The display device can include, by way of example but not limitation, a cathode ray tube (CRT), liquid crystal display (LCD), or some other applicable known or convenient display device. For simplicity, it is assumed that controllers of any devices not depicted in the example of
In operation, the computer system 1100 can be controlled by operating system software that includes a file management system, such as a disk operating system. One example of operating system software with associated file management system software is the family of operating systems known as Windows® from Microsoft Corporation of Redmond, Wash., and their associated file management systems. Another example of operating system software with its associated file management system software is the Linux™ operating system and its associated file management system. The file management system is typically stored in the non-volatile memory and/or drive unit and causes the processor to execute the various acts required by the operating system to input and output data and to store data in the memory, including storing files on the non-volatile memory and/or drive unit.
Some portions of the detailed description may be presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, members, symbols, characters, terms, numbers, or the like.
It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or “generating” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the methods of some embodiments. The required structure for a variety of these systems will appear from the description below. In addition, the techniques are not described with reference to any particular programming language, and various embodiments may thus be implemented using a variety of programming languages.
In alternative embodiments, the machine operates as a stand-alone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.
The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a laptop computer, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, an iPhone, a Blackberry, a processor, a telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.
While the machine-readable medium or machine-readable storage medium is shown in an exemplary embodiment to be a single medium, the term “machine-readable medium” and “machine-readable storage medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “machine-readable medium” and “machine-readable storage medium” shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies or modules of the presently disclosed technique and innovation.
In general, the routines executed to implement the embodiments of the disclosure, may be implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions referred to as “computer programs.” The computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and that, when read and executed by one or more processing units or processors in a computer, cause the computer to perform operations to execute members involving the various aspects of the disclosure.
Moreover, while embodiments have been described in the context of fully functioning computers and computer systems, those skilled in the art will appreciate that the various embodiments are capable of being distributed as a program product in a variety of forms, and that the disclosure applies equally regardless of the particular type of machine or computer-readable media used to actually effect the distribution.
Further examples of machine-readable storage media, machine-readable media, or computer-readable (storage) media include but are not limited to recordable type media such as volatile and non-volatile memory devices, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD ROMS), Digital Versatile Disks, (DVDs), etc.), among others, and transmission type media such as digital and analog communication links.
In some circumstances, operation of a memory device, such as a change in state from a binary one to a binary zero or vice-versa, for example, may comprise a transformation, such as a physical transformation. With particular types of memory devices, such a physical transformation may comprise a physical transformation of an article to a different state or thing. For example, but without limitation, for some types of memory devices, a change in state may involve an accumulation and storage of charge or a release of stored charge. Likewise, in other memory devices, a change of state may comprise a physical change or transformation in magnetic orientation or a physical change or transformation in molecular structure, such as from crystalline to amorphous or vice versa. The foregoing is not intended to be an exhaustive list in which a change in state for a binary one to a binary zero or vice-versa in a memory device may comprise a transformation, such as a physical transformation. Rather, the foregoing is intended as illustrative examples.
A storage medium typically may be non-transitory or comprise a non-transitory device. In this context, a non-transitory storage medium may include a device that is tangible, meaning that the device has a concrete physical form, although the device may change its physical state. Thus, for example, non-transitory refers to a device remaining tangible despite this change in state.
Remarks
The language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the invention be limited not by this Detailed Description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the embodiments, which is set forth in the following claims.
This application claims priority to the U.S. Provisional Patent Application Ser. No. 62/589,852, filed Nov. 22, 2017, and to the U.S. Provisional Patent Application Ser. No. 62/558,250, filed Sep. 13, 2017, all of which are incorporated herein by this reference in their entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6650357 | Richardson | Nov 2003 | B1 |
| 20150245767 | Northcott | Sep 2015 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20190082519 A1 | Mar 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| 62589852 | Nov 2017 | US | |
| 62558250 | Sep 2017 | US |
