Patent Grant
9846473
This relates generally to electronic devices, and more particularly, to electronic devices with components such as light sensors.
Electronic devices such as computers and cellular telephones may contain light-based components such as cameras and ambient light sensors. Light-based proximity sensors are used in cellular telephones to detect when a user has placed a cellular telephone against the user's head. These light-based components may be used to gather information on the environment in which a device is operating so that device functions can be adjusted appropriately.
It may be challenging to incorporate light-based components into an electronic device without disrupting the aesthetics and operation of the device. In devices with displays, for example, it may be desirable to form a window for a light sensor or camera so that the light sensor or camera can be mounted behind the display. If care is not taken, a window of this type may be overly conspicuous and or may not allow light-based components to function as desired. The light-based components that are included in a device may also be inadequate for gathering desired data on the operating environment of the device
It would therefore be desirable to be able to provide improved windows and light-based components for an electronic device such as an electronic device with a display.
An electronic device may have light-based components such as light sources, light detectors, and image sensors. The light-based components may be aligned with a window in the device. The window may be formed within an inactive border region of a display or within other device structures.
A window for light-based components may have one or more window members mounted within an opening in a display layer in an inactive portion of the display. Visible light blocking material such as chalcogenide glass may be incorporated into the window to provide the window with an opaque appearance that matches the opaque appearance of surrounding portions of the inactive portion of the display while simultaneously allowing infrared light such as light in a region of wavelengths between 5 and 14 microns or other suitable infrared light to pass to infrared light sensors aligned with the window. The infrared light sensors may be light detectors that contain a small number of individual light detecting elements or may be two-dimensional infrared image sensors.
In configurations in which the light-based components include a visible image sensor or other visible light detecting component, the window may be provided with a portion that is transparent at visible wavelengths. For example, the window may have an infrared-transparent ring that passes light to infrared detectors while blocking visible light and may have a central region of glass formed within the ring that is transparent at visible wavelengths.
Infrared detectors such as thermopile and pyroelectric detectors may be used to monitor for the presence of a user in the vicinity of the electronic device while consuming small amounts of power. Fresnel lenses or other light directing structures may be used to direct light to the infrared detectors. A visible light camera or other component can be awakened upon detection of the presence of a user or other suitable actions may be taken.
Electronic devices may include light-based components. Light-based components may be mounted behind one or more windows in the device. The windows may be formed in an inactive border region in a display or may be formed elsewhere in an electronic device. Configurations in which windows for light-based components are formed in displays are sometimes described herein as an example.
Illustrative electronic devices that may be provided with displays having windows for light-based components are shown in
Illustrative electronic device 10 of
The illustrative configurations for device 10 that are shown in
Housing 12 of device 10, which is sometimes referred to as a case, may be formed of materials such as plastic, glass, ceramics, carbon-fiber composites and other fiber-based composites, metal (e.g., machined aluminum, stainless steel, or other metals), other materials, or a combination of these materials. Device 10 may be formed using a unibody construction in which most or all of housing 12 is formed from a single structural element (e.g., a piece of machined metal or a piece of molded plastic) or may be formed from multiple housing structures (e.g., outer housing structures that have been mounted to internal frame elements or other internal housing structures).
Display 14 may be a touch sensitive display that includes a touch sensor or may be insensitive to touch. Touch sensors for display 14 may be formed from an array of capacitive touch sensor electrodes, a resistive touch array, touch sensor structures based on acoustic touch, optical touch, or force-based touch technologies, or other suitable touch sensor components.
Display 14 for device 10 may be a liquid crystal display, an organic light-emitting diode display, an electrophoretic display, a plasma display, or a display based on other display technologies.
A display cover layer may cover the surface of display 14 or a display layer such as a color filter layer, thin-film transistor layer, or other portion of a display may be used as the outermost (or nearly outermost) layer in display 14. The outermost display layer may be formed from a transparent glass sheet, a clear plastic layer, a layer of sapphire, ceramic, a transparent crystalline dielectric material, or other transparent member. Display 14 may have a planar profile, a curved profile, or other suitable cross-sectional shape. The outline of display 14 when viewed from the front of display 14 may be rectangular, may be circular, or may have other shapes. Configurations in which display 14 is planar and has a rectangular outline are sometimes described herein as an example. This is, however, merely illustrative. Display 14 may have any suitable shape.
Active area AA contains pixel array 92. Pixel array 92 contains an array of pixels such as pixels 94. Pixel array 92 may be controlled using control signals produced by display driver circuitry. The display driver circuitry may include one or more integrated circuits (e.g., timing controller integrated circuits) and/or thin-film transistor circuitry (e.g., data line demultiplexing circuitry and/or gate driver circuitry). The display driver circuitry of display 14 (e.g., the thin-film transistor circuitry such as the demultiplexer circuitry and gate driver circuitry) may be located in inactive area IA.
During operation of device 10, control circuitry in device 10 such as memory circuits, microprocessors, and other storage and processing circuitry may provide data to the display driver circuitry. The display driver circuitry may convert the data into signals for controlling rows and columns of display pixels 94 in pixel array 92.
Pixels 94 in pixel array 92 may contain thin-film transistor circuitry (e.g., polysilicon transistor circuitry, semiconducting-oxide transistor circuitry, etc.). In liquid crystal displays, pixel 94 may contain thin-film transistors and electrode structures for producing electric fields across a liquid crystal layer in display 14. In organic light-emitting diode displays, the pixel circuitry of pixels 94 may contain thin-film transistors and organic light-emitting diodes for generating images. Other types of display may contain other pixel structures (e.g., plasma pixels, electrophoretic pixels, etc.).
To block internal device components from view, inactive area IA of display 14 may contain opaque masking structures. For example, the inner surface of the outermost layer in display 14 (e.g., the inner border of a display cover layer for display 14) may be provided with an opaque layer of material such as black ink, white ink, ink of other colors, or other opaque masking material. The opaque masking layer may have one or more openings to accommodate light-based components. The opaque masking layer openings and optical structures that are aligned with the openings allow light to pass for light-based components and are sometimes referred to as windows, light-based component windows, optical windows, etc.
As shown in
A cross-sectional side view of device 10 is shown in
Layer 34 and other portions of display 14 may be mounted in housing 12. To hide internal components in device 10 from view from a user, the portion of layer 34 in inactive area IA (i.e., portion 341) may be provided with opaque masking structures that render this portion opaque to visible light. The portion of layer 34 in active area AA of display (i.e., portion 34AA) may be transparent to visible light, so that light 58 from pixel array 92 can pass to viewer 56. This allows viewer 56 to view images on display 14.
Window 32 may be aligned with light-based components 62. Light-based components 62 may include components that emit light 60 and/or components that sense light 60. Window 32 may be transparent to light 60, so that light 60 can pass from components 62 to the exterior of device 10 and/or so that light 60 can pass from the exterior of device 10 to components 62 in the interior of device 10.
Light 60 may include visible and/or infrared light. Infrared light in light 60 may have a wavelength longer than 700 nm, may have a wavelength longer than 7 microns, may have a wavelength of 5-14 microns, may have a wavelength of 8-14 microns, may have a wavelength less than 14 microns, or may have any other suitable wavelength. The human body emits radiant heat in the form of light that peaks at a wavelength of about 9.4 microns, so windows that transmit light at this wavelength may allow detection of the presence of a human body in the vicinity of display 14.
Components 62 may include visible-light components and/or infrared-light components. For example, components 62 may include a visible image sensor 40. Sensor 40 may be part of a visible-light camera (e.g., a camera with a resolution sufficient to capture digital images of users in the vicinity of device 10 for functions such as videoconferencing, face detection, etc.). Lenses 38 may be used to focus light 60 onto image sensor 40 and/or may form parts of other components 62.
If desired, components 62 may include an infrared image sensor such as infrared image sensor 42. Image sensor 42 may capture digital images at infrared wavelengths. Sensor 42 may have the same resolution as visible image sensor 40 or may have a higher or lower resolution. A low-resolution infrared sensor array 46 may be used to capture infrared data at a resolution that is lower than that of image sensor 42. Sensor 42 may, for example, be a 5 megapixel infrared camera sensor and sensor 46 may be an infrared image sensor having a two-dimensional array of image sensor elements that is sufficient to capture rough images at a resolution such as 20×20 pixels, 100×100 pixels, or 1000×1000 pixels. The resolution of sensor 46 may be less than 0.1 megapixels, less than 0.001 mega pixels, or other relatively low resolution. Restricting the resolution of sensor 46 may help enhance privacy in scenarios in which sensor 46 is always on (or nearly always on).
Infrared detectors 44 may gather infrared light without using an array of sensors (i.e., detectors 44 may include one or more solitary detectors or may contain a few sensor elements). Detectors 44 may, for example, include thermopile detectors, pyroelectric detectors, and other infrared detectors that are capable of detecting body heat from a user of device 10.
Visible detectors 48 may gather visible light using a relatively small number of sensor elements (e.g., one sensor element, two or more sensor elements, two to ten sensor elements, fewer than 10 elements, etc.). Detectors 48 may, for example, include an ambient light sensor.
In some situations, it may be desirable for components 62 to emit light (e.g., to help illuminate items in the vicinity of device 10). Illumination may be provided at visible wavelengths using visible light source 52 (e.g., a white light-emitting diode, a lamp, etc.) and may be provided at infrared wavelengths using infrared light source 52 (e.g., an infrared light-emitting diode, an infrared laser, an infrared lamp, etc.). In general, components 62 may include one or more of the components illustrated in
The visible and/or infrared sensing capabilities of components 62 may be used to detect the presence of users in the vicinity of device 10. For example, a camera may be used to capture images of users, image processing functions implemented on the control circuitry of device 10 may be used to detect user's faces, and other image processing operations may be performed to detect when a user is present and viewing display 14. As another example, an infrared sensor may detect the presence of one or more users (e.g., by detecting light at a wavelength between 5-14 microns or at a wavelength in a range of wavelengths from 8-14 microns using a sensor such as sensor 44, one or more of detectors 46, or other component 62). When a user is detected, device 10 can take appropriate action. For example, device 10 can activate display 14, can adjust audio playback, can adjust what type of content is presented to the user, can gather data, can turn on image processing functions (e.g., to gather video input from a user such as hand gestures), may activate input-output devices such as keyboard and touch pads, may wake up circuitry that is in a low-power sleep state, or may take other suitable actions.
Consider, as an example, a scenario of the type shown in
If desired, more detectors may be provided to detect the presence of users with a higher degree of angular discrimination. Based on knowledge of how many users are present and the location of those users, device 10 may make adjustments to display 14 (e.g., to optimize brightness, content, audio, etc.), may turn on or off device features, or may take other suitable action. As an example, device 10 may activate a visible image sensor in device 10 when it has been determined that one or more users is present in the vicinity of device 10. The visible image sensor can then be used to gather additional information from the user (hand gestures, images of the user for a video conference, face detection data indicating where the faces of the users are located, etc.).
At step 64, components 62 may be used to gather information on the operating environment of device 10. Image sensor components may, for example, capture visible or infrared images. To minimize power consumption, it may be desirable to minimize how many of components 62 are active. For example, it may be desirable to use only one or a few infrared sensors to detect whether or not a user is present in the vicinity of device 10, as described in connection with the illustrative sensor angular sensitivity ranges A, B, and C of
If, however, user activity is detected (e.g., if a low power infrared sensor detects that a user is present and/or that a user is moving within the vicinity of device 10), device 10 may take suitable action at step 68. For example, device 10 may wake up additional components 62 (e.g., a camera may be activate to capture images of the user and/or other sleeping components may be awoken), may use components 62 to gather more data (e.g., to gather more detailed images and/or light sensor readings than would otherwise be gathered), may adjust the operation of device 10 (e.g., to turn on audio that would otherwise be muted), or may take any other suitable action.
To prevent window 32 from having an unsightly appearance, it may be desirable to match the color and opacity of window 32 to surrounding structures in display 14. For example, if portion 34IA of outer display layer 34 is black, it may be desirable to provide window 32 with a black appearance. At the same time, window 32 should be transparent at wavelengths of interest for the operation of components 62. For example, if components 62 include infrared detectors operating at one or more wavelength ranges between 5 and 12 microns, window 32 should be transparent for those wavelength ranges (e.g., 5-12 microns, 8-12 microns, an appropriate subset of wavelengths between 5 and 12 microns, etc.).
An illustrative configuration for display 14 in which window 32 is transparent at infrared wavelengths and has an appearance that helps window 32 visually blend with portion 34IA of outer display layer 34 is shown in
Window 32 may include one or more window members in an opening in layer 34. For example, window 32 may include window member 76 and layer 78. Window member 76 may be formed from a material that is transparent at wavelengths of interest for operating components 62. If, for example, components 62 include an infrared detector operating at a range of wavelengths between 5-12 microns or 8-12 microns, window member 76 may be formed from a material that is transparent (e.g., transmitting 50% or more, 80% or more, or other suitable elevated amount) at 5-12 microns, 8-12 microns, or an appropriate subset of these ranges. Examples of material for window structures such as window member 76 include magnesium fluoride, calcium fluoride, zinc selenide, zinc sulfide, or infrared transparent plastic material with a spectral transmittance that is tuned to pass long infrared wavelengths (e.g., 8-14 microns, 5-14 microns, etc.). If desired, the external surface of member 76 may be coated with an antireflection layer or other coating to help ensure that the reflectivity and sheen of member 76 matches that of substrate 70. Window member 76 may have the shape of a circular disk or other suitable window shape.
Member 76 may be relatively hard, so it may be desirable to interpose a layer of polymer or other bonding gap fill material (e.g., adhesive or other relatively soft material) between member 76 and substrate 70, as illustrated by fill material 74. The index of refraction of material 74 may be configured to enhance reflectivity for light 60 along the walls of member 76. The index of refraction of material 74 may, for example, be lower than the index of refraction of member 76 to confine light 60 within member 76 through the principal of total internal reflection or the index of refraction of material 74 may be greater than the index of refraction of member 76 to create a reflective interface that helps reflect light 60 along the interior of member 76. A metal trim or other structure may also be inserted between member 76 and substrate 70 to help confine light 60.
Window 32 may be provided with a darkened appearance using a layer of material that transmits infrared light at 5-14 microns, 8-14 microns, or other suitable infrared wavelengths while blocking visible light (e.g., while transmitting less than 40%, less than 20%, less than 5%, or less than other suitable low amounts of visible light). Layer 78 may be, for example, a structure such as a disk-shaped member formed from black chalcogenide glass. Black chalcogenide glass may have long-wave infrared spectral transmittance properties that allow layer 78 to be transparent at infrared wavelengths of interest for operation of an infrared detector in components 62 (e.g., sensor 44 and/or detectors 46 operating at one or more wavelengths between 5 and 14 microns, 8 to 14 microns, or other suitable wavelength) while simultaneously exhibiting a black appearance or other dark appearance at visible wavelengths that matches the black color of layer 72. Other types of material that pass infrared light while blocking visible light may be used to form structure 78 if desired (e.g., polymers, thin-film interference filters, etc.). The use of chalcogenide glass is merely illustrative.
Transparent layer 70 may absorb more light at infrared wavelengths (e.g., 8-14 microns, etc.) than desired. Light transmittance at infrared wavelengths can be enhanced by locally thinning layer 70 in window 32. This type of arrangement is shown in
Another illustrative approach for forming infrared transparent windows that block visible light so that the windows appear similar to portion 34IA of display layer 34 is shown in
In some configurations, it may be desirable to incorporate both visible light and infrared light components in window 32. As an example, it may be desirable to incorporate infrared sensors to detect the presence of users. These sensors may be operated at a low power and may be constantly on or nearly constantly on. The components aligned with window 32 may also include a visible light component such as a digital image sensor in a camera. The camera may be used to conduct video calls, to gather hand gestures, to detect faces (e.g., as part of a face recognition algorithm to provide targeted services to particular uses and/or to determine when a user is actively viewing display 14, as part of a biometric identification scheme, etc.), to take pictures, to support operations in a game, etc. Operation of the video camera may consume more power than operation of the low-power infrared sensors, so power may be conserved by maintaining the camera in a low power sleep state until user activity is detected with the infrared sensor(s).
A window that is used to allow operation of a visible light camera may contain transparent structures and may therefore be difficult to completely hide from view from a user. To avoid creating more visible disruption than necessary, infrared sensor structures may be located under the same window as the visible light camera. As an example, one or more infrared light sensors may mounted adjacent to the visible light camera. The infrared light sensors may be arranged in a ring around the lens of the camera or may be mounted in the vicinity of the camera using other patterns. Window 32 may have a ring-shaped outer portion (or a portion of other shapes) that is transparent at infrared wavelengths to allow light to reach the infrared sensor(s) while being opaque a visible light wavelengths to help block the infrared sensors from view. The visible light camera may be mounted in alignment with a clear central disk of window material in window 32 (i.e., a glass window portion in the center of the ring-shaped outer portion).
A cross-sectional side view of an illustrative window arrangement that supports both infrared light components and visible light components is shown in
Circular window portion 32-2 may be formed from a material such as clear glass, transparent plastic, sapphire, or other material that is transparent at visible wavelengths. This allows visible wavelengths of light 60 to pass through window portion 32-2, through visible lens 38, and onto digital image sensor 40. Sensor 40 may have sufficient resolution (e.g., 1-20 megapixels or more or other suitable resolution) to form digital images of users in the vicinity of device 10. If desired, sensor 40 and lens 38 may be packaged in a package such as package 100 (e.g., to form a camera module). There is one sensor 40 in the illustrative configuration of
Infrared sensors 44 (e.g., an array of three infrared sensors or other suitable number of sensors that are sensitive at wavelengths of 5-14 microns 8-14 microns, or other suitable wavelengths) may be mounted in alignment with portion 32-1 of window. If, for example, window 32-1 has a circular ring shape, sensors 44 may be distributed evenly around the ring. Window layer 102 may be an infrared transparent ring member, a coating, or other material that is transparent to infrared light while blocking visible light. Window layer 102 may be a chalcogenide glass, a polymer coating, or other material that transmits infrared wavelengths of light 60 at a wavelength between 5-14 microns or other suitable wavelength range so that this light can be sensed by infrared sensors 44. The window coating 102 may also block visible light to help provide window portion 32-1 with an appearance that matches the opaque (e.g., black, white, etc.) appearance of portion 34IA of outer layer 34 in inactive region IA of display 14. Infrared sensors 44 may be mounted on package 100 (i.e., sensors 44 may be integrated with digital image sensor 40 to form a combined camera and infrared sensor assembly), may be mounted between the peripheral edges of lens 38 and window portion 32-1, or may be mounted elsewhere in the interior of device housing 12 in alignment with window 32.
Different types of sensors may be used for sensors 44-1, 44-2, and 44-3. For example, sensors 44-1 and 44-2 may be thermopile infrared sensors (i.e., sensors that include one or more thermocouple elements for measuring infrared light) and sensor 44-3 may be a pyroelectric sensor (i.e., a sensor that generates output in response to infrared light exposure through the pyroelectric effect). Other type of sensors may be used, if desired (e.g., compound semiconductor detectors, quantum dot detectors, other types of semiconductor detectors, etc.). The use of thermopile and pyroelectric sensors is merely illustrative.
The graphs of
By processing the signals from sensors such as sensors 44-1, 44-2, and 44-3, device 10 can monitor for the presence of users in the vicinity of device 10. The static signal at level T1 for sensors 44-1 and 44-2 can be considered background thermal noise. Changes to this signal can be considered signals of interest and an indication that a user has moved into range. Sensors 44-1 and 44-2 cover different angular ranges (e.g., to cover left and right sides of a room). When signal TP-1 goes high and signal TP-2 remains low, device 10 can conclude that a user is present in the left half of the room. Sensor 44-3 may monitor the entire room in which device 10 is located (e.g., by covering larger field of view 44-3F) and may detect changes of signal level PE due to movement of the user. By combining signals from multiple sensors, user presence detection accuracy can be enhanced and interference from stray light signals and other sources of ambient room noise can be minimized. If desired, additional sensors (e.g., a camera) may be activated and/or additional actions taken when a user's presence is detected (e.g., to provide further confirmation of the presence of a user, to provide user-specific features, to turn on display 14, to adjust audio playback, to identify a detected user, etc.). Sensors 44-1, 44-2, and 44-3 may be operated in a continuously on mode that consumes relatively small amounts of power. When sensors 44-1, 44-2, and 44-3 detect a user, higher power consumption components 62 can be used and additional device functions can be activated.
The foregoing is merely illustrative and various modifications can be made by those skilled in the art without departing from the scope and spirit of the described embodiments. The foregoing embodiments may be implemented individually or in any combination.
| Number | Name | Date | Kind |
|---|---|---|---|
| 7570329 | Huang et al. | Aug 2009 | B2 |
| 8878817 | Noma et al. | Nov 2014 | B2 |
| 8912480 | Pope et al. | Dec 2014 | B2 |
| 8981302 | Yang et al. | Mar 2015 | B2 |
| 20080284716 | Edwards et al. | Nov 2008 | A1 |
| 20120176668 | Saito | Jul 2012 | A1 |
| 20130048837 | Pope | Feb 2013 | A1 |
