Patent Application
20220116520
Portable computing devices, including laptop, notebook, and convertible computers, as well as tablet computing devices and smartphones, commonly have the capability to capture images of users while the users are viewing the displays of the devices. That is, a portable computing device typically has an image-capture device like a camera on the same side of the device as its display. Users are thus able to capture self portraits (i.e., “selfies”) and videos, as well as engage in video calls and conferences.
As noted in the background section, portable computing devices usually have cameras or other image-capture devices on the same sides as their displays. Traditionally, a portable computing device has had such an image-capture device disposed above the device's display, within a bezel surrounding the display. However, to maximize display size relative to device size or to decrease device size without correspondingly decreasing display size, as well as to provide for a more modern design “look,” manufacturers have been steadily shrinking the bezels surrounding the displays of their portable computing devices. In the most extreme cases, manufacturers have shrunk the bezels to effectively eliminate them, in so-called bezel-less devices.
To still accommodate image-capture devices within such portable computing devices, manufacturers have focused on two different strategies. The first is to reserve a “notch” or “punch hole” towards the top of a device's display for an image-capture device. There is no active display in this reserved area. However, the net result is strange looking to at least some consumers, and the portion of the display around the reserved area is more difficult to usably leverage. For example, when such a portable computing device displays a rectangular image, either a portion of the image corresponding to the “notch” or “punch hole” is not displayed, or the image is resized to fit a smaller rectangular portion of the display that does not overlap the “notch” or “punch hole.”
The second strategy is to position the image-capture device behind the display of a portable computing device. The display can thus still be rectangular, and the device can have little or no bezel. For this strategy to be workable, the display is at least partially transparent in the area behind which the image-capture device is located. Going forward, this second strategy is expected to become more popular than the first strategy as display technology improves and transparent displays decrease in cost, because the resulting portable computing devices have more usable displays and present a cleaner “look” than devices having distracting “notches” or “punch holes.”
However, capturing images using image-capture devices positioned behind displays of portable computing devices is more difficult than when using image-capture devices that are not positioned behind displays, particularly when the displays are currently displaying content. While the display itself may be transparent, light from the display that is used to display content or make the content more visible in an ambient environment can be reflected back into the image-capture device when capturing images. Therefore, captured images of the user of such a portable computing device can appear washed out, or can include artifacts resulting from the content that the device's display was showing when the images were captured.
To avoid this problem, techniques described herein interleave image frames of content with black frames for display on a display device when an image-capture device positioned behind the display device is being used to capture images. The image-capture device captures images just when the black frames are being displayed on the display device, and not when the image frames are being displayed. Therefore, there is no light from the display device that can be reflected back into the image-capture device, improving the quality of the captured images. The captured images can be still photos or images, as well as video (i.e., moving images) that include a successive number of such captured images.
The display device displays a current frame on the rows of the device on a frame line-by-frame line basis. Thus, when the current frame is to be displayed, the first line of the frame is displayed on the first row, then the second line is displayed on the second row, and so on, until the last line of the frame is displayed on the last row of the display. This process is repeated with the next frame, the frame after that, and so on. Therefore, if the sequence of frames to be displayed includes black frames interleaved with image frames, when a black frame is to be displayed following an image frame, the lines of the image frame are replaced with black frame lines on a line-by-line basis starting with the first row of the display, until the image frame has been completely replaced by the black frame lines.
The image-capture device is positioned behind specified rows of the display device. Therefore, the techniques described herein synchronize image capture by the image-capture device with the display of the black frames so that image capture occurs after black frame lines have been displayed on the display device's rows behind which the image-capture device is positioned. This ensures that no image frame lines are still being displayed when image capture occurs. Stated another way, image capture is synchronized with the display of black frames so that image capture occurs a specified delay after the display device has started displaying each black frame.
The display system 100 includes a display device 102, which can also be referred to as a display. The display device 102 can be a flat-panel display device. Examples of such flat-panel display devices include light-emitting diode (LED) display devices, as well as organic LED (OLED) display devices, micro LED display devices, quantum dot display devices like quantum LED (QLED) display devices, and so on.
The display system 100 includes an image-capture device 104 positioned behind a portion of the display device 102. The display device 102 is at least partially transparent in at least the portion behind which the image-capture device 104 is disposed, and may be completely transparent, to permit the image-capture device 100 to capture images from in front of the display device 102. The portion of the display device 102 behind which the image-capture device 104 is positioned is thus an active portion that actually displays content, and is not a bezel, “notch,” “punch hole,” etc., that does not display content. The image-capture device 104 can capture still images as well as moving images (i.e., video). The image-capture device 104 may be referred to as a camera, and may be implemented as a semiconductor image sensor, such as a charge-coupled device (CCD) or complementary metal oxide semiconductor (CMOS) image sensor.
In the example of
The display system 100 displays images on the display device 102 at a refresh rate, which is the number of times per second the display system 100 can redraw the image that is displayed on the display device 102. The refresh rate is measured as Hertz (Hz). The display device 102 may have a refresh rate of 60, 90, 120, 180, or 240, for example. This means that the display device 102 can display up to 60, 90, 120, 180, or 240 different image frames per second.
When the image-capture device 104 is not being used to capture images from in front of the display device 102, the display device 102 displays (just) image frames of content that the display system 100 generates or is provided. By comparison, when the image-capture device 104 is being used to capture images while the display device 102 is displaying content, black frames are interleaved with these image frames, with the image-capture device 104 specifically capturing images when the black frames are being displayed. That is, when the image-capture device 104 is being used to capture images, the image-capture device 104 specifically does not capture images when the image frames are being displayed.
A black frame is a frame in which the display device 102 does not display any content. For example, a display device 102 typically includes a number of pixel elements, which may each be made up of red, green, and blue sub-pixel elements. When the display device 102 displays content (i.e., an image frame), the sub-pixel elements of each pixel elements are controlled according to a corresponding image frame pixel, so that the image frame is displayed on the display device 102. For an n-bit display device 102—such as an 8-bit or a 24-bit display device 102—each red, green, and blue sub-pixel element of a pixel element can have a value between zero and 2n−1, so that the corresponding intensities of the red, green, and blue colors of the pixel element realize the full color of the corresponding image frame pixel.
In such an implementation, when the display device 102 displays a black frame, none of the pixel elements of the display device 102 are turned on. This is because the black frame corresponds to when no content is being displayed. The sub-pixel element of each pixel element of the display is turned off (i.e., has a value of zero). If the display device 102 is of a type that has backlighting, such as panel-wide liquid crystal display (LCD) backlighting, full-array LED backlighting, edge LED backlighting, cold-cathode fluorescent lamp (CCFL) backlighting, and/or local-zone dimmable backlighting, the backlighting may be turned off or set to its minimal value.
In
In
How black frames are interleaved with image frames when the image-capture device 104 is to capture images can depend on the refresh rate of the display device 102, the desired effective maximum frame rate at which the display device 102 is to display image frames while the image-capture device 104 is capturing images, and the desired capture rate at which the image-capture device 104 is to capture images. As one example, if the display device 102 has a relatively high refresh rate, such as 120 Hz or more, then interleaving image and black frames on a 1:1 basis may permit image capture at a high capture rate while still displaying image frames at a high effective frame rate. As a second example, if the display device 102 has a relatively low refresh rate, such as 60 Hz, then image frame and black frames may be interleaved on a 2:1 or even 3:1 basis to ensure that image frames are still displayed relatively fluidly such that most users will not be able to perceive the decrease in frame rate, although the capture rate may be slightly less fluid and thus perceivable.
A refresh rate of n Hz means that the display device 102 takes 1/n seconds to completely display the frame. Therefore, there is a 1/n second delay between when the display device 102 starts displaying the first frame line on the first row 302A until the device 102 finishes displaying the last frame line on the last row 302N. If there are N rows 302, a frame line of a current frame is displayed every 1/Nn seconds on a corresponding row 302, until all the frame lines of the current frame have been displayed.
The image-capture device 104 is positioned behind a number of the rows 302 of the display device 102. For example, the image-capture device 104 may be positioned behind ten rows 302, such as the third through thirteenth rows 302 of the display device 102. In this respect, Applicant notes that the
The display system 100, in addition to the display device 102 and the image-capture device 104, includes gates 304A, 304B, . . . , 304N, which are collectively referred to as the gates 304. Each gate 304 corresponds to a different row 302 of the display device 102. Selecting (i.e., selectively activating) a given gate 304 causes a current frame line to be loaded into (i.e., displayed by) the row 302 to which the given gate 304 corresponds. Therefore, the gates 304 are successively individually activated to display the frame lines of the current frame on the rows 302 of the display device 102.
The display system 100 also includes a display controller 306. The display controller 306 is implemented in hardware, and thus can be an application-specific integrated circuit (ASIC), for instance. The display controller 306 may include a hardware processor that executes microcode or other program code. However, that the display controller 306 is said to execute program code includes implementation in which the display controller 306 is an ASIC or other integrated circuit designed to perform the functionality described herein.
The display controller 306 in the example of
For example, the source driver 308 loads the first frame line of the current frame, and then gate driver 310 activate the gate 304A to cause the display device 102 to display this first frame line in the row 302A by transferring or loading the first frame line from the source driver 308 to the row 302A. The source driver 308 then loads the second frame line of the current frame, with the gate driver 310 activating the gate 304B to transfer the second frame line from the source driver 308 to the row 302B. This process continues on a frame line-by-frame line basis to display the frame lines of the current frame on the rows 302 of the display device 102. Display of the current frame on the display device 102 finishes with the source driver 308 loading the last frame line of the current frame and the gate driver 310 activating the gate 304N to transfer the last frame line from the source driver 308 to the row 302N.
The timing controller 312 outputs a sequence of clock pulses that controls loading of frame lines by the source driver 308 and activation of the gates 304 by the gate driver 310 to successively display the frame lines on the rows 302 of the display device 102. At each clock pulse, the source driver 308 loads a frame line of the current frame, and the gate driver 310 activates a gate 304 to transfer the loaded frame line into the row 302 corresponding to the gate 304. Therefore, if the display device 102 has a refresh rate of n Hz and has N rows 302, the timing controller 312 outputs clock pulses every 1/nN seconds, so that a gate 304 is activated every 1/nN seconds to display an image frame line on a row 302 every 1/nN seconds.
In
Within the time period 400 is a time period 404 during which black frame lines are displayed on the rows 302 behind which the image-capture device 104 is disposed. The time period 404 has a start time 408 and an end time 410. The start time 408 corresponds to the clock pulse 402 at which a black frame line is displayed on the first row 302 behind which the image-capture device 104 is disposed, and the end time 410 corresponds to the clock pulse 402 at which a black frame line is displayed on the last row 302 behind which the image-capture device 104 is disposed.
For example, if the image-capture device 104 is disposed behind the third through thirteenth rows 302 of the display device 102, the start time 408 of the time period 404 corresponds to the third clock pulse 402 at which a black frame line is displayed on the third row 302. The end time 410 of the time period 404 in this example corresponds to the thirteenth clock pulse 402 at which a black frame line is displayed on the thirteenth row 302. It is noted that
The display controller 306 does not cause the image-capture device 104 to begin capturing an image until after the clock pulse 402 corresponding to the end time 410 of the time period 404. That is, the display controller 306 does not cause the image-capture device 104 to begin capturing an image until black frame lines have been displayed on all the rows 302 behind which the image-capture device 104 is positioned. Stated another way, the display controller 306 displays the capture of an image by the image-capturing device 104 until black frame lines have been displayed on all the rows 302 behind which the image-capture device 104 is positioned. Therefore, even when a black frame is being displayed on the display device 102—in that the display device 102 has begun to display a black frame—the image-capture device 104 does not begin image capture until black frame lines have been displayed on the rows 302 behind which the device 104 is disposed.
In
The start time 412 of the time period 406 during which image capture occurs is after the end time 410 of the time period 404 during which black frame lines are displayed on the rows 302 behind which the image-capture device 104 is disposed. In the specific example of
In general, then, the start time 412 of the time period 406 during which image capture occurs can be configured or specified to coincide with a given clock pulse 402 after the clock pulse 402 corresponding to the end time 410 of the time period 404. That is, the start time 412 can begin at the first clock pulse 402 after the clock pulse 402 corresponding to the end time 410 of the time period 404, or at a specified number of clock pulses 402 after the clock pulse 402 corresponding to the end time 410. It can thus be said that the image capture by the image-capture device 104 is synchronized with the display of black frames so that image capture begins a specified delay after the display device 102 has started displaying each black frame. This specified delay is the length of time between the start of the first clock pulse 402A and the start time 412, which is configurable so long the start time 412 is after the end time 410. The specified delay thus corresponds to the display device 102 having already displayed black frame lines on at least the rows of the display device 102 behind which the image-capture device 104 is positioned.
The end time 414 of the time period 406 during which image capture occurs is likewise configurable, and may depend on how long it takes for the image-capture device 104 to fully capture an image once the device 104 has begun capturing the image at the start time 412. The end time 414 may be configured or specified to correspond to any clock pulse 402 through the last clock pulse 402N of the time period 400 in which the black frame is displayed. Therefore, at the start time 412 of the time period 406, the hardware display controller 306 may turn on or otherwise cause the image-capture device 104 to begin image capture, and at the end time 414 the display controller 306 may turn off or otherwise cause the image-capture device 104 to stop image capture.
At each clock pulse 402, then, the hardware display controller 306 displays a corresponding line of a current frame on the row 302 of the display device 102 corresponding to the clock pulse 402. When the current frame is a black frame, the display controller 306 causes the image-capture device 104 to begin image capture at the clock pulse 402 corresponding to a first selected row 302 of the display device 102. This first selected row can be the row 302 of the display device 102 immediately following the last row 302 behind which the image-capture device 104 is positioned, or one or more rows 302 after the last row 302 behind which the image-capture device 104 is positioned.
The display controller 306, again when the current frame is a black frame, causes the image-capture device 104 to end image capture at the clock pulse 402 corresponding to a second selected row 302 of the display device 102. This second selected row 302 is after the first selected row 302, since image capture is stopped after it has been started. The selected row 302 is further before the last row 302 of the display.
The techniques that have been described herein provide for image capture by an image-capture device 104 positioned behind a display device 102 at an active display area of the device 102. This is achieved by interleaving black frames with image frames of the content that the display device 102 is displaying, and causing the image-capture device 104 to capture images just when the display device 102 is displaying black frames. More specifically, when the display device 102 has begun displaying the frames lines of a black frame, the image-capture device 104 does not being capturing an image until the display device 102 has displayed the black frame lines corresponding to the rows 302 of display device 102 behind which the image-capture device 104 is positioned.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2019/020777 | 3/5/2019 | WO | 00 |
