Patent Application
20230143443
This application relates generally to the creation of high-dynamic-range (HDR) videos and more specifically to creation of HDR videos with a high number of frames per second (e.g., 32 frames per second or higher) by fusing computer-generated image frames with captured image frames to achieve the high number of frames per second.
Conventional approaches for generating a high-dynamic-range (HDR) video use multiple captured image frames to generate a single HDR frame that becomes a part of an HDR video. Use of multiple captured image frames result in HDR videos with frame rates that are lower than the maximum number of frames per second that an image sensor can capture (e.g., the HDR video can end up with 16 frames per second while the image sensor can be capable of capturing 32 frames per second). Stated another way, conventional approaches for generating HDR videos can result in frame rates that are half the frame rates for non-HDR videos. The reduced frame rates result in lower-quality HDR videos. Additionally, in some embodiments, the maximum frame rate achievable by the conventional approaches is limited by the need to have at least two captured image frames available before each HDR output frame is created, e.g., because processing of HDR output frames must occur after at least two captured image frames are available which can hinder ability of a camera system to generate a higher number frames per second since the system must always wait for capturing of another image frame to complete.
As such, there is a need for systems and methods that can create HDR videos with higher frame rates.
The disclosed methods and systems make use of computer-generated predicted image frames to allow for creation of a high-dynamic-range (HDR) video having a high number of frames per second (fps) than is currently achievable. In particular, rather than requiring a camera or imaging system to wait for at least two image frames to be captured before creating an HDR output frame, the techniques described herein provide for quick generation of computer-generated predicted image frames, which then allows for quicker creation of HDR output frames since the system need not constantly wait for capturing of pairs of captured image frames to complete before and HDR output frame can be created. This helps make efficient use of processing resources and avoids limiting processing downtime of some existing approaches for HDR video processing. This is explained in more detail with reference to the processes/techniques described with reference to
(A1) In accordance with some embodiments, a method of using computer-generated predicted image frames to create a high-dynamic-range video having a high number of frames per second is provided. The method includes receiving, at one or more processors that are in communication with an image sensor configured to capture image frames used to produce an HDR video, a first captured image frame and a second captured image frame captured via the image sensor. The first captured image frame represents a scene in the real-world at a first point in time and the second captured image frame represents the scene in the real-world at a second point in time that is after the first point in time (the first and second points in time can represent respective points in time at which exposures used to capture with the first and second captured image frames end). The method further includes, in accordance with a determination that the first captured image frame and the second captured image frame will be used to produce an HDR video, generating, via the one or more processors and based on the first captured image frame, a computer-generated predicted image frame representing the scene in the real-world at a time between the first point in time and the second point in time. The method also includes fusing (or merging) the second captured image frame with the computer-generated predicted image frame to generate an HDR frame (also referred to herein as an HDR output frame, as a fused HDR frame, or simply as a fused frame, all of which interchangeably refer to the frame produced after fusing or merging the second image frame with the computer-generated predicted image frame) for the HDR video.
(A2) In some embodiments of A1, the method further includes repeating the receiving, generating, and fusing operations to produce respective HDR frames for the HDR video, such that the HDR video has at least 32 frames per second. Stated another way, additional captured image frames are received at the receiving operation, additional computer-generated predicted image frames are then generated for respective points in time that are between respective points in time for two captured image frames, and then additional HDR output frames are created through fusing operations between capturing and computer-generated image frames until at least 32 HDR output frames are produced per second. More specifically, at the receiving operation, a third captured image frame (representing the scene in the real-world at a third time that is after the first and second points in time) can be received from the image sensor; at the generating operation, a second computer-generated predicted image frame representing the scene in the real-world at a time between the second point in time and the third point in time is created; and, at the fusing operation, the third captured image frame can be fused/ merged with the second computer-generated predicted image frame to generate a second HDR frame for the HDR video. And this repeats for fourth, fifth, sixth, etc. captured image frames at the receiving operation, repeats for third, fourth, fifth, etc. computer-generated image frames at the generating operation, and repeats for fusing each respective pair of captured image frame with a computer-generated predicted image frame such that an HDR video with a high number of frames per second is produced.
(A3) In some embodiments of A1-A2, the HDR video includes: (i) a first HDR frame that was created by fusing two captured image frames, and (ii) a second HDR frame that was created by fusing two computer-generated predicted image frames. In other words, in addition to fusing captured image frames with computer-generated predicted image frames, fusing can also occur for two captured image frames and/or for two computer-generated predicted image frames.
(A4) In some embodiments of A1-A3, the method further includes after producing the HDR video, receiving captured image frames captured via the image sensor and producing a non-HDR video without generating or using any computer-generated predicted image frames. The HDR video has a first number of frames that is greater than or equal to a second number of frames for the non-HDR video. In other words, the image sensor (and a device associated therewith) can be configured to capture both HDR and non-HDR videos, but only the HDR video processing generates and makes use of computer-generated predicted image frames.
(A5) In some embodiments of A4, the first number of frames per second and the second number of frames per second is 32 frames.
(A6) In some embodiments of A1-A5, the computer-generated predicted image frame is generated while the second captured image frame is being captured by the image sensor. As compared to some techniques for producing HDR videos, being able to generate the computer-generated predicted image frame while the second captured image frame is being captured by the image sensor allows for producing more HDR output frames per second, as the system need not wait for capturing of pairs of captured image frames to be available before producing the next HDR output frame. Instead, the system, by applying the techniques described herein is able to quickly generate the computer-generated predicted image frame and fuse that predicted image frame with the second captured image frame, and can therefore repurpose processing time previously lost while waiting for the second captured image frame to be captured along with a next third captured image frame that would have been fused with the second captured image frame to produce an HDR output frame. Stated simply, processing time lost during capturing of the second and third captured image frames can be repurposed to generate a computer-generated predicted image frame for fusing with the second captured image frame, thereby producing more HDR output frames per second by avoiding the processing downtime of some existing approaches for HDR video processing.
(A7) In some embodiments of A1-A6, the one or more processors that are in communication with the image sensor receive a third captured image frame representing the scene in the real-world at a third point in time that is after the second point in time, and the third captured image frame is captured, in part, while the HDR frame is being generated. Again, in some embodiments, this helps to ensure that more HDR output frames can be produced per second as capturing of image frames can continue while HDR output frames are being created.
(A8) In some embodiments of A1-A7, the computer-generated predicted image frame is a first computer-generated predicted image frame and the HDR frame is a first HDR frame, and the method further includes receiving, at the one or more processors that are in communication with the image sensor, a third captured image frame captured via the image sensor. The third captured image frame represents the scene in the real-world at a third point in time that is after the second point in time (as with the first and second points in time, the third point in time can represent a point in time at which an exposure used to capture with the third captured image frame ends). In accordance with a determination that the third captured image frame will be used in conjunction with the first captured image frame and the second captured image frame to produce the HDR video, the method includes generating, via the one or more processors and based on the second captured image frame, a second computer-generated predicted image frame representing the scene in the real-world at the time between the second point in time and the third point in time and fusing the third image frame with the second computer-generated predicted image frame to generate a second HDR frame for the HDR video. This is another more specific example of the looping behavior described above with reference to A2, in which various operations are repeated to allow for creation of all the HDR frames for the HDR video.
(A9) In some embodiments of A1, the method further includes receiving, at the one or more processors that are in communication with the image sensor, a third captured image frame captured via the image sensor, the third captured image frame representing the scene in the real-world at a third point in time that is after the second point in time (as with the first and second points in time, the third point in time can represent a point in time at which an exposure used to capture with the third captured image frame ends). In accordance with a determination that the third captured image frame will be used in conjunction with the first captured image frame and the second captured image frame to produce the HDR video, the method includes generating, via the one or more processors and based on the first captured image frame and the third captured image frame, the computer-generated predicted image frame representing the scene in the real-world at the time between the first point in time and the second point in time and fusing the computer-generated predicted image frame with the second image frame and the computer-generated predicted image frame to generate the HDR frame for the HDR video.
(A10) In some embodiments of A9, the computer-generated predicted image frame is a first computer-generated predicted image frame and the HDR frame is a first HDR frame, and the method further includes receiving, at the one or more processors that are in communication with the image sensor, a fourth captured image frame captured via the image sensor. The fourth captured image frame represents the scene in the real-world at a fourth point in time that is after the third point in time. In accordance with a determination that the fourth captured image frame will be used in conjunction with the first captured image frame, the second captured image frame, and the third captured image frame to produce the HDR video, the method includes generating, via the one or more processors and based on the second captured image frame and the fourth captured image frame, a second computer-generated predicted image frame representing the scene in the real-world at the time between the third point in time and the second point in time and fusing the third image frame with the second computer-generated predicted image frame to generate a second HDR frame for the HDR video.
(A11) In some embodiments of A1-A10, the first captured image frame is a first type of image frame, the second captured image frame is a second type of image frame, and the first type of image frame is distinct from the second type of image frame.
(A12) In some embodiments of A11, the first type of image frame has a short exposure duration, and the second type of image frame has a long exposure duration that is greater than the short exposure duration.
(A13) In some embodiments of A1-A12, the computer-generated predicted image frame is generated via a machine-learning system that has been trained using a training set consisting of a variety of image frames captured by an image sensor viewing different scenes in the real world.
(A14) In some embodiments of A1-A13, the HDR video has a number of frames per second (fps) that is at least equal to a maximum fps achievable by the one or more processors when using captured image frames to produce a video.
(A15) In some embodiments of A14, the HDR video has a fps greater than a maximum fps achievable by the one or more processors when using captured image frames to produce a video.
(A16) In some embodiments of A1-A15, the image sensor is part of a security camera, smartphone, smart watch, tablet, or AR glasses. As non-limiting examples, the image sensor can be a component of a imaging system 204 for a security camera (e.g., example depicted in
(B1) In another aspect, a system for generating an HDR video is provided. The system includes an image sensor configured to capture image frames used to produce a high-dynamic range (HDR) video and one or more processors that are in communication with the image sensor. The one or more processors are configured to receive a first captured image frame and a second captured image frame captured via the image sensor. The first captured image frame represents a scene in the real-world at a first point in time and the second captured image frame represents the scene in the real-world at a second point in time that is after the first point in time. The one or more processors, in accordance with a determination that the first captured image frame and the second captured image frame will be used to produce an HDR video, generate, via the one or more processors and based on the first captured image frame, a computer-generated predicted image frame representing the scene in the real-world at a time between the first point in time and the second point in time and fuse the second image frame with the computer-generated predicted image frame to generate an HDR frame for the HDR video. Non-limiting examples of the system include any one of a security camera, smartphone, smart watch, tablet, or AR glasses.
(B2) In some embodiments of B1, the one or more processors are further configured to perform the method of any of claims A2-A15.
(C1) In one other aspect, a non-transitory computer-readable storage medium including instructions for generating an HDR video is provided. The instructions for generating an HDR video, when executed by a device (e.g., non-limiting examples of the device include any one of a security camera, smartphone, smart watch, tablet, or AR glasses) that includes an image sensor, cause the device to perform the method of any of claims A2-A16.
(D1) In yet a further aspect, a device including an image sensor is provided. Non-limiting examples of the device include any one of a security camera, smartphone, smart watch, tablet, or AR glasses. The device including the image sensor is configured to perform the method of any of claims A2-A15.
(E1) In another aspect, means for performing the method any of A1-A15 are provided. The means can include software algorithms (e.g., algorithms implementing the flowcharts that are described below) performed on general-purpose hardware or application-specific integrated circuits (or a combination of both) configured to perform the algorithms described herein (e.g., algorithms implementing the flowcharts that are described below). The general-purpose hardware and/or application-specific integrated circuits can be integrated with a device, such as a security camera, smartphone, smart watch, tablet, or AR glasses.
Note that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described in the specification are not all inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Moreover, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes and may not necessarily have been selected to delineate or circumscribe the subject matter described herein.
So that the present disclosure can be understood in greater detail, a more particular description may be had by reference to the features of various embodiments, some of which are illustrated in the appended drawings. The appended drawings illustrate pertinent example features of the present disclosure and are therefore not to be considered limiting, for the description may admit to other effective features as the person of skill in this art will appreciate upon reading this disclosure.
In accordance with common practice, the various features illustrated in the drawings may not be drawn to scale. Accordingly, the dimensions of the various features may be arbitrarily expanded or reduced for clarity. In addition, some of the drawings may not depict all of the components of a given system, method, or device. Finally, like reference numerals may be used to denote like features throughout the specification and figures.
Numerous details are described herein in order to provide a thorough understanding of the example embodiments illustrated in the accompanying drawings. However, some embodiments may be practiced without many of the specific details, and the scope of the claims is only limited by those features and aspects specifically recited in the claims. Furthermore, well-known processes, components, and materials have not been described in exhaustive detail so as to avoid obscuring pertinent aspects of the embodiments described herein.
As will become apparent to a person of skill in this art upon reading this disclosure, the various embodiments provide systems and methods of using computer-generated predicted image frames to enable the generation of a high-dynamic-range (HDR) video having a high number of frames per second (fps).
An HDR video can be created after combining multiple image frames with different exposures taken close together in time to result in a dynamic range higher than those of individually-captured image frames. For example, referring again to
In the approach shown in
The adjacent pairs of captured image frames are used by the approach of
Unlike the approach of
In some embodiments, the communications interface 215 is configured to communicatively couple the imaging system 204 to one or more computing devices, such as a phone, a tablet, a computer, a server, a wrist-wearable device 750, a head-worn wearable device, etc. The communication interface 215 is used to establish wired or wireless connections between the imaging system 204 and the one or more computing devices. In some embodiments, the communication interface 215 includes hardware capable of data communications using any of a variety of custom or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, or MiWi), custom or standard wired protocols (e.g., Ethernet or HomePlug), and/or any other suitable communication protocol. Additional information on wired and wireless communication is provided below in reference to
In some embodiments, the display 220 of imaging system 204 is configured to present information to a user, such as one or more user interfaces, images, and video. In some embodiments, the display 220 is a touch display configured to receive one or more inputs from the user. Additional information on the display 220 is provided below in reference to
In some embodiments, the one or more image sensors 225 are components of an ultra-wide camera (e.g., imaging system 204), wide camera, telephoto camera, depth-sensing cameras, or other types of cameras. For example, the image sensors 225 can include a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). In some embodiments, the one or more image sensors 225 (in conjunction with a lens and aperture of the camera) are used to capture still image data for photographs and captured image data for use in creating videos via the imaging system 204, which captured image data can then be processed by one or more image signal processors (e.g., one of the processor(s) 250 can be an image signal processor) of the imaging system 204 to generate images and/or video that is then presented to a user for viewing. As explained in more detail below, inference or interpolation HDR processing of the captured image data ensures that a created HDR video has a number of frames per second (also reference simply herein as “fps”) equal to at least the original fps achieved while the captured image data was captured (such that the quality of an HDR video is not degraded during processing). The imaging system 204 can include multiple different modes for capturing image data or and multiple different modes for video data. For example, these modes can include an HDR image capture mode, a low light image capture mode, burst image capture mode, a panoramic capture mode, and other modes.
In some embodiments, a particular mode is automatically selected based on environmental conditions (e.g., lighting, movement of the device, etc.) when image data is being captured. For example, an imaging system 204 with available modes for HDR image capture mode and a low light image capture mode can automatically select the appropriate mode based on the environmental conditions (e.g., dark lighting may result in the use of low light image capture mode instead of HDR image capture mode). In some embodiments, a user selects the mode that they desire to use (e.g., the user selects a toggle to cause the imaging system 204 to produce HDR videos, even if environmental conditions might have suggested use of a different mode). The image data and/or video data captured by the one or more image sensors 225 is stored in memory 260 (which can include volatile and non-volatile memory such that the image data and/or video data can be temporarily or permanently stored, as needed depending on the circumstances).
In some embodiments, the one or more applications 235 include social-media applications, banking applications, messaging applications, web browsers, imaging applications, etc., which applications can be used to permit users to share captured images and processed HDR videos with other users using the one or more applications 235. User interfaces associated with the one or more applications can be displayed on imaging system 204.
In some embodiments, the machine-learning image processing hardware and software 245 is configured to apply one or more algorithms to at least a subset of the one or more images frames 270 captured or obtained by the imaging system 204. As its name implies, the machine-learning image processing hardware and software 245 can include a combination of hardware and software that is used in conjunction with applying the machine-learning image processing techniques described herein (e.g., computer generating predicting image frames using the inference or interpolation processes described in more detail below). For example, the machine-learning image processing hardware and software 245 can apply one or more algorithms (including making use of the inference and interpolation techniques described herein to create computer-generated predicted image frames that are then fused with other image frames (predicted or captured) to create fused HDR output frames, as described below in reference to
The one or more processors 250 can include and/or be implemented as any kind of computing device, such as an integrated SoC, a CPU, microcontroller, a field programmable gate array (FPGA), a microprocessor, a graphics processing unit (GPU), image signal processor (ISP), digital signal processor (DSP), neural processing unit (NPU), and/or other application specific integrated circuits (ASICs). The one or more processors 250 may operate in conjunction with memory 260, e.g., to access instruction sets to assist in the performance of the techniques and processes described herein. The memory 260 may be or include random-access memory (RAM), read-only memory (ROM), dynamic random-access memory (DRAM), static random-access memory (SRAM) and magneto resistive random-access memory (MRAM), and may include firmware, such as static data or fixed instructions, basic input/output system (BIOS), system functions, configuration data, and other routines used during the operation of the imaging system 204 and the one or more processors 250. The memory 260 also provides a storage area for data and instructions associated with applications and data handled by the one or more processors 250.
In some embodiments, the memory 260 stores at least an image database 262 (including the example captured image frames 270 (
In
The imaging system 204 is configured to create the HDR video 280 based on the captured image frames 270 and computer-generated predicted image frames (e.g., predicted frames 310,
HDR output frames that are used in the HDR video 280. The HDR video 280 created using the machine-learning image processing hardware and software 245 is configured to have an fps equal to or greater than the predetermined fps of the image frames 270. The HDR video 280 created using the machine-learning image processing hardware and software 245 can have at least twice the fps as compared to fps for an HDR video created using the approach shown and described above in reference to
Moving on from the high-level illustration of
The first, second, and third captured image frames 302-1, 302-2 and 302-3 are representations of a scene in the real-world at a respective point in time. For example, the first captured image frame 302-1 represents a scene in the real-world at a first point in time, the second captured image frame 302-2 represents a scene in the real-world at a second point in time, and the third captured image frame 302-3 represents a scene in the real-world at a third point in time. In some embodiments, the first point in time is followed by the second point in time and the second point in time is followed by the third point in time. As described above in reference to
The example first, second, and third computer-generated predicted image frames 304-1, 304-2, and 304-3 (
The first, second, and third computer-generated predicted image frames 304-1, 304-2, and 304-3 represent predictions as to how a scene in the real-world should look at respective points in time, as determined by the appropriate machine-learning model being utilized. In some embodiments, the first computer-generated predicted image frame 304-1 represents a scene in the real-world at a point in time between respective points in time associated with the first and second captured image frames 302-1 and 302-2, the second computer-generated predicted image frame 304-2 represents a scene in the real-world at a point in time between respective points in time associated with the second and third captured image frames 302-2 and 302-3, and the third computer-generated predicted image frame 304-3 represents a scene in the real-world at a point in time between respective points in time associated with the third captured image frame 302-3 and a fourth captured image frame (not shown). Each computer-generated predicted image frame 304 is an estimation or approximation of an image frame before the subsequent image frame. For example, the first computer-generated predicted image frame 304-1 is an estimated or approximated frame before the second captured image frame 302-2, the second computer-generated predicted image frame 304-2 is an estimated or approximated frame before the third captured image frame 302-3, and so forth. In some embodiments, the computer-generated predicted image frames represent the scene in the real-world at a next point in time associated with a next captured image frame rather than representing the scene at intermediary points in time (e.g., instead of computer-generated predicted image frame 304-2 representing a scene in the real-world at a point in time between respective points in time associated with the second and third captured image frames 302-2 and 302-3, it can instead represent the scene at the third point in time associated with the third captured image frame 302-3). Some embodiments can make use computer-generated predicted image frames associated with intermediary points in time in addition to computer-generated predicted image frames associated with a next point in time associated with a next captured image frame.
In some embodiments, a computer-generated predicted image frame 304 is generated while a subsequent captured image frame 302 in the sequence of successive captured frames 302 is being captured by the imaging system 204. For example, in some embodiments, while the first computer-generated predicted image frame 304-1 is being generated, the imaging system 204 can begin capturing the second captured image frame 302-2. In this way, the imaging system 204 (or other device performing the inference-processing technique) is able to make efficient use of the available computing resources and continues to capture image frames 302 while also producing additional computer-generated frames 304.
The inference-processing technique generates the first, second, and third HDR output frames 306-1, 306-2, and 306-3 based on a fusion (or combination or merging) of the subsequent image frame (e.g., captured frame 302-2) with the computer-generated predicted image frame (e.g., predicted frame 304-1) that was generated in accordance with the inference-processing technique based on the previous image frame (e.g., captured frame 302-1). For example, the second HDR image frame 306-2 is generated based on a fusion of the second captured image frame 302-2 (the subsequent image frame in this example) with the first computer-generated predicted image frame 304-1 (the computer-generated predicted image frame based on the previous captured image frame 302-1 in this example), and the third fused frame 306-3 is generated based a fusion of the third captured image frame 302-3 (the subsequent image frame in this example) with the second computer-generated predicted image frame 304-2 (the computer-generated predicted image frame based on the previous captured image frame 302-2 in this example). Creation of the first HDR image frame 306-1 is followed by the second HDR image frame 306-2, which is followed by the third HDR image frame 306-3. The first, second, and third HDR output frames 306-1, 306-2, and 306-3 eventually form the complete HDR video. In some embodiments, as shown in
In some embodiments, the imaging system 204 (or other device performing the inference-processing technique) repeatedly receives captured image frames 302, generates computer-generated predicted image frames 304, and fuses respective pairs of captured image frames 302 and computer-generated predicted image frames 304 to produce respective HDR output frames 306 for the HDR video (e.g., HDR video 280;
In other words, by using the inference-processing technique (e.g., fusion of captured image frames 302 and computer-generated predicted image frames 304), the imaging system 204 (or other device performing the inference-processing technique) is able to produce HDR videos having higher fps than is normally achievable using captured image frames alone (due to the limitations of the capabilities of the one or more processors 250 of the imaging systems as described above in reference to
In some embodiments, the captured image frames 308 have the same or different exposures. In some embodiments, a first set of the captured image frames 308 have a short exposure and a second set of the captured image frames 308 have a long exposure. For example, captured image frames 308-1, 308-3, 308-5, 308-7 are captured with a short exposure and captured image frames 308-2, 308-4, 308-6 and 308-8 are captured with a long exposure. In some embodiments, the captured image frames 308 alternate between using short and long exposures. The descriptions provided above concerning exposures, and in particular regarding short and long exposures provided in reference to
The inference-processing technique is configured to generate the computer-generated predicted image frames 310 (represented as “AI frames (F),” because the predicted frames are generated using machine-learning models, which are also referred to as a form of artificial intelligence (AI)) using the captured image frames 308. In some embodiments, the computer-generated predicted image frames 310 have the same or different exposures. In some embodiments, a first set of the computer-generated predicted image frames 310 have a short exposure and a second set of the computer-generated predicted image frames 310 have a long exposure. For example, in some embodiments, computer-generated predicted image frames 310-1, 310-3, 310-5, 310-7 have a short exposure and computer-generated predicted image frames 310-2, 310-4, 310-6 and 310-8 have a long exposure. In some embodiments, the exposure of the computer-generated predicted image frames 310 is based on the exposure of the respective captured image frames 308 used by the inference-processing technique to generate the computer-generated predicted image frames 310. Alternatively, in some embodiments, the exposure of the computer-generated predicted image frames 310 is determined by the inference-processing technique (i.e., the inference-processing technique can select an appropriate exposure to increase the overall accuracy of the computer-generated predicted image frames).
As was explained above and as is shown in
In some embodiments, the inference-processing technique can be used on an existing or stored video (which can either be a previously recorded video or can be a video received from another user) as was described above to allow for a form of retrofitting those existing or stored videos to allow them to have higher numbers of frames per second (e.g., by extracting the individual captured image frames, creating the computer-generated predicting image frames, and then fusing pairs of each to create a new HDR video with a higher number of frames per second).
While the technique 301 that involves use of the inference-processing technique for predicting image frames is shown and described in reference to
As described above in reference to
The interpolation-processing technique uses the captured image frames 302 to generate the computer-generated predicted image frames 404. In particular, the interpolation-processing technique uses at least two captured image frames 302 to generate a computer-generated predicted image frame 404. For example, the first computer-generated predicted image frame 404-1 is based on application of the interpolation-processing technique to the first captured image frame 302-1 and the third captured image frame 302-3, the second computer-generated predicted image frame 404-2 is based on application of the interpolation-processing technique to the second captured image frame 302-2 and a fourth captured image frame (not shown), and the third computer-generated predicted image frame 404-3 is based on application of the interpolation-processing technique to the third captured image frame 302-3 and a fifth captured image frame (not shown). Although not shown, fourth captured image frame is captured at a fourth point in time that is after the third point of time of the third captured image frame 302-3, and the fourth captured image frame is followed by the fifth captured image frame that is captured at a fifth point in time. Additionally, each captured image frame 302 is used to interpolate at least a forward and backward looking computer-generated predicted image frame 404. For example, the third captured image frame 302-3 is used to generate the first and third computer-generated predicted image frame 404-1 and 404-3. In some embodiments, the interpolation-processing technique is part of a machine-learning system (e.g., the machine-learning image processing hardware and software 245,
The first, second, and third computer-generated predicted image frames 404-1, 404-2, and 404-3 represent predictions as to how a scene in the real-world at respective points in time, as determined by the appropriate machine-learning model being utilized. In some embodiments, the first computer-generated predicted image frame 404-1 represents a scene in the real-world at a point in time between respective points in time associated with the first and second captured image frames 302-1 and 302-2, the second computer-generated predicted image frame 404-2 represents a scene in the real-world at a point in time between respective points in time associated with the second and third captured image frames 302-2 and 302-3, and the third computer-generated predicted image frame 404-3 represents a scene in the real-world at a point in time between respective points in time associated with the third captured image frame 302-3 and the fourth captured image frame. Each computer-generated predicted image frame 404 is an estimation or approximation of an image frame before a subsequent image frame. For example, the first computer-generated predicted image frame 404-1 is an estimated or approximated frame before the second captured image frame 302-2, the second computer-generated predicted image frame 404-2 is an estimated or approximated frame before the third captured image frame 302-3, and so forth. Because the interpolation-processing technique uses at least two captured image frames (e.g., one captured image frame at a first point in time and one captured image frame at a second point in time greater that the first point in time) to generate the computer-generated predicted image frame 404, the interpolation-processing technique generates estimated or approximated frames with higher accuracy (or less error) than the inference-processing technique. The increased accuracy and/or reduced error improves the overall quality of the created HDR video relative to the inference-processing technique.
In other words, the interpolation-processing technique can be used to further enhance the accuracy of the computer-generated predicted image frames by using both backward-looking (e.g., using the first captured image frame 302-1) and forward-looking (e.g., using the third captured image frame 302-3) inputs to predict what an image frame between the first point in time and the second point in time should looked like (e.g., the first computer-generated predicted image frame 404-1 represents a scene in the real-world at a point in time between the first and second captured image frames 302-1 and 302-2). Thus, an interpolative approach can be added to the inferential technique described above, which further enhances accuracy of the predicted image frames. In some embodiments, the computer-generated predicted image frames represent the scene in the real-world at a next point in time associated with a next captured image frame rather than representing the scene at intermediary points in time (e.g., instead of computer-generated predicted image frame 404-2 representing a scene in the real-world at a point in time between respective points in time associated with the second and third captured image frames 302-2 and 302-3, it can instead represent the scene at the third point in time associated with the third captured image frame 302-3). Some embodiments can make use computer-generated predicted image frames associated with intermediary points in time in addition to computer-generated predicted image frames associated with a next point in time associated with a next captured image frame.
The interpolation-processing technique generates the first, second, and third HDR output frames 406-1, 406-2, and 406-3 based on a fusion (or combination or merging) of the subsequent image frame (e.g., captured frame 302-2) with the computer-generated predicted image frame (e.g., predicted frame 404-1) that was generated in accordance with the interpolation-processing technique based on the previous image frame and a future image frame (e.g., first and third captured image frames 302-1 and 302-3). For example, the second HDR image frame 406-2 is generated based on a fusion of the second captured image frame 302-2 (the subsequent image frame in this example) with the first computer-generated predicted image frame 404-1 (the computer-generated predicted image frame based on the first and third captured image frames 302-1 and 302-3 in this example), and the third HDR image frame 406-3 is generated based a fusion of the third captured image frame 302-3 (the subsequent image frame in this example) with the second computer-generated predicted image frame 404-2 (the computer-generated predicted image frame based on the second and fourth captured image frames in this example). Creation of the first HDR image frame 406-1 is followed by the second HDR image frame 406-2, which is followed by the third HDR image frame 406-3. The first, second, and third HDR output frames 406-1, 406-2, and 406-3 eventually form the complete HDR video. In some embodiments, as shown in
In some embodiments, the imaging system 204 (or other device performing the inference-processing technique) repeatedly receives captured image frames 302, generates computer-generated predicted image frames 404, and fuses respective pairs of captured image frames 302 and computer-generated predicted image frames 404 to produce respective HDR output frames 406 for the HDR video (e.g., HDR video 280;
In other words, the interpolation-processing technique, similar to the inference-processing technique, is able to produce HDR videos having higher fps than is normally achievable using captured image frames alone (due to the limitations of the capabilities of the one or more processors 250 of the imaging systems as described above in reference to
In some embodiments, the captured image frames 308 have the same or different exposures. In some embodiments, a first set of the captured image frames 308 have a short exposure and a second set of the captured image frames 308 have a long exposure. For example, captured image frames 308-1, 308-3, 308-5, 308-7 are captured with a short exposure and captured image frames 308-2, 308-4, 308-6 and 308-8 are captured with a long exposure. In some embodiments, the captured image frames 308 alternate between using short and long exposures. The descriptions provided above concerning exposures, and in particular regarding short and long exposures provided in reference to
The interpolation-processing technique is configured to generate the computer-generated predicted image frames 410 (represented as “AI frames (F),” because the predicted frames are generated using machine-learning models, which are also referred to as a form of AI) using at least two captured image frames 308. In some embodiments, the computer-generated predicted image frames 410 have the same or different exposures. In some embodiments, a first set of the computer-generated predicted image frames 410 have a short exposure and a second set of the computer-generated predicted image frames 410 have a long exposure. For example, in some embodiments, computer-generated predicted image frames 410-1, 410-3, 410-5, 410-7 have a short exposure and computer-generated predicted image frames 410-2, 410-4, 410-6 and 410-8 have a long exposure. In some embodiments, the exposure of the computer-generated predicted image frames 410 is based on the exposure of the respective captured image frames 308 used by the interpolation-processing technique to generate the computer-generated predicted image frames 410. In some embodiments, the computer-generated predicted image frames 410 can have a mixed exposure (e.g., a computer-generated predicted image frame 410 can be generated using a short exposure captured image frame and a long exposure captured image frame). Alternatively, in some embodiments, the exposure of the computer-generated predicted image frames 410 is determined by the interpolation-processing technique (i.e., the interpolation-processing technique can select an appropriate exposure to increase the overall accuracy of the computer-generated predicted image frames).
As was explained above and as is shown in
In some embodiments, the interpolation-processing technique can be used on an existing or stored video (which can either be a previously recorded video or can be a video received from another user) as was described above to allow for a form of retrofitting those existing or stored videos to allow them to have higher numbers of frames per second (e.g., by extracting the individual captured image frames, creating the computer-generated predicting image frames, and then fusing pairs of each to create a new HDR video with a higher number of frames per second).
While the technique 401 that involves use of the interpolation-processing technique for predicting image frames is shown and described in reference to
For example, in some embodiments, a first HDR image frame can be generated using the inference-processing technique described above in reference to
As shown in
After receiving the captured image frames, the process 500 determines (502) whether an HDR video is requested by a user. In accordance with a determination that an HDR video is not requested (“No” at operation 502), the process 500 goes on to generate (504) non-HDR video frames using the captured image frames. Alternatively, in accordance with a determination that an HDR video is requested (“Yes” at operation 502), the process 500 goes on to generate the HDR video as discussed below. In some embodiments, the determination that an HDR video is requested is based on an HDR mode of the imaging system 204 being enabled or disabled. When the HDR mode is disabled, the process 500 makes a determination that an HDR video is not requested (“No” at operation 502) and goes on to generate (504) a non-HDR video based on the captured image frames. Alternatively, when the HDR mode is enabled, the process 500 makes a determination that an HDR video is requested (“Yes” at operation 502) and goes on to generate the HDR video as discussed below. In other words, in some embodiments, the process 500 determines (or checks) whether an HDR-video mode is enabled for the imaging system 204, such as a user having selected to produce HDR videos at a device that is used to manage an image sensor 225 (
In some embodiments, after the process 500 makes a determination that an HDR video is requested (“Yes” at operation 502), the process 500 receives (506) from the user a selected HDR technique. The selected technique can be one or more of the inference-processing technique, the interpolation-processing technique, the technique of
At operation 508, the process 500 determines (508) whether the inference technique for predicting image frames is active. In accordance with a determination that the inference technique for predicting image frames is active (“Yes” at operation 508), the process 500, using the inference-processing technique, generates (514) a computer-generated predicted image frame based on one of the captured image frames. For example, as shown and described above in reference to
Returning to operation 508, in accordance with a determination that the inference technique for predicting image frames is not active (“No” at operation 508), the process 500 determines (518) whether the interpolation technique for predicting image frames is active. In accordance with a determination that the interpolation technique for predicting image frames is active (“Yes” at operation 518), the process 500, using the interpolation-processing technique, generates (524) a computer-generated predicted image frame based on at least two captured image frames. For example, as shown and described above in reference to
Returning to operation 518, in accordance with a determination that the interpolation technique for predicting image frames is not active (“No” at operation 518), the process 500 goes on to generate a fused HDR output frame based on the technique of
Process 500 continues to operate until an HDR video is created. The created HDR video has a predetermined frame rate of equal to or greater than the frame rate of the captured image frames. While the primary examples use computer-generated predicted image frames to improve generation of HDR videos, as was noted above, other video-processing modes can also take advantage of the use of predicted image frames as described herein, e.g., for video-processing modes including low-light video processing, slow-motion mode, etc. to allow for improving frame rates for these other video-processing modes, which would make use of a similar flow diagram as that shown for
The method 600 includes receiving (610), at one or more processors that are in communication with an image sensor configured to capture image frames used to produce an HDR video, a first captured image frame and a second captured image frame captured via the image sensor. The first captured image frame represents a scene in the real-world at a first point in time and the second captured image frame represents the scene in the real-world at a second point in time that is after the first point in time. For example, the one or more processors 250 of the imaging system 204 that are in communication with an image sensor 225 (
The method 600 includes, in accordance with a determination (620) that the first captured image frame and the second captured image frame will be used to produce an HDR video, generating (630), via the one or more processors and based on the first captured image frame, a computer-generated predicted image frame representing the scene in the real-world at a time between the first point in time and the second point in time. For example, as shown and described above in reference to
In some embodiments, the first captured image frame is a first type of image frame, the second captured image frame is a second type of image frame, and the first type of image frame is distinct from the second type of image frame. For example, in some embodiments, the first type of image frame is a first format (e.g., PNG, JPEG, RAW, etc.) and the second type of image frame is a second file format distinct from the first file format. Alternatively, in some embodiments, the first type of image frame has a first resolution (e.g., 1080p, 1440p, 4K, etc.) and the second type of image frame is a second resolution distinct from the first resolution. The types of image frames can include different layout (portrait, landscape, etc.), images captured using different lenses or image sensors, and/or different capture modes (e.g., low light capture mode, burst mode, panoramic mode, etc.). In some embodiments, the first captured image frame is a short exposure frame, and the second captured image frame is a long exposure frame.
Method 600 further includes fusing 640 the second image frame with the computer-generated predicted image frame to generate an HDR frame for the HDR video. For example, as shown and described above in reference to
In some embodiments, the method 600 includes repeating the receiving captured image frames, generating computer-generated predicted image frames, and fusing captured image frames and computer-generated predicted image frame to produce respective HDR frames for the HDR video, such that the HDR video has at least 32 frames per second. In some embodiments, the method 600 includes a first HDR frame that was created by fusing two captured image frames, and a second HDR frame that was created by fusing two computer-generated predicted image frames.
In some embodiments, the method 600 includes after producing the HDR video, receiving captured image frames captured via the image sensor and producing a non-HDR video without using any computer-generated predicted image frames, wherein the HDR video has a first number of frames that is greater than or equal to a second number of frames for the non-HDR video. In some embodiments, the first number of frames and the second number of frames is 32 frames. The technique of
In some embodiments, the computer-generated predicted image frame is a first computer-generated predicted image frame and the HDR frame is a first HDR frame, and the method 600 further includes receiving, at the one or more processors that are in communication with the image sensor, a third captured image frame captured via the image sensor. The third captured image frame represents the scene in the real-world at a third point in time that is after the second point in time. The method 600 further includes, in accordance with a determination that the third captured image frame will be used in conjunction with the first captured image frame and the second captured image frame to pro-duce the HDR video, generating, via the one or more processors and based on the second captured image frame, a second computer-generated predicted image frame representing the scene in the real-world at the time between the second point in time and the third point in time and fusing the third image frame with the second computer-generated predicted image frame to generate a second HDR frame for the HDR video. The second HDR frame follows the first HDR frame in a sequence of HDR frames that eventually forms the complete HDR video.
In some embodiments, the method 600 further includes receiving, at the one or more processors that are in communication with the image sensor, a third captured image frame captured via the image sensor, the third captured image frame representing the scene in the real-world at a third point in time that is after the second point in time. In accordance with a determination that the third captured image frame will be used in conjunction with the first captured image frame and the second captured image frame to produce the HDR video, the method 600 further includes generating, via the one or more processors and based on the first captured image frame and the third captured image frame, the computer-generated predicted image frame representing the scene in the real-world at the time between the first point in time and the second point in time, and fusing the computer-generated predicted image frame with the second image frame and the computer-generated predicted image frame to generate the HDR frame for the HDR video.
In other words, an interpolation technique can be used to further enhance the accuracy of the computer-generated predicted image frames by using both backward-looking (using the first captured image frame) and forward-looking (using the third captured image frame) inputs to predict what an image frame between the first point in time and the second point in time should have looked like (e.g., as shown and described above in reference to
In some embodiments, the computer-generated predicted image frame is a first computer-generated predicted image frame and the HDR frame is a first HDR frame, and the method 600 further includes receiving, at the one or more processors that are in communication with the image sensor, a fourth captured image frame captured via the image sensor, the fourth captured image frame representing the scene in the real-world at a fourth point in time that is after the third point in time. The method 600 further includes, in accordance with a determination that the fourth captured image frame will be used in conjunction with the first captured image frame; the second captured image frame; and the third captured image frame to produce the HDR video, generating, via the one or more processors and based on the second captured image frame and the fourth captured image frame, a second computer-generated predicted image frame representing the scene in the real-world at the time between the third point in time and the second point in time. The method 600 further includes fusing the third image frame with the second computer-generated predicted image frame to generate a second HDR frame for the HDR video. In some embodiments, each frame is used to interpolate a forward and backward looking computer-generated predicted image frame (as described above in reference to
In some embodiments, the computer-generated predicted image frame is generated via a machine-learning system that has been trained using a training set consisting of a variety of image frames captured by an image sensor viewing different scenes in the real world.
In some embodiments, the HDR video has a number of fps that is at least equal to a maximum fps achievable by the one or more processors when using captured image frames to produce a video (e.g., as described above in reference to
The wrist-wearable device 750 can perform various functions and operations with reference to
The watch band 762 can be configured to be worn by a user such that an inner surface of the watch band 762 is in contact with the user's skin. When worn by a user, sensor 764 is in contact with the user's skin. The sensor 764 can be a biosensor that senses a user's heart rate, saturated oxygen level, temperature, sweat level, muscle intentions, or a combination thereof. The watch band 762 can include multiple sensors 764 that can be distributed on an inside and/or an outside surface of the watch band 762. Additionally, or alternatively, the watch body 754 can include the same or different sensors than the watch band 762 (or the watch band 762 can include no sensors at all in some embodiments). For example, multiple sensors can be distributed on an inside and/or an outside surface of watch body 754. As described below with reference to
The watch band 762 and/or watch body 754 can include a haptic device 766 (e.g., a vibratory haptic actuator) that is configured to provide haptic feedback (e.g., a cutaneous and/or kinesthetic sensation, etc.) to the user's skin. The sensor 764 and/or haptic device 766 can be configured to operate in conjunction with multiple applications including, without limitation, health monitoring, social media, game playing, and artificial reality (e.g., the applications associated with artificial reality).
In some examples, the watch band 762 can include a neuromuscular sensor 765 (e.g., an electromyography (EMG) sensor, a mechanomyogram (MMG) sensor, a sonomyography (SMG) sensor, etc.). Neuromuscular sensor 765 can sense a user's intention to perform certain motor actions. The sensed muscle intention can be used to control certain user interfaces displayed on the display 220 of the wrist-wearable device 750 and/or can be transmitted to device responsible for rendering an artificial-reality environment (e.g., a head-worn wearable device) to perform an action in an associated artificial-reality environment, such as to control the motion of a virtual device displayed to the user.
Signals from neuromuscular sensor 765 can be used to provide a user with an enhanced interaction with a physical object and/or a virtual object in an artificial-reality application generated by an artificial-reality system (e.g., user interface objects presented on the display 220, or another computing device (e.g., a head-worn wearable device, such as smart glasses)). Signals from neuromuscular sensor 765 can be obtained (e.g., sensed and recorded) by one or more neuromuscular sensors 765 of watch band 762. Although
The wrist-wearable device 750 can include a coupling mechanism (also referred to as a cradle) for detachably coupling watch body 754 to watch band 762. A user can detach watch body 754 from watch band 762 in order to reduce the encumbrance of the wrist-wearable device 750 to the user. The wrist-wearable device 750 can include a coupling surface 756 on the watch body 754 and/or coupling mechanism(s) 760 (e.g., a cradle, a tracker band, a support base, a clasp). A user can perform any type of motion to couple watch body 754 to watch band 762 and to decouple watch body 754 from watch band 762. For example, a user can twist, slide, turn, push, pull, or rotate watch body 754 relative to watch band 762, or a combination thereof, to attach watch body 754 to watch band 762 and to detach watch body 754 from watch band 762.
As shown in the example of
The wrist-wearable device 750 can include a single release mechanism 770 or multiple release mechanisms 770 (e.g., two release mechanisms 770 positioned on opposing sides of the wrist-wearable device 750 such as spring-loaded buttons). As shown in
In some examples, watch body 754 can be decoupled from watch body interface 780 by actuation of a release mechanism. The release mechanism can include, without limitation, a button, a knob, a plunger, a handle, a lever, a fastener, a clasp, a dial, a latch, or a combination thereof. In some examples, the wristband system functions can be executed independently in watch body 754, independently in watch body interface 780, and/or in communication between watch body 754 and watch body interface 780. Watch body interface 780 can be configured to operate independently (e.g., execute functions independently) from watch body 754. Additionally, or alternatively, watch body 754 can be configured to operate independently (e.g., execute functions independently) from watch body interface 780. As will be described in more detail below with reference to the block diagram of
In this example, watch body interface 780 can include all of the electronic components of watch band 762. In additional examples, one or more electronic components can be housed in watch body interface 780 and one or more other electronic components can be housed in portions of watch band 762 away from watch body interface 780.
For example, watch body 804 can include capacitive sensor 877, magnetic field sensor, antenna return-loss (RL) sensor, biometric sensor, battery 828, CPU 826, storage 802, heart rate sensor 858, EMG sensor 846, SpO2 sensor 854, altimeter 848, IMU 842, random access memory 803, charging input 830 and communication devices NFC 815, LTE 818, and WiFi/Bluetooth 820. Similarly, watch band 812 can include battery 838, microcontroller unit 852, memory 850, heart rate sensor 858, EMG sensor 846, SpO2 sensor 854, altimeter 848, IMU 842, charging input 834 and wireless transceiver 840. In some examples, a level of functionality of at least one of watch band 812 or watch body 804 can be modified when watch body 804 is detached from watch band 812. The level of functionality that can be modified can include the functionality of at least one sensor (e.g., heart rate sensor 858, EMG sensor 846, etc.). Each of watch body 804 and watch band 812 can execute instructions stored in storage 802 and memory 850 respectively that enables at least one sensor (e.g., heart rate sensor 858, EMG sensor 846, etc.) in watch band 812 to acquire data when watch band 812 is detached from watch body 804 and when watch band 812 is attached to watch body 804.
Watch body 804 and watch band 812 can further execute instructions stored in storage 802 and memory 850 respectively that enables watch band 812 to transmit the acquired data to watch body 804 (or other computing device such as a head mounted display or other computing device communicatively coupled to the wrist-wearable device system 800) using wired communications 827 and/or wireless transceiver 840. For example, watch body 804 can display visual content to a user on touchscreen display 813 (e.g., an instance of display 220) and play audio content on speaker 874. Watch body 804 can receive user inputs such as audio input from microphone 872 and touch input from buttons 824. Watch body 804 can also receive inputs associated with a user's location and/or surroundings. For example, watch body 804 can receive location information from GPS 816 and/or altimeter 848 of watch band 812.
Watch body 804 can receive image data (e.g., captured image frames) from at least one image sensor 135 (e.g., a camera). Image sensor 135 can include front-facing image sensor 725A (
Watch body 804 and watch band 812 can receive a charge using a variety of techniques. In some embodiments, watch body 804 and watch band 812 can use a wired charging assembly (e.g., power cords) to receive the charge. Alternatively, or in addition, watch body 804 and/or watch band 812 can be configured for wireless charging. For example, a portable charging device can be designed to mate with a portion of watch body 804 and/or watch band 812 and wirelessly deliver usable power to a battery of watch body 804 and/or watch band 812.
Watch body 804 and watch band 812 can have independent power and charging sources to enable each to operate independently. Watch body 804 and watch band 812 can also share power (e.g., one can charge the other) via power management IC 832 in watch body 804 and power management IC 836 in watch band 812. Power management IC 832 and power management IC 836 can share power over power and ground conductors and/or over wireless charging antennas.
Wrist-wearable device system 800 can operate in conjunction with a health monitoring application that acquires biometric and activity information associated with the user. The health monitoring application can be designed to provide information to a user that is related to the user's health. For example, wrist-wearable device system 800 can monitor a user's physical activity by acquiring data from IMU 842 while simultaneously monitoring the user's heart rate via heart rate sensor 858 and saturated blood oxygen levels via SpO2 sensor 854. CPU 826 can process the acquired data and display health related information to the user on touchscreen display 813.
Wrist-wearable device system 800 can detect when watch body 804 and watch band 812 are connected to one another (e.g., mechanically connected and/or electrically or magnetically connected) or detached from one another. For example, pin(s), power/ground connections 860, wireless transceiver 840, and/or wired communications 827, can detect whether watch body 804 and watch band 812 are mechanically and/or electrically or magnetically connected to one another (e.g., detecting a disconnect between the one or more electrical contacts of power/ground connections 860 and/or wired communications 827). In some examples, when watch body 804 and watch band 812 are mechanically and/or electrically disconnected from one another (e.g., watch body 812 has been detached from watch band 812 as described with reference to
Modifying the level of functionality (e.g., reducing the functionality in watch body 804 and/or watch band 812) can reduce power consumption in battery 828 and/or battery 838. For example, any of the sensors (e.g., heart rate sensor 858, EMG sensor 846, SpO2 sensor 854, altimeter 848, etc.), processors (e.g., CPU 826, microcontroller unit 852, etc.), communications elements (e.g., NFC 815, GPS 816, LTE 818, WiFi/Bluetooth™ 820, etc.), or actuators (e.g., haptics 822, 849, etc.) can reduce functionality and/or power consumption (e.g., enter a sleep mode) when watch body 804 and watch band 812 are mechanically and/or electrically disconnected from one another. Watch body 804 and watch band 812 can return to full functionality when watch body 804 and watch band 812 are mechanically and/or electrically connected to one another. The level of functionality of each of the sensors, processors, actuators, and memory can be independently controlled.
As described above, wrist-wearable device system 800 can detect when watch body 804 and watch band 812 are coupled to one another (e.g., mechanically connected and/or electrically connected) or decoupled from one another. In some examples, watch body 804 can modify a level of functionality (e.g., activate and/or deactivate certain functions) based on whether watch body 804 is coupled to watch band 812. For example, CPU 826 can execute instructions that detect when watch body 804 and watch band 812 are coupled to one another and activate front-facing image sensor 725A. CPU 826 can activate front-facing image sensor 725A based on receiving user input (e.g., a user touch input from touchscreen display 813, a user voice command from microphone 872, a user gesture recognition input from EMG sensor 846, etc.).
When CPU 826 detects that watch body 804 and watch band 812 are decoupled from one another, CPU 826 can modify a level of functionality (e.g., activate and/or deactivate additional functions). For example, CPU 826 can detect when watch body 804 and watch band 812 are decoupled from one another and activate rear-facing image sensor 725B. CPU 826 can activate rear-facing image sensor 725B automatically (e.g., without user input) and/or based on receiving user input (e.g., a touch input, a voice input, an intention detection, etc.). Automatically activating rear-facing image sensor 725B can allow a user to take wide-angle images without having to provide user input to activate rear-facing image sensor 725B.
In some examples, rear-facing image can be activated based on an image capture criterion (e.g., an image quality, an image resolution, etc.). For example, rear-facing image sensor 725B can receive an image (e.g., a test image). CPU 826 and/or rear-facing image sensor 725B can analyze the received test image data and determine whether the test image data satisfies the image capture criterion (e.g., the image quality exceeds a threshold, the image resolution exceeds a threshold, etc.). Rear-facing image sensor 725B can be activated when the test image data satisfies the image capture criterion. Additionally, or alternatively, rear-facing image sensor 725B can be deactivated when the test image data fails to satisfy the image capture criterion.
In some examples, CPU 826 can detect when watch body 804 is coupled to watch band 812 and deactivate rear-facing image sensor 725B. CPU 826 can deactivate rear-facing image sensor 725B automatically (e.g., without user input) and/or based on receiving user input (e.g., a touch input, a voice input, an intention detection, etc.). Deactivating rear-facing image sensor 725B can automatically (e.g., without user input) reduce the power consumption of watch body 804 and increase the battery charge time in watch body 804. In some examples, wrist-wearable device system 800 can include a coupling sensor 807 that senses whether watch body 804 is coupled to or decoupled from watch band 812. Coupling sensor 807 can be included in any of watch body 804, watch band 812, or watch band coupling mechanism 760 of
It will be understood that, although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the claims. As used in the description of the embodiments and the appended claims, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and/or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
As used herein, the term “if” can be construed to mean “when” or “upon” or “in response to determining” or “in accordance with a determination” or “in response to detecting,” that a stated condition precedent is true, depending on the context. Similarly, the phrase “if it is determined [that a stated condition precedent is true]” or “if [a stated condition precedent is true]” or “when [a stated condition precedent is true]” can be construed to mean “upon determining” or “in response to determining” or “in accordance with a determination” or “upon detecting” or “in response to detecting” that the stated condition precedent is true, depending on the context.
The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the claims to the precise forms disclosed. Many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain principles of operation and practical applications, to thereby enable others skilled in the art.
This application claims priority to U.S. Provisional Patent Application No. 63/277,065, filed Nov. 8, 2021, titled “Systems And Methods Of Fusing Computer-Generated Predicted Image Frames With Captured Images Frames To Create A High-Dynamic-Range Video Having A High Number Of Frames Per Second,” which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63277065 | Nov 2021 | US |
